Yes, you absolutely must have ground connected between the boards. Otherwise, what do you expect any of the voltage levels to mean? You should expect current to go in through the Vdd pin; where would it come back?
The reason it worked for you is probably that you always had at least one of the control inputs low, so the current could return through there. But that’s definitely not something to count on or do on purpose.
- Jan