As I said in my first post you don’t really need the regulator if you can tolerate the reduced max voltage which in turn reduces your max speed.
Here is the difference between you using the regulator at 5V to power the motor vs the direct battery.
Case 1: Suppose motor has 15 ohm resistance.
Apply 5V, motor will draw 330mA (5V/15 ohms). This means 330mA *( 5V / 4V source) / 75% efficiency = 550mA from the battery.This maybe is too much.
Case 2:
Apply 4V battery direct: motor will draw 260mA from the battery. (4V /15 ohms).This is OK.
There are more reasons why the battery can maybe power the motor directly. It also can absorb a surge without acting as weirdly as a regulator might.
But to be honest it hasn’t been very clear what you are doing and why you are doing it, which sometimes is No.1 to help troubleshoot.