Could you clarify what you meant by the “motors start to flip”?
In general, the step rate is limited by the stepper motor. Based on the datasheet you gave for the PF35T, 500 steps per second seems like a reasonable maximum speed. However, I expect you to get more than 500 steps per second from the nema 8 hybrid bipolar stepper motor (#1204). Do you have a load on your stepper motor? This graph, for the nema 8 hybrid bipolar stepper motor, should give you an idea what pulse rates you can expect form it at certain loads. Most stepper motor datasheets have similar pull out torque curves that can give a good approximation of the maximum speed of a stepper motor at certain loads
It sounds like you might have not set the current limit appropriately. Please refer to the “Current limiting” section under the “Description” tab on A4988 driver product page for instructions on how to set the current limit using Vref. You might also consider using a power supply with a higher voltage. By using a higher voltage along with active current limiting, the current is able to ramp up faster, which lets you achieve higher step rates than you could using the rated voltage. In addition, since the vexta stepper motor you mentioned has a rated voltage of 12V, you will only be able to get the maximum amount of torque and speed from that motor when using a supply voltage of at least 12V.