As I mentioned before, the maximum current limit that the chip allows you to set is around 18A (4A per phase is the limit that our board can do without additional cooling). From the values you posted, it looks like you’re setting the current limit to over 9A per phase.
From page 14 of the DRV8711 datasheet the equation for calculating the current limit is:
Where I_{CHOP} is the current limit in amps and R_{ISENSE} is the current sense resistor in ohms. For the #3730 High-Power Stepper Motor Driver 36v4, R_{ISENSE} is 30 milliohms. So, to set the current limit to 2A per phase for your motor:
Rearranging to solve for TORQUE in terms of ISGAIN:
Our library tries to use the highest ISGAIN setting (which is 40 and achieved by setting both ISGAIN bits high), then checks to see if that results in a TORQUE value higher than 0xFF. If it does, then we need to use a lower ISGAIN so the TOQRUE value fits within 8 bits. However, in your case, to set it to 2A per phase on the High-Power Motor Driver 36v4, an ISGAIN of 40 is fine since if we plug 40 into our equation TORQUE is less than 255:
So, I recommend setting the TORQUE bits of the TORQUE register to 0b000011011111 (i.e. 223 in decimal) and the CTRL register bits to 0b111100010001 (which should set the ISGAIN to 40 and enable the driver).
As far as testing to see if your board is still working, you might start by seeing if you can just read some of the registers.
Brandon