For tilt calibration to work, you must calibrate all three axes of the magnetometer, and the simple min/max axial approach does not always work well. You might find this post informative, which reviews and applies a far more powerful and effective procedure: Correcting the Balboa magnetometer
Also, unless you are extremely careful with and understand the (at least six) different angular system conventions, this formula is probably wrong:
//--- Final data
magXheading = magXcomp * cos(pitch) + magZcomp * sin(pitch);
magYheading = magXcomp * sin(roll) * sin(pitch) + magYcomp * cos(roll) - magZcomp * sin(roll) * cos(pitch);
It is much, much easier and safer to use the vector approach to tilt compensation, published by the Pololu engineers. See the last entry in the above Pololu forum thread for their code, modified for the Balboa compass mounting arrangement.