I am currently trying on tilt compensate Pololu AltIMU-10 v4 to give me the heading with respect to magnetic north. However, the compensated value is far more unstable compared to the original value.
I didn’t see any issues with your code at a quick glance. One debugging strategy might be to print out intermediate values within the calculation to see if they make sense. What is the type of the variables ax, mx, etc? If they are integers, you might need to cast them to wider integer types or floats to avoid overflows when you multiply them or losing precision when you divide them.
You might also want to take a look at the way we compute a tilt-compensated heading in our LSM303 Arduino library and its Heading example. It uses a different approach than the ST app note, though.
Sorry for the late reply. I just solved the issue today. The problem is I did get the offset of the magnetometer at my home last time. However, I did not do so for my workplace. Therefore, when I work at my workplace, it is inaccurate.
The unstable readings might be due to I actually placed the IMU on my phone to check the readings. It might be the magnetic field inside the phone interfering with the IMU, causing the reading to be very incorrect.
By the way, do we need to calibrate every time we switched places using this magnetometer?