If you want to follow the datasheet specification (never a bad idea), if your ATMega168 is going to be powered by less than 4.5V, it should be running off of a 10MHz or slower clock at startup, before your code even starts running. The clock source will need to be selected and scaled with fuses.
If you don’t want to replace the external 20MHz resonator on your Baby O, you can set the “divide clock by 8 internally” fuse to run at 2.5MHz, or switch over to the internal 8MHz oscillator (or do both, to run at 1MHz).
-Adam