Just yellow on is normal. If you hold the serial input low while turning the servo controller on, you should see all LEDs on. It doesn’t sound like you’re doing that, though. Once you let the line go (or drive it high), it should go high because it has a pull-up, and you should see just the yellow LED until you try sending something down the line.
There is just a little problem left, it seems that i’m not at all familiar with the electronical english jargon which means that i don’t really know what “holding low” or “driving high” means.
Holding and driving are about the same, but holding implies a bit longer time or until something else happens. Low and high are the logic levels, which are about 0 V and 5 V in this case. Driving something high therefore means making it high, or applying 5 V to something. Holding it high would mean continuing to drive high. Driving implies a low-impedance output that makes sure the output is what you want it to be, as opposed to what a pull-up resistor does: weakly influencing a line toward one side instead of another. If you have a line with a pull-up, it will be high if you do nothing else, but driving it low will overcome the effect of the pull-up.
(The other term you will commonly see is for no connection: this is often referred to as floating or high-z (z for impedance).)