If you are compiling your program for the A-Star using the Arduino IDE, then the USB library provided by the Arduino core code will watch for the baud rate to be changed to 1200 and for the DTR signal to be set to 0. When that happens, it resets the AVR in a way that causes the bootloader to start.
If you are not compiling using the Arduino IDE, you can still program the A-Star to behave in the same way. You might consider importing the Arduino core code into your project, or using some other USB stack such as LUFA.
When you click the "Upload" button in the Arduino IDE, it opens the selected serial port, sets the baud rate to 1200, and closes it. Then it monitors the serial ports on your system for a while, waiting for a new one to appear. When a new serial port appears, it assumes that the serial port is for the A-Star's bootloader, and then it launches AVRDUDE in order to upload your program using that serial port.
You might consider using libserialport to write a command-line utility in C that does the same thing as the Arduino IDE. Alternatively, most programming languages have libraries for listing the serial ports, opening them, and setting their baud rates, so you could a language other than C.