I am glad that the Wixel has worked well for you.
The Wixel’s Wireless Serial App uses over 2000 lines of code when you count all the libraries it depends on, and it only takes about 8.2 kB, so I wonder why your application would take up so much more space.
In case you have not done so already, I recommend looking at the .map file generated by SDCC so that you can see how much space each function is taking. This might give you an idea of which areas to optimize further. If you are using pointers, you might consider adding the XDATA qualifier to them to avoid using SDCC’s 3-byte generic pointers, which have more overhead than necessary.
If that does not help, you can free up an extra 3 kB by getting a CC2511F32 programmer and overwriting the Wixel’s USB bootloader. Note that you would have to remove the code in
fixed.s that refers to data structures in the bootloader (such as the USB serial number), and provide your own versions of any needed data structures.
I am not aware of any chips like the CC2511 with more flash space. You might consider getting the CC2500 transceiver, which is compatible with the CC2511, and controlling it with a separate microcontroller that has more flash space.