Thank you for your interest in the Baby Orangutan, and for the detailed and organized explanation of your problem.
Yes, the OrangutanPulseIn and PololuWheelEncoders components of the Pololu USB AVR C/C++ Library define their own interrupt service routine (ISR) for the pin-change interrupts on the AVR. The AVR can only have one ISR for each interrupt, so these libraries conflict.
You can probably make your program work by modifying the library. I would recommend that you change the ISRs in OrangutanPulseIn.cpp and PololuWheelEncoders.cpp to just be normal functions instead of real ISRs. You would have to get rid of all the lines that refer to interrupt vectors PCINT0_vect, PCINT1_vect, PCINT2_vect, and PCINT3_vect. Also, define your ISRs with
extern "C" so you can call them from your C program. So the ISR definition in each file would look something like:
extern "C" void pulse_in_isr(void)
// ISR code here
main.c, you need to define an actual ISR that calls these functions. Something like this should work:
Compiling the modified library
We usually compile the library on Ubuntu using Ubuntu's AVR GCC cross compiler and standard tools like Bash and GNU Make. However, I think it will be easier for you to instead just copy all the library files that you need into your own Atmel Studio project and compile them as part of that project. You should not use our templates; you would need to make a new project by selecting File -> New Project -> C/C++ -> GCC C++ Executable Project. This kind of project can have both C and C++ source files in it. I recommend that you add your
main.c file to it, try to build the project, and then fix compilation and linker errors by adding needed .cpp and .h files from the source code of the library to your project until everything builds. If you have some familiarity with C or C++ already, I think the errors will not be too hard to fix, and you can post the errors here if you need help.
While you are developing this, it might be good to move or rename the "pololu" folder in Atmel Studio's AVR GCC toolchain so that you can make sure that you are using your own copies of the Pololu AVR C/C++ Library header files instead of using the main library's header files. However, since you don't need to modify any of those header files, it will not be too bad if you accidentally use them. The "pololu" folder that gets installed by our library in Atmel Studio's toolchain can be found here, assuming you installed Atmel Studio in the default location:
C:\Program Files (x86)\Atmel\Studio\7.0\toolchain\avr8\avr8-gnu-toolchain\avr\include\pololu
Since your new ISR would call functions in other source files, the compiler might be forced to save/restore many registers that are not actually used, and this would slow down the ISR. There is a good chance that your application will work fine anyway, but if you want to speed up the ISRs, you might consider enabling GCC's link-time optimization (LTO) feature. However, I have not looked into whether link-time optimization is actually supported in Atmel Studio. (It is supported in the Arduino IDE, another environment for AVR programming that is compatible with the Baby Orangutans.)