Both approaches are viable options. Generally, stepping down the voltage like you described would result in longer run times before the battery drains, but will likely weigh more since it involves a second battery cell.
We have several step-down regulators that might work. For example, the #2859 D24V22F6 can handle around 2.5A, which should be fine as long as you avoid stalling your motors (which you should try to do anyway since doing so can quickly damage the motors).
If you want to use the 3.7V battery and step-up the voltage, the most powerful 6V step-up regulator we currently have is the #2566 U3V50F6, which can only handle an output of around 1.5A when using a 1S LiPo battery as an input. Since the combined stall current of the motors at 6V is around 3.2A, I would probably recommend taking some precautions against drawing too much current if you choose this option (such as ramping up the speed or limiting the maximum duty cycle).
Using one of our switching regulators to power the QTR-8A reflectance sensor should be fine; however, please note that if you power it from 6V, the output could raise as high as 6V, so you should be sure that your microcontroller can handle signals at that voltage.