A few other things to consider in addition to what Paul recommended:
1) I like to start with just the proportional term, P, at a low speed to get something that behaves reasonably. You can calculate a ballpark value for Kp by determining what line position value from your sensor should correspond to a maximum correction from your motors (i.e. at what point do you want one motor turning full speed while the other motor is stopped)? Pick a value for Kp that makes that happen.
For example, let's say that your sensor returns line positions from 0 to 100, where 0 is full left and 100 is full right, and let's say your motor speeds vary from -100 (full reverse) to 100 (full forward). To get a maximum correction, you want a motor differential of, say, -100 when your sensor reading is 0 and +100 when your sensor reading is 100:
correction = Kp * (line_position - 50) // ignore D for now
right_motor_speed = 100 - correction
left_motor_speed = 100 + correction
To convert appropriately from line sensor values to motor speeds as defined above, you can see that Kp should be around 2. If your line sensor values or motor speeds have different domains, then the ideal Kp will be different. The point is to pick a Kp that converts line sensor readings into motor speeds in a way that gives you just the right amount of responsiveness. Too much and you'll overshoot on turns. Too little and you'll lose the line on sharp turns.
2) Once you have a good sense for Kp, try adding in the differential term, D. The key thing to note is that the Kd needs to be much bigger than Kp for it to have any effect on the correction computation. This is because Kd is being multiplied by the difference of two similar terms:
correction = Kp * line_position + Kd * (line_position - prev_line_position)
The difference is likely to be very small relative to the value of line_position itself. I recommend starting with Kd = 20 * Kp, but the actual value of Kd will depend on your particular implementation. For example, the faster your main loop runs, the bigger Kd should be.
3) Make sure you have the signs right in your equation. If you have a sign error somewhere, your trial and error tests are likely to be very frustrating.