Yes, to get smooth line following, you need to tune your PID constants. In general, we recommend starting by just using a proportional (Kp) term, i.e.:
int16_t speedDifference = error / 4;
Then you can start experimenting by trial and error with the value of that coefficient (try a series of slightly bigger and smaller terms), and find the value that makes the Zumo follow the line best. Then, you can add in the derivative (Kd) term. For example:
int16_t speedDifference = error / 4 + 6 * (error - lastError);
Note that Kd is being multiplied by the difference in successive errors, which will generally be much smaller than the error itself, so Kd usually needs to be much bigger than Kp for it to have a comparable effect on the motor speed differential. Starting with a Kd constant that is 10 to 20 times bigger than Kp is reasonable. You can read more about choosing a good Kp and Kd in this blog post by Ben.
I also recommend reading the Wikipedia page on PID control.
Additionally, the paneled floor you placed your line on top of might be interfering with the Zumo’s ability to detect the line (i.e. it could be getting false readings from the darker parts of your floor). It would help to create a course on a surface that has a stark contrast to the black line (i.e. a reflective white surface).
-Jon