Let’s say we have the following data, and we learn the following piecewise line as a regression predictor:
Now, let’s say we get the following test data (visualized as the white data points):
As we can see, the learned regression function is very specific to the training data (black points) without generalizing to the test data (white points). This is a general problem in machine learning called overfitting.
If we instead used a plain old straight line to fit the data, we would get a much more generalizable fit:
The goal of the methods we will cover in this chapter are to generate a generalized fit to the training data, like the line of above, rather than a super specific fit that will not generalize to other data points. We aim to accomplish this in an algorithmic way that is specific to the dataset.