What is linear regression?
Suppose you are thinking of selling your home. Different sized homes around you have sold for different amounts:
Your home is 3000 square feet. How much should you sell it for? You have to look at the existing data and predict a price for your home. This is called linear regression.
Here’s an easy way to do it. Look at the data you have so far:
Each point represents one home. Now you can eyeball it and roughly draw a line that gets pretty close to all of these points:
Then look at the price shown by the line, where the square footage is 3000:
Boom! Your home should sell for $260,000.
That’s all there is to it. You plot your data, eyeball a line, and use the line to make predictions. You need to make sure your line fits the data well:
But of course we don’t want to eyeball a line, we want to compute the exact line that best “fits” our data.
How do you decide what line is good? Here’s a bad line:
This line is way off. For example, according to this line, a 1000 sq foot house should sell for $310,000, whereas we know it actually sold for $200,000:
Maybe a drunk person drew this line…it is $110,000 dollars off the mark for that house. It is also far off all the other values:
On average, this line is $73,333 off ($110,000 + $70,000 + $40,000 / 3).
Here’s a better line:
This line is an average of $8,333 dollars off…much better. This $8,333 is called the cost of using this line. The “cost” is how far off the line is from the real data. The best line is the one that is the least off from the real data. To find out what line is the best line, we need to use a cost function.
The cost function
First some foundational math. We are drawing a line. Here’s what the equation of a line looks like:
The first number tells you how high the line should be at the start:
The second number tells you the angle of the line:
So we need to come up with two pretty good numbers that make the line fit the data. Here’s how we do it:
- Pick two starting numbers. These are traditionally called
. Zero and zero are a good first guess for these.
- Draw a line using those numbers:
(Yes, it is a line where all the predicted values are zero. It is still a fine first guess, we are going to keep improving it).
- Calculate how far off we are, on average, from the actual data:
This is called the cost function. You pass
into the function, and it tells you how far off that line is (i.e. the cost of using that line).
I’ll also show the math formula they use for this. It is pretty much the same thing:
In this formula,
represents the predicted value:
It’s just the formula for our line! If we plug in a specific value for
, we will get a value for
gives us the difference between the predicted value and the actual value.
For a given value of
, the cost function will tell you how good those values are (i.e. it will tell you how far off your predictions were from the actual data). But what do we do based on that information? How do we find the values of
that will draw the best line? By using gradient descent.
Let me start with a simpler version of gradient descent, and then move on to the real version. Suppose we decide to leave
at zero. So we experiment with what value
should be, but
is always zero. Now you can try various values for
, and you will end up with different costs. You can plot all of these costs on a graph:
So for example, using a
value of 75 is better than a value of 160. The cost is lower:
Here are the corresponding lines (remember,
is zero in these lines):
We can see that the line on the left seems to fit the data better than the line on the right, so it makes sense that the cost of that line is lower. And from this graph it looks like
gives us the lowest cost overall:
Since it is the lowest point in this graph. So with all the costs graphed out like this, we just need to find the lowest point on the graph, and that will give us the optimal value of
Gradient descent helps us find the lowest point on this graph. You start with a value for
, and update it iteratively till you arrive at the best value. So you can start at
. Then you have to ask, should I go left or right?
Well, we want to go down, so lets go right a small step:
This is the new value for
. Again we ask, should we go left or right? At each step, you need to head downward, till you get to a point where you’re as low as you can go:
This is gradient descent: going down bit by bit till you hit the bottom. How do you figure out which way is down? The answer will be obvious to calculus experts but not so obvious for the rest of us: you take the derivative at that point. This is the part that I’ll gloss over and just give you a formula to apply. But the important bit to know is, if you take the current value of
and add the derivative at that point, you will go down. You just do that a bunch of times (say 1000 times) and you will hit bottom!
That was the simpler view, now lets get back to the original problem. In this problem, we need to know the optimal values for both
. So the graph looks more like this:
So you can see having
would be really bad, but
would be okay.
It is still a bowl with a low point, it is just in 3d because now we are considering
as well. But the idea stays the same: start somewhere in the bowl and just keep taking steps till you are at the bottom!
This is how you find the line with the lowest cost with gradient descent. You find the
values that give you the best fitting line by starting with a guess and incrementally updating that guess to make the cost lower and lower.
Here’s the formula for gradient descent:
They are partial derivatives used in two update rules: one for
and another for
. The two are almost the same, except the
derivative has that extra
at the end.
Putting it all together
I used something called feature scaling here, so the square feet values will be
as expected. This just makes gradient descent work better here.
We started with these data points:
Using gradient descent, we find that
give us the best fitting line, which costs $7,142.9 and looks like this:
(our eyeballed line was really close, it had a cost of $8,333.33).
And know you know that you can sell your 3000 square foot home for $271,430 dollars!
- Linear regression is used to predict a value (like the sale price of a house).
- Given a set of data, first try to fit a line to it.
- The cost function tells you how good your line is.
- You can use gradient descent to find the best line.