Sign in to continue learning
Applied Statistics for Data Analytics
34h44m
Correlation is a way to quantify the relationship between two numerical features. You've seen scatter plots that visually represent how two features are related. Now, you'll formalize that relationship. There are multiple ways of measuring correlation, but the most common is the Pearson correlation coefficient, which is often represented by lowercase r. r is always a number between minus 1 and 1. Let's start with the intuition behind this measure. Say you have a scatter plot of two numerical features. On the x-axis, you have the number of TV episodes watched, and on the y-axis, the total number of hours of TV watched. You can think of r as measuring how well you can fit a straight line through these points, called a line of best fit. More technically, r measures the strength and direction of a linear relationship between two features. Let's take the perfect case. Say you have these points for your scatter plot. The straight line has a consistent slope, meaning for every one unit change in x, y changes by a consistent amount. That would make sense for a TV show where every episode is exactly the same length, say 45 minutes. If you know how many episodes of TV someone watched, you also know exactly how long they've been watching TV for. In this example, you have a line at y equals 0.75x, meaning that each episode is three quarters of an hour. So this line has a slope of positive 0.75, and for every change of 1 in x, y changes by 0.75. In this case, the correlation coefficient r equals positive 1. You can say that x and y are perfectly correlated because changes in x produce exactly predictable changes in y. Now imagine your data is less perfect with this set of points. It's not as easy to fit a line through the middle, but it is definitely possible to get a good approximation. Each point is either a little above or a little below the line, but as x goes up, y also goes up, and it's relatively predictable how much it will go up. You can tell that the more TV episodes you watch, the more time you've watched TV, but the episodes aren't all the same length. The correlation coefficient might be close to positive 0.85. In this scatter plot, the line of best fit is the same as the previous one, but the correlation isn't as strong. Here's another example where as x goes up, y goes down. In this case, x is the number of TV episodes watched and y is the number of hours spent studying. So the more TV a person watches, the less time they might have to study. In this case, the line of best fit has a negative slope. However, the correlation is equally as strong as the previous example. The correlation coefficient would be around negative 0.85. Remember, you should think of correlation as essentially about how well you can fit a straight line through the data, regardless of whether the relationship is positive or negative. So when x goes up, y goes down, and it goes down a roughly predictable amount. This correlation is as strong as the previous example. It's just called negative, like the slope of the line of best fit. Here's what a scatter plot would look like with no correlation, which means the correlation coefficient is equal to 0. This graph represents the number of TV episodes watched and the amount of water each person drank in a day. You can drink water whether you watch TV or not, so there doesn't appear to be any relationship. The line of best fit has a slope of 0, because as x goes up, y doesn't go up or down in any predictable way. So the line of best fit just predicts the mean value of y, no matter the x value. That's the best possible guess, and it's not very good. Calculating R by hand can get complicated, but computers nowadays make it much easier. Let's examine how to interpret R, which tells you two important things about the relationship between two features. First, the strength of the correlation. Here's a general guide for interpreting R based on its absolute value or distance from 0. Positive or negative values between 0 and 0.3 indicate a weak correlation. Between 0.3 and 0.7, positive or negative suggests a moderate correlation. Between 0.7 and 1, positive or negative indicates a strong correlation. Second, R tells you the direction of the correlation. A positive R means both features tend to increase or decrease together. A negative R means that as one feature increases, the other tends to decrease. Notice that interpreting R is a similar process to interpreting skew, which also has a strength and a direction. Here are three scatter plots. What do you think the R is for each? Here's a hint. See how they all look like a straight line? These each have an R of 1. You can perfectly fit a line to this data, and as one feature goes up, so does the other. The slope of the line is positive, no matter how big or how small, and the relationship is highly predictable. What about these three scatter plots? What R do you think they have? These each have an R of negative 1. You can fit a line perfectly through them. However, as X goes up, Y actually goes down, making the correlation negative. Here are two scatter plots with correlations of 0.8 and minus 0.8. Can you tell which is which? The one on the left is positive 0.8, while the one on the right is negative 0.8. Okay, last one. What's your guess for the correlation of this scatter plot? It's zero. There's no discernible relationship here, and the straight line here would just be flat. Great work studying correlations. It's always exciting to calculate a number that maps onto your intuition about the data. Join me in the next video to learn more about the difference between correlation and correlation. Thanks for watching.