What is it?
The formula for covariance is this:
Looking at this, let’s imagine for a moment that every value for dev(xi) and for dev(yi) is either +1 or -1. That means that the only possible values for the product dev(xi)×dev(yi) are also either -1 or +1:
they are +1 when xi and yi share the same sign
they are -1 when xi and yi have opposite signs
Overall, the covariance therefore will tend to be:
+ve when more than half the xi and yi have the same sign
-ve when more than half the xi and yi have opposite signs
zero when same sign and opposite sign are balanced
The sign of the covariance tells us what sort of relationship there is between x and y is:
+ve: when x increases, so does y
-ve: when x increases, y decreases
zero: when x increases, y is unchanged
The value of the covariance tells us something about how strong the relationship is. Consider this:
This says that yi is given by the value of the corresponding xi times b plus something else ei, where b is a constant (has the same for all i). This something else, ei, is how we are going to judge the strength of the relationship between x and y. If ei is typically much smaller than b×xi, then yi depends mostly on xi the relationship between x and y is strong. If ei is typically much larger than b×xi, then yi depends mostly on ei and the relationship between x and y is weak.
We are going to keep this simple, by saying that y and x have the same variance. This means that b now goes from -1 through 0 to +1.
We are interested in this quantity, which we will call a residual:
The variance of the residual is given by our formula from before:
We can now do something a bit sneaky. If we move the b outside of the cov(y,bx) and var(bx), then we have this:
So by re-arranging, we get:
But we also know that var(e) = var(y) – var(bx), so
which then simplifies to:
So, we reach this:
This has shown us how the covariance of two variables is a direct measure of how strong the relationship is between them.