Matrix Form of Regression Model

Finding the Least Squares Estimator

See Section 5 (Multiple Linear Regression) of Derivations of the Least Squares Equations for Four Models for technical details.

If you prefer, you can read Appendix B of the textbook for technical details.

Model:	y = Xβ + ε
Normal Equation:	X^TXβ^ = X^Ty (Obtained by computing vector derivative and setting to zero vector)
Estimated Parameters:	β^ = (X^TX)^-1 X^Ty (Least Squares Estimator)
Covariance of Parameters:	Cov(β^) = (X^TX)^-1(MSE), where MSE = SSE/DFE = SSE/(n-p)
Standard Deviations:	Square roots of diagonal elements of the covariance matrix.

Use the following values to express the horizontal regression model ( y_i = μ + ε_i ) in matrix form.
The mean of these y-values is 30.6.

Recall these parameter estimates for the SpringReg Example:

Model 1: Horizontal Line Regression
     
       Analysis of Variance
            Sum of   Mean
Source  DF  Squares  Square  
Model   1     75200    75200 
Error   4   3.86667  0.96667
U Total 5     75204
             Parameter Estimates
                           Parameter Standard
Variable  Label        DF  Estimate  Error
disp      Displacement  1  50.06667  0.17951
 

Model 2: Simple Linear Regression

       Analysis of Variance
            Sum of   Mean
Source  DF  Squares  Square  
Model    1    25200    25200
Error    3  3.60000  1.20000
C Total  4    25204 
             Parameter Estimates
                           Parameter Standard
Variable  Label        DF  Estimate  Error
Intercept Intercept     1  -0.40000  0.84853
disp      Displacement  1  50.20000  0.34641

Compute the regression parameters for the SpringReg Example using the matrix form of the regression model:
See the SAS source code and output of the MatrixSpring Example

Matrix Prerequisites

Matrix Prereqs: if A and B are matrices, v and w are vectors, and c is a scalar constant,
1. A(v + w) = Av + Aw
2. A c B = c AB
3. AI = IA = A
4. A A^-1 = A^-1 A = I
5. (AB)^-1 = B^-1 A^-1
6. (A^T)^T = A
7. (AB)^T = B^T A^T
8. (A^-1)^T = (A^T)^-1
9. A + 0 = 0 + A = A
10. A 0 = 0

Random Vector Prerequisites

Random Vector Prereqs: if x and y are random vectors, and v is a constant vector,
1. E(x + y) = E(x) + E(y)
2. E(Ax) = A E(x)
3. E(v) = v
4. Cov(x + v) = Cov(x)
5. Cov(Ax) = A Cov(x) A^T
We also use the fact that E(MSE) = σ²

Unbiased Property of Parameter Estimates

Assuming that the residuals are unbiased, (E(ε) = 0), the parameter estimates are also unbiased (E(β^) = β):

E(β^)	=	E((X^T X)^-1 X^T Y)	Matrix formula for β^
	=	E((X^T X)^-1 X^T (Xβ + ε))	Matrix regression model for y
	=	E((X^T X)^-1 X^T Xβ + (X^TX)^-1 X^Tε)	Property 1 of Matrix Prereqs
	=	E(β + (X^TX)^-1 X^Tε)	Property 4 of Matrix Prereqs
	=	E(β) + E((X^TX)^-1 X^Tε)	Property 1 of Random Vector Prereqs
	=	E(β) + (X^TX)^-1 X^T E(ε)	Property 2 of Random Vector Prereqs
	=	E(β) + (X^TX)^-1 X^T 0	ε is unbiased
	=	E(β) + 0	Property 10 of Matrix Prereqs
	=	E(β)	Property 9 of Matrix Prereqs
	=	β	Property 3 of Random Vector Prereqs

Thus, β^ is unbiased for β.

Covariance Matrix of Parameter Estimates

Assuming that the residuals are homoscecastic and uncorrelated (Cov(ε) = σ² I), we derive the covarance matrix of β^.

Cov(β^)	=	Cov((X^T X)^-1 X^T Y)	Matrix formula for β^
	=	Cov((X^T X)^-1 X^T (Xβ + ε))	Matrix regression model for y
	=	Cov((X^T X)^-1 (X^T X)β + (X^T X)^-1 X^Tε)	Property 1 of Matrix Prereqs
	=	Cov(β + (X^T X)^-1 X^Tε)	Property 4 of Matrix Prereqs
	=	Cov((X^T X)^-1 X^Tε)	Property 4 of Random Vector Prereqs
	=	(X^T X)^-1 X^T Cov(ε) ((X^T X)^-1 X^T)^T	Property 5 of Random Vector Prereqs
	=	(X^T X)^-1 X^T Cov(ε) (X^T)^T ((X^T X)^-1)^T	Property 7 of Matrix Prereqs
	=	(X^T X)^-1 X^T Cov(ε) X ((X^T X)^-1)^T	Property 6 of Matrix Prereqs
	=	(X^T X)^-1 X^T Cov(ε) X ((X^T X)^T)^-1	Property 8 of Matrix Prereqs
	=	(X^T X)^-1 X^T Cov(ε) X (X^T (X^T)^T)^-1	Property 7 of Matrix Prereqs
	=	(X^T X)^-1 X^T Cov(ε) X (X^T X)^-1	Property 6 of Matrix Prereqs
	=	(X^T X)^-1 X^T σ² I X (X^T X)^-1	ε is homoscedastic and uncorrelated
	=	σ²(X^T X)^-1 X^T I X (X^T X)^-1	Property 2 of Matrix Prereqs
	=	σ²(X^T X)^-1 X^TX (X^T X)^-1	Property 3 of Matrix Prereqs
	=	σ²(X^T X)^-1	Property 4 of Matrix Prereqs

^-1

The Hat Matrix

Since β^ = (X^T X)^-1 X^T y and y^ = Xβ^, y^ = X (X^T X)^-1 X^T y.
Define H = X (X^T X)^-1 X^T.
Because H puts a hat on y, H is called the hat matrix: (y^ = Hy)
Properties of H and I - H:
The symmetric and itempotent properties are important, because they make H a perpendicular projection matrix that projects the observation vector y from the n-dimentional observation space into the p dimensional parameter space.
These properties also make I - H a projection matrix that projects y perpendicularly onto the n - p dimensional residual space.
We can use the symmetric and itempotent properties of H to find the covariance matrix of y^:
As usual, we use the MSE to estimate σ² in the expression for the covariance matrix of y^:
The square roots of the diagonal elements of Cov(y^) give us the estimated standard errors of the predicted values.
A predicted value y_i^ and the standard error of that predicted value SE(y_i) can be used to compute a confidence interval for the true or expected value of y at that point:

Confidence and Prediction Intervals

Define a to be a vector of new independent variable settings:
The predicted value y^, given the new independent variable settings a:
The 100(1 - α)% Confidence Interval for y^ for the new independent variable settings a is
The 100(1 - α)% Prediction Interval for a new y-value for the independent variable settings a is
Here is a SAS gplot graph of the 95% confidence and prediction intervals for the Bears dataset. Here is the SAS source code.
See Page 8 the SAS output for the SpringMatrix data for 95% confidence and prediction intervals at the values of the independent variable.
Practice Problem: Suppose that a new hamster, living under the conditions of the hamsters described in the Hamster dataset, hibernates 20% of the time. Find
1. The predicted value of Lifetime, given the independent variable setting Hiber=20.
2. A 95% confidence interval for the predicted value in part a.
3. A 95% prediction interval for a new observation with Hiber=20.
Obtain MSE and (X^T X)^-1 from the SAS output in the Hamster Example.