To Lecture Notes

CSC 423 -- 7/31/17

Review Exercises

1. What is an R package? How do you install an R package?
   Ans: An R package is a collection of R functions, data, and compiled code that performs a task useful to R users. You load an R package with the install.packages function. For example, to install the leaps package:
   

   install.packages("leaps")
   

2. If x is a random variable, what does E(x) mean?
   Ans: E(x) is the expected value of x, which is the long run average of the outputs of x. If x is a discrete random variable defined by a probability distribution function like this:
    
   

   Outcome	Probability
   x1	P(x1)
   x2	P(x2)
   ...	...
   xn	P(xn)

   
   The expected value of x is defined by
    
   
   E(x) = Σ_i=1ⁿ x_i P(x_i)
   If p is the probability density of a continuous random variable x, compute the expected value with this integral:
    
   
   E(x) = ∫_i=-∞^∞ t p(t) dt
   
   
3. What does the term unbiased mean?
   Ans: A sample statistic T is unbiased for a population parameter θ if E(T) = θ. Here are four examples of unbiased statistics that we have seen:
   
   1. Sample mean: E(y) = μ (The sample mean is unbiased for the population mean.)
   2. Sample Variance: E(s_y²) = σ² (The sample variance is unbiased for the population variance.)
   3. Estimated Regression Parameter: E(β_j^{^}) = β_j (Each estimated regression parameter is unbiased for the true population parameter.
   4. Estimated Value: E(y_i^{^}) = y_i
   a, b, c, and d are based on the assumption that the residuals are unbiased: E(ε_i) = 0.
    
4. What is a residual plot? a partial residual plot?
   Ans: It is a plot of the residuals plotted vs. one of the independent variables. If x is an independent variable here are the SAS and R
   

   * In SAS proc reg plot statement;
   plot r.*x;
   
   # R plot statement:
   r = resid(model)
   plot(x, r)
   

5. What are the definitions of these statistics?
    
   
   SSM   SSE   SST   DFM   DFE   DFT
   
   Ans:
   SSM = Sum of Squares for Model = Σ_i=1ⁿ (y_i^{^} - y)²
   SSE = Sum of Squares for Error = Σ_i=1ⁿ (y_i - y_i^{^})²
   SSE = Total Sum of Squares = Σ_i=1ⁿ (y_i - y)²
   DFM = Degrees of Freedom for Model = p - 1
   DFE = Degrees of Freedom for Error = n - p
   DFT = Total Degrees of Freedom = n - 1
    
6. What is the overall F-test for regression tell you?
   Ans: The overall F-test tests the null hypothesis that all of the non-intercept regression parameters are equal to zero:
    
   
   H₀: β₁ = β₂ = ... = β_p-1
   
   The test statistic is
    
   
   F = (SSM / DFM) / (SSE / DFE) = MSM / MSE
   
   Look up the confidence interval for the F statistic, assuming the null hypothesis, in an F table with (DFM, DFE) degrees of freedom.
    
7. Given these statistics for a regression problem:
    
   
   n = 18   p = 4   SSM = 59.1   SSE = 130.8
   
   Should you accept or reject the null hypotheses for the overall F-test:
    
   
   H₀: β₁ = β₂ = ... = β_p-1?
   
   Ans: DFM = p - 1 = 4 - 1 = 3, DFE = n - p = 18 - 4 = 14,
    
   
   F = (SSM / DFM) / (SSE / DFE) = (59.1 / 3) / (130.8 / 14) = 19.7 / 9.343 = 2.109
   
   The 95% confidence interval for the F statistic with (3, 14) degrees of freedom is (0, 3.344), so accept the null hypothesis.
    
8. For the regression model in Question 4, compute R² and R²
   Ans: R² = SSM / (SSM + SSE) = 59.1 / (59.1 + 130.8) = 0.311. Then
   R² = 1 - (1 - R²)((n - 1) / (n - p)) = 1 - (1 - 0.311)(17 / 14) = 0.163.
    
9. What are some SAS options for the model statement of proc reg?
   Ans: influence noint tol vif. Today, we will see the options p and r.
    
10. What is the difference between a standardized residual and an externally studentized residual?
    Ans: A standardized variable is another name for the z-score of a variable:
    z_i = (x_i - x) / s_x, where x and s_x are computed from the entire dataset. For the externally studentized variable, the sample mean and standard deviation are computed from the dataset with the ith observation deleted:
    z_i^* = (x_i - x_(i)) / s_x,(i). In the case of externally studentized residuals:
    z_i^* = (ε_i - ε_(i)) / s_ε,(i).
     
11. What is an influence point? List five measures of influence?
    Ans: It is an observation that has an unusually high influence on the regression model; if the observation is deleted from the dataset, the regression model changes dramatically.
     
12. What is multicollinarity? List two measures of multicollinarity?
    Ans: Multicollinarity exists in a multiple regression model if at least one of the independent variables is highly correlated with another independent variable or with a linear combination of some of the other independent variables. Two measures of multicollinarity are the tolerance and the variance inflation factor (VIF).

Using R² to Pick the Best Regression Model

• One way to choose the best independent variables to use in a regression model is pick the model with the largest adjusted R-squrared (R²) value.
   
• Look at the CPUReg3 Example.

Matrix Form of the Regression Model

• Look at the CrudeOil Example, which explains how to set up a regression model in matrix form:
   
  
  y = X^Tβ + ε.
  
  The least squares estimator for this regression equation is
   
  
  β^{^} = (X^TX)^-1 X^Ty.

Confidence and Prediction Bands

• Even though we think of the regression equation
   
  
  y = β₀^ + β₁^ x₁ + ... + β_p-1^ x_p-1
  
  as the best equation for predicting the dependent variable from the independent variables, this equation is actually a random equation because the estimated parameters are random variables.
   
• Although the true regression parameters are unknown, they are unbiased if the residuals are unbiased.
   
• If the residuals are also normal and homoscedastic, SAS and R can compute a confidence interval for the "true" value of y for values of the independent variables not necessarily in the original dataset. The "true" value of y is the expected value of y for given values of the independent variables.
   
• SAS and R can also compute a confidence interval for a new observation. This confidence interval is always wider than the confidence interval for the true value of y because it takes into account the standard deviation of the new observation.
   
• Some SAS proc reg options for obtaining information about residuals and predicted values:
   
  

  Statistic	Proc Reg Option
  Residual	r
  Predicted Value	p
  Confidence Band for True Predicted Value	clm
  Predicted Value of New Observation	cli

  
  
• Look at the Prediction Example.
   
• The SAS plots of 95% confidence and prediction intervals for two examples:
   
  
  Bears   Hamster
  
  
• Although the confidence and prediction bands produced by the SAS proc reg options clm and cli are valid for multiple regression models, graphs showing this data is only possible works for models with one independent variable.
   
• Computational Details for Confidence and Prediction Bands
   
• Practice Problem:
   
  
  1. Using the CrudeOil Dataset, find 99% confidence intervals for the true predicted values at the data points.
      
  2. Find a 99% prediction interval for the new observation at pressure=1700, angle=10.

• Even a single outlier can drastically change the correlation of a bivariate dataset.
   
• Conduct an R simulation to see how a single outlier can drastically affect the correlation of an otherwise uncorrelated dataset. See the InfluenceSimulation Example.
   
• A point that almost single-handedly determines the correlation or the regression model is called an influence point.
   
• The situation where an independent variable is highly correlated with another independent variable or with a linear combination of several independent variables.
   
• These tables summarize popular measures of influence and multicollinarity. See Influence Points and Multicollinarity for more details.
   
  

  Measures of Influence
  Statistic	Symbol	Cutoff	Description
  Deleted Studentized Residual	zi*	2 to 3	z-score of residual, where the mean and SD are computed with the observation deleted
  Leverage Statistic, or Hat-value	hii	0.2 to 0.5 or 2p/n	Diagonal values of the hat matrix H 0 = no influence, 1 = completely determines the model.
  DFBETAS Distance	|DFBETASij|	2 / √n	Change in each βi^ when deleting the ith observation.
  DFFITS Distance	|DFFITSij|	2√p / n	Change in scaled predicted value when deleting the ith observation.
  Cook's D	Di	4/n	Scaled average squared difference of predicted values with and without the ith observation. The scale factor is 1 / (p MSE)

   
  n is the number of observations; p is the number of independent variables.
   
• Although there are many difference ways to measure influence, most all of them are based on two ideas: leverage and the jackknife:
   
  
  1. The leverage of the ith observation for the predicted value y_i^ is defined as the value of h_ii in the linear equation
      
     
     y_i^ = h_i1y₁ + h_i2y₂ + ... + h_iiy_i + ... + h_iny_n
     
     where the h_1i, h_2i, ... ,h_ii, ... , h_ni are the entries of the ith row of the hat matrix H = X^T (X^T X)^-1 X.
     h_ii tells how much influence the actual ith value y_i has on the ith predicted value y_i^. It is h_ii, the ith diagonal element of the hat matrix H. The leverage statistic h_ii often disagrees with other influence measures.
      
     The hat matrix H = X (X^T X)^-1 X^T computed from from the CrudeOil Dataset is
     

     /  0.444  0.278  0.111  0.278 0.111 -0.056  0.111 -0.056 -0.222 \
     |  0.278  0.278  0.278  0.111 0.111  0.111 -0.056 -0.056 -0.056 | 
     |  0.111  0.278  0.444 -0.056 0.111  0.278 -0.222 -0.056  0.111 | 
     |  0.278  0.111 -0.056  0.278 0.111 -0.056  0.278  0.111 -0.056 | 
     |  0.111  0.111  0.111  0.111 0.111  0.111  0.111  0.111  0.111 | 
     | -0.056  0.111  0.278 -0.056 0.111  0.278 -0.056  0.111  0.278 | 
     |  0.111 -0.056 -0.222  0.278 0.111 -0.056  0.444  0.278  0.111 | 
     | -0.056 -0.056 -0.056  0.111 0.111  0.111  0.278  0.278  0.278 | 
     \ -0.222 -0.056  0.111 -0.056 0.111  0.278  0.111  0.278  0.444 /

     Verify that the diagonal values of H are the leverage statistics h_ii. Here is the R script for computing the hat matrix H.
      
  2. A jacknife estimate is an estimate that is computed from the data with ith observation deleted. That estimate is then compared to the estimator obtained using the entire dataset. Some examples of jackknife statistics are:
      
     
     DFBETAS_ij is computed for each estimated parameter. It indicates how much change there is between the estimated parameter for the dataset and the jackknife estimate of the parameter (with the ith observation deleted). The ith observation is considered to be influential with this measure if |DFBETAS_ij| > 2 / √n
      
     DFFITS_i is the change in the predicted value y_i^ if the ith observation is omitted. The jackknife estimate of the predicted value is denoted by y_(i)^. The ith observation is considered to be influential with this measure if |DFFITS_i| > 2 √p / n
      
     Cook's D statistic is another popular jackknife influence statistic. Use the proc reg model option r for obtaining the residuals, not the influence option, to obtain Cook's D.
      
     
     D_i = [1/(p MSE)] Σ_j=1ⁿ (y_j^ - y_j(i)^)²
     
     
• The most intuitive influence measure is the deleted studentized residual.
   
• Don't automatically delete influence points; they might be the most important points in the dataset.
   
• R automatically marks what it considers to be influence points with asterisks ( * ). Use the R function call
  

  summary(influence.measures(model))
  

  to show which influence measures R considers to be influential, for points that are marked as influence points using
  

  influence.measures(model)
  

• Possible remedies for outliers or influence points:
   
  
  1. Check for errors in the influential observations.
      
  2. Check if important independent variables have been omitted from the model.
      
  3. Transform the dependent variable with a log or square root transform.
      
  4. Collect more data with independent variables close to those of the influence points.
  
  
• Look at the BodyBrain Example checking for influence points

Dummy Variables

• A dummy variable is an independent variable in a regression model that takes on only values 0 and 1.
   
• Dummy variables are used to represent a categorical variable.
   
• For the WLoss3 Example, the categorical variable is center (weight loss center).  The levels of center are A, B, C, D, and E.  We assign four dummy variables a, b, c, and d like this:
   
  

  Levels of center	a	b	c	d
  A	1	0	0	0
  B	0	1	0	0
  C	0	0	1	0
  D	0	0	0	1
  E	0	0	0	0

  
  
• Level E is the base level because it has no associated dummy variable.
   
• If a categorical variable has k levels, then only k - 1 dummy variables are needed.
   
• Only k - 1 dummy variables are used for a categorical variable with k levels to allow the model to be solved for the parameter estimates.
   
• Sales Example
   
• Movies Example

Project 3

• Discuss Project 3.

Stepwise Regression and Model Validation

• Model Selection: All Subsets, Backward, Forward, and Stepwise Selection. Here are some details about model selection criteria.
   
• PropVal1 Example
   
• PropVal2 Example
   
• External Model Validation
   
• To split a dataset into training and test sets, you can use this SAS or R code. Use the propval.txt dataset.
   
• PropVal3 Example
   
• Crime Example
   
• GLMSelect Example
   
• A commonly used jackknife statistic for validating a model is the PRESS statistic, which is the leave-one-out sum of squares:
   
  
  PRESS = Σ_i=1ⁿ (y_i - y_(i)^{^})²
  
  y_(i)^{^} is the predicted value obtained from the regression model, which leaves out the ith observation.
   
• SAS automatically computes PRESS when the r option is used for the proc reg model statement.
   
• R computes the PRESS statistic with the PRESS function in the qpcR package. Supply the lm model object as the input argument to PRESS.
   
• As a rule of thumb, the PRESS statistic indicates a good cross-validated regression model if PRESS < 1.5 or 2 SSE.
   
• We will see how to get SAS and R to compute the PRESS statistic on Wednesday, August 3.

Final Project

• Sample Power Point Presentation for Final Project: red-wine.zip.

A Quadratic Model

• Sometimes a residual plot or partial residual plot will show a biased pattern where the expected values of the residuals follow a parabolic curve.
   
• In this case, the fit of the model may be improved by including a quadratic term (x_j*x_j).
   
• See the Olympic Example.  The quadratic term fat * fat is added to the model.  The variable fat represents the daily fat intake of olympic athletes

Interaction Terms

• A commonly used regression model is the additive model:
   
  
  y = β₀ + β₁ x₁ + ... + β_jx_j + ... + β_p-1 x_p-1
  
  If all independent variables are held constant except x_j, the change c in x_j produces the change c β_j in y.
   
• Sometimes two or more independent variables interact with each other. In this case the amount of change in y due to a change in x_j (all other independent variables are held equal) may depend on the value of a third variable. There is called a two-way interaction between x_j and the third variable.
   
• In the GFClocks Example, the dataset gfclocks.txt contains the prices at auction of grandfather clocks (dependent variable).  The independent variables are the age of the clock and the number of bidders.  There is a two-way interaction between the variables age and nbids.  This is modeled by adding the term age * nbids.

Transformations

• The Pendulum Example was not discussed in class. However, the BodyBrain Example was discussed in class.
   
• The Pendulum Example is an interesting example of using transformations in regression models because we know the true regression equation from physics: the equation that predicts the period per of a pendulum given its length len is
   
  
  per = 2π √len / g = (2 * 3.14159265 / √9.80665) * sqrtlen + 0 = 2.00064 * sqrtlen + 0
  
  
• If the residuals are heteroscedastic, one remedy is to use a data transformation of the dependent variable, less often the independent variable. Try one of these transformations in this order:
   
  
  1. Log Transform:   u = log(y)   (only if y is non-zero)
      
  2. Square Root Transform:   u = sqrt(y)   (only if y is positive)
      
  3. Logit Transform:   u = log(y / (1 - y))
      
  4. Reciprocal Transform:   u = 1 / y,   only if y ≠ 0
      
  5. The Box-Cox Power Transform:
      
     
     u = [(y + c)^λ - 1] / λ,   if λ ≠ 0
      
     u = log(y + c),   λ = 0
     
     The constant c is chosen so that y + c is positive for all y-values.
  
  
• Look at the residual plot of the transformed model to see if the residuals have become homoscedastic and normal.
   
• Use a log or square root transform if a variable varies by many orders of magnitude from low values to high values.
   
• If you are interested, read the Pendulum Example. Here is a summary table of the SAS or R output:
   
  

  Model	Independent Variable	Parameter Estimates	Standard Error	95% Confidence Interval
  Linear	Intercept	0.42081	0.02494	(0.36842, 0.47320)
  	len	2.12796	0.08195	(1.95578, 2.30014)
  Quadratic	Intercept	0.29426	0.01204	(0.26886, 0.31965)
  	len	3.48676	0.10392	(3.26751, 3.70601)
  	lenlen	-2.54743	0.18924	(-2.94670, -2.14816)
  Square Root	Intercept	0.01165	0.00621	(-0.00141, 0.02471)
  	sqrtlen	1.98756	0.01203	(1.96228, 2.01285)
  Log-Log	Intercept	0.68005	0.00702	(0.66530, 0.69480)
  	loglen	0.48618	0.00402	(0.47773, 0.49463)

  
  The physics formula for the period per of a pendulum with length len is
   
  
  per = 2π √len / g = (2 * 3.14159265 / √9.80665) * sqrtlen + 0 = 2.00064 * sqrtlen + 0
   
  The value of 2.00064 is within the 95% confidence interval for the parameter of sqrtlen. The true intercept 0 is also within the confidence interval for the Intercept parameter.
   
  Now suppose we don't know that we need to take the square root in the physics formula.  Use a log-log model (take logs of both the independent and dependent variables).  Locate the confidence intervals for the Intercept and the parameter for loglen.  They are (0.66530, 0.6948) and (0.47773, 0.49463).  The true regression equation is obtained by taking the log of both sides of the physics equation above:
   
  
  log per = log(2π √len / g) = [log(2π) - (1/2) log g)] + (1/2) loglen = 0.5 * loglen + 0.69635.
  
  The true values of 0.69635 and 0.5 of the regression parameters are barely outside of the confidence intervals.  Presumably, if the experiment were conducted with more precision, these true values would lie in the confidence intervals as well.
   
• Transforms can also ameliorate the effects of influence points, which we will discuss later.
   
• BodyBrain Example
   
• What is the predicted brain weight for an animal with body weight 50 kg?
   
  
  1. Use Model 1: the untransformed model.
      
  2. Use Model 2: the log-log model.