To Lecture Notes

CSC 423 -- 7/27/16

Review Questions

1. What are the test statistics for (a) the one-sample t-test, (b) the paired sample t-test, and (c) the independent two-sample t-test? For this last test, see The Independent Two-sample t-test, for the test when the variances of the two groups are assumed to be equal. See Question 2 when the variances are not assumed to be equal. Ans:
   (a) t = (x - μ) / SE_xbar
   (b) if d_i = x_1i - x_2i , t = (d - 0) / SE_dbar
   (c) See the Independent Two-sample t-test document
    
2. What is the Welsh-Satterthwaite test?
   Ans: The Welch-Satterthwaite test (1947) is the independent 2-sample t-test when the variances of the two groups are not assumed to be equal. The Indepent two-sample t-test with pooled variance is used when the variances are assumed to be equal.  Welch-Satterthwaite test uses an approximation with fractional degrees of freedom. The Behrens-Fisher Problem is the problem of finding a good independent 2-sample t-test when the variances of the two groups are not assumed to be equal. The Welsh-Satterthwaite test is generally accepted to be the best solution to the Behrens-Fisher Problem, there are other tests as well. These include the test by Chapman (1950), the Prokof’yev-Shishkin Test (1974), and the Dudewicz-Ahmed Test (1998).
    
3. For a regression model, what is the difference between y_i and y_i^{^}?   between μ and μ^{^}?  In other words, what does the hat ( ^ ) mean?
   Ans: The hat means "estimated": μ^{^} is the estimated value of the unknown parameter μ, y_i^{^} is the estimated value for the ith value of the independent variable.
    
4. What is a least squares estimator (LSE) for a regression model?
   Ans: It is the regression equation that minimizes the sum of squares for error (SSE) over all possible regression equations. TO find the LSE, rhe sum of the squares of the residuals (SSE) is differentiated with respect to each of the parameters being estimated. Each of these expressions is then set to 0 and the set of simultaneous equations is solved for the parameters. We discussed the two simplest cases of horizontal line regression and regression through the origin in class. See the document Derivations of the LSE for Four Regression Models. You don't need to know the derivations for the final exam, only the resulting regression equations in Exercise 5.
    
5. What are the estimated regression equations for (a) horizontal line regression, (b), regression through the origin, and (c) simple straight line regression. Use the LSE in each case.
   Ans: See Sections 2, 3, and 4 in the Technical Details document: Derivations of the LSE for Four Regression Models. Here are the three regression equations:
   
   1. Horizontal Regression Equation: y = y
   2. Regression through the Origin: y = a^{^}x, where a^{^} = (x₁y₁ + ... + x_ny_n) / (x₁² + ... + x_n²).
   3. Simple Linear Regression: y - y = (r_xy s_y / s_x) (x - x)
   
   
6. What does the R² value tell you about a regression model?
   Ans: The larger R², the better the regression model. R² tells you the fraction of variation of the dependent variable that is due to the variation in the independent variables.
    
7. What are the requirements for a good regression model?
   Ans: A good regression model must be accurate (have a large R² value), but it must also be parsimonious (as few independent variables as possible). In addition, its residuals must be well behaved (unbiased, homoscedastic, normal).
    
8. Use SAS or R to compute x, y, s_x, s_y, and r for the following bivariate dataset:
    
   

   x:	1	2	3	4
   y:	1	3	2	4

   
   Then use these statistics to compute the simple straight line regression equation and the regression through the origin equation for this dataset.
   Ans: Here are the R and SAS scripts. The computed statistics are
   
   x = 2.5   y = 2.5   s_x = 1.290994   s_y = 1.290994   r_xy = 0.8.
   Then sub these values into the simple linear regression equation:
   
   y - y = (r_xy s_y / s_x) (x - x)
   y - 2.5 = (0.8 * 1.290994 / 1.290994)(x - 2.5)
   y - 2.5 = 0.8 * x - 0.8 * 2.5
   y = 0.8x + 0.5
   
   Here is the calculation of the regression through the origin equation:
   
   a^{^} = (sum of x_i y_i) / (sum of x_i²)
   = (1 · 1 + 2 · 3 + 3 · 2 + 4 · 4) / (1² + 2² + 3² + 4²) = 0.9666667,
   so that the regression through the origin equation is y = 0.9666667 x.
    
9. Add an options statement to the following SAS script that will modify the listing (typewriter output) as follows:
   
   1. Suppress the date,
   2. Suppress "The SAS System",
   3. Start numbering the pages at 1,
   4. Set the page width to 70 characters,
   5. Use only keyboard characters.

   data test;
      do i = 1 to 100;
         x = rand("normal");
         output;
      end;
      
   proc means n mean std p1 p5 p10 
      p25 p50 p75 p90 p95 p99;
   

   Hint: use this option to force only keyboard characters to be used in the output:
   

   formchar = '|----|+|---+=|-/<>*';
   

   Ans: Add this options statement to the top of the SAS script:
   

   options linesize=70 nodate pageno=1 
      formchar = '|----|+|---+=|-/<>*';
   

   Also add the statement
   

   title "";
   

   to the data step or proc for which you want to suppress the default title.

More About Linear Regression

• The data for regression can be experimental or observational.
   
• Experimental data are generated by designed experiments where the values of the independent variables are controlled or planned in advance. With human or animal subjects, the subjects are randomly assigned to treatment groups in a planned experiment.
   
• Observational data are collected as-is and not specified in advance. Observational data are more likely to be affected by confounding factors, which are independent variables that are unknown or, at least, not included in the model.
   
• Here is an article that discusses some of the dangers of using observational studies:
   
  
  Young and Karr, Deming, data and observational studies. A process out of control and needing fixing
  
  
• The F-test for Linear Regression and Adjusted R² (R²)
   
• The overall F-test is used to test whether any of the independent variables are significant in a regression model. The null hypothesis for the overall F-test is that all of the regression parameters are zero except for the intercept parameter β₀.
   
• Locate the following for the Hamster Example on the SAS output:
   
  
  SSM   SSE   SST   DFM   DFE   DFT
  MSM   MSE   F   p-value for F
  
  
  Ans for SAS:
  SSM = 3378694; SSE = 8922952; SST = 12301646;
  DFM = 1; DFE = 134; DFT = 135;
  MSM = 3378694; MSE = 66589;
  F = 50.74; p-value < 0.0001
   
  Ans for R:
  Read these values directly from the R summary(model) output:
  DFM = 1; DFE = 134; F = 50.74; p-value = 5.843e-11 = 5.843 × 10^-11
  SSM, SSE, MSM, and MSE can be computed using R like this:
  

  ssm = sum((predict(model) - mean(hamster$Lifetime))^2)
  sse = sum(resid(model)^2)
  msm = ssm / 1
  mse = sse / 134
  cat("SSM = ", ssm, " ", "SSE = ", sse, "\n")
  cat("MSM = ", msm, " ", "MSE = ", mse, "\n")
  Output: 
  SSM =  3378694   SSE =  8922952 
  MSM =  3378694   MSE =  66589.19 
  

• Look at the CPUReg2 Example (1982 Dataset)
   
  
  1. Obtain the parameter estimates and standard errors of the complete regression model with all regressors CardsIn, LinesOut, Steps, and MountedDevices.
      
  2. Omit the regressor with the least significant independent variable and obtain the resulting model.
      
  3. Create the residual plot and normal plot of the residuals for the resulting model.
      
  4. Omit the regressor with the next least significant independent variable and obtain the resulting model.
      
  5. Create the residual plot and normal plot of the residuals for the resulting model.
  
  
• Locate the following for the CPUReg2 Example on the SAS and R outputs, as we did with the Hamster dataset.
   
  
  SSM   SSE   SST   DFM   DFE   DFT
  MSM   MSE   MST   F   p-value for F
   
  Ans for SAS:
  SSM = 0.59705; SSE = 0.04067; SST = 0.63772;
  DFM = 3; DFE = 34; DFT = 37;
  MSM = 0.19902; MSE = 0.00120;
  F = 166.38; p-value = 2.2e-16
   
  Ans for R:
  Read these values directly from the R summary(model) output:
  DFM = 3; DFE = 34; F = 166.4; p-value = 2.2e-16 = 2.2 × 10^-16
  Compute SSM, SSE, MSM, and MSE as we did in the Hamster Example above:
  

  ssm = sum((predict(model2) - mean(cpu$CpuTime))^2)
  sse = sum(resid(model2)^2)
  msm = ssm / 1
  mse = sse / 134
  cat("SSM = ", ssm, " ", "SSE = ", sse, "\n")
  cat("MSM = ", msm, " ", "MSE = ", mse, "\n")
  Output: 
  SSM =  0.5970542   SSE =  0.04067083 
  MSM =  0.1990181   MSE =  0.001196201 
  

The Adjusted R² Value

• Look at the The R² and Adjusted R² Values section of The F-Test for Linear Regression document.
   
• Practice Problem:   Compute the R-squared and adjusted R-squared values for the regression model if
   
  
  SSE = 94.82   SSM = 23.34   n = 43   p = 5
  
  Ans: R² = 1 - SSE/SST = 1 - SSE/(SSE + SSM) = 1 - 94.82/(94.82+23.34) = 0.198
  R² = 1 - (1 - R²)(n-1)/(n-p) = 1 - (1 - 0.198)(42/38) = 0.114.

Influence Points

• In some cases, a single point, usually an outlier in the independent variables, can drastically change the estimated regression parameters of a bivariate dataset.
   
• Example: this graph shows the effect of an influence point on the regression line. The regression line with the influence point included is the solid red line; the regression line with the influence point omitted is the dotted red line. Here is the R Script that produced this graph.
   
• A point that almost single-handedly determines the correlation or the regression model is called an influence point, also called a leverage point, although some authors reserve the term leverage point to refer to points with a large leverage statistic h_ii (see table below).
   
• The following table summarizes popular measures of influence. See Influence Points and Multicollinarity for more details.
   
  

  Measures of Influence
  Statistic	Symbol	Cutoff	Description
  Externally Studentized Residual	zi*	2 to 3	z-score of residual, where the mean and SD are computed with the observation deleted
  Leverage Statistic, or Hat-value	hii	0.2 to 0.5 or 2p/n	Diagonal values of the hat matrix H 0 = no influence, 1 = completely determines the model.
  Cook's Distance D	Di	1 or 4/n	Measures the average effect of deleting the observation.
  DFBETAS Distance	|DFBETASij|	2 / √n	Change in each β^ when deleting the ith observation.
  DFFITS Distance	|DFFITSij|	2√p / n	zi*√[hii / (1 - hii)]

   
  n is the number of observations; p is the number of independent variables.
   
• The most intuitive influence measure is the externally studentized residual.
   
• Another popular influence measure is the leverage statistic, although it often disagrees with the other measures.
   
• Don't automatically delete influence points; they might be the most important points in the dataset.
   
• R automatically marks what it considers to be influence points with asterisks ( * ).
   
• Sales Example

Multicollinarity

• Multicollinarity occurs when at least one independent variable is highly correlated with a linear combination of the other independent variables.
   
• Here are the two most commonly used measures of multicollinarity:
   
  

  Measures of Multicollinarity
  Statistic	Symbol	Cutoff
  Tolerance	1 - Rj2	< 0.2
  Variance Inflation Factor	VIF = 1 / (1 - Rj2)	> 5.0

  
  
• R_j is computed like this:
   
  
  1. Consider the jth predictor as the dependent variable,
      
  2. find the linear regression model for predicting x_j from all the other predictors.
      
  3. R_j is the R-squared value of the regression model in Step 2.
  
  
• Sales Example

Regresson Diagnostic Plots

• SAS contains the Output Delivery System (ODS) that produces suites of diagnostic plots with a minimum of source code.
   
• In conjuction with proc reg, ODS produces a suite of eleven regression diagnostics plots. See the SAS column of the Diagnostics Example.
   
• These ODS plots are automatically created for proc reg in SAS 9.4.
   
• R can also produce regression diagnostic plots.  
   
• The model object returned from the lm function can be plotted with the plot function to produce six regression diagnostic plots.
  Use plot(model) to get six regression and diagnostic plots.  See the R column of the Diagnostics Example.