To Lecture Notes

CSC 423 -- 8/17/15

Review Exercises

1. Here is a hypothetical dataset. x_i is the distance from the basket that a shot is taken in a basketball game, y indicates whether the shot is made (1=made, 0=missed).
   

   x:	2	4	8	10	12	14	16	18	20	22
   y:	1	1	0	1	1	0	0	0	0	0

   
   Construct a logistic regression model for predicting the probability of making the shot from a given distance.
    
2. In SAS, how are proc reg and proc glm similar and dissimilar?
    
3. Why are residuals difficult to interpret in logistic regression?
    
4. What is an interaction term? How do you implement an interaction term in a regression model.

The Pendulum Example

• If the residuals are heteroscedastic, one remedy is to use a data transformation of the dependent variable, less often the independent variable. Try one of these transformations in this order:
   
  
  1. Log Transform:   u = log(y)   (only if y is positive)
      
  2. Square Root Transform:   u = sqrt(y)   (only if y is non-negative)
      
  3. Logit Transform:   u = log(y / (1 - y)) (only if y is between 0 and 1)
  
  
• Look at the residual plot of the transformed model to see if the residuals have become homoscedastic and normal.
   
• Use a log or square root transform if a variable varies by many orders of magnitude from low values to high values.
   
• Consider a log or square root transform if a variable is skewed to the right.  Use a log transform if the variable is extremely skewed; use a square root transform if the variable is moderately skewed.
   
• Discuss the Pendulum Example. Here is a summary table of the SAS or R output:
   
  

  Model	Independent Variable	Parameter Estimates	Standard Error	95% Confidence Interval
  Linear	Intercept	0.42081	0.02494	(0.36842, 0.47320)
  	len	2.12796	0.08195	(1.95578, 2.30014)
  Quadratic	Intercept	0.29426	0.01204	(0.26886, 0.31965)
  	len	3.48676	0.10392	(3.26751, 3.70601)
  	lenlen	-2.54743	0.18924	(-2.94670, -2.14816)
  Square Root	Intercept	0.01165	0.00621	(-0.00141, 0.02471)
  	sqrtlen	1.98756	0.01203	(1.96228, 2.01285)
  Log-Log	Intercept	0.68005	0.00702	(0.66530, 0.69480)
  	loglen	0.48618	0.00402	(0.47773, 0.49463)

  
  The physics formula for the period per of a pendulum with length len is
   
  
  per = 2π √len / g = (2 * 3.14159265 / √9.80665) * sqrtlen + 0 = 2.00064 * sqrtlen + 0
   
  The value of 2.00064 is within the 95% confidence interval for the parameter of sqrtlen. The true intercept 0 is also within the confidence interval for the Intercept parameter.
   
  Now suppose we don't know that we need to take the square root in the physics formula.  Use a log-log model (take logs of both the independent and dependent variables).  Locate the confidence intervals for the Intercept and the parameter for loglen.  They are (0.66530, 0.6948) and (0.47773, 0.49463).  The true regression equation is obtained by taking the log of both sides of the physics equation above:
   
  
  log per = log(2π √len / g) = [log(2π) - (1/2) log g)] + (1/2) loglen = 0.5 * loglen + 0.69635.
  
  The true values of 0.69635 and 0.5 of the regression parameters are barely outside of the confidence intervals.  Presumably, if the experiment were conducted with more precision, these true values would lie in the confidence intervals as well.
   
• Transforms can also ameliorate the effects of influence points.

More about Logistic Regression

• Chi-squared statistics
   
  
  Sections 9, 10, 11

One-way ANOVA

• ANOVA means Analysis of Variance.
   
• Some names for a categorical regression parameter are factor, grouping variable, discrete independent variable, or treatment. We use the term factor.
   
• A one-way ANOVA model is a model with one factor.
   
• ANOVA is another name for a regression model with all categorical variables, which, in practice, is implemented with dummy variables:
   
  
  y_i = μ + α₁x_i1 + ... + α_jx_ij + ... + α_kx_i,J-1 + ε_i;   i = 1 ... n;   j = 1 ... J-1.
  
  or
  y_i = μ + Σ_j=1^J-1 α_jx_ij + ε_i
  
  n is the number of observations, J is the number of levels, and x_ij are dummy variables. Recall that with J levels, there are J - 1 dummy variables.
   
• For a controlled experiment, a one-way ANOVA model is sometimes called a completely randomized design because treatments are randomly assigned to the subjects.
   
• Previously the WLoss3 Example implemented with explicit dummy variables. The factor center (weight loss center) having 5 levels A, B, C, D, and E was implemented with the four dummy variables A, B, C, and D according to this table:
   
  

  	Dummy Variables for Factor Center
  factor levels	A	B	C	D
  A	1	0	0	0
  B	0	1	0	0
  C	0	0	1	0
  D	0	0	0	1
  E	0	0	0	0

  
  
• Here is the design matrix for the WLoss Examples. It shows the settings of the dummy variables for each observation.
   
• The WLoss1 and WLoss2 examples show how to use SAS and R to convert the discrete factors A, B, C, D, and E to dummy variables.
   
• The WLoss4 Example shows how to use the SAS procs anova and glm to analyze an ANOVA model.  These procs automatically convert the factor to dummy variables.  R can also automatically convert factors to dummy variables.
   
• As with a regression model using dummy variables, the overall F-test tests the null hypotheses that all regression coefficients except the intercept are zero. The alternative hypothesis is that at least one of the coefficients is non-zero, but it does not tell use which one or ones are significant.
   
• The F-test for one-way ANOVA compares the between variance (MSM = SSM / DFM = variance among the group means) to the within variance (MSE = SSE / DFE = variance within the groups). If MSM is significantly larger than MSE or MSM / MSE is significantly greater than 1, we reject H₀ that there is no variation between the groups.
   
• If the factor only has two levels, the independent two-sample t-test that assumes equal variances is completely equivalent to one-way ANOVA. Here is a demonstration with R using only the rows in wloss4.txt with levels A or B.
   
• To see which tests levels are significantly different from which other levels, see the section Multiple Comparison Tests below.

Two-way ANOVA

• Two-way ANOVA is an ANOVA model with two factors α and β.
   
• Look at the Orange2 Example, which has two treatments: orange tree variety and type of pesticide.
   
• Here is a diagram showing the orange tree yields (oranges per tree) for all the possible combinations of variety and pesticide:
  

                          Pesticide
                1       2       3       4       
            +-------+-------+-------+-------+
          1 | 49 39 | 50 55 | 43 38 | 43 58 |
            +-------+-------+-------+-------+
  Variety 2 | 55 41 | 67 58 | 53 42 | 85 73 |
            +-------+-------+-------+-------+
          3 | 66 68 | 85 92 | 69 62 | 85 89 |
            +-------+-------+-------+-------+
  

• Denote the number of levels of α by I and the number of levels of β by J.
   
• If there is no interaction between the effects, we can assume the additive two-way model
   
  
  y_i = μ + Σ_j=1^J-1 α_j x_ij + Σ_k=1^K-1 β_k x_ik^' + ε_j .
  
  n is the number of observations, J is the number of levels of α, K is the number of levels of β, and are the dummy variables are x_ij and x_ij^'. Similarly to the one-way case, there are I - 1 dummy variables for α and J - 1 dummy variables for β.
   
• If there are interactions between α and β, we have the two-way interaction model:
   
  
  y_i = μ + Σ_j=1^J-1 α_j x_ij + Σ_k=1^K-1 β_k x_ik^' + Σ_j=1^J-1 Σ_k=1^K-1 (αβ)_ik x_ijk^" + ε_j .
  
  n is the number of observations, J is the number of levels of α, K is the number of levels of β, and JK is the number of interactions.  The dummy variables are x_ij, x_ik^', and x_ijk^". There are J - 1 dummy variables for the main effect parameters α_j, K - 1 dummy variables for the main effect parameters β_k, and (J - 1)*(K - 1) dummy variables for the interaction parameters (αβ)_jk.
   
• We need at least 2 observations per cell for the interaction model to work. If there is only one observation per cell, all the degrees of freedom are used up and there are none left for the degrees of freedom for error:
   
  
  DFE = n - DFM = n - (1 + I - 1 + J - 1 + (I - 1)*(J - 1) )
        = n - (1 + I - 1 + J - 1 + IJ - I - J + 1) = n - IJ = 0
  
  
• Here are the design matrices for the Orange Examples. Both the additive and saturated models are shown.
   
• Look at the Orange2 Example. SAS automatically creates the interaction plot, which plots the predicted values of yield vs. variety using pesticide as the plotting symbol. The yields for the same level of pesticide are connected with line segments to form polylines. Parallel or near parallel polylines show no interaction. Nonparallel polylines show that the interaction variety*pesticide is present.
   
• Consider the additive model in the Orange2 Example. To test whether the pesticide effect is significant, we perform this partial F-test:
   
  
  F(pesticide)
  = [(SSE_{variety only} - SSE_additive) / (DFE_{variety only} - DFE_additive)] / MSE(additive)
  = (2964.375 - 971.917)/(5 - 2) / 53.995 = 12.30
  
  This F-value can also be found on the SAS output for the pesticide effect in the additive model. Since the p-value for F is 0.0001 < 0.05, the pesticide effect is significant.
   
• This partial F-test can also be performed in R:
  

  model1 <- lm(yield ~ variety, data=orange)
  model2 <- lm(yield ~ variety + pesticide, 
               data=orange)
  library(car)
  anova(model2, model1)
  # Output:
   Res.Df RSS Df Sum of Sq F Pr(>F) 
  1 18 971.92 
  2 21 2964.38 -3 -1992.5 12.3 0.0001295 ***

  The F-value for the pesticide-only one-way model is 2.86, which has the p-value 0.0625. This is larger than the p-value from the partial F-test in the previous bullet point. This is because variety effect in the additive model takes away variation due to error and moves it to variation due to the variety effect.
   
• To really know how proc glm performs F-tests, one needs to understand the different types of sums of squares (SS) that proc glm computes. By default, the Type II and Type III SS are shown, but Type I and Type IV SS can also be requested. R can compute Type II and Type III SS. (To my knowledge, it does not compute Type I and Type IV SS.) Here is a brief description of each:
   
  
  Type I SS:   Fully sequential. The SS for an effect is adjusted for all of the effects preceding it in the model statement. This is usually the preferred method for regression models.
   
  Type II SS:   Method of fitting constants. The SS for an effect is adjusted for all effects listed in the model statement that do not contain that effect. For example, if the model statement is
  

  model A B A*B;
  

  the SS for A and B will be adjusted for each other, but A*B will be adjusted for neither A nor B. A*B will be adjusted for both A and B.
   
  Type III SS: Weighted squares of means. Effects in the model will be adjusted for all other effects in the model, regardless of whether or not they contain that effect. For example, for the model statement
  

  model A B A*B;
  

  the effect A is adjusted for both B and A*B, even though A is contained in A*B.
   
  Type IV SS:   Goodnight's method. Type III and IV SS are the same unless some cells are empty. Then different a different strategy than Type III is used to compensate for the empty cell.
   
  
  If the design is balanced, that is, all the cells contain the same number of observations, then all four types of SS have the same value.

Case Studies

1. Factors Affecting the Sale Price of Condominium Units Sold at Auction
    
   
   • This case study is discussed on pages 447 to 465 of the Mendenhall and Sincich textbook.
      
   • Look at the Condo Example.
      
   • Here are the variables columns of the condo.txt dataset:
      
     

     Variable	Meaning
     PRICE100	Dependent variable. Price ($100)
     FLOOR	Floor height of the unit (1 to 8)
     DIST	Distance (feet) from the elevator
     VIEW	Dummy variable = 1 if unit has an ocean view, 0 otherwise
     END	Dummy variable = 1 if unit is on end of floor with blocked view
     FURNISH	Dummy variable = 1 if unit is furnished

     
     
   • Here are the models considered in this example:
      
     
     Model 1: All predictors are linear:
     

     PRICE100 = FLOOR DIST VIEW END FURNISH
     

     Model 2: Quadratic terms are added for floor and distance:
     

     FLOOR*FLOOR  DIST*DIST  FLOOR*DIST
     

     Model 3: Interaction terms with view are added to the floor and distance terms:
     

     FLOOR*VIEW  DIST*VIEW  FLOOR*FLOOR*VIEW
     
     DIST*FLOOR*VIEW  DIST*DIST*VIEW
     

     Model 4: Same as Model 3, but floor is converted to a factor using 7 dummy variables. (The factor FLOOR has 8 levels.) Use proc glm because setting up interaction terms with dummy variables is much easier than with proc reg.
     
     
   • Here is a table summarizing these four models:
      
     

     Model	DFM	SSE	DFE	MSE	R2
     1	5	120802	203	595.08	0.4819
     2	8	100163	200	500.81	0.5639
     3	13	88827	195	455.52	0.6034
     4	42	62150	166	374.40	0.6756

     
     
   • Perform the partial F-test to see if adding the quadratic terms in Model 2 is a significant improvement over Model 1:
      
     
     F = [(SSE_Model1 - SSE_Model2) / (DFE_Model1 / DFE_Model2)] / MSE_Model2
     = [(120802 - 100163) / (203 - 200)] / 500.81 = 13.74
     
     
   • Perform the partial F-test to see if adding the interaction terms with FLOOR in Model 3 is a significant improvement over Model 2:
      
     
     F = [(SSE_Model2 - SSE_Model3) / (DFE_Model2 / DFE_Model3)] / MSE_Model3
     = [(100163 - 88827) / (200 - 195)] / 455.52 = 4.98
     
     
2. Deregulation of the Intrastate Trucking Industry

 

• See the discussion on pages 345 to 354 in the textbook.
   
• Here are the trucking datasets: trucking.txt (134 observations), trucking4.txt (448 observations).
   
• The variables of the Trucking Datasets:
   
  

  Variable	Meaning
  LNPRICE	Dependent variable. Log of supply price
  DIST	Miles traveled (100 miles)
  WGT	Weight of product shipped (1000 lbs.)
  PCTLD	Percent of truck load capacity
  ORIG	City of origin (JAX or MIA)
  MARKET	Size of market destination (LARGE or SMALL)
  DEREG	Deregulation in effect (YES or NO)
  PROD	Product classification (100, 150, 200)

  
  
• The textbook explains that a stepwise regression was performed on these variables. The result was that four of these variables were selected:
  

  DIST  WGT  DEREG  ORIG
  

• Practice Problem: Implement and compare the following regression models using the trucking4.txt dataset:
   
  
  Model1: Complete quadratic model in DIST and WGT; two and three way interactions of quadratic terms with DEREG and ORIG.
  

  DIST  WGT  DIST*DIST  WGT*WGT  DIST*WGT 
  
  DEREG  DIST*DEREG  WGT*DEREG  DIST*DIST*DEREG
  
  WGT*WGT*DEREG  DIST*WGT*DEREG
  
  ORIG  DIST*ORIG  WGT*ORIG  DIST*DIST*ORIG
  
  WGT*WGT*ORIG  DIST*WGT*ORIG
  
  DEREG*ORIG  DIST*DEREG*ORIG  WGT*DEREG*ORIG  
  
  DIST*DIST*DEREG*ORIG  WGT*WGT*DEREG*ORIG  
  
  DIST*WGT*DEREG*ORIG
  

  Model2: Same as Model1, except that quadratic terms involving DIST and WGT are dropped, leaving.
  

  DIST  WGT  DEREG  DIST*DEREG  WGT*DEREG  
  
  ORIG  DIST*ORIG  WGT*ORIG  DEREG*ORIG
  
  DIST*DEREG*ORIG  WGTDEREG*ORIG  
  

  Model3: Same as Model1, except that quantitative-qualitative interaction terms are omitted. These are the terms left in Model3:
  

  DIST  WGT  DIST*DIST  WGT*WGT  DIST*WGT 
  
  DEREG  ORIG  DEREG*ORIG  
  

  Model4: Same as Model1, except that the interactions between quadratic terms with DEREG and ORIG are dropped. (Interactions between DEREG, ORIG, and DIST*WGT are kept.) The remaining terms in Model4 are:
  

  DIST  WGT  DIST*DIST  WGT*WGT  DIST*WGT 
  
  DEREG  DIST*DEREG  WGT*DEREG  DIST*DIST*DEREG
  
  DIST*WGT*DEREG  ORIG  DIST*ORIG  WGT*ORIG  
  
  DIST*WGT*ORIG  DEREG*ORIG  DEREG*ORIG  
  
  DIST*DEREG*ORIG  WGT*DEREG*ORIG
  
  DIST*WGT*DEREG*ORIG
  

Multiple Comparison Tests

• Multiple comparison tests allow a data analyst to see which levels of a factor are significantly different from which other levels.
   
• If there are k different levels, there are C = k(k - 1) / 2 different pairwise comparisons that can be made. A naive method to see which levels are significantly different from which other levels is to perform C different t-tests at the α level. However, the chance that at least one of the comparisons is significant is greater than α. If the comparisons are pairwise independent, the probability that at least one comparison is significant is 1 - (1 - α)^C. This is the multiple comparisons problem.
   
• A conservative solution is the Bonferroni test, which adjusts the significance level to be α/C, so that
   
  
  Prob(at least one comparison is significant) = 1 - (1 - α/C)^C ∼ α
  
  if α is small and C is large.
   
• There are many available multiple comparison tests, most of which are more sophisticated than the Bonferroni test. Here are the tests available with SAS:
   
  

  BON  DUNCAN  GABRIEL  REGWQ  SIDAK  
  
  SCHEFFE  SMM  T  TUKEY
  

  Personally, I like Duncan's test. However, everything really reduces to whether you prefer a conservative test or liberal test.
  
  
• The Bonferroni test, specified in proc anova or proc glm as means center / bon;, is the most conservative of the tests.
   
• R performs pairwise multiple comparisons with the pairwise.t.test functions. It has these available parameter settings for adjusting the p-value:
   
  

  "BH"  "bonferroni"  "BY"  "fdr"  
  
  "hochberg"  "holm"  "hommel"  "none"

• See the R source code and output for the WLoss4 Example.