To Documents

Model Selection

Model Selection Criteria

• We have seen how the R-squared statistic can be used to compare regression models.
   
• A model with a larger R-squared value means that the independent variables explain a larger percentage of the variation in the independent variable.
   
• However, this may conflict with parsimony. The adjusted R-squared value takes this into account:
   
  
  R² = 1 - MSE/TSE = 1 - (1 - R²)[(n - 1) / (n - p - 1)]
  
  R² decreases if there are too many regressors in the model.
   
• Here are some other model selection criteria. n is the number of observations, p is the number of model parameters, including the intercept. σ^² is an estimate of the pure error variance from fitting the full model.
   
  

  Symbol	Name	Definition
  AIC	Akaike Information Criterion	n ln(SSE / n) + 2p
  AICC	Corrected Akaike Information Criterion	1 + ln(SSE/n) + 2(p+1)/(n-p-2)
  AIC	Baysian Information Criterion	n ln(SSE/n) + 2(p+2)s-2q2, where q = nσ^2/SSE
  C(p) or Cp	Mallows C(p)	SSE/σ^2 + 2p - n
  SBC	Schwartz Bayesian Information Criterion	n ln(SSE/n) + p ln(n)

  
  
• These criteria represent various approaches for balancing model accuracy with parsimony.
   
• Recommendation: use the adjusted R-squared value.

Model Selection Strategies

• Maximum R-squared Regression   Compute the regression model for each subset of the independent variables. For each p (number of independent variables), select the model that optimizes the selection criterion being used.
   
• Backward Selection
   
  
  1. Start with all regressors in the model.
      
  2. Throw out the regressor with the largest p-value that is greater than SLSTAY. The SAS default value for SLSTAY is 0.10.
      
  3. Repeat Step 2 until there are no regressors with a p-value greater than SLSTAY.
  
  
• Forward Selection
   
  
  1. Start with no regressors in the model.
      
  2. Add the regressor with the largest p-value that is less than SLENTER. The SAS default value for SLENTER is 0.50.
      
  3. Repeat Step 2 until all regressors have p-values less than SLENTER.
  
  
• Stepwise Selection
   
  
  1. Start with no regressors in the model.
      
  2. Add the regressor with the largest p-value that is less than SLENTER. The SAS default value for SLENTER is 0.15.
      
  3. Throw out any regressors have a p-value that has become larger than SLSTAY.  The SAS default value for SLSTAY is 0.15.
      
  4. Repeat Step 2 until all regressors have p-values less than SLSTAY and all regressors not in the model have p-values greater than SLENTER.
  
  
• LASSO Selection (1996)   Minimize SSE where the absolute values of the regression parameters are contrained to be less than or equal to a parameter t. This is computed as a quadratic programming problem. LASSO means Least Absolute Shrinkage and Selection Operator.
   
• Least Angle Regression   Another modern variable selection method, related to the LASSO method. It also uses linear and quadratic programming techniques to find the best subset of variables.