Skip to contents

Introduction

All Possible Regression

All subset regression tests all possible subsets of the set of potential independent variables. If there are K potential independent variables (besides the constant), then there are \(2^{k}\) distinct subsets of them to be tested. For example, if you have 10 candidate independent variables, the number of subsets to be tested is \(2^{10}\), which is 1024, and if you have 20 candidate variables, the number is \(2^{20}\), which is more than one million.

model <- lm(mpg ~ disp + hp + wt + qsec, data = mtcars)
ols_step_all_possible(model)
##    Index N      Predictors  R-Square Adj. R-Square Mallow's Cp
## 3      1 1              wt 0.7528328     0.7445939  0.70869536
## 1      2 1            disp 0.7183433     0.7089548  0.67512054
## 2      3 1              hp 0.6024373     0.5891853  0.50969578
## 4      4 1            qsec 0.1752963     0.1478062  0.07541973
## 8      5 2           hp wt 0.8267855     0.8148396  0.78108710
## 10     6 2         wt qsec 0.8264161     0.8144448  0.77856272
## 6      7 2         disp wt 0.7809306     0.7658223  0.72532105
## 5      8 2         disp hp 0.7482402     0.7308774  0.69454380
## 7      9 2       disp qsec 0.7215598     0.7023571  0.66395284
## 9     10 2         hp qsec 0.6368769     0.6118339  0.52014395
## 14    11 3      hp wt qsec 0.8347678     0.8170643  0.78199548
## 11    12 3      disp hp wt 0.8268361     0.8082829  0.76789526
## 13    13 3    disp wt qsec 0.8264170     0.8078189  0.76988533
## 12    14 3    disp hp qsec 0.7541953     0.7278591  0.68301440
## 15    15 4 disp hp wt qsec 0.8351443     0.8107212  0.77102968

The plot method shows the panel of fit criteria for all possible regression methods.

model <- lm(mpg ~ disp + hp + wt + qsec, data = mtcars)
k <- ols_step_all_possible(model)
plot(k)

Best Subset Regression

Select the subset of predictors that do the best at meeting some well-defined objective criterion, such as having the largest R2 value or the smallest MSE, Mallow’s Cp or AIC.

model <- lm(mpg ~ disp + hp + wt + qsec, data = mtcars)
ols_step_best_subset(model)
##    Best Subsets Regression    
## ------------------------------
## Model Index    Predictors
## ------------------------------
##      1         wt              
##      2         hp wt           
##      3         hp wt qsec      
##      4         disp hp wt qsec 
## ------------------------------
## 
##                                                    Subsets Regression Summary                                                    
## ---------------------------------------------------------------------------------------------------------------------------------
##                        Adj.        Pred                                                                                           
## Model    R-Square    R-Square    R-Square     C(p)        AIC        SBIC        SBC         MSEP       FPE       HSP       APC  
## ---------------------------------------------------------------------------------------------------------------------------------
##   1        0.7528      0.7446      0.7087    12.4809    166.0294    74.2916    170.4266    296.9167    9.8572    0.3199    0.2801 
##   2        0.8268      0.8148      0.7811     2.3690    156.6523    66.5755    162.5153    215.5104    7.3563    0.2402    0.2091 
##   3        0.8348      0.8171       0.782     3.0617    157.1426    67.7238    164.4713    213.1929    7.4756    0.2461    0.2124 
##   4        0.8351      0.8107       0.771     5.0000    159.0696    70.0408    167.8640    220.8882    7.9497    0.2644    0.2259 
## ---------------------------------------------------------------------------------------------------------------------------------
## AIC: Akaike Information Criteria 
##  SBIC: Sawa's Bayesian Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
##  MSEP: Estimated error of prediction, assuming multivariate normality 
##  FPE: Final Prediction Error 
##  HSP: Hocking's Sp 
##  APC: Amemiya Prediction Criteria

The plot method shows the panel of fit criteria for best subset regression methods.

model <- lm(mpg ~ disp + hp + wt + qsec, data = mtcars)
k <- ols_step_best_subset(model)
plot(k)

Stepwise Forward Regression

Build regression model from a set of candidate predictor variables by entering predictors based on p values, in a stepwise manner until there is no variable left to enter any more. The model should include all the candidate predictor variables. If details is set to TRUE, each step is displayed.

Variable Selection

# stepwise forward regression
model <- lm(y ~ ., data = surgical)
ols_step_forward_p(model)
## 
## 
##                               Stepwise Summary                              
## --------------------------------------------------------------------------
## Step    Variable         AIC        SBC       SBIC        R2       Adj. R2 
## --------------------------------------------------------------------------
##  0      Base Model     802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs            730.620    744.543    579.638    0.78091    0.75808 
## --------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------

Plot

model <- lm(y ~ ., data = surgical)
k <- ols_step_forward_p(model)
plot(k)

Detailed Output

# stepwise forward regression
model <- lm(y ~ ., data = surgical)
ols_step_forward_p(model, details = TRUE)
## Forward Selection Method 
## ------------------------
## 
## Candidate Terms: 
## 
## 1. bcs 
## 2. pindex 
## 3. enzyme_test 
## 4. liver_test 
## 5. age 
## 6. gender 
## 7. alc_mod 
## 8. alc_heavy 
## 
## 
## Step   => 0 
## Model  => y ~ 1 
## R2     => 0 
## 
## Initiating stepwise selection... 
## 
##                      Selection Metrics Table                      
## -----------------------------------------------------------------
## Predictor      Pr(>|t|)    R-Squared    Adj. R-Squared      AIC   
## -----------------------------------------------------------------
## liver_test      0.00000        0.455             0.444    771.875 
## enzyme_test     0.00000        0.334             0.322    782.629 
## pindex          0.00155        0.177             0.161    794.100 
## alc_heavy       0.00172        0.174             0.158    794.301 
## bcs             0.01025        0.120             0.103    797.697 
## alc_mod         0.19286        0.032             0.014    802.828 
## gender          0.20972        0.030             0.011    802.956 
## age             0.39073        0.014            -0.005    803.834 
## -----------------------------------------------------------------
## 
## Step      => 1 
## Selected  => liver_test 
## Model     => y ~ liver_test 
## R2        => 0.455 
## 
##                      Selection Metrics Table                      
## -----------------------------------------------------------------
## Predictor      Pr(>|t|)    R-Squared    Adj. R-Squared      AIC   
## -----------------------------------------------------------------
## alc_heavy       0.00065        0.567             0.550    761.439 
## enzyme_test     0.00089        0.562             0.544    762.077 
## pindex          0.07087        0.489             0.469    770.387 
## alc_mod         0.10979        0.481             0.461    771.141 
## gender          0.79395        0.455             0.434    773.802 
## age             0.83908        0.455             0.434    773.831 
## bcs             0.93062        0.455             0.433    773.867 
## -----------------------------------------------------------------
## 
## Step      => 2 
## Selected  => alc_heavy 
## Model     => y ~ liver_test + alc_heavy 
## R2        => 0.567 
## 
##                      Selection Metrics Table                      
## -----------------------------------------------------------------
## Predictor      Pr(>|t|)    R-Squared    Adj. R-Squared      AIC   
## -----------------------------------------------------------------
## enzyme_test     0.00057        0.659             0.639    750.509 
## pindex          0.00961        0.622             0.599    756.125 
## bcs             0.55687        0.570             0.544    763.063 
## age             0.58269        0.569             0.544    763.110 
## alc_mod         0.91757        0.567             0.541    763.428 
## gender          0.93799        0.567             0.541    763.433 
## -----------------------------------------------------------------
## 
## Step      => 3 
## Selected  => enzyme_test 
## Model     => y ~ liver_test + alc_heavy + enzyme_test 
## R2        => 0.659 
## 
##                     Selection Metrics Table                     
## ---------------------------------------------------------------
## Predictor    Pr(>|t|)    R-Squared    Adj. R-Squared      AIC   
## ---------------------------------------------------------------
## pindex          1e-04        0.750             0.730    735.715 
## bcs           0.21294        0.670             0.643    750.782 
## alc_mod       0.75743        0.660             0.632    752.403 
## age           0.77290        0.660             0.632    752.416 
## gender        0.99197        0.659             0.631    752.509 
## ---------------------------------------------------------------
## 
## Step      => 4 
## Selected  => pindex 
## Model     => y ~ liver_test + alc_heavy + enzyme_test + pindex 
## R2        => 0.75 
## 
##                     Selection Metrics Table                     
## ---------------------------------------------------------------
## Predictor    Pr(>|t|)    R-Squared    Adj. R-Squared      AIC   
## ---------------------------------------------------------------
## bcs           0.01248        0.781             0.758    730.620 
## age           0.86220        0.750             0.724    737.680 
## gender        0.96390        0.750             0.724    737.712 
## alc_mod       0.97040        0.750             0.724    737.713 
## ---------------------------------------------------------------
## 
## Step      => 5 
## Selected  => bcs 
## Model     => y ~ liver_test + alc_heavy + enzyme_test + pindex + bcs 
## R2        => 0.781 
## 
##                     Selection Metrics Table                     
## ---------------------------------------------------------------
## Predictor    Pr(>|t|)    R-Squared    Adj. R-Squared      AIC   
## ---------------------------------------------------------------
## age           0.74164        0.781             0.754    732.494 
## gender        0.80666        0.781             0.753    732.551 
## alc_mod       0.94086        0.781             0.753    732.614 
## ---------------------------------------------------------------
## 
## 
## No more variables to be added.
## 
## Variables Selected: 
## 
## => liver_test 
## => alc_heavy 
## => enzyme_test 
## => pindex 
## => bcs
## 
## 
##                               Stepwise Summary                              
## --------------------------------------------------------------------------
## Step    Variable         AIC        SBC       SBIC        R2       Adj. R2 
## --------------------------------------------------------------------------
##  0      Base Model     802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs            730.620    744.543    579.638    0.78091    0.75808 
## --------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------

Stepwise Backward Regression

Build regression model from a set of candidate predictor variables by removing predictors based on p values, in a stepwise manner until there is no variable left to remove any more. The model should include all the candidate predictor variables. If details is set to TRUE, each step is displayed.

Variable Selection

# stepwise backward regression
model <- lm(y ~ ., data = surgical)
ols_step_backward_p(model)
## 
## 
##                              Stepwise Summary                              
## -------------------------------------------------------------------------
## Step    Variable        AIC        SBC       SBIC        R2       Adj. R2 
## -------------------------------------------------------------------------
##  0      Full Model    736.390    756.280    586.665    0.78184    0.74305 
##  1      alc_mod       734.407    752.308    584.276    0.78177    0.74856 
##  2      gender        732.494    748.406    581.938    0.78142    0.75351 
##  3      age           730.620    744.543    579.638    0.78091    0.75808 
## -------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## ------------------------------------------------------------------------------------------------

Plot

model <- lm(y ~ ., data = surgical)
k <- ols_step_backward_p(model)
plot(k)

Detailed Output

# stepwise backward regression
model <- lm(y ~ ., data = surgical)
ols_step_backward_p(model, details = TRUE)
## Backward Elimination Method 
## ---------------------------
## 
## Candidate Terms: 
## 
## 1. bcs 
## 2. pindex 
## 3. enzyme_test 
## 4. liver_test 
## 5. age 
## 6. gender 
## 7. alc_mod 
## 8. alc_heavy 
## 
## 
## Step   => 0 
## Model  => y ~ bcs + pindex + enzyme_test + liver_test + age + gender + alc_mod + alc_heavy 
## R2     => 0.782 
## 
## Initiating stepwise selection... 
## 
## Step     => 1 
## Removed  => alc_mod 
## Model    => y ~ bcs + pindex + enzyme_test + liver_test + age + gender + alc_heavy 
## R2       => 0.78177 
## 
## Step     => 2 
## Removed  => gender 
## Model    => y ~ bcs + pindex + enzyme_test + liver_test + age + alc_heavy 
## R2       => 0.78142 
## 
## Step     => 3 
## Removed  => age 
## Model    => y ~ bcs + pindex + enzyme_test + liver_test + alc_heavy 
## R2       => 0.78091 
## 
## 
## No more variables to be removed.
## 
## Variables Removed: 
## 
## => alc_mod 
## => gender 
## => age
## 
## 
##                              Stepwise Summary                              
## -------------------------------------------------------------------------
## Step    Variable        AIC        SBC       SBIC        R2       Adj. R2 
## -------------------------------------------------------------------------
##  0      Full Model    736.390    756.280    586.665    0.78184    0.74305 
##  1      alc_mod       734.407    752.308    584.276    0.78177    0.74856 
##  2      gender        732.494    748.406    581.938    0.78142    0.75351 
##  3      age           730.620    744.543    579.638    0.78091    0.75808 
## -------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## ------------------------------------------------------------------------------------------------

Stepwise Regression

Build regression model from a set of candidate predictor variables by entering and removing predictors based on p values, in a stepwise manner until there is no variable left to enter or remove any more. The model should include all the candidate predictor variables. If details is set to TRUE, each step is displayed.

Variable Selection

# stepwise regression
model <- lm(y ~ ., data = surgical)
ols_step_both_p(model)
## 
## 
##                                 Stepwise Summary                                
## ------------------------------------------------------------------------------
## Step    Variable             AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------------
##  0      Base Model         802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test (+)     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy (+)      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test (+)    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex (+)         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs (+)            730.620    744.543    579.638    0.78091    0.75808 
## ------------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------

Plot

model <- lm(y ~ ., data = surgical)
k <- ols_step_both_p(model)
plot(k)

Detailed Output

# stepwise regression
model <- lm(y ~ ., data = surgical)
ols_step_both_p(model, details = TRUE)
## Stepwise Selection Method 
## -------------------------
## 
## Candidate Terms: 
## 
## 1. bcs 
## 2. pindex 
## 3. enzyme_test 
## 4. liver_test 
## 5. age 
## 6. gender 
## 7. alc_mod 
## 8. alc_heavy 
## 
## 
## Step   => 0 
## Model  => y ~ 1 
## R2     => 0 
## 
## Initiating stepwise selection... 
## 
## Step      => 1 
## Selected  => liver_test 
## Model     => y ~ liver_test 
## R2        => 0.455 
## 
## Step      => 2 
## Selected  => alc_heavy 
## Model     => y ~ liver_test + alc_heavy 
## R2        => 0.567 
## 
## Step      => 3 
## Selected  => enzyme_test 
## Model     => y ~ liver_test + alc_heavy + enzyme_test 
## R2        => 0.659 
## 
## Step      => 4 
## Selected  => pindex 
## Model     => y ~ liver_test + alc_heavy + enzyme_test + pindex 
## R2        => 0.75 
## 
## Step      => 5 
## Selected  => bcs 
## Model     => y ~ liver_test + alc_heavy + enzyme_test + pindex + bcs 
## R2        => 0.781 
## 
## 
## No more variables to be added or removed.
## 
## 
##                                 Stepwise Summary                                
## ------------------------------------------------------------------------------
## Step    Variable             AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------------
##  0      Base Model         802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test (+)     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy (+)      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test (+)    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex (+)         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs (+)            730.620    744.543    579.638    0.78091    0.75808 
## ------------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------

Stepwise AIC Forward Regression

Build regression model from a set of candidate predictor variables by entering predictors based on Akaike Information Criteria, in a stepwise manner until there is no variable left to enter any more. The model should include all the candidate predictor variables. If details is set to TRUE, each step is displayed.

Variable Selection

# stepwise aic forward regression
model <- lm(y ~ ., data = surgical)
ols_step_forward_aic(model)
## 
## 
##                               Stepwise Summary                              
## --------------------------------------------------------------------------
## Step    Variable         AIC        SBC       SBIC        R2       Adj. R2 
## --------------------------------------------------------------------------
##  0      Base Model     802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs            730.620    744.543    579.638    0.78091    0.75808 
## --------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------

Plot

model <- lm(y ~ ., data = surgical)
k <- ols_step_forward_aic(model)
plot(k)

Detailed Output

# stepwise aic forward regression
model <- lm(y ~ ., data = surgical)
ols_step_forward_aic(model, details = TRUE)
## Forward Selection Method 
## ------------------------
## 
## Candidate Terms: 
## 
## 1. bcs 
## 2. pindex 
## 3. enzyme_test 
## 4. liver_test 
## 5. age 
## 6. gender 
## 7. alc_mod 
## 8. alc_heavy 
## 
## 
## Step     => 0 
## Model    => y ~ 1 
## AIC      => 802.606 
## 
## Initiating stepwise selection... 
## 
##                        Table: Adding New Variables                        
## -------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2  
## -------------------------------------------------------------------------
## liver_test      1    771.875    777.842    616.009    0.45454     0.44405 
## enzyme_test     1    782.629    788.596    626.220    0.33435     0.32154 
## pindex          1    794.100    800.067    637.196    0.17680     0.16097 
## alc_heavy       1    794.301    800.268    637.389    0.17373     0.15784 
## bcs             1    797.697    803.664    640.655    0.12010     0.10318 
## alc_mod         1    802.828    808.795    645.601    0.03239     0.01378 
## gender          1    802.956    808.923    645.725    0.03009     0.01143 
## age             1    803.834    809.801    646.572    0.01420    -0.00476 
## -------------------------------------------------------------------------
## 
## Step     => 1 
## Added    => liver_test 
## Model    => y ~ liver_test 
## AIC      => 771.8753 
## 
##                       Table: Adding New Variables                        
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## alc_heavy       1    761.439    769.395    605.506    0.56674    0.54975 
## enzyme_test     1    762.077    770.033    606.090    0.56159    0.54440 
## pindex          1    770.387    778.343    613.737    0.48866    0.46861 
## alc_mod         1    771.141    779.097    614.435    0.48147    0.46113 
## gender          1    773.802    781.758    616.901    0.45528    0.43391 
## age             1    773.831    781.787    616.928    0.45498    0.43361 
## bcs             1    773.867    781.823    616.961    0.45462    0.43323 
## ------------------------------------------------------------------------
## 
## Step     => 2 
## Added    => alc_heavy 
## Model    => y ~ liver_test + alc_heavy 
## AIC      => 761.4394 
## 
##                       Table: Adding New Variables                        
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## enzyme_test     1    750.509    760.454    595.297    0.65900    0.63854 
## pindex          1    756.125    766.070    600.225    0.62163    0.59892 
## bcs             1    763.063    773.008    606.379    0.56975    0.54394 
## age             1    763.110    773.055    606.421    0.56938    0.54354 
## alc_mod         1    763.428    773.373    606.704    0.56683    0.54084 
## gender          1    763.433    773.378    606.709    0.56679    0.54080 
## ------------------------------------------------------------------------
## 
## Step     => 3 
## Added    => enzyme_test 
## Model    => y ~ liver_test + alc_heavy + enzyme_test 
## AIC      => 750.5089 
## 
##                      Table: Adding New Variables                       
## ----------------------------------------------------------------------
## Predictor    DF      AIC        SBC       SBIC        R2       Adj. R2 
## ----------------------------------------------------------------------
## pindex        1    735.715    747.649    582.943    0.75015    0.72975 
## bcs           1    750.782    762.716    595.377    0.66973    0.64277 
## alc_mod       1    752.403    764.337    596.743    0.65967    0.63189 
## age           1    752.416    764.350    596.755    0.65959    0.63180 
## gender        1    752.509    764.443    596.833    0.65900    0.63116 
## ----------------------------------------------------------------------
## 
## Step     => 4 
## Added    => pindex 
## Model    => y ~ liver_test + alc_heavy + enzyme_test + pindex 
## AIC      => 735.7146 
## 
##                      Table: Adding New Variables                       
## ----------------------------------------------------------------------
## Predictor    DF      AIC        SBC       SBIC        R2       Adj. R2 
## ----------------------------------------------------------------------
## bcs           1    730.620    744.543    579.638    0.78091    0.75808 
## age           1    737.680    751.603    585.012    0.75030    0.72429 
## gender        1    737.712    751.635    585.036    0.75016    0.72413 
## alc_mod       1    737.713    751.636    585.037    0.75015    0.72413 
## ----------------------------------------------------------------------
## 
## Step     => 5 
## Added    => bcs 
## Model    => y ~ liver_test + alc_heavy + enzyme_test + pindex + bcs 
## AIC      => 730.6204 
## 
##                      Table: Adding New Variables                       
## ----------------------------------------------------------------------
## Predictor    DF      AIC        SBC       SBIC        R2       Adj. R2 
## ----------------------------------------------------------------------
## age           1    732.494    748.406    581.938    0.78142    0.75351 
## gender        1    732.551    748.463    581.978    0.78119    0.75325 
## alc_mod       1    732.614    748.526    582.023    0.78093    0.75297 
## ----------------------------------------------------------------------
## 
## 
## No more variables to be added.
## 
## Variables Selected: 
## 
## => liver_test 
## => alc_heavy 
## => enzyme_test 
## => pindex 
## => bcs
## 
## 
##                               Stepwise Summary                              
## --------------------------------------------------------------------------
## Step    Variable         AIC        SBC       SBIC        R2       Adj. R2 
## --------------------------------------------------------------------------
##  0      Base Model     802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs            730.620    744.543    579.638    0.78091    0.75808 
## --------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------

Stepwise AIC Backward Regression

Build regression model from a set of candidate predictor variables by removing predictors based on Akaike Information Criteria, in a stepwise manner until there is no variable left to remove any more. The model should include all the candidate predictor variables. If details is set to TRUE, each step is displayed.

Variable Selection

# stepwise aic backward regression
model <- lm(y ~ ., data = surgical)
k <- ols_step_backward_aic(model)
k
## 
## 
##                              Stepwise Summary                              
## -------------------------------------------------------------------------
## Step    Variable        AIC        SBC       SBIC        R2       Adj. R2 
## -------------------------------------------------------------------------
##  0      Full Model    736.390    756.280    586.665    0.78184    0.74305 
##  1      alc_mod       734.407    752.308    583.884    0.78177    0.74856 
##  2      gender        732.494    748.406    581.290    0.78142    0.75351 
##  3      age           730.620    744.543    578.844    0.78091    0.75808 
## -------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## ------------------------------------------------------------------------------------------------

Plot

model <- lm(y ~ ., data = surgical)
k <- ols_step_backward_aic(model)
plot(k)

Detailed Output

# stepwise aic backward regression
model <- lm(y ~ ., data = surgical)
ols_step_backward_aic(model, details = TRUE)
## Backward Elimination Method 
## ---------------------------
## 
## Candidate Terms: 
## 
## 1. bcs 
## 2. pindex 
## 3. enzyme_test 
## 4. liver_test 
## 5. age 
## 6. gender 
## 7. alc_mod 
## 8. alc_heavy 
## 
## 
## Step     => 0 
## Model    => y ~ bcs + pindex + enzyme_test + liver_test + age + gender + alc_mod + alc_heavy 
## AIC      => 736.3899 
## 
## Initiating stepwise selection... 
## 
##                    Table: Removing Existing Variables                    
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## alc_mod         1    734.407    752.308    584.276    0.78177    0.74856 
## gender          1    734.478    752.379    584.323    0.78148    0.74823 
## age             1    734.544    752.445    584.367    0.78121    0.74792 
## liver_test      1    735.878    753.779    585.255    0.77574    0.74162 
## bcs             1    741.677    759.577    589.203    0.75032    0.71233 
## alc_heavy       1    749.210    767.111    594.541    0.71294    0.66926 
## pindex          1    756.624    774.525    600.014    0.67070    0.62059 
## enzyme_test     1    763.557    781.458    605.318    0.62559    0.56861 
## ------------------------------------------------------------------------
## 
## Step     => 1 
## Removed  => alc_mod 
## Model    => y ~ bcs + pindex + enzyme_test + liver_test + age + gender + alc_heavy 
## AIC      => 734.4068 
## 
##                    Table: Removing Existing Variables                    
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## gender          1    732.494    748.406    581.938    0.78142    0.75351 
## age             1    732.551    748.463    581.978    0.78119    0.75325 
## liver_test      1    733.921    749.833    582.951    0.77556    0.74691 
## bcs             1    739.677    755.589    587.106    0.75032    0.71845 
## alc_heavy       1    750.486    766.398    595.217    0.69499    0.65605 
## pindex          1    754.759    770.671    598.530    0.66987    0.62773 
## enzyme_test     1    761.595    777.507    603.950    0.62532    0.57749 
## ------------------------------------------------------------------------
## 
## Step     => 2 
## Removed  => gender 
## Model    => y ~ bcs + pindex + enzyme_test + liver_test + age + alc_heavy 
## AIC      => 732.4942 
## 
##                    Table: Removing Existing Variables                    
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## age             1    730.620    744.543    579.638    0.78091    0.75808 
## liver_test      1    732.339    746.262    580.934    0.77382    0.75026 
## bcs             1    737.680    751.603    585.012    0.75030    0.72429 
## alc_heavy       1    748.486    762.409    593.500    0.69499    0.66322 
## pindex          1    752.777    766.700    596.959    0.66976    0.63536 
## enzyme_test     1    759.596    773.518    602.553    0.62532    0.58629 
## ------------------------------------------------------------------------
## 
## Step     => 3 
## Removed  => age 
## Model    => y ~ bcs + pindex + enzyme_test + liver_test + alc_heavy 
## AIC      => 730.6204 
## 
##                    Table: Removing Existing Variables                    
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## liver_test      1    730.924    742.858    579.087    0.77136    0.75269 
## bcs             1    735.715    747.649    582.943    0.75015    0.72975 
## alc_heavy       1    747.181    759.114    592.362    0.69104    0.66582 
## pindex          1    750.782    762.716    595.377    0.66973    0.64277 
## enzyme_test     1    757.971    769.905    601.477    0.62270    0.59190 
## ------------------------------------------------------------------------
## 
## 
## No more variables to be removed.
## 
## Variables Removed: 
## 
## => alc_mod 
## => gender 
## => age
## 
## 
##                              Stepwise Summary                              
## -------------------------------------------------------------------------
## Step    Variable        AIC        SBC       SBIC        R2       Adj. R2 
## -------------------------------------------------------------------------
##  0      Full Model    736.390    756.280    586.665    0.78184    0.74305 
##  1      alc_mod       734.407    752.308    583.884    0.78177    0.74856 
##  2      gender        732.494    748.406    581.290    0.78142    0.75351 
##  3      age           730.620    744.543    578.844    0.78091    0.75808 
## -------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## ------------------------------------------------------------------------------------------------

Stepwise AIC Regression

Build regression model from a set of candidate predictor variables by entering and removing predictors based on Akaike Information Criteria, in a stepwise manner until there is no variable left to enter or remove any more. The model should include all the candidate predictor variables. If details is set to TRUE, each step is displayed.

Variable Selection

# stepwise aic regression
model <- lm(y ~ ., data = surgical)
ols_step_both_aic(model)
## 
## 
##                                 Stepwise Summary                                
## ------------------------------------------------------------------------------
## Step    Variable             AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------------
##  0      Base Model         802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test (+)     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy (+)      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test (+)    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex (+)         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs (+)            730.620    744.543    579.638    0.78091    0.75808 
## ------------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------

Plot

model <- lm(y ~ ., data = surgical)
k <- ols_step_both_aic(model)
plot(k)

Detailed Output

# stepwise aic regression
model <- lm(y ~ ., data = surgical)
ols_step_both_aic(model, details = TRUE)
## Stepwise Selection Method 
## -------------------------
## 
## Candidate Terms: 
## 
## 1. bcs 
## 2. pindex 
## 3. enzyme_test 
## 4. liver_test 
## 5. age 
## 6. gender 
## 7. alc_mod 
## 8. alc_heavy 
## 
## 
## Step     => 0 
## Model    => y ~ 1 
## AIC      => 802.606 
## 
## Initiating stepwise selection... 
## 
##                        Table: Adding New Variables                        
## -------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2  
## -------------------------------------------------------------------------
## bcs             1    797.697    803.664    640.655    0.12010     0.10318 
## pindex          1    794.100    800.067    637.196    0.17680     0.16097 
## enzyme_test     1    782.629    788.596    626.220    0.33435     0.32154 
## liver_test      1    771.875    777.842    616.009    0.45454     0.44405 
## age             1    803.834    809.801    646.572    0.01420    -0.00476 
## gender          1    802.956    808.923    645.725    0.03009     0.01143 
## alc_mod         1    802.828    808.795    645.601    0.03239     0.01378 
## alc_heavy       1    794.301    800.268    637.389    0.17373     0.15784 
## -------------------------------------------------------------------------
## 
## Step     => 1 
## Added    => liver_test 
## Model    => y ~ liver_test 
## AIC      => 771.8753 
## 
##                       Table: Adding New Variables                        
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## bcs             1    773.867    781.823    616.961    0.45462    0.43323 
## pindex          1    770.387    778.343    613.737    0.48866    0.46861 
## enzyme_test     1    762.077    770.033    606.090    0.56159    0.54440 
## age             1    773.831    781.787    616.928    0.45498    0.43361 
## gender          1    773.802    781.758    616.901    0.45528    0.43391 
## alc_mod         1    771.141    779.097    614.435    0.48147    0.46113 
## alc_heavy       1    761.439    769.395    605.506    0.56674    0.54975 
## ------------------------------------------------------------------------
## 
## Step     => 2 
## Added    => alc_heavy 
## Model    => y ~ liver_test + alc_heavy 
## AIC      => 761.4394 
## 
##                   Table: Removing Existing Variables                    
## -----------------------------------------------------------------------
## Predictor     DF      AIC        SBC       SBIC        R2       Adj. R2 
## -----------------------------------------------------------------------
## liver_test     1    794.301    800.268    637.389    0.17373    0.15784 
## alc_heavy      1    771.875    777.842    616.009    0.45454    0.44405 
## -----------------------------------------------------------------------
## 
##                       Table: Adding New Variables                        
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## bcs             1    763.063    773.008    606.379    0.56975    0.54394 
## pindex          1    756.125    766.070    600.225    0.62163    0.59892 
## enzyme_test     1    750.509    760.454    595.297    0.65900    0.63854 
## age             1    763.110    773.055    606.421    0.56938    0.54354 
## gender          1    763.433    773.378    606.709    0.56679    0.54080 
## alc_mod         1    763.428    773.373    606.704    0.56683    0.54084 
## ------------------------------------------------------------------------
## 
## Step     => 3 
## Added    => enzyme_test 
## Model    => y ~ liver_test + alc_heavy + enzyme_test 
## AIC      => 750.5089 
## 
##                    Table: Removing Existing Variables                    
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## liver_test      1    773.555    781.511    616.671    0.45777    0.43650 
## alc_heavy       1    762.077    770.033    606.090    0.56159    0.54440 
## enzyme_test     1    761.439    769.395    605.506    0.56674    0.54975 
## ------------------------------------------------------------------------
## 
##                      Table: Adding New Variables                       
## ----------------------------------------------------------------------
## Predictor    DF      AIC        SBC       SBIC        R2       Adj. R2 
## ----------------------------------------------------------------------
## bcs           1    750.782    762.716    595.377    0.66973    0.64277 
## pindex        1    735.715    747.649    582.943    0.75015    0.72975 
## age           1    752.416    764.350    596.755    0.65959    0.63180 
## gender        1    752.509    764.443    596.833    0.65900    0.63116 
## alc_mod       1    752.403    764.337    596.743    0.65967    0.63189 
## ----------------------------------------------------------------------
## 
## Step     => 4 
## Added    => pindex 
## Model    => y ~ liver_test + alc_heavy + enzyme_test + pindex 
## AIC      => 735.7146 
## 
##                    Table: Removing Existing Variables                    
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## liver_test      1    748.167    758.112    593.257    0.67347    0.65388 
## alc_heavy       1    755.099    765.044    599.321    0.62875    0.60647 
## enzyme_test     1    756.125    766.070    600.225    0.62163    0.59892 
## pindex          1    750.509    760.454    595.297    0.65900    0.63854 
## ------------------------------------------------------------------------
## 
##                      Table: Adding New Variables                       
## ----------------------------------------------------------------------
## Predictor    DF      AIC        SBC       SBIC        R2       Adj. R2 
## ----------------------------------------------------------------------
## bcs           1    730.620    744.543    579.638    0.78091    0.75808 
## age           1    737.680    751.603    585.012    0.75030    0.72429 
## gender        1    737.712    751.635    585.036    0.75016    0.72413 
## alc_mod       1    737.713    751.636    585.037    0.75015    0.72413 
## ----------------------------------------------------------------------
## 
## Step     => 5 
## Added    => bcs 
## Model    => y ~ liver_test + alc_heavy + enzyme_test + pindex + bcs 
## AIC      => 730.6204 
## 
##                    Table: Removing Existing Variables                    
## ------------------------------------------------------------------------
## Predictor      DF      AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------
## liver_test      1    730.924    742.858    579.087    0.77136    0.75269 
## alc_heavy       1    747.181    759.114    592.362    0.69104    0.66582 
## enzyme_test     1    757.971    769.905    601.477    0.62270    0.59190 
## pindex          1    750.782    762.716    595.377    0.66973    0.64277 
## bcs             1    735.715    747.649    582.943    0.75015    0.72975 
## ------------------------------------------------------------------------
## 
##                      Table: Adding New Variables                       
## ----------------------------------------------------------------------
## Predictor    DF      AIC        SBC       SBIC        R2       Adj. R2 
## ----------------------------------------------------------------------
## age           1    732.494    748.406    581.938    0.78142    0.75351 
## gender        1    732.551    748.463    581.978    0.78119    0.75325 
## alc_mod       1    732.614    748.526    582.023    0.78093    0.75297 
## ----------------------------------------------------------------------
## 
## 
## No more variables to be added or removed.
## 
## Variables Selected: 
## 
## => liver_test 
## => alc_heavy 
## => enzyme_test 
## => pindex 
## => bcs
## 
## 
##                                 Stepwise Summary                                
## ------------------------------------------------------------------------------
## Step    Variable             AIC        SBC       SBIC        R2       Adj. R2 
## ------------------------------------------------------------------------------
##  0      Base Model         802.606    806.584    646.794    0.00000    0.00000 
##  1      liver_test (+)     771.875    777.842    616.009    0.45454    0.44405 
##  2      alc_heavy (+)      761.439    769.395    605.506    0.56674    0.54975 
##  3      enzyme_test (+)    750.509    760.454    595.297    0.65900    0.63854 
##  4      pindex (+)         735.715    747.649    582.943    0.75015    0.72975 
##  5      bcs (+)            730.620    744.543    579.638    0.78091    0.75808 
## ------------------------------------------------------------------------------
## 
## Final Model Output 
## ------------------
## 
##                            Model Summary                            
## -------------------------------------------------------------------
## R                         0.884       RMSE                 184.276 
## R-Squared                 0.781       MSE                38202.426 
## Adj. R-Squared            0.758       Coef. Var             27.839 
## Pred R-Squared            0.700       AIC                  730.620 
## MAE                     137.656       SBC                  744.543 
## -------------------------------------------------------------------
##  RMSE: Root Mean Square Error 
##  MSE: Mean Square Error 
##  MAE: Mean Absolute Error 
##  AIC: Akaike Information Criteria 
##  SBC: Schwarz Bayesian Criteria 
## 
##                                  ANOVA                                  
## -----------------------------------------------------------------------
##                    Sum of                                              
##                   Squares        DF    Mean Square      F         Sig. 
## -----------------------------------------------------------------------
## Regression    6535804.090         5    1307160.818    34.217    0.0000 
## Residual      1833716.447        48      38202.426                     
## Total         8369520.537        53                                    
## -----------------------------------------------------------------------
## 
##                                       Parameter Estimates                                        
## ------------------------------------------------------------------------------------------------
##       model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
## ------------------------------------------------------------------------------------------------
## (Intercept)    -1178.330       208.682                 -5.647    0.000    -1597.914    -758.746 
##  liver_test       58.064        40.144        0.156     1.446    0.155      -22.652     138.779 
##   alc_heavy      317.848        71.634        0.314     4.437    0.000      173.818     461.878 
## enzyme_test        9.748         1.656        0.521     5.887    0.000        6.419      13.077 
##      pindex        8.924         1.808        0.380     4.935    0.000        5.288      12.559 
##         bcs       59.864        23.060        0.241     2.596    0.012       13.498     106.230 
## ------------------------------------------------------------------------------------------------