Calibration and Validation of Random Forest Models

Calibration and Validation of Random Forest Models

<p>Supplemental Methods</p><p>Calibration and validation of random forest models</p><p>Random Forests The random forests (RF) approach involves producing multiple regression trees, which are then combined to make a single consensus prediction for a given observation (Breiman L, 2001). We generated the SNP RF model and the CSM RF model using the R package randomForest. The RF model is composed of an aggregate collection of regression trees, each created from boostrapped training samples: each branch is selected from a random subset of a given number (denoted be mtry) of the input variables (data columns). The two main parameters are mtry and ntree, the number of trees in the forest. We used the mean squared error (abbreviated MSE) as a measure of the prediction accuracy of the RF model.</p><p>RF models have the advantage of giving a summary of the importance of each variable based on the randomized variable selection process used to grow the RF. Two MSE error estimates are used in the validation procedure: the OOB error and the cross-validation error. An important feature of RFs is its use of out-of-bag (OOB) samples. An OOB sample is the set of observations which are not used for building the current tree, and can be used to estimate the MSE error; it can be shown that an OOB error estimate is almost identical to that obtained by K-fold cross-validation. </p><p>Model Calibration We first tuned the two parameters mtry and ntree of the RF method. Figure 1A and 2A shows the OOB error progression on 500 trees for random forests using different parameters mtry. MSE errors stabilize at about 400 trees, so we see that ntree=500 (default value) was sufficient to give good performance for the SNP model and for the CSM model. In a regression framework, the default value of mtry is [p/3] where p is the number of variables. The case mtry=p corresponds to bagging (or bootstrap aggregation), a general –purpose procedure for reducing the variance of a statistical learning method. Notice that bigger mtry is better for the SNP data, according to the MSE error (Figure 1A and Figure 2A), we choose mtry=15 for the SNP model and mtry=20 for the CSM model of ccRCC.</p><p>Model Validation RFs were grown with ntree=500 for all models. We used mtry=15 for the SNP model and mtry=20 for the CSM model.The two model were trained using 54 explanatory variables. The validation of the two models is given in terms of MSE. We used 10-fold cross-validation to compute the prediction error. We compared the prediction error of the RF model to the prediction error obtained training a multiple regression linear model with the same input </p><p>1 variables involved. We see that RFs outperform a linear model for the SNP and cancer mutation data (Figure1B and 2B).</p><p>References</p><p>Breiman, L. (2001), Random Forests, Machine Learning 45(1), 5-32 CSM model</p><p>Figure1 A.MSE sensitivity to ntree and mtry (CSM model) B.MSE for linear regression model(Lmr) and Random forest (RF) with 10-fold cross validation (CSM model); C. observed and predicted cancer somatic mutation densities with 10-fold cross validation; D. the default number of trees in the RF model minimizes the OOB error.</p><p>2 SNP model </p><p>Figure2 A.MSE sensitivity to ntree and mtry (SNP model) B.MSE for linear regression model(Lmr) and Random forest (RF) with 10-fold cross validation (SNP model); C. observed and predicted SNP densities with 10-fold cross validation; D. the default number of trees in the RF model minimizes the OOB error.</p><p>3</p>

View Full Text

Details

  • File Type
    pdf
  • Upload Time
    -
  • Content Languages
    English
  • Upload User
    Anonymous/Not logged-in
  • File Pages
    3 Page
  • File Size
    -

Download

Channel Download Status
Express Download Enable

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

  • Not to be reproduced or distributed without explicit permission.
  • Not used for commercial purposes outside of approved use cases.
  • Not used to infringe on the rights of the original creators.
  • If you believe any content infringes your copyright, please contact us immediately.

Support

For help with questions, suggestions, or problems, please contact us