{
  "id": 23321,
  "title": "Why Decision Tree beats Random Forest ??",
  "url": "/competitions/bosch-production-line-performance/discussion/23321",
  "author_name": "",
  "post_date": "2016-09-03T17:22:19.707Z",
  "votes": 2,
  "comment_count": 7,
  "views": 1338,
  "content": "<p>I am surprised to find the single decision tree classifiers have better than Random Forest Classifiers. </p>\n\n<p>The result I got are:</p>\n\n<p>RandomForestClassifier&#8230;</p>\n\n<p>Train time (secs): 39.636</p>\n\n<p>train score: 0.14864609152107056\ntime (secs): 5.049</p>\n\n<p>test score: 0.09539416995188536\ntime (secs): 2.246</p>\n\n<hr>\n\n<p>DecisionTreeClassifier&#8230;</p>\n\n<p>Train time (secs): 91.615</p>\n\n<p>train score: 0.2816306193360963\ntime (secs): 4.388</p>\n\n<p>test score: 0.12805531718219185\ntime (secs): 1.948</p>\n\n<p>The parameters I used are :</p>\n\n<p>clf1 = RandomForestClassifier(n_estimators = 10,\n                             max_features = 50,\n                             min_samples_leaf = 8,\n                             max_depth = 12)</p>\n\n<p>clf2 = DecisionTreeClassifier(max_depth=12,\n                             min_samples_leaf = 8)</p>\n\n<p>I use Matthew's correlation coefficient for the scoring. \nThe single decision tree has both the higher training score (train and test on the training test ) and the higher testing score (train on the training test and test on the testing test ). So the problem doesn&#8217;t seem to be overfitting. \nAnother thing that struck me training a single decision tree takes longer to train.\nPS. the data is only form train_numberic, shuffled. NaNs are replaced with the column mean\nTrain set size: 400000\nTest set size: 177563</p>",
  "messages": [
    {
      "id": "133766",
      "postDate": "09/03/2016 17:22:19",
      "content": "<p>I am surprised to find the single decision tree classifiers have better than Random Forest Classifiers. </p>\n\n<p>The result I got are:</p>\n\n<p>RandomForestClassifier&#8230;</p>\n\n<p>Train time (secs): 39.636</p>\n\n<p>train score: 0.14864609152107056\ntime (secs): 5.049</p>\n\n<p>test score: 0.09539416995188536\ntime (secs): 2.246</p>\n\n<hr>\n\n<p>DecisionTreeClassifier&#8230;</p>\n\n<p>Train time (secs): 91.615</p>\n\n<p>train score: 0.2816306193360963\ntime (secs): 4.388</p>\n\n<p>test score: 0.12805531718219185\ntime (secs): 1.948</p>\n\n<p>The parameters I used are :</p>\n\n<p>clf1 = RandomForestClassifier(n_estimators = 10,\n                             max_features = 50,\n                             min_samples_leaf = 8,\n                             max_depth = 12)</p>\n\n<p>clf2 = DecisionTreeClassifier(max_depth=12,\n                             min_samples_leaf = 8)</p>\n\n<p>I use Matthew's correlation coefficient for the scoring. \nThe single decision tree has both the higher training score (train and test on the training test ) and the higher testing score (train on the training test and test on the testing test ). So the problem doesn&#8217;t seem to be overfitting. \nAnother thing that struck me training a single decision tree takes longer to train.\nPS. the data is only form train_numberic, shuffled. NaNs are replaced with the column mean\nTrain set size: 400000\nTest set size: 177563</p>",
      "rawMarkdown": "I am surprised to find the single decision tree classifiers have better than Random Forest Classifiers. \r\n\r\nThe result I got are:\r\n \r\nRandomForestClassifier…\r\n\r\nTrain time (secs): 39.636\r\n\r\ntrain score: 0.14864609152107056\r\ntime (secs): 5.049\r\n\r\ntest score: 0.09539416995188536\r\ntime (secs): 2.246\r\n\r\n------------------------------------------\r\nDecisionTreeClassifier…\r\n\r\nTrain time (secs): 91.615\r\n\r\ntrain score: 0.2816306193360963\r\ntime (secs): 4.388\r\n\r\ntest score: 0.12805531718219185\r\ntime (secs): 1.948\r\n\r\n\r\nThe parameters I used are :\r\n\r\nclf1 = RandomForestClassifier(n_estimators = 10,\r\n                             max_features = 50,\r\n                             min_samples_leaf = 8,\r\n                             max_depth = 12)\r\n\r\nclf2 = DecisionTreeClassifier(max_depth=12,\r\n                             min_samples_leaf = 8)\r\n\r\n\r\nI use Matthew's correlation coefficient for the scoring. \r\nThe single decision tree has both the higher training score (train and test on the training test ) and the higher testing score (train on the training test and test on the testing test ). So the problem doesn’t seem to be overfitting. \r\nAnother thing that struck me training a single decision tree takes longer to train.\r\nPS. the data is only form train_numberic, shuffled. NaNs are replaced with the column mean\r\nTrain set size: 400000\r\nTest set size: 177563",
      "votes": null
    },
    {
      "id": "133851",
      "postDate": "09/04/2016 03:05:25",
      "content": "<p>I'm not familiar with this data set but it might be because the random forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>",
      "rawMarkdown": "I'm not familiar with this data set but it might be because the random forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features.",
      "votes": null
    },
    {
      "id": "133857",
      "postDate": "09/04/2016 04:07:10",
      "content": "<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>",
      "rawMarkdown": "[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).",
      "votes": null
    },
    {
      "id": "133861",
      "postDate": "09/04/2016 05:07:54",
      "content": "<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>",
      "rawMarkdown": "[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting.",
      "votes": null
    },
    {
      "id": "133877",
      "postDate": "09/04/2016 08:54:57",
      "content": "<p>[quote=AiO;133861]</p>\n\n<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>\n\n<p>[/quote]</p>\n\n<p>Which is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.</p>",
      "rawMarkdown": "[quote=AiO;133861]\r\n\r\n[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting. \r\n\r\n[/quote]\r\n\r\nWhich is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \r\nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.",
      "votes": null
    },
    {
      "id": "133930",
      "postDate": "09/04/2016 14:07:53",
      "content": "<p>[quote=AiO;133861]</p>\n\n<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>\n\n<p>[/quote]</p>\n\n<p>The ensemble of Random &quot;Overfitted&quot; Trees makes it less prone to overfit (because the trees are pretty different from each other).  Series of random procedures, such as bagging of training set and random attribute selection drawn from a randomly chosen subset of features, produce an ensemble of low-correlated decision trees. Trees with reduced correlation is one of the key aspects that guarantee the high effectiveness of RF classifier. Another fundamental aspect is necessary to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth. By considering the average of several decorrelated trees as the model prediction, it can be shown that the RF classifier reduces variance while keeping its bias unchanged. It typically brings up better models, with higher generalization power.</p>",
      "rawMarkdown": "[quote=AiO;133861]\r\n\r\n[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting. \r\n\r\n[/quote]\r\n\r\nThe ensemble of Random \"Overfitted\" Trees makes it less prone to overfit (because the trees are pretty different from each other).  Series of random procedures, such as bagging of training set and random attribute selection drawn from a randomly chosen subset of features, produce an ensemble of low-correlated decision trees. Trees with reduced correlation is one of the key aspects that guarantee the high effectiveness of RF classifier. Another fundamental aspect is necessary to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth. By considering the average of several decorrelated trees as the model prediction, it can be shown that the RF classifier reduces variance while keeping its bias unchanged. It typically brings up better models, with higher generalization power.",
      "votes": null
    },
    {
      "id": "133936",
      "postDate": "09/04/2016 14:17:26",
      "content": "<p>[quote=Malte Nalenz;133877]</p>\n\n<p>[quote=AiO;133861]</p>\n\n<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>\n\n<p>[/quote]</p>\n\n<p>Which is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.\n[/quote]</p>\n\n<p>Another problem also might be due to the number of trees you are using (usually the more the better). 10 is too little. I would use over 100.</p>",
      "rawMarkdown": "[quote=Malte Nalenz;133877]\r\n\r\n[quote=AiO;133861]\r\n\r\n[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting. \r\n\r\n[/quote]\r\n\r\nWhich is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \r\nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.\r\n[/quote]\r\n\r\nAnother problem also might be due to the number of trees you are using (usually the more the better). 10 is too little. I would use over 100.",
      "votes": null
    },
    {
      "id": "133953",
      "postDate": "09/04/2016 15:23:16",
      "content": "<p>Thanks a lot for the discussion. I will try to increase maximum features and tree depth to overfit the model a little bit.</p>\n\n<p>Another thing. When caculate the score, I just used the sklearn package  &quot;matthews_corrcoef(y_true, y_pred)&quot; to get the score. The threshold probability is not fine tuned. I later found, the score can be higher if the threshold is optimized for the matthew corrcoef.</p>",
      "rawMarkdown": "Thanks a lot for the discussion. I will try to increase maximum features and tree depth to overfit the model a little bit.\r\n\r\nAnother thing. When caculate the score, I just used the sklearn package  \"matthews_corrcoef(y_true, y_pred)\" to get the score. The threshold probability is not fine tuned. I later found, the score can be higher if the threshold is optimized for the matthew corrcoef.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 133851,
      "author_name": "notaapple",
      "author_url": "",
      "post_date": "09/04/2016 03:05:25",
      "content": "<p>I'm not familiar with this data set but it might be because the random forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133857,
      "author_name": "raphaelrcampos",
      "author_url": "",
      "post_date": "09/04/2016 04:07:10",
      "content": "<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133861,
      "author_name": "notaapple",
      "author_url": "",
      "post_date": "09/04/2016 05:07:54",
      "content": "<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133877,
      "author_name": "malten",
      "author_url": "",
      "post_date": "09/04/2016 08:54:57",
      "content": "<p>[quote=AiO;133861]</p>\n\n<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>\n\n<p>[/quote]</p>\n\n<p>Which is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133930,
      "author_name": "raphaelrcampos",
      "author_url": "",
      "post_date": "09/04/2016 14:07:53",
      "content": "<p>[quote=AiO;133861]</p>\n\n<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>\n\n<p>[/quote]</p>\n\n<p>The ensemble of Random &quot;Overfitted&quot; Trees makes it less prone to overfit (because the trees are pretty different from each other).  Series of random procedures, such as bagging of training set and random attribute selection drawn from a randomly chosen subset of features, produce an ensemble of low-correlated decision trees. Trees with reduced correlation is one of the key aspects that guarantee the high effectiveness of RF classifier. Another fundamental aspect is necessary to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth. By considering the average of several decorrelated trees as the model prediction, it can be shown that the RF classifier reduces variance while keeping its bias unchanged. It typically brings up better models, with higher generalization power.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133936,
      "author_name": "raphaelrcampos",
      "author_url": "",
      "post_date": "09/04/2016 14:17:26",
      "content": "<p>[quote=Malte Nalenz;133877]</p>\n\n<p>[quote=AiO;133861]</p>\n\n<p>[quote=RaphaelCampos;133857]</p>\n\n<p>[quote=AiO;133851]</p>\n\n<p>I'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. </p>\n\n<p>[/quote]</p>\n\n<p>The Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).</p>\n\n<p>[/quote]</p>\n\n<p>Seems like it would cause overfitting. </p>\n\n<p>[/quote]</p>\n\n<p>Which is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.\n[/quote]</p>\n\n<p>Another problem also might be due to the number of trees you are using (usually the more the better). 10 is too little. I would use over 100.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133953,
      "author_name": "dariusalaska",
      "author_url": "",
      "post_date": "09/04/2016 15:23:16",
      "content": "<p>Thanks a lot for the discussion. I will try to increase maximum features and tree depth to overfit the model a little bit.</p>\n\n<p>Another thing. When caculate the score, I just used the sklearn package  &quot;matthews_corrcoef(y_true, y_pred)&quot; to get the score. The threshold probability is not fine tuned. I later found, the score can be higher if the threshold is optimized for the matthew corrcoef.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "133766": "I am surprised to find the single decision tree classifiers have better than Random Forest Classifiers. \r\n\r\nThe result I got are:\r\n \r\nRandomForestClassifier…\r\n\r\nTrain time (secs): 39.636\r\n\r\ntrain score: 0.14864609152107056\r\ntime (secs): 5.049\r\n\r\ntest score: 0.09539416995188536\r\ntime (secs): 2.246\r\n\r\n------------------------------------------\r\nDecisionTreeClassifier…\r\n\r\nTrain time (secs): 91.615\r\n\r\ntrain score: 0.2816306193360963\r\ntime (secs): 4.388\r\n\r\ntest score: 0.12805531718219185\r\ntime (secs): 1.948\r\n\r\n\r\nThe parameters I used are :\r\n\r\nclf1 = RandomForestClassifier(n_estimators = 10,\r\n                             max_features = 50,\r\n                             min_samples_leaf = 8,\r\n                             max_depth = 12)\r\n\r\nclf2 = DecisionTreeClassifier(max_depth=12,\r\n                             min_samples_leaf = 8)\r\n\r\n\r\nI use Matthew's correlation coefficient for the scoring. \r\nThe single decision tree has both the higher training score (train and test on the training test ) and the higher testing score (train on the training test and test on the testing test ). So the problem doesn’t seem to be overfitting. \r\nAnother thing that struck me training a single decision tree takes longer to train.\r\nPS. the data is only form train_numberic, shuffled. NaNs are replaced with the column mean\r\nTrain set size: 400000\r\nTest set size: 177563",
    "133851": "I'm not familiar with this data set but it might be because the random forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features.",
    "133857": "[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).",
    "133861": "[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting.",
    "133877": "[quote=AiO;133861]\r\n\r\n[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting. \r\n\r\n[/quote]\r\n\r\nWhich is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \r\nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.",
    "133930": "[quote=AiO;133861]\r\n\r\n[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting. \r\n\r\n[/quote]\r\n\r\nThe ensemble of Random \"Overfitted\" Trees makes it less prone to overfit (because the trees are pretty different from each other).  Series of random procedures, such as bagging of training set and random attribute selection drawn from a randomly chosen subset of features, produce an ensemble of low-correlated decision trees. Trees with reduced correlation is one of the key aspects that guarantee the high effectiveness of RF classifier. Another fundamental aspect is necessary to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth. By considering the average of several decorrelated trees as the model prediction, it can be shown that the RF classifier reduces variance while keeping its bias unchanged. It typically brings up better models, with higher generalization power.",
    "133936": "[quote=Malte Nalenz;133877]\r\n\r\n[quote=AiO;133861]\r\n\r\n[quote=RaphaelCampos;133857]\r\n\r\n[quote=AiO;133851]\r\n\r\nI'm not familiar with this data set but it might be because the randomly forest is looking at only 50 features when looking for best split. For the decision tree it's looking at all the features. I'm not sure how many features this dataset has. But if the decision tree is using more features at each node then it would explain the different computation time and the difference in accuracy. If you want to improve the score of you random forest, increase the number of max_features. \r\n\r\n[/quote]\r\n\r\nThe Random Forest requires to build trees with prediction capabilities better than random guessing, which is typically achieved by growing them to their maximum depth (set max_depth=None).\r\n\r\n\r\n[/quote]\r\n\r\nSeems like it would cause overfitting. \r\n\r\n[/quote]\r\n\r\nWhich is how randomForest works. In general ensemble methods work well, if the weak learners explain  different parts of the variation of y. This is why tree methods are often used: through their individual overfitting it is ensured that you get very different trees in the forest. And by averaging over the trees you get a stable classifier. In theory that is. \r\nI think here the problem is, that only a small portion of the covariates is informative at all. By sampling only a small amount of covariates you can easily end up with no informative covariates at all. The resulting tree from this iteration will distort the overall fit, as it will model spurious relationships. Especially the setting of max_features = 50 seems very low. As AiO said increasing the max_features or better to remove clearly uninformative predictors should help the randomForest quite a bit.\r\n[/quote]\r\n\r\nAnother problem also might be due to the number of trees you are using (usually the more the better). 10 is too little. I would use over 100.",
    "133953": "Thanks a lot for the discussion. I will try to increase maximum features and tree depth to overfit the model a little bit.\r\n\r\nAnother thing. When caculate the score, I just used the sklearn package  \"matthews_corrcoef(y_true, y_pred)\" to get the score. The threshold probability is not fine tuned. I later found, the score can be higher if the threshold is optimized for the matthew corrcoef."
  },
  "source": "meta"
}