{
  "id": 10681,
  "title": "How to drop more than 500 places!",
  "content": "<p>-Well it started all with Abisheks Benchmark Code and the transformation into R - thanks for sharing!</p>\n<p><br>-Then I set up my own cross validation with the help of the great caret package!<br>#5-fold-crossvalidation<br>fitControl&lt;-trainControl(method=&quot;repeatedcv&quot;,number=5,repeats=2)</p>\n<p><br>optimizing c for each Parameter- (example for Ca)<br>svmFitCa2&lt;-train(x=train2,y=labels[,1],method=&quot;svmRadial&quot;,trControl=fitControl,tuneLength=10)<br>leading to good RMSE between 0.3 and 0.4 with the exception of Phosphor RMSE=0.841.</p>\n<p><br>Adding the untransformed satellite data improved the RMSE for Ph and Sand a little bit. (so I combined these predictions with the above ones = my first final submission) -&gt; leading to public rank in the last days about 350th</p>\n<p><br>The biggest problem was &quot;P&quot; so I decided to run a random forest for it. The prediction quality was in the same category as the svm -&gt; RMSE=0.8893 so i ensembled svm with rf leading to bad public leaderborad score 0.49078 but hoping for a better private score. (my second final submission)</p>\n<p><br>Well the final results are a complete knock down! The mass with unchanged benchmark code scored better than me. (I hoped for 25 % with a bit luck 10% but ended below 50%). Is this overfit? I thought I am relatively secure due to the cross validation.</p>\n<p><br>What happened? I am not sure. One fault was to rely on more data. I should stripped off the CO2 and maybe a lot more variables (saving a lot computing time). Does anyone see another major mistake?</p>\n<p>A big hug to all who dropped more than 50 places, grats to the winners and thank you for all the sharing in the forums-</p>\n<p>Best regards, Vielauge</p>",
  "comments": [
    {
      "id": 56502,
      "author_name": "mlandry",
      "author_url": "",
      "post_date": "2014-10-22T14:22:17.640000",
      "content": "<p>Vielauge, using caret's&nbsp;method for cross-fold is generally a good idea, but in this competition random CV&nbsp;can be dangerous. Since the test data was known to be ordered by location,&nbsp;test data shared no locations, and you could clearly see that the high-range values for a few of the targets clustered together, a straight-forward cross-validation might&nbsp;not be&nbsp;the way to go. It's possible that isn't your problem, but that seems the first point of suspicion to me.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 56474,
      "author_name": "lewis",
      "author_url": "",
      "post_date": "2014-10-22T10:20:49.240000",
      "content": "<p>I share some of your pain...</p>\n<p>Based on the 'solution sharing' post maybe we should have focused on broader ensembles rather than tuning. I think I made the mistake of dismissing options like gbm just because they weren't as good as svm according to my CV scores. I didn't pursue combinations as much as I might have done.</p>\n<p>I like this explanation of the benefits of that approach</p>\n<p><a href=\"http://www.overkillanalytics.net/more-is-always-better-the-power-of-simple-ensembles/\">http://www.overkillanalytics.net/more-is-always-better-the-power-of-simple-ensembles/</a></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 56739,
      "author_name": "Vielauge",
      "author_url": "",
      "post_date": "2014-10-25T12:56:26.283000",
      "content": "<p>Thank you all for answering this private thread!</p>\n<p>Yes, some things learned:</p>\n<p>1. Cross validation can be misleading.. and there were a lot of other competitors who stepped into this trap, as we can read in many other forum entries. You are right mlandry, I guess this was the main issue.</p>\n<p>2. I already knew that ensembles are powerful, but maybe they are even more powerful than I thought. Thank you for the fine article lewis ml!</p>\n<p>We will see us in one of the next kaggle adventures.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 56478,
      "author_name": "Bluefool",
      "author_url": "",
      "post_date": "2014-10-22T10:39:28.127000",
      "content": "<p>[quote=lewis ml;56477]</p>\n<p>[quote=ACS69;56475]</p>\n<p>On First Derivative data, you could get .44 with a GBM</p>\n<p>[/quote]</p>\n<p>Thanks, yes - I'd got that far or somewhere similar. But I made the mistake of thinking it wasn't close enough to the svm score of 0.40 to be worth merging the two.</p>\n<p>[/quote]</p>\n<p>oh no - rule of data analysis - ensemble in everything. Even if you did a (svm * 0.75) + (gbm * 0.25)</p>\n\n<p>EDIT: I even ensembled in scores of 0.47</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 56477,
      "author_name": "lewis",
      "author_url": "",
      "post_date": "2014-10-22T10:28:14.573000",
      "content": "<p>[quote=ACS69;56475]</p>\n<p>On First Derivative data, you could get .44 with a GBM</p>\n<p>[/quote]</p>\n<p>Thanks, yes - I'd got that far or somewhere similar. But I made the mistake of thinking it wasn't close enough to the svm score of 0.40 to be worth merging the two.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 56475,
      "author_name": "Bluefool",
      "author_url": "",
      "post_date": "2014-10-22T10:21:52.210000",
      "content": "<p>On First Derivative data, you could get .44 with a GBM</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 56470,
      "author_name": "Bluefool",
      "author_url": "",
      "post_date": "2014-10-22T10:08:47.937000",
      "content": "<p>All the benchmarks on the forum overfitted.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 7,
  "messages": [
    {
      "id": 56502,
      "postDate": "2014-10-22T14:22:17.640Z",
      "content": "<p>Vielauge, using caret's&nbsp;method for cross-fold is generally a good idea, but in this competition random CV&nbsp;can be dangerous. Since the test data was known to be ordered by location,&nbsp;test data shared no locations, and you could clearly see that the high-range values for a few of the targets clustered together, a straight-forward cross-validation might&nbsp;not be&nbsp;the way to go. It's possible that isn't your problem, but that seems the first point of suspicion to me.</p>",
      "votes": 1
    },
    {
      "id": 56474,
      "postDate": "2014-10-22T10:20:49.240Z",
      "content": "<p>I share some of your pain...</p>\n<p>Based on the 'solution sharing' post maybe we should have focused on broader ensembles rather than tuning. I think I made the mistake of dismissing options like gbm just because they weren't as good as svm according to my CV scores. I didn't pursue combinations as much as I might have done.</p>\n<p>I like this explanation of the benefits of that approach</p>\n<p><a href=\"http://www.overkillanalytics.net/more-is-always-better-the-power-of-simple-ensembles/\">http://www.overkillanalytics.net/more-is-always-better-the-power-of-simple-ensembles/</a></p>",
      "votes": 1
    },
    {
      "id": 56739,
      "postDate": "2014-10-25T12:56:26.283Z",
      "content": "<p>Thank you all for answering this private thread!</p>\n<p>Yes, some things learned:</p>\n<p>1. Cross validation can be misleading.. and there were a lot of other competitors who stepped into this trap, as we can read in many other forum entries. You are right mlandry, I guess this was the main issue.</p>\n<p>2. I already knew that ensembles are powerful, but maybe they are even more powerful than I thought. Thank you for the fine article lewis ml!</p>\n<p>We will see us in one of the next kaggle adventures.</p>"
    },
    {
      "id": 56478,
      "postDate": "2014-10-22T10:39:28.127Z",
      "content": "<p>[quote=lewis ml;56477]</p>\n<p>[quote=ACS69;56475]</p>\n<p>On First Derivative data, you could get .44 with a GBM</p>\n<p>[/quote]</p>\n<p>Thanks, yes - I'd got that far or somewhere similar. But I made the mistake of thinking it wasn't close enough to the svm score of 0.40 to be worth merging the two.</p>\n<p>[/quote]</p>\n<p>oh no - rule of data analysis - ensemble in everything. Even if you did a (svm * 0.75) + (gbm * 0.25)</p>\n\n<p>EDIT: I even ensembled in scores of 0.47</p>"
    },
    {
      "id": 56477,
      "postDate": "2014-10-22T10:28:14.573Z",
      "content": "<p>[quote=ACS69;56475]</p>\n<p>On First Derivative data, you could get .44 with a GBM</p>\n<p>[/quote]</p>\n<p>Thanks, yes - I'd got that far or somewhere similar. But I made the mistake of thinking it wasn't close enough to the svm score of 0.40 to be worth merging the two.</p>"
    },
    {
      "id": 56475,
      "postDate": "2014-10-22T10:21:52.210Z",
      "content": "<p>On First Derivative data, you could get .44 with a GBM</p>"
    },
    {
      "id": 56470,
      "postDate": "2014-10-22T10:08:47.937Z",
      "content": "<p>All the benchmarks on the forum overfitted.</p>"
    },
    {
      "id": 56468,
      "postDate": "2014-10-22T10:04:16.290Z",
      "content": "<p>-Well it started all with Abisheks Benchmark Code and the transformation into R - thanks for sharing!</p>\n<p><br>-Then I set up my own cross validation with the help of the great caret package!<br>#5-fold-crossvalidation<br>fitControl&lt;-trainControl(method=&quot;repeatedcv&quot;,number=5,repeats=2)</p>\n<p><br>optimizing c for each Parameter- (example for Ca)<br>svmFitCa2&lt;-train(x=train2,y=labels[,1],method=&quot;svmRadial&quot;,trControl=fitControl,tuneLength=10)<br>leading to good RMSE between 0.3 and 0.4 with the exception of Phosphor RMSE=0.841.</p>\n<p><br>Adding the untransformed satellite data improved the RMSE for Ph and Sand a little bit. (so I combined these predictions with the above ones = my first final submission) -&gt; leading to public rank in the last days about 350th</p>\n<p><br>The biggest problem was &quot;P&quot; so I decided to run a random forest for it. The prediction quality was in the same category as the svm -&gt; RMSE=0.8893 so i ensembled svm with rf leading to bad public leaderborad score 0.49078 but hoping for a better private score. (my second final submission)</p>\n<p><br>Well the final results are a complete knock down! The mass with unchanged benchmark code scored better than me. (I hoped for 25 % with a bit luck 10% but ended below 50%). Is this overfit? I thought I am relatively secure due to the cross validation.</p>\n<p><br>What happened? I am not sure. One fault was to rely on more data. I should stripped off the CO2 and maybe a lot more variables (saving a lot computing time). Does anyone see another major mistake?</p>\n<p>A big hug to all who dropped more than 50 places, grats to the winners and thank you for all the sharing in the forums-</p>\n<p>Best regards, Vielauge</p>"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:35:20Z",
  "competition_id": 3966,
  "competition_slug": "afsis-soil-properties",
  "team_id": 118041,
  "solution_url": "/competitions/afsis-soil-properties/writeups/vielauge-how-to-drop-more-than-500-places"
}