{
  "id": 14616,
  "title": "New model scores 0.9232 but I have no quota left today!",
  "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14616",
  "author_name": "",
  "post_date": "2015-06-08T16:53:49.413Z",
  "votes": null,
  "comment_count": 23,
  "views": 3740,
  "content": "<p>Just tried a new feature by which my model was significantly improved from 0.91011 to&nbsp;0.92321 based on 10 times/10 folds validation. Hooray!</p>\n<p>But wait one second, I have no quota of submission left for today!!</p>\n<p>What a tragedy!!!</p>",
  "messages": [
    {
      "id": "81241",
      "postDate": "06/08/2015 16:53:49",
      "content": "<p>Just tried a new feature by which my model was significantly improved from 0.91011 to&nbsp;0.92321 based on 10 times/10 folds validation. Hooray!</p>\n<p>But wait one second, I have no quota of submission left for today!!</p>\n<p>What a tragedy!!!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81256",
      "postDate": "06/08/2015 18:49:10",
      "content": "<p>I also had a jump on validation score today, but when I submit it, the leader board score is however a little bit&nbsp;worse. Turn out I am overfitting to the training data...</p>\n<p>I just realized because the training size is small, the positive instances size is only 103, it is very easy to overfit to the training data, but if we want to perform well, do we need to overfit to the LB and test data?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81257",
      "postDate": "06/08/2015 19:14:46",
      "content": "<p>[quote=SkyLibrary;81256]</p>\n<p>I also had a jump on validation score today, but when I submit it, the leader board score is however a little bit&nbsp;worse. Turn out I am overfitting to the training data...</p>\n<p>I just realized because the training size is small, the positive instances size is only 103, it is very easy to overfit to the training data, but if we want to perform well, do we need to overfit to the LB and test data?</p>\n<p>[/quote]</p>\n<p>You are right. I have a model which gives 0.97 mean AUC with 0.01 SD in 10 fold CV repeated 10 times. It gives 0.8 on LB.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81266",
      "postDate": "06/08/2015 20:18:59",
      "content": "<p>I've trained models that have given me up to 0.97 but they end up being like 0.89 on the leaderboard. I wouldn't worry too much about it. My final submission had a CV of 0.925 and a final leaderboard of ~ 0.92</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81271",
      "postDate": "06/08/2015 20:48:40",
      "content": "<p>The cv process for this comp soundly defeated me.</p>\n\n<p>but I learned a ton- excited to see the final leaderboard&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81272",
      "postDate": "06/08/2015 20:55:14",
      "content": "<p>In my case, CV scored pretty close to LB that's why I feel a bit regret about having no more chance to submit&nbsp;it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81273",
      "postDate": "06/08/2015 20:58:27",
      "content": "<p>Maybe but in my case my CV and leaderboard were almost exactly the same until 0.91-0.915. After that they started to diverge. CV started to get more but leaderboard barely moved or got worse. I hit 0.91 on my 3rd real&nbsp;attempt and then struggled after that.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81274",
      "postDate": "06/08/2015 21:04:47",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81275",
      "postDate": "06/08/2015 21:29:02",
      "content": "<p>[quote=ask788;81273]</p>\n<p>Maybe but in my case my CV and leaderboard were almost exactly the same until 0.91-0.915. After that they started to diverge. CV started to get more but leaderboard barely moved or got worse. I hit 0.91 on my 3rd real&nbsp;attempt and then struggled after that.&nbsp;</p>\n<p>[/quote]</p>\n<p>Same here. Starting from&nbsp;some point, my local score continued to improve while I couldn't beat my best&nbsp;leaderboard score anymore.&nbsp;Whatever technique I applied since then&nbsp;(parameters tuning, ensembling, feature elimination etc) - it helped to improve the CV score but inevitably damaged the public score. I even tried to rework my model completely and compounded a minimalistic set of features - that yielded around 0.931 locally and still significantly reduced the LB score. A bit frustrating, given that I like the&nbsp;&quot;best local score&quot; model more than the&nbsp;&quot;best LB score&quot; model - the former contains less&nbsp;features and they&nbsp;look more logical and reasonable.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81276",
      "postDate": "06/08/2015 22:00:12",
      "content": "<p>Lessons learned, &nbsp;struggling with overfitting on a small dataset isn't fun at all.&nbsp;</p>\n<p>I am not sure why the dataset is so small,&nbsp;are there suppose to be plenty of data for online bidding, or it is just because there is not enough labeled data...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81280",
      "postDate": "06/08/2015 23:04:24",
      "content": "<p>It turns out the last feature I tried had a significant improvement on the models</p>\n\n<p>Avg(time - last_bid_time_of_auction)/(last_bid_time_of_auction -&nbsp;first_bid_time_of_auction))</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81281",
      "postDate": "06/08/2015 23:05:11",
      "content": "<p>the final leaderboard may surprise you: I saw many cases that CV looked bad on public LB but eventually pushed you up to a very high score in the private LB. For me, I trust CV more, although I didn't spent much time on this competition.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81282",
      "postDate": "06/08/2015 23:09:57",
      "content": "<p>The rational behind this is the bots may tend to make bids as close as possible to auction deadline to get a better chance to win without overpricing the auction by making to many unnecessary bids.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81284",
      "postDate": "06/08/2015 23:20:43",
      "content": "<p>that sounds reasonable, but since the data is cut, I am not sure if we see the end of the auctions at all. And when you look at the time series, then it seems that most of the bids are made at the beginning of each of the periods, which is counter intuitive. So the time might actually be reversed.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81285",
      "postDate": "06/08/2015 23:41:23",
      "content": "<p>&nbsp;It's actually possible the time serials are reversed - that's why I calculated the average distance of bids between the time span. It turns out bots tend to have lower &quot;time&quot; values than human. That can only be explained by reversed time. Either way, the engineered feature yields the difference so it helped a lot</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81288",
      "postDate": "06/09/2015 00:12:45",
      "content": "<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81308",
      "postDate": "06/09/2015 02:33:52",
      "content": "<p>Kudos ask788!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81322",
      "postDate": "06/09/2015 05:16:05",
      "content": "<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81325",
      "postDate": "06/09/2015 07:15:41",
      "content": "<p>[quote=cfh;81322]</p>\n<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>\n<p>[/quote]</p>\n<p>10 Fold Stratified worked very well for me. Negligible difference b/w private and CV scores.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81332",
      "postDate": "06/09/2015 09:14:07",
      "content": "<p>[quote=cfh;81322]</p>\n<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>\n<p>[/quote]</p>\n<p>I completely understand you. :) Just shamefully moved from 25th to 209th. However, this is a good lesson for me to avoid chasing for any minuscule improvement in the LB score and trust the CV score more. I was following the &quot;let's see if it works&quot; approach and concentrated on&nbsp;engineering any features that worked, regardless of their meaning. Some of them didn't have a sound rationale behind, they just worked and that's why I left them in the input set. It seems like I ended up with a model that was overfit to the public fragment of testing data and worked bad in terms of generalization. Yet ~0.922 isn't a bad score overall. :)</p>\n<p><a href=\"http://www.chioka.in/how-to-select-your-final-models-in-a-kaggle-competitio/\">Here is an article</a>&nbsp;written by a kaggler that provides some hints to avoid the drop we've just experienced. And congrats to all the winners! :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81363",
      "postDate": "06/09/2015 16:02:32",
      "content": "<p>In my case, SK's&nbsp;GridSearchCV didn't work very well in terms of the accuracy on &quot;auc_roc&quot; scoring. That's why I had to manually code my grid search function which uses 10*10 CVs for each combination of parameters and gave me much better results.</p>\n\n<p>[quote=cfh;81322]</p>\n<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81366",
      "postDate": "06/09/2015 16:08:04",
      "content": "<p>Just a quick update:</p>\n<p>With the new feature I mentioned, the model ended up with a score of&nbsp;0.92609 in PB, and would have ranked #154, if I could have had the chance to submit it, which would have been a significant improvement as compared my actual results:&nbsp;0.92028 and #245.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81367",
      "postDate": "06/09/2015 16:13:04",
      "content": "<p>I disagree that gridsearchCV gives meaningless results if you train on a big chunk of the ensemble. If you apply manual cuts however, I agree that one has to be careful in interpreting the score obtained on a subset of the data. For me, grid search cv with 5 fold cv and roc auc scorer was almost always on spot with my prediction.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81506",
      "postDate": "06/10/2015 18:06:10",
      "content": "<p>[quote=ChrisCC;81241]</p>\n<p>Just tried a new feature by which my model was significantly improved from 0.91011 to&nbsp;0.92321 based on 10 times/10 folds validation. Hooray!</p>\n<p>But wait one second, I have no quota of submission left for today!!</p>\n<p>What a tragedy!!!</p>\n<p>[/quote]</p>\n<p>So what did it score?</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 81256,
      "author_name": "skylibrary",
      "author_url": "",
      "post_date": "06/08/2015 18:49:10",
      "content": "<p>I also had a jump on validation score today, but when I submit it, the leader board score is however a little bit&nbsp;worse. Turn out I am overfitting to the training data...</p>\n<p>I just realized because the training size is small, the positive instances size is only 103, it is very easy to overfit to the training data, but if we want to perform well, do we need to overfit to the LB and test data?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81257,
      "author_name": "thakurrajanand",
      "author_url": "",
      "post_date": "06/08/2015 19:14:46",
      "content": "<p>[quote=SkyLibrary;81256]</p>\n<p>I also had a jump on validation score today, but when I submit it, the leader board score is however a little bit&nbsp;worse. Turn out I am overfitting to the training data...</p>\n<p>I just realized because the training size is small, the positive instances size is only 103, it is very easy to overfit to the training data, but if we want to perform well, do we need to overfit to the LB and test data?</p>\n<p>[/quote]</p>\n<p>You are right. I have a model which gives 0.97 mean AUC with 0.01 SD in 10 fold CV repeated 10 times. It gives 0.8 on LB.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81266,
      "author_name": "ask788",
      "author_url": "",
      "post_date": "06/08/2015 20:18:59",
      "content": "<p>I've trained models that have given me up to 0.97 but they end up being like 0.89 on the leaderboard. I wouldn't worry too much about it. My final submission had a CV of 0.925 and a final leaderboard of ~ 0.92</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81271,
      "author_name": "scharf",
      "author_url": "",
      "post_date": "06/08/2015 20:48:40",
      "content": "<p>The cv process for this comp soundly defeated me.</p>\n\n<p>but I learned a ton- excited to see the final leaderboard&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81272,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "06/08/2015 20:55:14",
      "content": "<p>In my case, CV scored pretty close to LB that's why I feel a bit regret about having no more chance to submit&nbsp;it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81273,
      "author_name": "ask788",
      "author_url": "",
      "post_date": "06/08/2015 20:58:27",
      "content": "<p>Maybe but in my case my CV and leaderboard were almost exactly the same until 0.91-0.915. After that they started to diverge. CV started to get more but leaderboard barely moved or got worse. I hit 0.91 on my 3rd real&nbsp;attempt and then struggled after that.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81274,
      "author_name": "ramanand",
      "author_url": "",
      "post_date": "06/08/2015 21:04:47",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 81275,
      "author_name": "dmitryvorobyev",
      "author_url": "",
      "post_date": "06/08/2015 21:29:02",
      "content": "<p>[quote=ask788;81273]</p>\n<p>Maybe but in my case my CV and leaderboard were almost exactly the same until 0.91-0.915. After that they started to diverge. CV started to get more but leaderboard barely moved or got worse. I hit 0.91 on my 3rd real&nbsp;attempt and then struggled after that.&nbsp;</p>\n<p>[/quote]</p>\n<p>Same here. Starting from&nbsp;some point, my local score continued to improve while I couldn't beat my best&nbsp;leaderboard score anymore.&nbsp;Whatever technique I applied since then&nbsp;(parameters tuning, ensembling, feature elimination etc) - it helped to improve the CV score but inevitably damaged the public score. I even tried to rework my model completely and compounded a minimalistic set of features - that yielded around 0.931 locally and still significantly reduced the LB score. A bit frustrating, given that I like the&nbsp;&quot;best local score&quot; model more than the&nbsp;&quot;best LB score&quot; model - the former contains less&nbsp;features and they&nbsp;look more logical and reasonable.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81276,
      "author_name": "skylibrary",
      "author_url": "",
      "post_date": "06/08/2015 22:00:12",
      "content": "<p>Lessons learned, &nbsp;struggling with overfitting on a small dataset isn't fun at all.&nbsp;</p>\n<p>I am not sure why the dataset is so small,&nbsp;are there suppose to be plenty of data for online bidding, or it is just because there is not enough labeled data...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81280,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "06/08/2015 23:04:24",
      "content": "<p>It turns out the last feature I tried had a significant improvement on the models</p>\n\n<p>Avg(time - last_bid_time_of_auction)/(last_bid_time_of_auction -&nbsp;first_bid_time_of_auction))</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81281,
      "author_name": "phunter",
      "author_url": "",
      "post_date": "06/08/2015 23:05:11",
      "content": "<p>the final leaderboard may surprise you: I saw many cases that CV looked bad on public LB but eventually pushed you up to a very high score in the private LB. For me, I trust CV more, although I didn't spent much time on this competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81282,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "06/08/2015 23:09:57",
      "content": "<p>The rational behind this is the bots may tend to make bids as close as possible to auction deadline to get a better chance to win without overpricing the auction by making to many unnecessary bids.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81284,
      "author_name": "tkurth",
      "author_url": "",
      "post_date": "06/08/2015 23:20:43",
      "content": "<p>that sounds reasonable, but since the data is cut, I am not sure if we see the end of the auctions at all. And when you look at the time series, then it seems that most of the bids are made at the beginning of each of the periods, which is counter intuitive. So the time might actually be reversed.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81285,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "06/08/2015 23:41:23",
      "content": "<p>&nbsp;It's actually possible the time serials are reversed - that's why I calculated the average distance of bids between the time span. It turns out bots tend to have lower &quot;time&quot; values than human. That can only be explained by reversed time. Either way, the engineered feature yields the difference so it helped a lot</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81288,
      "author_name": "ask788",
      "author_url": "",
      "post_date": "06/09/2015 00:12:45",
      "content": "<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81308,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "06/09/2015 02:33:52",
      "content": "<p>Kudos ask788!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81322,
      "author_name": "cfhlearns",
      "author_url": "",
      "post_date": "06/09/2015 05:16:05",
      "content": "<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81325,
      "author_name": "thakurrajanand",
      "author_url": "",
      "post_date": "06/09/2015 07:15:41",
      "content": "<p>[quote=cfh;81322]</p>\n<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>\n<p>[/quote]</p>\n<p>10 Fold Stratified worked very well for me. Negligible difference b/w private and CV scores.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81332,
      "author_name": "dmitryvorobyev",
      "author_url": "",
      "post_date": "06/09/2015 09:14:07",
      "content": "<p>[quote=cfh;81322]</p>\n<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>\n<p>[/quote]</p>\n<p>I completely understand you. :) Just shamefully moved from 25th to 209th. However, this is a good lesson for me to avoid chasing for any minuscule improvement in the LB score and trust the CV score more. I was following the &quot;let's see if it works&quot; approach and concentrated on&nbsp;engineering any features that worked, regardless of their meaning. Some of them didn't have a sound rationale behind, they just worked and that's why I left them in the input set. It seems like I ended up with a model that was overfit to the public fragment of testing data and worked bad in terms of generalization. Yet ~0.922 isn't a bad score overall. :)</p>\n<p><a href=\"http://www.chioka.in/how-to-select-your-final-models-in-a-kaggle-competitio/\">Here is an article</a>&nbsp;written by a kaggler that provides some hints to avoid the drop we've just experienced. And congrats to all the winners! :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81363,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "06/09/2015 16:02:32",
      "content": "<p>In my case, SK's&nbsp;GridSearchCV didn't work very well in terms of the accuracy on &quot;auc_roc&quot; scoring. That's why I had to manually code my grid search function which uses 10*10 CVs for each combination of parameters and gave me much better results.</p>\n\n<p>[quote=cfh;81322]</p>\n<p>[quote=ask788;81288]</p>\n<p>OH WOW!! I ended up at 8th place. Jumping a whole 81 positions. This was only my 3rd Kaggle competition and I must say I'm surprised. Only a few hours ago I was telling myself I'd be happy if I didn't get kicked out of the top 10%. I thought I didn't have a chance here :)</p>\n<p>[/quote]</p>\n<p>I must say I envy you: I was sitting at 50 in the public LB, hoping to secure my first 10% finish without much sweat, and what happened? I fell down to rank 369. Very disappointed to say the least.</p>\n<p>I'll have to find out my CV procedure was flawed (which is hard to imagine since I just used sklearn components, 10-fold stratified CV), or if there was truly that much randomness in the change from public to private LB.</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81366,
      "author_name": "chriscc",
      "author_url": "",
      "post_date": "06/09/2015 16:08:04",
      "content": "<p>Just a quick update:</p>\n<p>With the new feature I mentioned, the model ended up with a score of&nbsp;0.92609 in PB, and would have ranked #154, if I could have had the chance to submit it, which would have been a significant improvement as compared my actual results:&nbsp;0.92028 and #245.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81367,
      "author_name": "tkurth",
      "author_url": "",
      "post_date": "06/09/2015 16:13:04",
      "content": "<p>I disagree that gridsearchCV gives meaningless results if you train on a big chunk of the ensemble. If you apply manual cuts however, I agree that one has to be careful in interpreting the score obtained on a subset of the data. For me, grid search cv with 5 fold cv and roc auc scorer was almost always on spot with my prediction.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81506,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "06/10/2015 18:06:10",
      "content": "<p>[quote=ChrisCC;81241]</p>\n<p>Just tried a new feature by which my model was significantly improved from 0.91011 to&nbsp;0.92321 based on 10 times/10 folds validation. Hooray!</p>\n<p>But wait one second, I have no quota of submission left for today!!</p>\n<p>What a tragedy!!!</p>\n<p>[/quote]</p>\n<p>So what did it score?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "81241": "",
    "81256": "",
    "81257": "",
    "81266": "",
    "81271": "",
    "81272": "",
    "81273": "",
    "81274": "",
    "81275": "",
    "81276": "",
    "81280": "",
    "81281": "",
    "81282": "",
    "81284": "",
    "81285": "",
    "81288": "",
    "81308": "",
    "81322": "",
    "81325": "",
    "81332": "",
    "81363": "",
    "81366": "",
    "81367": "",
    "81506": ""
  },
  "source": "meta"
}