Our team decided not to use any leakage data and results were terrible. From the top 30 on LB (before people started to use leakages we fall down to 1300+ place, 1000+ on private LB).
\n\nSo the question is what is the best PRIVATE score for your models (with leaks or without it) ? And what are that scores?
\n\nP.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.
", "messages": [ { "id": "705055", "postDate": "12/28/2019 11:05:17", "content": "Our team decided not to use any leakage data and results were terrible. From the top 30 on LB (before people started to use leakages we fall down to 1300+ place, 1000+ on private LB).
\n\nSo the question is what is the best PRIVATE score for your models (with leaks or without it) ? And what are that scores?
\n\nP.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.
", "rawMarkdown": "Our team decided not to use any leakage data and results were terrible. From the top 30 on LB (before people started to use leakages we fall down to 1300+ place, 1000+ on private LB).\n\nSo the question is what is the best PRIVATE score for your models (with leaks or without it) ? And what are that scores?\n\n\n\nP.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.", "votes": null }, { "id": "705061", "postDate": "12/28/2019 11:16:12", "content": " , Public , Private\nWith leak: 0.9419735342595975,1.2610372245679247\nWithout leak: 1.064485714837533, 1.3233004688722467
", "rawMarkdown": ", Public , Private\nWith leak: 0.9419735342595975,1.2610372245679247\nWithout leak: 1.064485714837533, 1.3233004688722467", "votes": null }, { "id": "705071", "postDate": "12/28/2019 11:33:24", "content": "I never trained with leaks so I can't tell you. But my private LB is 1.242-1.243. Honestly, the only thing useful was using leak to validate, but at some point using it to validate caused some overfitting to those sites. The extent however was mitigated for me.
", "rawMarkdown": "I never trained with leaks so I can't tell you. But my private LB is 1.242-1.243. Honestly, the only thing useful was using leak to validate, but at some point using it to validate caused some overfitting to those sites. The extent however was mitigated for me.", "votes": null }, { "id": "705080", "postDate": "12/28/2019 11:40:31", "content": "So you've got 0.93 on LB without any usage of leakage?
", "rawMarkdown": "So you've got 0.93 on LB without any usage of leakage?", "votes": null }, { "id": "705086", "postDate": "12/28/2019 11:45:38", "content": "I replaced the test labels just to see how far I would get. Bare in mind, I posted all my scores with no leak whatsoever here. I also made a post on why I chose not to train using any leaks in my solution. Replacing submission file with sites 0,1,2,4,15 gave about 0.095 - 0.1 improvement on public LB (it did nothing for private LB, but I wasn't concerned about that)
", "rawMarkdown": "I replaced the test labels just to see how far I would get. Bare in mind, I posted all my scores with no leak whatsoever [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/118214). I also made a post on why I chose not to train using any leaks in my [solution](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123496). Replacing submission file with sites 0,1,2,4,15 gave about 0.095 - 0.1 improvement on public LB (it did nothing for private LB, but I wasn't concerned about that)", "votes": null }, { "id": "705273", "postDate": "12/28/2019 17:43:58", "content": "I did not use leak data for neither train nor valid as I promised when It occurred. I blended 1.06 1.07 1.08 1.09 models from mine, so that I made blended prediction set to be high correlations for all 4 models(meter-wise), and That was my best. We did not know which site which meter reading will be crucial, so I think this is safe way not to shake down drastically.
", "rawMarkdown": "I did not use leak data for neither train nor valid as I promised when It occurred. I blended 1.06 1.07 1.08 1.09 models from mine, so that I made blended prediction set to be high correlations for all 4 models(meter-wise), and That was my best. We did not know which site which meter reading will be crucial, so I think this is safe way not to shake down drastically.", "votes": null }, { "id": "706183", "postDate": "12/30/2019 03:14:17", "content": "May be local overfitting can lead to high scores. This is especially true when you mix models with the highest local CV scores with your best LB scores.In private i get 20+,but in lb it is 1000+(1.06 + 1.07).i don't use leakage to train
", "rawMarkdown": "May be local overfitting can lead to high scores. This is especially true when you mix models with the highest local CV scores with your best LB scores.In private i get 20+,but in lb it is 1000+(1.06 + 1.07).i don't use leakage to train", "votes": null }, { "id": "706210", "postDate": "12/30/2019 04:32:34", "content": "Yeah, There must be some lucky needed in this competition. Same approach but different results lead to whom to won gold , whom to fall down to drastically XD . you might be right. Some slightly Lucky over fitting to private set area could lead to high score. This complex theoretical - like competition no reason might be needed first… attaching it after XD. Otherwise, good trial and their own solid approach are worth regardless of what results may come. Good outcomes depends on somewhat other things also competition host?? XD Just naive thinking.
", "rawMarkdown": "Yeah, There must be some lucky needed in this competition. Same approach but different results lead to whom to won gold , whom to fall down to drastically XD . you might be right. Some slightly Lucky over fitting to private set area could lead to high score. This complex theoretical - like competition no reason might be needed first… attaching it after XD. Otherwise, good trial and their own solid approach are worth regardless of what results may come. Good outcomes depends on somewhat other things also competition host?? XD Just naive thinking.", "votes": null }, { "id": "706227", "postDate": "12/30/2019 05:12:33", "content": "I did not train with leak data but used it to validate sites 0,1,2,4,15. My best score without leak is around 1.03+
\n\nI agree with @teeyee314 that after getting your model to a certain strength, you face overfitting to those sites.
", "rawMarkdown": "I did not train with leak data but used it to validate sites 0,1,2,4,15. My best score without leak is around 1.03+ \n\nI agree with @teeyee314 that after getting your model to a certain strength, you face overfitting to those sites.", "votes": null }, { "id": "706234", "postDate": "12/30/2019 05:25:13", "content": ">P.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.
\n\nThe organizers already acknowledged that leak data can be used in any way: https://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357
\n\nThey also mentioned here (seems like the permalink isn't working as expected. You can just scroll down the same discussion thread) why they decided not to add it as part of official dataset.
", "rawMarkdown": ">P.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.\n\nThe organizers already acknowledged that leak data can be used in any way: https://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357\n\nThey also mentioned [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#682012) (seems like the permalink isn't working as expected. You can just scroll down the same discussion thread) why they decided not to add it as part of official dataset.", "votes": null } ], "comments": [ { "id": 705061, "author_name": "klloke", "author_url": "", "post_date": "12/28/2019 11:16:12", "content": " , Public , Private\nWith leak: 0.9419735342595975,1.2610372245679247\nWithout leak: 1.064485714837533, 1.3233004688722467
", "votes": null, "replies": [] }, { "id": 705071, "author_name": "teeyee314", "author_url": "", "post_date": "12/28/2019 11:33:24", "content": "I never trained with leaks so I can't tell you. But my private LB is 1.242-1.243. Honestly, the only thing useful was using leak to validate, but at some point using it to validate caused some overfitting to those sites. The extent however was mitigated for me.
", "votes": null, "replies": [ { "id": 705080, "author_name": "isaienkov", "author_url": "", "post_date": "12/28/2019 11:40:31", "content": "So you've got 0.93 on LB without any usage of leakage?
", "votes": null, "replies": [] }, { "id": 705086, "author_name": "teeyee314", "author_url": "", "post_date": "12/28/2019 11:45:38", "content": "I replaced the test labels just to see how far I would get. Bare in mind, I posted all my scores with no leak whatsoever here. I also made a post on why I chose not to train using any leaks in my solution. Replacing submission file with sites 0,1,2,4,15 gave about 0.095 - 0.1 improvement on public LB (it did nothing for private LB, but I wasn't concerned about that)
", "votes": null, "replies": [] } ] }, { "id": 705273, "author_name": "hanjoonchoe", "author_url": "", "post_date": "12/28/2019 17:43:58", "content": "I did not use leak data for neither train nor valid as I promised when It occurred. I blended 1.06 1.07 1.08 1.09 models from mine, so that I made blended prediction set to be high correlations for all 4 models(meter-wise), and That was my best. We did not know which site which meter reading will be crucial, so I think this is safe way not to shake down drastically.
", "votes": null, "replies": [] }, { "id": 706183, "author_name": "xhzs2333", "author_url": "", "post_date": "12/30/2019 03:14:17", "content": "May be local overfitting can lead to high scores. This is especially true when you mix models with the highest local CV scores with your best LB scores.In private i get 20+,but in lb it is 1000+(1.06 + 1.07).i don't use leakage to train
", "votes": null, "replies": [ { "id": 706210, "author_name": "hanjoonchoe", "author_url": "", "post_date": "12/30/2019 04:32:34", "content": "Yeah, There must be some lucky needed in this competition. Same approach but different results lead to whom to won gold , whom to fall down to drastically XD . you might be right. Some slightly Lucky over fitting to private set area could lead to high score. This complex theoretical - like competition no reason might be needed first… attaching it after XD. Otherwise, good trial and their own solid approach are worth regardless of what results may come. Good outcomes depends on somewhat other things also competition host?? XD Just naive thinking.
", "votes": null, "replies": [] } ] }, { "id": 706227, "author_name": "sheriytm", "author_url": "", "post_date": "12/30/2019 05:12:33", "content": "I did not train with leak data but used it to validate sites 0,1,2,4,15. My best score without leak is around 1.03+
\n\nI agree with @teeyee314 that after getting your model to a certain strength, you face overfitting to those sites.
", "votes": null, "replies": [] }, { "id": 706234, "author_name": "rohanrao", "author_url": "", "post_date": "12/30/2019 05:25:13", "content": ">P.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.
\n\nThe organizers already acknowledged that leak data can be used in any way: https://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357
\n\nThey also mentioned here (seems like the permalink isn't working as expected. You can just scroll down the same discussion thread) why they decided not to add it as part of official dataset.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "705055": "Our team decided not to use any leakage data and results were terrible. From the top 30 on LB (before people started to use leakages we fall down to 1300+ place, 1000+ on private LB).\n\nSo the question is what is the best PRIVATE score for your models (with leaks or without it) ? And what are that scores?\n\n\n\nP.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.", "705061": ", Public , Private\nWith leak: 0.9419735342595975,1.2610372245679247\nWithout leak: 1.064485714837533, 1.3233004688722467", "705071": "I never trained with leaks so I can't tell you. But my private LB is 1.242-1.243. Honestly, the only thing useful was using leak to validate, but at some point using it to validate caused some overfitting to those sites. The extent however was mitigated for me.", "705080": "So you've got 0.93 on LB without any usage of leakage?", "705086": "I replaced the test labels just to see how far I would get. Bare in mind, I posted all my scores with no leak whatsoever [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/118214). I also made a post on why I chose not to train using any leaks in my [solution](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/123496). Replacing submission file with sites 0,1,2,4,15 gave about 0.095 - 0.1 improvement on public LB (it did nothing for private LB, but I wasn't concerned about that)", "705273": "I did not use leak data for neither train nor valid as I promised when It occurred. I blended 1.06 1.07 1.08 1.09 models from mine, so that I made blended prediction set to be high correlations for all 4 models(meter-wise), and That was my best. We did not know which site which meter reading will be crucial, so I think this is safe way not to shake down drastically.", "706183": "May be local overfitting can lead to high scores. This is especially true when you mix models with the highest local CV scores with your best LB scores.In private i get 20+,but in lb it is 1000+(1.06 + 1.07).i don't use leakage to train", "706210": "Yeah, There must be some lucky needed in this competition. Same approach but different results lead to whom to won gold , whom to fall down to drastically XD . you might be right. Some slightly Lucky over fitting to private set area could lead to high score. This complex theoretical - like competition no reason might be needed first… attaching it after XD. Otherwise, good trial and their own solid approach are worth regardless of what results may come. Good outcomes depends on somewhat other things also competition host?? XD Just naive thinking.", "706227": "I did not train with leak data but used it to validate sites 0,1,2,4,15. My best score without leak is around 1.03+ \n\nI agree with @teeyee314 that after getting your model to a certain strength, you face overfitting to those sites.", "706234": ">P.S. I think organizers could deal with leakage in some better way. For example collect and add them to datasets to allow people to use them in free way without any additional actions.\n\nThe organizers already acknowledged that leak data can be used in any way: https://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357\n\nThey also mentioned [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#682012) (seems like the permalink isn't working as expected. You can just scroll down the same discussion thread) why they decided not to add it as part of official dataset." }, "source": "meta" }