Where do I start with - I guess I got several good submission files but I missed two of them (got me at least 0.517~0.518 in private LB), which during that time I am a bit hesitate to select as I am not confident they will produce robust results... Sad story for me as this shakeup happened (also I am fortunate not to select the other files that produce even worse private LB score). Nobody to blame, this is really the opportunity to learn and grow.
\n\nSo to many experts, masters+, if you have gone through such incident before, could you please share your journey & experience in terms of building robust data, model, submission pipelines or any other aspects? Also, if your score jumped quite a bit on the private LB, feel free to share your thoughts as well. I believe most folks will benefit from this in the future.
\n\nThanks!
", "messages": [ { "id": "268959", "postDate": "01/16/2018 01:33:31", "content": "Where do I start with - I guess I got several good submission files but I missed two of them (got me at least 0.517~0.518 in private LB), which during that time I am a bit hesitate to select as I am not confident they will produce robust results... Sad story for me as this shakeup happened (also I am fortunate not to select the other files that produce even worse private LB score). Nobody to blame, this is really the opportunity to learn and grow.
\n\nSo to many experts, masters+, if you have gone through such incident before, could you please share your journey & experience in terms of building robust data, model, submission pipelines or any other aspects? Also, if your score jumped quite a bit on the private LB, feel free to share your thoughts as well. I believe most folks will benefit from this in the future.
\n\nThanks!
", "rawMarkdown": "Where do I start with - I guess I got several good submission files but I missed two of them (got me at least 0.517~0.518 in private LB), which during that time I am a bit hesitate to select as I am not confident they will produce robust results... Sad story for me as this shakeup happened (also I am fortunate not to select the other files that produce even worse private LB score). Nobody to blame, this is really the opportunity to learn and grow. \n\nSo to many experts, masters+, if you have gone through such incident before, could you please share your journey & experience in terms of building robust data, model, submission pipelines or any other aspects? Also, if your score jumped quite a bit on the private LB, feel free to share your thoughts as well. I believe most folks will benefit from this in the future. \n\nThanks!", "votes": null }, { "id": "268980", "postDate": "01/16/2018 02:43:07", "content": "Trust on your knowledge and analysis. Don't be affected by other's performance on public board. This is what I learned.
", "rawMarkdown": "Trust on your knowledge and analysis. Don't be affected by other's performance on public board. This is what I learned.", "votes": null }, { "id": "268986", "postDate": "01/16/2018 02:57:32", "content": "Thanks Jacob. Valuable lesson learned!
", "rawMarkdown": "Thanks Jacob. Valuable lesson learned!", "votes": null }, { "id": "268996", "postDate": "01/16/2018 03:07:04", "content": "Same with me, and I finally decided to avarage all model that get good result on local test to make it more robust. This is a right decision but just too late. I miss the deadline 2 minute. So sad.
", "rawMarkdown": "Same with me, and I finally decided to avarage all model that get good result on local test to make it more robust. This is a right decision but just too late. I miss the deadline 2 minute. So sad.", "votes": null }, { "id": "269001", "postDate": "01/16/2018 03:27:53", "content": "I am sorry to hear that... One day, our chance will turn back!
", "rawMarkdown": "I am sorry to hear that... One day, our chance will turn back!", "votes": null }, { "id": "269015", "postDate": "01/16/2018 04:02:34", "content": "Actually, it is not that easy as I thoutht before. It is really like lottery just as sjv said. \nMy best lgb single model get 0.518 on private and 0.514 on public.\nMy best nn single model get 0.519 on private and 0.515 on public.\nIf I average them, I can get 0.515 on private but 0.511 on public. (My glod medal<(=′⌒`=)>）
\n\nBut!! Since I had tons of model get public score 0.510 0.511... and get similar score in local test. I would never average these two model even give me 10 more chances. So, I believe that lots of kagglers in that competition would regret that they had not select their best model for final score. The public leaderboard is really misleading! Just check the data quality before commiting to the competition otherwise it would be really painful to evaluate and get a big shake-up.
", "rawMarkdown": "Actually, it is not that easy as I thoutht before. It is really like lottery just as sjv said. \nMy best lgb single model get 0.518 on private and 0.514 on public.\nMy best nn single model get 0.519 on private and 0.515 on public.\nIf I average them, I can get 0.515 on private but 0.511 on public. (My glod medal<(=′⌒`=)>）\n\nBut!! Since I had tons of model get public score 0.510 0.511... and get similar score in local test. I would never average these two model even give me 10 more chances. So, I believe that lots of kagglers in that competition would regret that they had not select their best model for final score. The public leaderboard is really misleading! Just check the data quality before commiting to the competition otherwise it would be really painful to evaluate and get a big shake-up.", "votes": null } ], "comments": [ { "id": 268980, "author_name": "creatrol", "author_url": "", "post_date": "01/16/2018 02:43:07", "content": "Trust on your knowledge and analysis. Don't be affected by other's performance on public board. This is what I learned.
", "votes": null, "replies": [ { "id": 268986, "author_name": "dongxu027", "author_url": "", "post_date": "01/16/2018 02:57:32", "content": "Thanks Jacob. Valuable lesson learned!
", "votes": null, "replies": [] } ] }, { "id": 268996, "author_name": "yedthon", "author_url": "", "post_date": "01/16/2018 03:07:04", "content": "Same with me, and I finally decided to avarage all model that get good result on local test to make it more robust. This is a right decision but just too late. I miss the deadline 2 minute. So sad.
", "votes": null, "replies": [ { "id": 269001, "author_name": "dongxu027", "author_url": "", "post_date": "01/16/2018 03:27:53", "content": "I am sorry to hear that... One day, our chance will turn back!
", "votes": null, "replies": [] }, { "id": 269015, "author_name": "yedthon", "author_url": "", "post_date": "01/16/2018 04:02:34", "content": "Actually, it is not that easy as I thoutht before. It is really like lottery just as sjv said. \nMy best lgb single model get 0.518 on private and 0.514 on public.\nMy best nn single model get 0.519 on private and 0.515 on public.\nIf I average them, I can get 0.515 on private but 0.511 on public. (My glod medal<(=′⌒`=)>）
\n\nBut!! Since I had tons of model get public score 0.510 0.511... and get similar score in local test. I would never average these two model even give me 10 more chances. So, I believe that lots of kagglers in that competition would regret that they had not select their best model for final score. The public leaderboard is really misleading! Just check the data quality before commiting to the competition otherwise it would be really painful to evaluate and get a big shake-up.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "268959": "Where do I start with - I guess I got several good submission files but I missed two of them (got me at least 0.517~0.518 in private LB), which during that time I am a bit hesitate to select as I am not confident they will produce robust results... Sad story for me as this shakeup happened (also I am fortunate not to select the other files that produce even worse private LB score). Nobody to blame, this is really the opportunity to learn and grow. \n\nSo to many experts, masters+, if you have gone through such incident before, could you please share your journey & experience in terms of building robust data, model, submission pipelines or any other aspects? Also, if your score jumped quite a bit on the private LB, feel free to share your thoughts as well. I believe most folks will benefit from this in the future. \n\nThanks!", "268980": "Trust on your knowledge and analysis. Don't be affected by other's performance on public board. This is what I learned.", "268986": "Thanks Jacob. Valuable lesson learned!", "268996": "Same with me, and I finally decided to avarage all model that get good result on local test to make it more robust. This is a right decision but just too late. I miss the deadline 2 minute. So sad.", "269001": "I am sorry to hear that... One day, our chance will turn back!", "269015": "Actually, it is not that easy as I thoutht before. It is really like lottery just as sjv said. \nMy best lgb single model get 0.518 on private and 0.514 on public.\nMy best nn single model get 0.519 on private and 0.515 on public.\nIf I average them, I can get 0.515 on private but 0.511 on public. (My glod medal<(=′⌒`=)>）\n\nBut!! Since I had tons of model get public score 0.510 0.511... and get similar score in local test. I would never average these two model even give me 10 more chances. So, I believe that lots of kagglers in that competition would regret that they had not select their best model for final score. The public leaderboard is really misleading! Just check the data quality before commiting to the competition otherwise it would be really painful to evaluate and get a big shake-up." }, "source": "meta" }