Can the hosts update the supplemental training set with the new data during the evaluation phase? It would be nice for people who got worse scores than expected or got an error in their submission to check their models on the new data. Also, one of the updates to the supplemental training set had incorrect data in it. This was figured out by someone in the competition and not by the hosts. This move would ensure the integrity of this competition and provide transparency that I think the competitors in this competition deserve after months of working hard.
\nAlso, thanks to the hosts for hosting this competition and thanks to all the great ideas produced by other competitors.
", "messages": [ { "id": "1695012", "postDate": "02/17/2022 22:38:38", "content": "Can the hosts update the supplemental training set with the new data during the evaluation phase? It would be nice for people who got worse scores than expected or got an error in their submission to check their models on the new data. Also, one of the updates to the supplemental training set had incorrect data in it. This was figured out by someone in the competition and not by the hosts. This move would ensure the integrity of this competition and provide transparency that I think the competitors in this competition deserve after months of working hard.
\nAlso, thanks to the hosts for hosting this competition and thanks to all the great ideas produced by other competitors.
", "rawMarkdown": "Can the hosts update the supplemental training set with the new data during the evaluation phase? It would be nice for people who got worse scores than expected or got an error in their submission to check their models on the new data. Also, one of the updates to the supplemental training set had incorrect data in it. This was figured out by someone in the competition and not by the hosts. This move would ensure the integrity of this competition and provide transparency that I think the competitors in this competition deserve after months of working hard.\n\nAlso, thanks to the hosts for hosting this competition and thanks to all the great ideas produced by other competitors.", "votes": null }, { "id": "1695036", "postDate": "02/17/2022 22:49:22", "content": "It would be nice to have it, but even one of my submission which does not use supplemental, and simply does some silly calculation like Vwap * high/close for the prediction has also failed.. So I think the issue must be somewhere else, I find it hard to believe every single submission I made has failed including the simple baseline ones..
\n@cstein06 @sohier would be nice to hear if these issues are at least being looked into?
", "rawMarkdown": "It would be nice to have it, but even one of my submission which does not use supplemental, and simply does some silly calculation like Vwap * high/close for the prediction has also failed.. So I think the issue must be somewhere else, I find it hard to believe every single submission I made has failed including the simple baseline ones..\n\n@cstein06 @sohier would be nice to hear if these issues are at least being looked into?", "votes": null }, { "id": "1695043", "postDate": "02/17/2022 22:59:12", "content": "@petersk20 this is a great idea - it would be nice to validate our results and would solve all issues around integrity of final score
", "rawMarkdown": "petersk20 this is a great idea - it would be nice to validate our results and would solve all issues around integrity of final score", "votes": null }, { "id": "1695090", "postDate": "02/18/2022 00:15:01", "content": "Coundn't agree more, @petersk20 !
\nI would love to learn why my models with supplemental data has experienced a huge drop from non-leak LB. This might be because my models overfitted.
\nAnother possible cause is supplemental training set could have an error again as many people suspect. Full transparency will guarantee the integrity of the competition and maximize our learning opportunities.
\nAlso, I would love to know if there is anyone who can get high score without supplemental.csv. If yes, that indicates the supplemental.csv might be correct and vice-versa.
", "rawMarkdown": "Coundn't agree more, @petersk20 !\n\nI would love to learn why my models with supplemental data has experienced a huge drop from non-leak LB. This might be because my models overfitted. \n\nAnother possible cause is supplemental training set could have an error again as many people suspect. Full transparency will guarantee the integrity of the competition and maximize our learning opportunities. \n\nAlso, I would love to know if there is anyone who can get high score **without supplemental.csv**. If yes, that indicates the supplemental.csv might be correct and vice-versa.", "votes": null }, { "id": "1696599", "postDate": "02/19/2022 00:37:54", "content": "We can use late submissions to test our models after the competition ends in 3 months. It means that we need to hold our curiosity for 3 months to understand why our models went wrong.
", "rawMarkdown": "We can use late submissions to test our models after the competition ends in 3 months. It means that we need to hold our curiosity for 3 months to understand why our models went wrong.", "votes": null } ], "comments": [ { "id": 1695036, "author_name": "julianmukaj", "author_url": "", "post_date": "02/17/2022 22:49:22", "content": "It would be nice to have it, but even one of my submission which does not use supplemental, and simply does some silly calculation like Vwap * high/close for the prediction has also failed.. So I think the issue must be somewhere else, I find it hard to believe every single submission I made has failed including the simple baseline ones..
\n@cstein06 @sohier would be nice to hear if these issues are at least being looked into?
", "votes": null, "replies": [] }, { "id": 1695043, "author_name": "tomforbes", "author_url": "", "post_date": "02/17/2022 22:59:12", "content": "@petersk20 this is a great idea - it would be nice to validate our results and would solve all issues around integrity of final score
", "votes": null, "replies": [] }, { "id": 1695090, "author_name": "tommy1028", "author_url": "", "post_date": "02/18/2022 00:15:01", "content": "Coundn't agree more, @petersk20 !
\nI would love to learn why my models with supplemental data has experienced a huge drop from non-leak LB. This might be because my models overfitted.
\nAnother possible cause is supplemental training set could have an error again as many people suspect. Full transparency will guarantee the integrity of the competition and maximize our learning opportunities.
\nAlso, I would love to know if there is anyone who can get high score without supplemental.csv. If yes, that indicates the supplemental.csv might be correct and vice-versa.
", "votes": null, "replies": [] }, { "id": 1696599, "author_name": "yus002", "author_url": "", "post_date": "02/19/2022 00:37:54", "content": "We can use late submissions to test our models after the competition ends in 3 months. It means that we need to hold our curiosity for 3 months to understand why our models went wrong.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1695012": "Can the hosts update the supplemental training set with the new data during the evaluation phase? It would be nice for people who got worse scores than expected or got an error in their submission to check their models on the new data. Also, one of the updates to the supplemental training set had incorrect data in it. This was figured out by someone in the competition and not by the hosts. This move would ensure the integrity of this competition and provide transparency that I think the competitors in this competition deserve after months of working hard.\n\nAlso, thanks to the hosts for hosting this competition and thanks to all the great ideas produced by other competitors.", "1695036": "It would be nice to have it, but even one of my submission which does not use supplemental, and simply does some silly calculation like Vwap * high/close for the prediction has also failed.. So I think the issue must be somewhere else, I find it hard to believe every single submission I made has failed including the simple baseline ones..\n\n@cstein06 @sohier would be nice to hear if these issues are at least being looked into?", "1695043": "petersk20 this is a great idea - it would be nice to validate our results and would solve all issues around integrity of final score", "1695090": "Coundn't agree more, @petersk20 !\n\nI would love to learn why my models with supplemental data has experienced a huge drop from non-leak LB. This might be because my models overfitted. \n\nAnother possible cause is supplemental training set could have an error again as many people suspect. Full transparency will guarantee the integrity of the competition and maximize our learning opportunities. \n\nAlso, I would love to know if there is anyone who can get high score **without supplemental.csv**. If yes, that indicates the supplemental.csv might be correct and vice-versa.", "1696599": "We can use late submissions to test our models after the competition ends in 3 months. It means that we need to hold our curiosity for 3 months to understand why our models went wrong." }, "source": "meta" }