Hi all,
\n\nJust wondering what people have been doing during model training... I've been training my model on 2003-2014 data, then applying it to the 2015-19 data as my test set and getting some fairly competitive results (but not standout results either), but am now wondering if it's best to train my model using 2003-2018 data, then fitting it to 2019 tourney and optimising log-loss there?
\n\nThanks in advance.
", "messages": [ { "id": "765712", "postDate": "03/07/2020 01:47:52", "content": "Hi all,
\n\nJust wondering what people have been doing during model training... I've been training my model on 2003-2014 data, then applying it to the 2015-19 data as my test set and getting some fairly competitive results (but not standout results either), but am now wondering if it's best to train my model using 2003-2018 data, then fitting it to 2019 tourney and optimising log-loss there?
\n\nThanks in advance.
", "rawMarkdown": "Hi all,\n\nJust wondering what people have been doing during model training... I've been training my model on 2003-2014 data, then applying it to the 2015-19 data as my test set and getting some fairly competitive results (but not standout results either), but am now wondering if it's best to train my model using 2003-2018 data, then fitting it to 2019 tourney and optimising log-loss there?\n\nThanks in advance.", "votes": null }, { "id": "765972", "postDate": "03/07/2020 13:09:44", "content": "Yes, either of those is acceptable. As you've probably figured out you would just have to change your training set for each year 2015-2019 to add the previous year (i.e. 2015 would still only be predicted using 2003-2014).
\n\nAlso as a side note, don't forget that a good portion of the leaderboard may not be representative of true performance without data leaks!
", "rawMarkdown": "Yes, either of those is acceptable. As you've probably figured out you would just have to change your training set for each year 2015-2019 to add the previous year (i.e. 2015 would still only be predicted using 2003-2014).\n\nAlso as a side note, don't forget that a good portion of the leaderboard may not be representative of true performance without data leaks!", "votes": null }, { "id": "766308", "postDate": "03/08/2020 01:11:53", "content": "I was more thinking about selecting a final model for submission; are you trying to optimise your log-loss score for the 2019 season that has just passed, or optimise it on the stage 1 leaderboard (obviously accounting for leakage of the stage 1 results)...
", "rawMarkdown": "I was more thinking about selecting a final model for submission; are you trying to optimise your log-loss score for the 2019 season that has just passed, or optimise it on the stage 1 leaderboard (obviously accounting for leakage of the stage 1 results)...", "votes": null }, { "id": "766712", "postDate": "03/08/2020 15:49:10", "content": "Got it. I would be hesitant to optimize to a single season. Things change a lot from year to year and I don't think there is much evidence that 2019 is inherently more representative of what the 2020 results would be than an aggregation of multiple seasons. This post might help explain the year to year variance a little better and why you don't want to be optimizing to a single year.
", "rawMarkdown": "Got it. I would be hesitant to optimize to a single season. Things change a lot from year to year and I don't think there is much evidence that 2019 is inherently more representative of what the 2020 results would be than an aggregation of multiple seasons. [This post](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131028#748701) might help explain the year to year variance a little better and why you don't want to be optimizing to a single year.", "votes": null }, { "id": "766802", "postDate": "03/08/2020 18:53:20", "content": "Great points! Thanks @rsa013!
", "rawMarkdown": "Great points! Thanks @rsa013!", "votes": null }, { "id": "768133", "postDate": "03/10/2020 13:45:32", "content": "You can also try leave-one-out method. Train for each year separately.
", "rawMarkdown": "You can also try leave-one-out method. Train for each year separately.", "votes": null } ], "comments": [ { "id": 765972, "author_name": "rsa013", "author_url": "", "post_date": "03/07/2020 13:09:44", "content": "Yes, either of those is acceptable. As you've probably figured out you would just have to change your training set for each year 2015-2019 to add the previous year (i.e. 2015 would still only be predicted using 2003-2014).
\n\nAlso as a side note, don't forget that a good portion of the leaderboard may not be representative of true performance without data leaks!
", "votes": null, "replies": [ { "id": 766308, "author_name": "jaseziv83", "author_url": "", "post_date": "03/08/2020 01:11:53", "content": "I was more thinking about selecting a final model for submission; are you trying to optimise your log-loss score for the 2019 season that has just passed, or optimise it on the stage 1 leaderboard (obviously accounting for leakage of the stage 1 results)...
", "votes": null, "replies": [] }, { "id": 766712, "author_name": "rsa013", "author_url": "", "post_date": "03/08/2020 15:49:10", "content": "Got it. I would be hesitant to optimize to a single season. Things change a lot from year to year and I don't think there is much evidence that 2019 is inherently more representative of what the 2020 results would be than an aggregation of multiple seasons. This post might help explain the year to year variance a little better and why you don't want to be optimizing to a single year.
", "votes": null, "replies": [] }, { "id": 766802, "author_name": "jaseziv83", "author_url": "", "post_date": "03/08/2020 18:53:20", "content": "Great points! Thanks @rsa013!
", "votes": null, "replies": [] } ] }, { "id": 768133, "author_name": "", "author_url": "", "post_date": "03/10/2020 13:45:32", "content": "You can also try leave-one-out method. Train for each year separately.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "765712": "Hi all,\n\nJust wondering what people have been doing during model training... I've been training my model on 2003-2014 data, then applying it to the 2015-19 data as my test set and getting some fairly competitive results (but not standout results either), but am now wondering if it's best to train my model using 2003-2018 data, then fitting it to 2019 tourney and optimising log-loss there?\n\nThanks in advance.", "765972": "Yes, either of those is acceptable. As you've probably figured out you would just have to change your training set for each year 2015-2019 to add the previous year (i.e. 2015 would still only be predicted using 2003-2014).\n\nAlso as a side note, don't forget that a good portion of the leaderboard may not be representative of true performance without data leaks!", "766308": "I was more thinking about selecting a final model for submission; are you trying to optimise your log-loss score for the 2019 season that has just passed, or optimise it on the stage 1 leaderboard (obviously accounting for leakage of the stage 1 results)...", "766712": "Got it. I would be hesitant to optimize to a single season. Things change a lot from year to year and I don't think there is much evidence that 2019 is inherently more representative of what the 2020 results would be than an aggregation of multiple seasons. [This post](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131028#748701) might help explain the year to year variance a little better and why you don't want to be optimizing to a single year.", "766802": "Great points! Thanks @rsa013!", "768133": "You can also try leave-one-out method. Train for each year separately." }, "source": "meta" }