{ "id": 440857, "title": "Validation strategy", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/440857", "author_name": "", "post_date": "2023-09-16T13:02:48.407195100Z", "votes": 4, "comment_count": 5, "views": 0, "content": "

Hello everyone, what is your validation strategy?

\n

I have gone through this notebook by @carlmcbrideellis and tried something like this

\n
\n

solution = pd.read_parquet(TR_FILE, columns=[\"series_id\", \"step\", \"awake\"],filters=[(\"series_id\", \"in\", test_series)])
\n solution = solution.rename(columns={\"awake\": \"event\"})
\n solution[\"event\"] = solution[\"event\"].replace({1: \"onset\",0: \"wakeup\"})

\n

tolerances = {
\n \"onset\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],
\n \"wakeup\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],
\n }

\n

series_id_column_name = \"series_id\"
\n time_column_name = \"step\"
\n event_column_name = \"event\"
\n score_column_name = \"score\"
\n use_scoring_intervals = None

\n

print(event_detection_ap(solution, sub, tolerances))

\n
\n

where sub is my model predictions, I am getting less than 0.1 but public leaderboard score is around 0.25, is there something I am missing here?

", "messages": [ { "id": "2441798", "postDate": "09/16/2023 13:02:48", "content": "

Hello everyone, what is your validation strategy?

\n

I have gone through this notebook by @carlmcbrideellis and tried something like this

\n
\n

solution = pd.read_parquet(TR_FILE, columns=[\"series_id\", \"step\", \"awake\"],filters=[(\"series_id\", \"in\", test_series)])
\n solution = solution.rename(columns={\"awake\": \"event\"})
\n solution[\"event\"] = solution[\"event\"].replace({1: \"onset\",0: \"wakeup\"})

\n

tolerances = {
\n \"onset\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],
\n \"wakeup\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],
\n }

\n

series_id_column_name = \"series_id\"
\n time_column_name = \"step\"
\n event_column_name = \"event\"
\n score_column_name = \"score\"
\n use_scoring_intervals = None

\n

print(event_detection_ap(solution, sub, tolerances))

\n
\n

where sub is my model predictions, I am getting less than 0.1 but public leaderboard score is around 0.25, is there something I am missing here?

", "rawMarkdown": "Hello everyone, what is your validation strategy?\n\nI have gone through this [notebook](https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric) by @carlmcbrideellis and tried something like this\n\n>solution = pd.read_parquet(TR_FILE, columns=[\"series_id\", \"step\", \"awake\"],filters=[(\"series_id\", \"in\", test_series)])\nsolution = solution.rename(columns={\"awake\": \"event\"})\nsolution[\"event\"] = solution[\"event\"].replace({1: \"onset\",0: \"wakeup\"})\n\n> tolerances = {\n \"onset\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],\n \"wakeup\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],\n}\n\n>series_id_column_name = \"series_id\"\ntime_column_name = \"step\"\nevent_column_name = \"event\"\nscore_column_name = \"score\"\nuse_scoring_intervals = None\n\n>print(event_detection_ap(solution, sub, tolerances))\n\nwhere sub is my model predictions, I am getting less than 0.1 but public leaderboard score is around 0.25, is there something I am missing here?", "votes": null }, { "id": "2442360", "postDate": "09/17/2023 00:25:51", "content": "

Hi @anilreddyvv , what is TR_FILE in your code? Is it the path of the train series?

\n

If so, please note that the score we get in public LB is evaluated with the hidden test set, not the train set or dummy test set one.

", "rawMarkdown": "Hi @anilreddyvv , what is `TR_FILE` in your code? Is it the path of the train series?\n\nIf so, please note that the score we get in public LB is evaluated with the hidden test set, not the train set or dummy test set one.", "votes": null }, { "id": "2442549", "postDate": "09/17/2023 05:37:15", "content": "

Yes, I am aware that scoring is on hidden test data. But the question is why is model score on validation data (used some series purely for validating the model) is very less compared to hidden test data? Am I using the scoring function incorrectly or is there an actual problem in the model?

", "rawMarkdown": "Yes, I am aware that scoring is on hidden test data. But the question is why is model score on validation data (used some series purely for validating the model) is very less compared to hidden test data? Am I using the scoring function incorrectly or is there an actual problem in the model?", "votes": null }, { "id": "2443327", "postDate": "09/17/2023 16:49:21", "content": "

Since the data is actually a time series, could it be better if we take the last T-steps window for each series as the validation data? Just like how the time-series CV split is done

", "rawMarkdown": "Since the data is actually a time series, could it be better if we take the last T-steps window for each series as the validation data? Just like how the time-series CV split is done", "votes": null }, { "id": "2443342", "postDate": "09/17/2023 16:54:06", "content": "

I mean, something like this, but done in each series ID

\n

\"Time

", "rawMarkdown": "I mean, something like this, but done in each series ID\n\n![Time series CV](https://i.stack.imgur.com/Q37Bn.png)", "votes": null }, { "id": "2444516", "postDate": "09/18/2023 10:07:11", "content": "

It might be useful in a way, but I feel measuring performance on completely new series_id (on which model is not trained) would be more appropriate

", "rawMarkdown": "It might be useful in a way, but I feel measuring performance on completely new series_id (on which model is not trained) would be more appropriate", "votes": null } ], "comments": [ { "id": 2442360, "author_name": "rimbax", "author_url": "", "post_date": "09/17/2023 00:25:51", "content": "

Hi @anilreddyvv , what is TR_FILE in your code? Is it the path of the train series?

\n

If so, please note that the score we get in public LB is evaluated with the hidden test set, not the train set or dummy test set one.

", "votes": null, "replies": [ { "id": 2442549, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/17/2023 05:37:15", "content": "

Yes, I am aware that scoring is on hidden test data. But the question is why is model score on validation data (used some series purely for validating the model) is very less compared to hidden test data? Am I using the scoring function incorrectly or is there an actual problem in the model?

", "votes": null, "replies": [ { "id": 2443327, "author_name": "rimbax", "author_url": "", "post_date": "09/17/2023 16:49:21", "content": "

Since the data is actually a time series, could it be better if we take the last T-steps window for each series as the validation data? Just like how the time-series CV split is done

", "votes": null, "replies": [] }, { "id": 2443342, "author_name": "rimbax", "author_url": "", "post_date": "09/17/2023 16:54:06", "content": "

I mean, something like this, but done in each series ID

\n

\"Time

", "votes": null, "replies": [ { "id": 2444516, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/18/2023 10:07:11", "content": "

It might be useful in a way, but I feel measuring performance on completely new series_id (on which model is not trained) would be more appropriate

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2441798": "Hello everyone, what is your validation strategy?\n\nI have gone through this [notebook](https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric) by @carlmcbrideellis and tried something like this\n\n>solution = pd.read_parquet(TR_FILE, columns=[\"series_id\", \"step\", \"awake\"],filters=[(\"series_id\", \"in\", test_series)])\nsolution = solution.rename(columns={\"awake\": \"event\"})\nsolution[\"event\"] = solution[\"event\"].replace({1: \"onset\",0: \"wakeup\"})\n\n> tolerances = {\n \"onset\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],\n \"wakeup\": [12, 36, 60, 90, 120, 150, 180, 240, 300, 360],\n}\n\n>series_id_column_name = \"series_id\"\ntime_column_name = \"step\"\nevent_column_name = \"event\"\nscore_column_name = \"score\"\nuse_scoring_intervals = None\n\n>print(event_detection_ap(solution, sub, tolerances))\n\nwhere sub is my model predictions, I am getting less than 0.1 but public leaderboard score is around 0.25, is there something I am missing here?", "2442360": "Hi @anilreddyvv , what is `TR_FILE` in your code? Is it the path of the train series?\n\nIf so, please note that the score we get in public LB is evaluated with the hidden test set, not the train set or dummy test set one.", "2442549": "Yes, I am aware that scoring is on hidden test data. But the question is why is model score on validation data (used some series purely for validating the model) is very less compared to hidden test data? Am I using the scoring function incorrectly or is there an actual problem in the model?", "2443327": "Since the data is actually a time series, could it be better if we take the last T-steps window for each series as the validation data? Just like how the time-series CV split is done", "2443342": "I mean, something like this, but done in each series ID\n\n![Time series CV](https://i.stack.imgur.com/Q37Bn.png)", "2444516": "It might be useful in a way, but I feel measuring performance on completely new series_id (on which model is not trained) would be more appropriate" }, "source": "meta" }