My model's accuracy or F1 score hangs around 0.97~0.98, but its predictions for the test set are counterintuitive. Predicted sleep periods range from a few minutes to a few hours. And there are too many sleep periods.
\nDo you guys also have similar problems?
How do you deal with the predicted labels?
How much does your model score for these metrics?
Do metrics like accuracy or F1 score for each row of the test set matter in this competition? (I believe not.)
What is your postprocessing technique?
My model's accuracy or F1 score hangs around 0.97~0.98, but its predictions for the test set are counterintuitive. Predicted sleep periods range from a few minutes to a few hours. And there are too many sleep periods.
\nDo you guys also have similar problems?
How do you deal with the predicted labels?
How much does your model score for these metrics?
Do metrics like accuracy or F1 score for each row of the test set matter in this competition? (I believe not.)
What is your postprocessing technique?
I suggest that you validate your models on mAP, since it is the competition's evaluation metric, you can find examples of implementation in public notebooks.
\nAs for post processing, there are plenty of things you can try, and it depends on your modeling strategy. But they all try to look for the most relevant signal that can be classified as transition time (onset or wakeup). You can also start by looking at public notebooks to get a general idea and build from there.
", "rawMarkdown": "I suggest that you validate your models on mAP, since it is the competition's evaluation metric, you can find examples of implementation in public notebooks.\n\nAs for post processing, there are plenty of things you can try, and it depends on your modeling strategy. But they all try to look for the most relevant signal that can be classified as transition time (onset or wakeup). You can also start by looking at public notebooks to get a general idea and build from there.", "votes": null }, { "id": "2517199", "postDate": "11/08/2023 09:27:26", "content": "Thank you, Catharis!
", "rawMarkdown": "Thank you, Catharis!", "votes": null }, { "id": "2522174", "postDate": "11/12/2023 12:56:10", "content": "Hello, try the competition metric and build a reliable validation (divide data by series_id). As for postprocessing, I suppose that there is not too much to do. You can try to average predictions from models, for example, two models give you steps 1000 and 1010 as a prediction, then take the 1005th step as a final prediction. Also remember that all answers are exactly on minutes with 0 seconds. You can also try to limit the number of events by day. For example in the train set there was only 1 day, where a person had 4 events, in other cases it is no more than 3.
", "rawMarkdown": "Hello, try the competition metric and build a reliable validation (divide data by series_id). As for postprocessing, I suppose that there is not too much to do. You can try to average predictions from models, for example, two models give you steps 1000 and 1010 as a prediction, then take the 1005th step as a final prediction. Also remember that all answers are exactly on minutes with 0 seconds. You can also try to limit the number of events by day. For example in the train set there was only 1 day, where a person had 4 events, in other cases it is no more than 3.", "votes": null }, { "id": "2522320", "postDate": "11/12/2023 15:34:15", "content": "It's a good idea! I appreciate it.
", "rawMarkdown": "It's a good idea! I appreciate it.", "votes": null }, { "id": "2522912", "postDate": "11/13/2023 06:43:11", "content": "You might want to try using NMS (non-maximal suppression)：
\nyou can find more information in:
\nhttps://www.kaggle.com/competitions/dfl-bundesliga-data-shootout/discussion/360236
I'll check on it. Appreciate it!
", "rawMarkdown": "I'll check on it. Appreciate it!", "votes": null } ], "comments": [ { "id": 2517074, "author_name": "fahmiayari", "author_url": "", "post_date": "11/08/2023 07:38:27", "content": "I suggest that you validate your models on mAP, since it is the competition's evaluation metric, you can find examples of implementation in public notebooks.
\nAs for post processing, there are plenty of things you can try, and it depends on your modeling strategy. But they all try to look for the most relevant signal that can be classified as transition time (onset or wakeup). You can also start by looking at public notebooks to get a general idea and build from there.
", "votes": null, "replies": [ { "id": 2517199, "author_name": "cgl029", "author_url": "", "post_date": "11/08/2023 09:27:26", "content": "Thank you, Catharis!
", "votes": null, "replies": [] } ] }, { "id": 2522174, "author_name": "manwithaflower", "author_url": "", "post_date": "11/12/2023 12:56:10", "content": "Hello, try the competition metric and build a reliable validation (divide data by series_id). As for postprocessing, I suppose that there is not too much to do. You can try to average predictions from models, for example, two models give you steps 1000 and 1010 as a prediction, then take the 1005th step as a final prediction. Also remember that all answers are exactly on minutes with 0 seconds. You can also try to limit the number of events by day. For example in the train set there was only 1 day, where a person had 4 events, in other cases it is no more than 3.
", "votes": null, "replies": [ { "id": 2522320, "author_name": "cgl029", "author_url": "", "post_date": "11/12/2023 15:34:15", "content": "It's a good idea! I appreciate it.
", "votes": null, "replies": [] } ] }, { "id": 2522912, "author_name": "jimmyisme1", "author_url": "", "post_date": "11/13/2023 06:43:11", "content": "You might want to try using NMS (non-maximal suppression)：
\nyou can find more information in:
\nhttps://www.kaggle.com/competitions/dfl-bundesliga-data-shootout/discussion/360236
I'll check on it. Appreciate it!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2516884": "My model's accuracy or F1 score hangs around 0.97~0.98, but its predictions for the test set are counterintuitive. Predicted sleep periods range from a few minutes to a few hours. And there are too many sleep periods.\n\n1. Do you guys also have similar problems? \n\n2. How do you deal with the predicted labels? \n\n3. How much does your model score for these metrics?\n\n4. Do metrics like accuracy or F1 score for each row of the test set matter in this competition? (I believe not.)\n\n5. What is your postprocessing technique?", "2517074": "I suggest that you validate your models on mAP, since it is the competition's evaluation metric, you can find examples of implementation in public notebooks.\n\nAs for post processing, there are plenty of things you can try, and it depends on your modeling strategy. But they all try to look for the most relevant signal that can be classified as transition time (onset or wakeup). You can also start by looking at public notebooks to get a general idea and build from there.", "2517199": "Thank you, Catharis!", "2522174": "Hello, try the competition metric and build a reliable validation (divide data by series_id). As for postprocessing, I suppose that there is not too much to do. You can try to average predictions from models, for example, two models give you steps 1000 and 1010 as a prediction, then take the 1005th step as a final prediction. Also remember that all answers are exactly on minutes with 0 seconds. You can also try to limit the number of events by day. For example in the train set there was only 1 day, where a person had 4 events, in other cases it is no more than 3.", "2522320": "It's a good idea! I appreciate it.", "2522912": "You might want to try using NMS (non-maximal suppression)：\nyou can find more information in:\n[https://www.kaggle.com/competitions/dfl-bundesliga-data-shootout/discussion/360236](https://www.kaggle.com/competitions/dfl-bundesliga-data-shootout/discussion/360236)", "2538675": "I'll check on it. Appreciate it!" }, "source": "meta" }