I looked at all data. There are a lot of wrong anotations, missing anotations. Incostistencies in anotations (very similar sequences of anglez/enmo result in different anotations).
\nMy findings:
\nIts more than 1 wrong/missing annotation per file.
\nIs it the same when it comes to unseen test set? I guess so, because LB is worse than CV by ~0.03-0.04 for most of us (https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/442552). So in the best case, test set quality of annotations is the same.
\nShouldn't test set it be fixed and cleared of the noise before final evaulation of this competition? Since top rankings are close, it should affect final rankings significantly.
", "messages": [ { "id": 2549668, "postDate": "2023-12-05T12:43:31.543Z", "content": "I looked at all data. There are a lot of wrong anotations, missing anotations. Incostistencies in anotations (very similar sequences of anglez/enmo result in different anotations).
\nMy findings:
\nIts more than 1 wrong/missing annotation per file.
\nIs it the same when it comes to unseen test set? I guess so, because LB is worse than CV by ~0.03-0.04 for most of us (https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/442552). So in the best case, test set quality of annotations is the same.
\nShouldn't test set it be fixed and cleared of the noise before final evaulation of this competition? Since top rankings are close, it should affect final rankings significantly.
", "rawMarkdown": "I looked at all data. There are a lot of wrong anotations, missing anotations. Incostistencies in anotations (very similar sequences of anglez/enmo result in different anotations).\n\nMy findings:\n- there are about 100 missing night anotations (onset and wakeup, so together 200)\n- there are about 115 wrong misplaced annotations. \n\nIts more than 1 wrong/missing annotation per file.\n\nIs it the same when it comes to unseen test set? I guess so, because LB is worse than CV by ~0.03-0.04 for most of us (https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/442552). So in the best case, test set quality of annotations is the same.\n\nShouldn't test set it be fixed and cleared of the noise before final evaulation of this competition? Since top rankings are close, it should affect final rankings significantly.", "votes": 2 }, { "id": 2550423, "postDate": "2023-12-06T03:06:43.410Z", "content": "If the noise in hidden test is completely random then it affected everyone equally and did not affect the outcome of the competition.
\nI am curious if there was a way to predict the bad annotators. For example if all bad annotations are from one annotator and a model can detect which test series_id will be labeled by the bad annotator then models that can locate bad annotators will achieve better LB scores (because they can modify the score prediction for these annnotators) and they will perform better in this competition but they will not help the host build better models.
Also much of the data anglez and enmo has been copy pasted from another time within the same series_id. If a model can use this information to achieve a better LB score, then this property of the model will not benefit the host either
I tried (pretty hard 😅) to see if I could predict if a series had a \"bad annotator\", and never came up with anything useful; so I think I would lean to it affecting all teams and all series about evenly…
\nMy model did use the repeating (copy/pasted) anglez though, and that was useful for CV/LB, but it probably wasn't as useful for a real world model, yes.
", "rawMarkdown": "I tried (pretty hard 😅) to see if I could predict if a series had a \"bad annotator\", and never came up with anything useful; so I think I would lean to it affecting all teams and all series about evenly... \n\nMy model did use the repeating (copy/pasted) anglez though, and that _was_ useful for CV/LB, but it probably wasn't as useful for a real world model, yes.", "votes": 1, "replies": [ { "id": 2550446, "postDate": "2023-12-06T03:26:34.380Z", "content": "\n\nI tried (pretty hard 😅) to see if I could predict if a series had a \"bad annotator\", and never came up with anything useful; so I think I would lean to it affecting all teams and all series about evenly…
\n
I also tried to detect this but could not (i.e. i never got boost in both CV and LB).
\n\n\nMy model did use the repeating (copy/pasted) anglez though, and that was useful for CV/LB, but it probably wasn't as useful for a real world model, yes.
\n
I created a few features based on this (i.e. about repeated anglez and/or enmo). It improved CV but did not improve public LB so I did not include it in my final model. However now that the competition ended, i see that my best unselected submission is one that used this feature so perhaps using this information did help on private LB.
I saw some inconsistencies in the annotations of some wakeup patterns - there seem to be two possibilities the wakeup event can belong to, and the 30min rule seems to be not strictly followed. I wonder if top teams calibrate the probabilities of their model by generating pseudo labels for these uncertain events (which I didn't do).
\nPredicting these events with 1/2 probability in the two possible locations should be beneficial to the AP metric.
", "rawMarkdown": "I saw some inconsistencies in the annotations of some wakeup patterns - there seem to be two possibilities the wakeup event can belong to, and the 30min rule seems to be not strictly followed. I wonder if top teams calibrate the probabilities of their model by generating pseudo labels for these uncertain events (which I didn't do).\n\nPredicting these events with 1/2 probability in the two possible locations should be beneficial to the AP metric.", "replies": [ { "id": 2550271, "postDate": "2023-12-06T00:20:48.137Z", "content": "You did really well, congrats :)
\nI tried re-labelling and fixing those labels myself, but can't say using such train_events.csv changed by much or improved my model. I think it dealt with such a noise in data quite well.
\nHowever, fixing labels improved my CV score by a lot (from 0.77 to 0.82/0.84).
", "rawMarkdown": "You did really well, congrats :) \n\nI tried re-labelling and fixing those labels myself, but can't say using such train_events.csv changed by much or improved my model. I think it dealt with such a noise in data quite well.\n\nHowever, fixing labels improved my CV score by a lot (from 0.77 to 0.82/0.84). ", "votes": 1, "replies": [ { "id": 2550281, "postDate": "2023-12-06T00:28:28.797Z", "content": "Thanks! What I did is to simply exclude the bad labels and series from training, which did stabilise my model's convergence.
\nI also suspect there might be some data leakage from the training set we can download to the public LB, which may explain such a shakeup.
\nI ran some matrix profile tests with each series id independently, and with the bad series ids combined together. It seemed that there are some steps where the matrix profile is 0 when analysed together, but >> 0 when analysed separately.
\nThis suggests that some missing labels are due to copy and paste across series.
", "rawMarkdown": "Thanks! What I did is to simply exclude the bad labels and series from training, which did stabilise my model's convergence. \n\nI also suspect there might be some data leakage from the training set we can download to the public LB, which may explain such a shakeup.\n\nI ran some matrix profile tests with each series id independently, and with the bad series ids combined together. It seemed that there are some steps where the matrix profile is 0 when analysed together, but >> 0 when analysed separately. \n\nThis suggests that some missing labels are due to copy and paste across series.", "votes": 1 } ] } ] }, { "id": 2549818, "postDate": "2023-12-05T14:44:20.137Z", "rawMarkdown": "", "votes": -3, "isDeleted": true } ], "comments": [ { "id": 2550423, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-12-06T03:06:43.410000", "content": "If the noise in hidden test is completely random then it affected everyone equally and did not affect the outcome of the competition.
\nI am curious if there was a way to predict the bad annotators. For example if all bad annotations are from one annotator and a model can detect which test series_id will be labeled by the bad annotator then models that can locate bad annotators will achieve better LB scores (because they can modify the score prediction for these annnotators) and they will perform better in this competition but they will not help the host build better models.
Also much of the data anglez and enmo has been copy pasted from another time within the same series_id. If a model can use this information to achieve a better LB score, then this property of the model will not benefit the host either
I tried (pretty hard 😅) to see if I could predict if a series had a \"bad annotator\", and never came up with anything useful; so I think I would lean to it affecting all teams and all series about evenly…
\nMy model did use the repeating (copy/pasted) anglez though, and that was useful for CV/LB, but it probably wasn't as useful for a real world model, yes.
", "votes": 1, "replies": [ { "id": 2550446, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-12-06T03:26:34.380000", "content": "\n\nI tried (pretty hard 😅) to see if I could predict if a series had a \"bad annotator\", and never came up with anything useful; so I think I would lean to it affecting all teams and all series about evenly…
\n
I also tried to detect this but could not (i.e. i never got boost in both CV and LB).
\n\n\nMy model did use the repeating (copy/pasted) anglez though, and that was useful for CV/LB, but it probably wasn't as useful for a real world model, yes.
\n
I created a few features based on this (i.e. about repeated anglez and/or enmo). It improved CV but did not improve public LB so I did not include it in my final model. However now that the competition ended, i see that my best unselected submission is one that used this feature so perhaps using this information did help on private LB.
I saw some inconsistencies in the annotations of some wakeup patterns - there seem to be two possibilities the wakeup event can belong to, and the 30min rule seems to be not strictly followed. I wonder if top teams calibrate the probabilities of their model by generating pseudo labels for these uncertain events (which I didn't do).
\nPredicting these events with 1/2 probability in the two possible locations should be beneficial to the AP metric.
", "votes": 0, "replies": [ { "id": 2550271, "author_name": "Daniel Kalicki", "author_url": "", "post_date": "2023-12-06T00:20:48.137000", "content": "You did really well, congrats :)
\nI tried re-labelling and fixing those labels myself, but can't say using such train_events.csv changed by much or improved my model. I think it dealt with such a noise in data quite well.
\nHowever, fixing labels improved my CV score by a lot (from 0.77 to 0.82/0.84).
", "votes": 1, "replies": [ { "id": 2550281, "author_name": "Chau YH", "author_url": "", "post_date": "2023-12-06T00:28:28.797000", "content": "Thanks! What I did is to simply exclude the bad labels and series from training, which did stabilise my model's convergence.
\nI also suspect there might be some data leakage from the training set we can download to the public LB, which may explain such a shakeup.
\nI ran some matrix profile tests with each series id independently, and with the bad series ids combined together. It seemed that there are some steps where the matrix profile is 0 when analysed together, but >> 0 when analysed separately.
\nThis suggests that some missing labels are due to copy and paste across series.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2549818, "author_name": "", "author_url": "", "post_date": "2023-12-05T14:44:20.137000", "content": "", "votes": -3, "replies": [] } ], "raw_markdown_by_id": { "2549668": "I looked at all data. There are a lot of wrong anotations, missing anotations. Incostistencies in anotations (very similar sequences of anglez/enmo result in different anotations).\n\nMy findings:\n- there are about 100 missing night anotations (onset and wakeup, so together 200)\n- there are about 115 wrong misplaced annotations. \n\nIts more than 1 wrong/missing annotation per file.\n\nIs it the same when it comes to unseen test set? I guess so, because LB is worse than CV by ~0.03-0.04 for most of us (https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/442552). So in the best case, test set quality of annotations is the same.\n\nShouldn't test set it be fixed and cleared of the noise before final evaulation of this competition? Since top rankings are close, it should affect final rankings significantly.", "2550423": "If the noise in hidden test is **completely random** then it affected everyone equally and did not affect the outcome of the competition.\n\nI am curious if there was a way to predict the bad annotators. For example if all bad annotations are from one annotator and a model can detect which `test series_id` will be labeled by the bad annotator then models that can locate bad annotators will achieve better LB scores (because they can modify the `score` prediction for these annnotators) and they will perform better in this competition but they will not help the host build better models.\n\nAlso much of the data `anglez` and `enmo` has been copy pasted from another time within the same series_id. If a model can use this information to achieve a better LB score, then this property of the model will not benefit the host either", "2550252": "I saw some inconsistencies in the annotations of some wakeup patterns - there seem to be two possibilities the wakeup event can belong to, and the 30min rule seems to be not strictly followed. I wonder if top teams calibrate the probabilities of their model by generating pseudo labels for these uncertain events (which I didn't do).\n\nPredicting these events with 1/2 probability in the two possible locations should be beneficial to the AP metric.", "2549818": "" } }