Hi everyone, I just saw that the previous discussion \"what are you imu-only scores\" from @gerwynng is not on the first page of discussion anymore, yet I think it is a very good discussion, feel free to continue share your imu-only score here or go to the previous discussion https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/583023#3249998
\nBtw, it will also be good to know how big is your model (e.g. file size, num of params)
", "messages": [ { "id": 3250079, "postDate": "2025-07-17T16:47:31.257Z", "content": "It's a shame IMO to see so many downvote, the old thread is outdated and would be really interesting to see some updated data. Would really like to know what is imu only for 8.5-8.6! I guess somewhere around 0.83 give or take 0.01
", "rawMarkdown": "It's a shame IMO to see so many downvote, the old thread is outdated and would be really interesting to see some updated data. Would really like to know what is imu only for 8.5-8.6! I guess somewhere around 0.83 give or take 0.01", "votes": 7, "replies": [ { "id": 3250151, "postDate": "2025-07-17T18:56:27.013Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 3250186, "postDate": "2025-07-17T20:50:43.803Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 3250017, "postDate": "2025-07-17T14:48:04.877Z", "content": "Hi everyone, I just saw that the previous discussion \"what are you imu-only scores\" from @gerwynng is not on the first page of discussion anymore, yet I think it is a very good discussion, feel free to continue share your imu-only score here or go to the previous discussion https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/583023#3249998
\nBtw, it will also be good to know how big is your model (e.g. file size, num of params)
", "rawMarkdown": "Hi everyone, I just saw that the previous discussion \"what are you imu-only scores\" from @gerwynng is not on the first page of discussion anymore, yet I think it is a very good discussion, feel free to continue share your imu-only score here or go to the previous discussion https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/583023#3249998\n\nBtw, it will also be good to know how big is your model (e.g. file size, num of params)", "votes": 8 }, { "id": 3250354, "postDate": "2025-07-18T06:51:59.450Z", "content": "0.825 cv, 0.82 lb
", "rawMarkdown": "0.825 cv, 0.82 lb", "votes": 3, "replies": [ { "id": 3261873, "postDate": "2025-08-02T12:18:36.620Z", "content": "May I ask what your model size/param count is?
", "rawMarkdown": "May I ask what your model size/param count is?", "replies": [ { "id": 3268079, "postDate": "2025-08-12T11:20:30.933Z", "content": "3M parameters
", "rawMarkdown": "3M parameters" } ] } ] }, { "id": 3250119, "postDate": "2025-07-17T17:52:02.803Z", "content": "My IMU only score at the moment is CV: 0.77x and LB: 0.78x. My validation strategy is the usual StratifiedGroupKFold with subjects as groups. I need to go back to this model and see if I can reach 0.80 both CV/LB. I am using raw data, no FE.
you can try the feature engineering in some public notebook as I think they boosted both my CV and LB score
", "rawMarkdown": "you can try the feature engineering in some public notebook as I think they boosted both my CV and LB score", "votes": -1 } ] }, { "id": 3250407, "postDate": "2025-07-18T09:43:34.500Z", "content": "0.803 cv, 0.81 lb
", "rawMarkdown": "0.803 cv, 0.81 lb", "votes": 2 }, { "id": 3250076, "postDate": "2025-07-17T16:39:54.280Z", "content": "Pro Tip: Just sort the discussions by \"Recent Comments\" to always see the latest news
\n(Or just Bookmark it yourself)
I know how to find it, but I just find that people are not so active for that post now, so it will be good to remind others.
", "rawMarkdown": "I know how to find it, but I just find that people are not so active for that post now, so it will be good to remind others.", "votes": -1 } ] }, { "id": 3253121, "postDate": "2025-07-24T03:56:23.110Z", "content": "@georgewang73 and others
\nHow is that you people manage to get a score of above 0.7.
\nMy local score has NOT crossed 0.5 (I am a begineer though, but willing to learn). Previously i used just raw IMU data now i did some feature engineering as suggested in this notebook.
This is my link to my colab notebook. WHY AM I NOT GETTING BETTER LOCAL SCORES EVEN
\n(i HAVE ATTACHED MY NOTEBOOK AS WELL)
\nAny feedback from you experts is highly appreciated..
", "rawMarkdown": "@georgewang73 and others\nHow is that you people manage to get a score of above 0.7.\nMy local score has NOT crossed 0.5 (I am a begineer though, but willing to learn). Previously i used just raw IMU data now i did some feature engineering as suggested [in this notebook](https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention).\n\nThis is my link to[ my colab notebook](https://colab.research.google.com/drive/1xS68dekfo0qU8OLQlQVsE8dX_epk3X3P?usp=sharing). WHY AM I NOT GETTING BETTER LOCAL SCORES EVEN\n\n(i HAVE ATTACHED MY NOTEBOOK AS WELL)\n\nAny feedback from you experts is highly appreciated..", "replies": [ { "id": 3253174, "postDate": "2025-07-24T06:46:44.440Z", "content": "I don't see any calculation of the competition metric in your notebook. You are only looking at the classification accuracy but never actually compute the competition metric everybody is talking about and which is used for the final scoring.
", "rawMarkdown": "I don't see any calculation of the competition metric in your notebook. You are only looking at the classification accuracy but never actually compute the competition metric everybody is talking about and which is used for the final scoring.", "replies": [ { "id": 3253184, "postDate": "2025-07-24T07:12:58.700Z", "content": "Agreed. But I did see notebooks that used IMU only to have accuracies around 0.7. Eg this notebook here.
\nI am also using very similar engineered features. But my local accuracy is very low (as low as tossing a coin)😂😂.
\nKindly help..
", "rawMarkdown": "Agreed. But I did see notebooks that used IMU only to have accuracies around 0.7. Eg this [notebook here](https://www.kaggle.com/code/jiazhuang/cmi-imu-only-lstm).\n\nI am also using very similar engineered features. But my local accuracy is very low (as low as tossing a coin)😂😂.\n\nKindly help.." }, { "id": 3253953, "postDate": "2025-07-25T14:40:54.747Z", "content": "First thing I think you haven't trained enough, you only trained for 20 epochs and the validation accuracy is still increasing, second I think you need to compute the competition score, which is the average of the binary and macro f1, the binary is near 1 (above 0.95 probably), so if you get a macro f1 of 0.5, you will have the score over 0.7 for sure.
", "rawMarkdown": "First thing I think you haven't trained enough, you only trained for 20 epochs and the validation accuracy is still increasing, second I think you need to compute the competition score, which is the average of the binary and macro f1, the binary is near 1 (above 0.95 probably), so if you get a macro f1 of 0.5, you will have the score over 0.7 for sure." }, { "id": 3254217, "postDate": "2025-07-26T03:25:18.073Z", "content": "Nice to hear that. Let me try doing that.
\nThanks a ton..
@georgewang73
\nThanks again.
\nI Implemented the Competition Metric as you suggested. Locally I was able to get 0.72 ish but Now after much struggle was able to submit my solution (based on other well written notebooks - idea credits are theirs) and my Leader board metric stands at 0.69….
Also check if your macro f1 is calculated correctly, you should collapse the non-gesture label into one label when calculating this, otherwise the score might be a bit higher. Also, you should do stratification for your cross validation based on gestures and group by subject.
", "rawMarkdown": "Also check if your macro f1 is calculated correctly, you should collapse the non-gesture label into one label when calculating this, otherwise the score might be a bit higher. Also, you should do stratification for your cross validation based on gestures and group by subject." }, { "id": 3263321, "postDate": "2025-08-05T03:54:48.793Z", "content": "Oh ok, So far My cross validation was on sequence_id / gestures. Now I shall try grouping by subject.
\nThanks..
", "rawMarkdown": "Oh ok, So far My cross validation was on sequence_id / gestures. Now I shall try grouping by subject.\n\nThanks..\n" } ] } ] }, { "id": 3250150, "postDate": "2025-07-17T18:55:25.383Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3250079, "author_name": "greySnow", "author_url": "", "post_date": "2025-07-17T16:47:31.257000", "content": "It's a shame IMO to see so many downvote, the old thread is outdated and would be really interesting to see some updated data. Would really like to know what is imu only for 8.5-8.6! I guess somewhere around 0.83 give or take 0.01
", "votes": 7, "replies": [ { "id": 3250151, "author_name": "", "author_url": "", "post_date": "2025-07-17T18:56:27.013000", "content": "", "votes": 0, "replies": [ { "id": 3250186, "author_name": "", "author_url": "", "post_date": "2025-07-17T20:50:43.803000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3250354, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-07-18T06:51:59.450000", "content": "0.825 cv, 0.82 lb
", "votes": 3, "replies": [ { "id": 3261873, "author_name": "Mahog", "author_url": "", "post_date": "2025-08-02T12:18:36.620000", "content": "May I ask what your model size/param count is?
", "votes": 0, "replies": [ { "id": 3268079, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2025-08-12T11:20:30.933000", "content": "3M parameters
", "votes": 0, "replies": [] } ] } ] }, { "id": 3250119, "author_name": "moth", "author_url": "", "post_date": "2025-07-17T17:52:02.803000", "content": "My IMU only score at the moment is CV: 0.77x and LB: 0.78x. My validation strategy is the usual StratifiedGroupKFold with subjects as groups. I need to go back to this model and see if I can reach 0.80 both CV/LB. I am using raw data, no FE.
you can try the feature engineering in some public notebook as I think they boosted both my CV and LB score
", "votes": -1, "replies": [] } ] }, { "id": 3250407, "author_name": "kss", "author_url": "", "post_date": "2025-07-18T09:43:34.500000", "content": "0.803 cv, 0.81 lb
", "votes": 2, "replies": [] }, { "id": 3250076, "author_name": "PatrickAIForFun", "author_url": "", "post_date": "2025-07-17T16:39:54.280000", "content": "Pro Tip: Just sort the discussions by \"Recent Comments\" to always see the latest news
\n(Or just Bookmark it yourself)
I know how to find it, but I just find that people are not so active for that post now, so it will be good to remind others.
", "votes": -1, "replies": [] } ] }, { "id": 3253121, "author_name": "Gnanasenthil", "author_url": "", "post_date": "2025-07-24T03:56:23.110000", "content": "@georgewang73 and others
\nHow is that you people manage to get a score of above 0.7.
\nMy local score has NOT crossed 0.5 (I am a begineer though, but willing to learn). Previously i used just raw IMU data now i did some feature engineering as suggested in this notebook.
This is my link to my colab notebook. WHY AM I NOT GETTING BETTER LOCAL SCORES EVEN
\n(i HAVE ATTACHED MY NOTEBOOK AS WELL)
\nAny feedback from you experts is highly appreciated..
", "votes": 0, "replies": [ { "id": 3253174, "author_name": "PatrickAIForFun", "author_url": "", "post_date": "2025-07-24T06:46:44.440000", "content": "I don't see any calculation of the competition metric in your notebook. You are only looking at the classification accuracy but never actually compute the competition metric everybody is talking about and which is used for the final scoring.
", "votes": 0, "replies": [ { "id": 3253184, "author_name": "Gnanasenthil", "author_url": "", "post_date": "2025-07-24T07:12:58.700000", "content": "Agreed. But I did see notebooks that used IMU only to have accuracies around 0.7. Eg this notebook here.
\nI am also using very similar engineered features. But my local accuracy is very low (as low as tossing a coin)😂😂.
\nKindly help..
", "votes": 0, "replies": [] }, { "id": 3253953, "author_name": "menglei", "author_url": "", "post_date": "2025-07-25T14:40:54.747000", "content": "First thing I think you haven't trained enough, you only trained for 20 epochs and the validation accuracy is still increasing, second I think you need to compute the competition score, which is the average of the binary and macro f1, the binary is near 1 (above 0.95 probably), so if you get a macro f1 of 0.5, you will have the score over 0.7 for sure.
", "votes": 0, "replies": [] }, { "id": 3254217, "author_name": "Gnanasenthil", "author_url": "", "post_date": "2025-07-26T03:25:18.073000", "content": "Nice to hear that. Let me try doing that.
\nThanks a ton..
@georgewang73
\nThanks again.
\nI Implemented the Competition Metric as you suggested. Locally I was able to get 0.72 ish but Now after much struggle was able to submit my solution (based on other well written notebooks - idea credits are theirs) and my Leader board metric stands at 0.69….
Also check if your macro f1 is calculated correctly, you should collapse the non-gesture label into one label when calculating this, otherwise the score might be a bit higher. Also, you should do stratification for your cross validation based on gestures and group by subject.
", "votes": 0, "replies": [] }, { "id": 3263321, "author_name": "Gnanasenthil", "author_url": "", "post_date": "2025-08-05T03:54:48.793000", "content": "Oh ok, So far My cross validation was on sequence_id / gestures. Now I shall try grouping by subject.
\nThanks..
", "votes": 0, "replies": [] } ] } ] }, { "id": 3250150, "author_name": "", "author_url": "", "post_date": "2025-07-17T18:55:25.383000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3250079": "It's a shame IMO to see so many downvote, the old thread is outdated and would be really interesting to see some updated data. Would really like to know what is imu only for 8.5-8.6! I guess somewhere around 0.83 give or take 0.01", "3250017": "Hi everyone, I just saw that the previous discussion \"what are you imu-only scores\" from @gerwynng is not on the first page of discussion anymore, yet I think it is a very good discussion, feel free to continue share your imu-only score here or go to the previous discussion https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/discussion/583023#3249998\n\nBtw, it will also be good to know how big is your model (e.g. file size, num of params)", "3250354": "0.825 cv, 0.82 lb", "3250119": "My IMU only score at the moment is CV: 0.77x and LB: 0.78x. My validation strategy is the usual `StratifiedGroupKFold` with subjects as groups. I need to go back to this model and see if I can reach 0.80 both CV/LB. I am using raw data, no FE.", "3250407": "0.803 cv, 0.81 lb", "3250076": "Pro Tip: Just sort the discussions by \"Recent Comments\" to always see the latest news\n(Or just Bookmark it yourself)", "3253121": "@georgewang73 and others\nHow is that you people manage to get a score of above 0.7.\nMy local score has NOT crossed 0.5 (I am a begineer though, but willing to learn). Previously i used just raw IMU data now i did some feature engineering as suggested [in this notebook](https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention).\n\nThis is my link to[ my colab notebook](https://colab.research.google.com/drive/1xS68dekfo0qU8OLQlQVsE8dX_epk3X3P?usp=sharing). WHY AM I NOT GETTING BETTER LOCAL SCORES EVEN\n\n(i HAVE ATTACHED MY NOTEBOOK AS WELL)\n\nAny feedback from you experts is highly appreciated..", "3250150": "" } }