![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Dec-2025/actions.png\")
I really enjoyed participating in this competition. Two months ago, I was immediately attracted when I saw that there were no public deep learning public notebooks because this challenge is exactly what deep learning loves! I joined this competition with the simple goal of beating the best public XGBoost notebook with an NN.
\nThank you Mahdi Ravaghi for the strong XGBoost baseline here (and thank you other Kagglers whose ideas are contained within). After changing that notebook to build one model per lab_id per action. And training on 100% train data. The notebook achieved CV = 0.475, Public LB = 0.477, Private LB 0.450, (88th place) top 100 rank, wow!
In this competition, we observe a near perfect CV LB relationship when using the correct validation score. (i.e. when XGB CV was 0.475 then XGB Public LB was 0.477). The correct CV is computed using only 15 lab_id which excludes labs MABe22, CalMS21, and CRIM13 because the host said these labs did not exist in test data. So all CV scores discussed in this write up, use the correct local validation score. (UPDATE: relationship with Private LB isn't as good)
The power of NN is transfer learning from one lab to another! To make NN shine, we need to preprocess the data in some invariant way because the labs are different. Also we need to define the target labels correctly. Lastly we use data augmentation to unlock NN's full potential 🔥
\nWe train our NN on all labs together. From seeing one lab, the NN will learn to make better predictions on another lab. To accomplish this, we need to make input features lab_id invariant. We can further encourage transfer learning by using data augmentation to minimize lab_id feature differences. (i.e. if mice are difference sizes between labs, then we data augment randomly scale sizes of mice up and down).
Our NN has 37 output neurons to predict the probability of each of the 37 possible actions. However not all videos nor labs contain all 37 actions as seen below:
\n
When training our NN, we need to define the targets correctly. For every video frame, the target=1 if train_annotation says 1. Otherwise it is either negative or masked. We set target=0 when train.csv says behavior_labeled but the target is missing in train_annotation. And we set target=MASK if an action is both missing from train_annotation and not included in the behavior_labeled list. \"Mask\" means that we do not include loss for this action in this video frame. (And t=0 and t=1 applies loss).
If we input the mice body parts x,y as they are in the train_tracking files, the NN will get confused. Because the arena sizes and shapes are different for each lab. The position x,y = 10,10 in one lab will not mean the same thing as position x,y = 10,10 in another lab. Below we see the variation among 3 lab arenas by making a scatter plot of 1 mouse's nose over 5 videos. We see that the labs are different shapes and sizes:
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Dec-2025/arenas.png\")
There are at least 3 ways to make invariant mouse body part positions for our NN:
\nears at (0,0) and rotate tail_base pointed down negative y axis. (All other mice in the video get shifted and rotated with the same transformation).x,y so that left side of arena is 0 and right side of arena is 1. And bottom is 0 and top is 1. (Also provide NN the pixels_per_cm feature).x,y. Instead of saying nose is at (10,10), we use the distance between nose and other body parts such as tail_base. We can compute distance as either a vector or a magnitude.The above bullet points make positions invariant. Regarding velocity, we note that velocity vector and velocity magnitude are already invariant. And we note that all distances (as vector or magnitude) between agent mouse body part and target mouse body part are already invariant.
\nBelow is an example of agent mouse nose position scatterplot after we transform with agent centric coordinate system. We observe in the scatter plot that all noses positions are now around (0,2) and when the mouse turns it head left it goes to (-2,2) and when it turns right it goes to (2,2):
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Dec-2025/agent_centric.png\")
Another challenge for transfer learning between labs is body parts. We observe that different labs track different body parts:
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Dec-2025/bodyparts.png\")
We observe that all labs have [\"nose\", \"ear_left\", \"ear_right\", \"tail_base\"]. Then about half of the labs use [\"hip_left\", \"hip_right\"] and the other half use [\"lateral_left\", \"lateral_right\"]. It turns out that hip and lateral are similar enough that we can rename them both [\"side_left\", \"side_right\"] to maximize transfer learning!
We train our NN using all labs together! Using the 8 body parts listed above and the method of positive, negative, masked loss. We build a CNN Transformer using invariant position, velocity, and distance features. The CNN layers come first and convolute in the time direction. They learn to make features based on how everything is changing. Next the NN feeds these features into layers of attention. Using cross (mouse) attention and self (mouse) attention, the transformer layers learn to compare agent and target mice and compare body parts within a single mouse. Finally the head layer outputs 37 outputs giving a probability for each of the 37 possible actions!
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/refs/heads/main/Dec-2025/cnn_trans_model.png\"){kind=spacer}
We train the same CNN Transformer multiple times and use a different length sliding window each time (i.e. vary from 2 seconds to 16 seconds). We also add more dilated convolution layers to the longer time windows to capture the larger time scale. Afterward we have four different trained NN and we can ensemble them:
\nUsing data augmentation gives a huge boost in performance. Data augmentation helps for two reasons. First is makes it harder for the NN to overfit (i.e. memorize instead of generalize the data) and we can train for more epochs. Second, it helps transfer learning between labs because all the labs start looking the same. For example if one lab had larger mice than another lab, after random scaling up and down, the model receives all sizes mice from every lab. We use
\nWe predict using the same slide window scheme that we train with. Then each video frame predicts 37 probabilities. Next we smooth probs in time and filter actions that were not in behaviors_labeled (per video_id, agent_id, target_id triplet). Lastly we apply a dictionary of action thresholds per lab_id. Each action under threshold gets set to prob=0. Lastly we take argmax to pick the frame action. (UPDATE: I have an unselected submission which achieves +0.004 private LB (rank 4th or 5th) that uses better tie breaking logic to decide what happens when multiple actions exceed threshold).
We take an equal weighted average of all our different time scale NN and achieve CV = 0.546 and Public LB = 0.551 and Private LB = 0.518 (rank 8th). In this ensemble we include 3 families of NN. Each family uses a different set of invariant features and a range of time windows. In total our final ensemble use 2xNN from family one, 5xNN from family two, and 3xNN from family three (for a total of 10 NN).
To boost our NN score, we train an XGB using one OOF from each of the 3 families of NN and all the features from Mahdi Ravaghi public notebook here. We train one XGB per lab per action. This XGB's CV achieves CV = 0.528 and Public LB = 0.536 and Private LB = 0.508 (rank 12th). And when we blend it with our Ensemble NN, the result is CV = 0.559 and Public LB = 0.562 and Private LB = 0.523 (rank 7th), wow!
I really enjoyed participating in this competition. Two months ago, I was immediately attracted when I saw that there were no public deep learning public notebooks because this challenge is exactly what deep learning loves! I joined this competition with the simple goal of beating the best public XGBoost notebook with an NN.
\nThank you Mahdi Ravaghi for the strong XGBoost baseline here (and thank you other Kagglers whose ideas are contained within). After changing that notebook to build one model per lab_id per action. And training on 100% train data. The notebook achieved CV = 0.475, Public LB = 0.477, Private LB 0.450, (88th place) top 100 rank, wow!
In this competition, we observe a near perfect CV LB relationship when using the correct validation score. (i.e. when XGB CV was 0.475 then XGB Public LB was 0.477). The correct CV is computed using only 15 lab_id which excludes labs MABe22, CalMS21, and CRIM13 because the host said these labs did not exist in test data. So all CV scores discussed in this write up, use the correct local validation score. (UPDATE: relationship with Private LB isn't as good)
The power of NN is transfer learning from one lab to another! To make NN shine, we need to preprocess the data in some invariant way because the labs are different. Also we need to define the target labels correctly. Lastly we use data augmentation to unlock NN's full potential 🔥
\nWe train our NN on all labs together. From seeing one lab, the NN will learn to make better predictions on another lab. To accomplish this, we need to make input features lab_id invariant. We can further encourage transfer learning by using data augmentation to minimize lab_id feature differences. (i.e. if mice are difference sizes between labs, then we data augment randomly scale sizes of mice up and down).
Our NN has 37 output neurons to predict the probability of each of the 37 possible actions. However not all videos nor labs contain all 37 actions as seen below:
\n
When training our NN, we need to define the targets correctly. For every video frame, the target=1 if train_annotation says 1. Otherwise it is either negative or masked. We set target=0 when train.csv says behavior_labeled but the target is missing in train_annotation. And we set target=MASK if an action is both missing from train_annotation and not included in the behavior_labeled list. \"Mask\" means that we do not include loss for this action in this video frame. (And t=0 and t=1 applies loss).
If we input the mice body parts x,y as they are in the train_tracking files, the NN will get confused. Because the arena sizes and shapes are different for each lab. The position x,y = 10,10 in one lab will not mean the same thing as position x,y = 10,10 in another lab. Below we see the variation among 3 lab arenas by making a scatter plot of 1 mouse's nose over 5 videos. We see that the labs are different shapes and sizes:
There are at least 3 ways to make invariant mouse body part positions for our NN:
\nears at (0,0) and rotate tail_base pointed down negative y axis. (All other mice in the video get shifted and rotated with the same transformation).x,y so that left side of arena is 0 and right side of arena is 1. And bottom is 0 and top is 1. (Also provide NN the pixels_per_cm feature).x,y. Instead of saying nose is at (10,10), we use the distance between nose and other body parts such as tail_base. We can compute distance as either a vector or a magnitude.The above bullet points make positions invariant. Regarding velocity, we note that velocity vector and velocity magnitude are already invariant. And we note that all distances (as vector or magnitude) between agent mouse body part and target mouse body part are already invariant.
\nBelow is an example of agent mouse nose position scatterplot after we transform with agent centric coordinate system. We observe in the scatter plot that all noses positions are now around (0,2) and when the mouse turns it head left it goes to (-2,2) and when it turns right it goes to (2,2):
Another challenge for transfer learning between labs is body parts. We observe that different labs track different body parts:
\n
We observe that all labs have [\"nose\", \"ear_left\", \"ear_right\", \"tail_base\"]. Then about half of the labs use [\"hip_left\", \"hip_right\"] and the other half use [\"lateral_left\", \"lateral_right\"]. It turns out that hip and lateral are similar enough that we can rename them both [\"side_left\", \"side_right\"] to maximize transfer learning!
We train our NN using all labs together! Using the 8 body parts listed above and the method of positive, negative, masked loss. We build a CNN Transformer using invariant position, velocity, and distance features. The CNN layers come first and convolute in the time direction. They learn to make features based on how everything is changing. Next the NN feeds these features into layers of attention. Using cross (mouse) attention and self (mouse) attention, the transformer layers learn to compare agent and target mice and compare body parts within a single mouse. Finally the head layer outputs 37 outputs giving a probability for each of the 37 possible actions!
\n
We train the same CNN Transformer multiple times and use a different length sliding window each time (i.e. vary from 2 seconds to 16 seconds). We also add more dilated convolution layers to the longer time windows to capture the larger time scale. Afterward we have four different trained NN and we can ensemble them:
\nUsing data augmentation gives a huge boost in performance. Data augmentation helps for two reasons. First is makes it harder for the NN to overfit (i.e. memorize instead of generalize the data) and we can train for more epochs. Second, it helps transfer learning between labs because all the labs start looking the same. For example if one lab had larger mice than another lab, after random scaling up and down, the model receives all sizes mice from every lab. We use
\nWe predict using the same slide window scheme that we train with. Then each video frame predicts 37 probabilities. Next we smooth probs in time and filter actions that were not in behaviors_labeled (per video_id, agent_id, target_id triplet). Lastly we apply a dictionary of action thresholds per lab_id. Each action under threshold gets set to prob=0. Lastly we take argmax to pick the frame action. (UPDATE: I have an unselected submission which achieves +0.004 private LB (rank 4th or 5th) that uses better tie breaking logic to decide what happens when multiple actions exceed threshold).
We take an equal weighted average of all our different time scale NN and achieve CV = 0.546 and Public LB = 0.551 and Private LB = 0.518 (rank 8th). In this ensemble we include 3 families of NN. Each family uses a different set of invariant features and a range of time windows. In total our final ensemble use 2xNN from family one, 5xNN from family two, and 3xNN from family three (for a total of 10 NN).
To boost our NN score, we train an XGB using one OOF from each of the 3 families of NN and all the features from Mahdi Ravaghi public notebook here. We train one XGB per lab per action. This XGB's CV achieves CV = 0.528 and Public LB = 0.536 and Private LB = 0.508 (rank 12th). And when we blend it with our Ensemble NN, the result is CV = 0.559 and Public LB = 0.562 and Private LB = 0.523 (rank 7th), wow!
Great work!
", "rawMarkdown": "Great work!", "votes": null }, { "id": "3377235", "postDate": "12/16/2025 00:18:45", "content": "can not agree more
", "rawMarkdown": "can not agree more", "votes": null }, { "id": "3377244", "postDate": "12/16/2025 00:35:39", "content": "Congratulations Chris! And thanks a lot for the writeup!
\nYour NN-only score is super impressive! If you don't mind a few questions..
\nThanks!
", "rawMarkdown": "Congratulations Chris! And thanks a lot for the writeup!\n\nYour NN-only score is super impressive! If you don't mind a few questions..\n\n1. How did you deal with the sparsity in the data? Did you interpolate the sensor keypoints or applied any other preproc?\n2. After making the input features lab invariant, did you train directly on all labs? Or did you first pre-train on all labs to later finetune on each individual lab?\n\nThanks!", "votes": null }, { "id": "3377250", "postDate": "12/16/2025 00:47:28", "content": "Hi, congratulations winning 4th! I train a single NN on all 847 labeled videos. The secret is invariant features and data augmentations.
\nMy NN can train on one lab and successfully infer a good CV score on different lab because all the labs look the same to my NN. Furthermore, I have one NN that only uses nose, ear_left, ear_right, and tail_base and scores close to gold by itself (using different time scales).
\nSo already those 4 features exist for every video and perform well. Then i either add all the other body parts and mark them as missing if a lab doesn't have them. Or you can just call \"hips\" and \"lateral\" both \"sides\". Then every video has \"sides\". And with data augmentation all \"sides\" look alike too. So all these extra body parts improve upon only using nose, ear, tail_base.
", "rawMarkdown": "Hi, congratulations winning 4th! I train a single NN on all 847 labeled videos. The secret is invariant features and data augmentations. \n\nMy NN can train on one lab and successfully infer a good CV score on different lab because all the labs look the same to my NN. Furthermore, I have one NN that only uses nose, ear_left, ear_right, and tail_base and scores close to gold by itself (using different time scales). \n\nSo already those 4 features exist for every video and perform well. Then i either add all the other body parts and mark them as missing if a lab doesn't have them. Or you can just call \"hips\" and \"lateral\" both \"sides\". Then every video has \"sides\". And with data augmentation all \"sides\" look alike too. So all these extra body parts improve upon only using nose, ear, tail_base.", "votes": null }, { "id": "3377251", "postDate": "12/16/2025 00:47:53", "content": "Totally agree 👍 👏
", "rawMarkdown": "Totally agree 👍 👏", "votes": null }, { "id": "3377252", "postDate": "12/16/2025 00:48:51", "content": "Hey Chris, could you explain the \"After changing that notebook to build one model per lab_id per action. And training on 100% train data.\" to me? I personally did one model per (section, action) across all labs, and used group‑aware CV to prevent leakage between videos. Using groups_action = meta['lab_id'] + '_' + meta['video_id']. Are you saying for this example the model is a full train (1 fold) with one model per lab_id per action?
", "rawMarkdown": "Hey Chris, could you explain the \"After changing that notebook to build one model per lab_id per action. And training on 100% train data.\" to me? I personally did one model per (section, action) across all labs, and used group‑aware CV to prevent leakage between videos. Using groups_action = meta['lab_id'] + '_' + meta['video_id']. Are you saying for this example the model is a full train (1 fold) with one model per lab_id per action?", "votes": null }, { "id": "3377256", "postDate": "12/16/2025 01:02:46", "content": "Congrats Chris! Solo gold and 7x GM, well deserved! Your NN performance seems great. It was especially challenging in this competition due to sparsity of some labels and differences between the labs.
", "rawMarkdown": "Congrats Chris! Solo gold and 7x GM, well deserved! Your NN performance seems great. It was especially challenging in this competition due to sparsity of some labels and differences between the labs.", "votes": null }, { "id": "3377265", "postDate": "12/16/2025 01:34:06", "content": "Congratulations ！ Your solution is truly impressive，I learned a lot. \nif I might ask a few questions regarding the training setup：
\nDuring training, for self actions, is the input still structured as an agent–target pair?\nIn that case, are the agent and target features identical？
If there are 4 mice in a video, then within a single time window there are theoretically 4×4 possible agent–target combinations.\nDo you actually enumerate all of these pairs during training, or do you apply some form of filtering?
For time windows with no annotated behavior, do you include them as a negative class during training, or is training restricted only to windows that contain at least one labeled behavior?\nThanks again for the excellent work!
Congrats! I knew NNs would be in the winning solutions but as I joined late I didn't have time to explore them
\n\n\nAfter changing that notebook to build one model per lab_id per action. And training on 100% train data. The notebook achieved CV = 0.475, Public LB = 0.477, Private LB 0.450, (88th place) top 100 rank, wow!
\n
I have 2 questions about this part:
\nThanks.
\nYes, the model always receives agent and target features. For self actions, it receives duplicates of all these features (and the relationship differences are all zero).
I only train on the pairs that are identified in the train.csv behaviors_labeled column. Whenever we see mouseX,mouseY in behaviors_labeled for a specific video, then we train for the full length of the video on that pair.
We train with all windows (both with and without 1 label). However in family three of my NN, for each video I duplicate all the windows with at least 1 labeled behavior. (i.e. upsample positives by 2x)
=======
\nRegarding #3, i did try a trick from object detection but it didn't work better. I built two models. The first model is only trained on windows with at least 1 positive label. Then then second model predicts when a label exists or not. This trick works great for rare image object detection.
\nSince model 1 always see positive targets, it learns very well to pick the right class. But of course it will always predict a class all the time. So the second model's purpose is to remove a prediction if the second model believes the window doesn't have an action. This has won object detection competitions, but i didn't get it to work here.
", "rawMarkdown": "Thanks.\n1. Yes, the model always receives agent and target features. For self actions, it receives duplicates of all these features (and the relationship differences are all zero).\n\n2. I only train on the pairs that are identified in the `train.csv` behaviors_labeled column. Whenever we see `mouseX,mouseY` in behaviors_labeled for a specific video, then we train for the full length of the video on that pair.\n\n3. We train with all windows (both with and without 1 label). However in family three of my NN, for each video I duplicate all the windows with at least 1 labeled behavior. (i.e. upsample positives by 2x)\n\n=======\n\nRegarding #3, i did try a trick from object detection but it didn't work better. I built two models. The first model is only trained on windows with at least 1 positive label. Then then second model predicts when a label exists or not. This trick works great for rare image object detection. \n\nSince model 1 always see positive targets, it learns very well to pick the right class. But of course it will always predict a class all the time. So the second model's purpose is to remove a prediction if the second model believes the window doesn't have an action. This has won object detection competitions, but i didn't get it to work here.", "votes": null }, { "id": "3377273", "postDate": "12/16/2025 02:02:12", "content": "Hi. There is no need to use group aware CV (if your only goal is to submit to LB) because i didn't tune anything. I kept the XGB parameters as they were. Instead of training on 9 sections like the public notebook, I train on 20 sections. Each lab_id is its own section. And AdaptableSnail is 2 sections because I actually train on lab_id + body parts.
\nFor for each lab_id, we train one XGB model for 275 iterations on 100% train data (with XGB parameters same as public notebook). That's it. Then during inference, we use these models to infer each test lab_id action pair and get public LB 0.477
\nTo compute the CV score and OOF for ensemble, then I did use GroupKFold based on video_id. I created my own 8 folds and trained 8 fold models and inferred the 8 validation subsets. The CV score was 0.475.
", "rawMarkdown": "Hi. There is no need to use group aware CV (if your only goal is to submit to LB) because i didn't tune anything. I kept the XGB parameters as they were. Instead of training on 9 sections like the public notebook, I train on 20 sections. Each lab_id is its own section. And AdaptableSnail is 2 sections because I actually train on lab_id + body parts.\n\nFor for each lab_id, we train one XGB model for 275 iterations on 100% train data (with XGB parameters same as public notebook). That's it. Then during inference, we use these models to infer each test lab_id action pair and get public LB 0.477\n\nTo compute the CV score and OOF for ensemble, then I did use GroupKFold based on video_id. I created my own 8 folds and trained 8 fold models and inferred the 8 validation subsets. The CV score was 0.475.", "votes": null }, { "id": "3377275", "postDate": "12/16/2025 02:07:15", "content": "Thanks @aerdem4 Congratulations winning 4th place Cash Gold! I remember how you stayed same or greater on private LB in CIBMTR comp. I am impressed once again how your team stays the same or improves on private LB! Great job!
\nI actually removed one of my post process here because i was scared of shake down 😀 If I didn't remove that PP, I would finish with same private LB 0.527 like your team 😀 (It was PP to decide ties when model predicts 2+ actions per frame).
", "rawMarkdown": "Thanks @aerdem4 Congratulations winning 4th place Cash Gold! I remember how you stayed same or greater on private LB in CIBMTR comp. I am impressed once again how your team stays the same or improves on private LB! Great job!\n\nI actually removed one of my post process here because i was scared of shake down 😀 If I didn't remove that PP, I would finish with same private LB 0.527 like your team 😀 (It was PP to decide ties when model predicts 2+ actions per frame).", "votes": null }, { "id": "3377276", "postDate": "12/16/2025 02:09:16", "content": "Thank you! That one is going to hurt for a bit. My plan since about month 1 of the competition was to make tabular ensemble, add tabm, and then move into CNN. I knew I had something missing with cv but was trying things far less straight forward thinking people had found some magic. We actually had a model with only .45 LB that we managed to boost with post process and some ensemble tweaks. Thanks for your walkthrough, the CNN dive is super insightful!
", "rawMarkdown": "Thank you! That one is going to hurt for a bit. My plan since about month 1 of the competition was to make tabular ensemble, add tabm, and then move into CNN. I knew I had something missing with cv but was trying things far less straight forward thinking people had found some magic. We actually had a model with only .45 LB that we managed to boost with post process and some ensemble tweaks. Thanks for your walkthrough, the CNN dive is super insightful!", "votes": null }, { "id": "3377277", "postDate": "12/16/2025 02:14:59", "content": "Hi @mahoganybuttstrings , to submit to LB we don't need to compute the CV. We just train the notebook as is with 100% train. To force the notebook to only put 1 lab in each section, we just do this. We add the lab as a \"body part\":
\nLABS = list( train.lab_id.unique() )\ntrain[\"body_parts_tracked\"] = train[\"body_parts_tracked\"].str.replace(']','')\ntrain[\"body_parts_tracked\"] = train[\"body_parts_tracked\"] + ',\"' + train.lab_id + '\"]'\n\ntest[\"body_parts_tracked\"] = test[\"body_parts_tracked\"].str.replace(']','')\ntest[\"body_parts_tracked\"] = test[\"body_parts_tracked\"] + ',\"' + test.lab_id + '\"]'\nand then later we remove these labs from body parts with
\ndrop_body_parts += LABS\nThat's it. Afterward, the notebook will train 20 sections instead of 9 sections. But the notebook trains with 3 KFold, so each model is only trained with 66% data. We need to write our own KFold function to pass to function Trainer in library koolbox. So I wrote a KFold function that just yields 100% data for each of the 3 folds with different shuffled order each time.
", "rawMarkdown": "Hi @mahoganybuttstrings , to submit to LB we don't need to compute the CV. We just train the notebook as is with 100% train. To force the notebook to only put 1 lab in each section, we just do this. We add the lab as a \"body part\":\n\n LABS = list( train.lab_id.unique() )\n train[\"body_parts_tracked\"] = train[\"body_parts_tracked\"].str.replace(']','')\n train[\"body_parts_tracked\"] = train[\"body_parts_tracked\"] + ',\"' + train.lab_id + '\"]'\n\n test[\"body_parts_tracked\"] = test[\"body_parts_tracked\"].str.replace(']','')\n test[\"body_parts_tracked\"] = test[\"body_parts_tracked\"] + ',\"' + test.lab_id + '\"]'\n\nand then later we remove these labs from body parts with\n\n drop_body_parts += LABS\n\nThat's it. Afterward, the notebook will train 20 sections instead of 9 sections. But the notebook trains with 3 KFold, so each model is only trained with 66% data. We need to write our own KFold function to pass to function Trainer in library koolbox. So I wrote a KFold function that just yields 100% data for each of the 3 folds with different shuffled order each time.", "votes": null }, { "id": "3377278", "postDate": "12/16/2025 02:18:32", "content": "Congrats Chris, your engineering on the target building and data transformation developments are really impressed. Also really thanks for the write-up I learn a new lesson for tabular competition.
", "rawMarkdown": "Congrats Chris, your engineering on the target building and data transformation developments are really impressed. Also really thanks for the write-up I learn a new lesson for tabular competition.", "votes": null }, { "id": "3377279", "postDate": "12/16/2025 02:22:58", "content": "Ah, ok, thanks for the explanation!
", "rawMarkdown": "Ah, ok, thanks for the explanation!", "votes": null }, { "id": "3377285", "postDate": "12/16/2025 02:48:52", "content": "@mahoganybuttstrings One other comment. We need to use the thresholds.pkl file that was generated by the original public notebook because when we train 100% the new thresholds.pkl will be meaningless.
", "rawMarkdown": "mahoganybuttstrings One other comment. We need to use the thresholds.pkl file that was generated by the original public notebook because when we train 100% the new thresholds.pkl will be meaningless.", "votes": null }, { "id": "3377286", "postDate": "12/16/2025 02:50:11", "content": "@cody11null There is one more important detail I forgot to say. We need to use the thresholds.pkl file that was generated by the original 3 KFold public notebook because when we train 100% the new thresholds.pkl will be meaningless.
\nOr if you do separately compute an authentic CV OOF set, you can find better thresholds using that.
", "rawMarkdown": "cody11null There is one more important detail I forgot to say. We need to use the thresholds.pkl file that was generated by the original 3 KFold public notebook because when we train 100% the new thresholds.pkl will be meaningless.\n\nOr if you do separately compute an authentic CV OOF set, you can find better thresholds using that.", "votes": null }, { "id": "3377342", "postDate": "12/16/2025 06:29:52", "content": "Congrats on the 7th place!
\nYour approach to handling lab variations using invariant features (especially the agent-centric coordinate system) is brilliant. The architecture combining CNN-Transformer with a multi-scale ensemble strategy looks very robust. Also, stacking it with XGBoost clearly gave a great boost.
\nThanks for the detailed write-up and sharing your insights!
", "rawMarkdown": "Congrats on the 7th place!\n\nYour approach to handling lab variations using invariant features (especially the agent-centric coordinate system) is brilliant. The architecture combining CNN-Transformer with a multi-scale ensemble strategy looks very robust. Also, stacking it with XGBoost clearly gave a great boost.\n\nThanks for the detailed write-up and sharing your insights!", "votes": null }, { "id": "3377456", "postDate": "12/16/2025 11:15:26", "content": "Thank you Musa
", "rawMarkdown": "Thank you Musa", "votes": null }, { "id": "3377601", "postDate": "12/16/2025 16:02:53", "content": "@arc144 One other thing I did was add positive weights to rare actions in loss calculation to boost them. And I used 37 multi-label loss instead of 38 multi-class loss. Even though this is a 38 multi-class (i.e. only 1 class per frame which includes \"no action\"), there are so many \"no action\" frames that multi-class gets too many updates from the 38th \"no action\" class.
def masked_bce_with_logits(logits, targets, mask, pos_weight=None):\n \"\"\"\n logits, targets, mask: [B,T,C]\n pos_weight: [C] or None (scales positive examples only)\n \"\"\"\n loss = F.binary_cross_entropy_with_logits(logits, targets, reduction='none') # [B,T,C]\n if pos_weight is not None:\n # scale positives by pos_weight (like BCEWithLogitsLoss(pos_weight))\n scale = 1 + (pos_weight.view(1,1,-1) - 1) * targets\n loss = loss * scale\n loss = loss * mask\n denom = mask.sum().clamp_min(1.0)\n return loss.sum() / denom\nCongratulations @cdeotte, Thanks a lot for the detailed write-up!! There are a lot of things to learn from this write-up and the discussions.
", "rawMarkdown": "Congratulations @cdeotte, Thanks a lot for the detailed write-up!! There are a lot of things to learn from this write-up and the discussions.", "votes": null }, { "id": "3377712", "postDate": "12/16/2025 20:25:42", "content": "Thank you
", "rawMarkdown": "Thank you", "votes": null }, { "id": "3377783", "postDate": "12/16/2025 23:57:58", "content": "very good plots.
", "rawMarkdown": "very good plots.", "votes": null }, { "id": "3378128", "postDate": "12/17/2025 15:25:59", "content": "Is this single NN a CV-fold training or a single run? How did you prevent overfitting/know when to stop when you trained on the full 847labeled videos?
", "rawMarkdown": "Is this single NN a CV-fold training or a single run? How did you prevent overfitting/know when to stop when you trained on the full 847labeled videos?", "votes": null }, { "id": "3378144", "postDate": "12/17/2025 15:48:00", "content": "My submissions to LB train on 100% train data. And for each NN, I ensemble 6 of the same NN trained with different seeds. I compute the CV using 3 GroupKFold (by video_id). After finding the optimal number of epochs, I then train with 100% using 2/3 of this number epochs. Because the data is enlarging from 66% to 100%. (We could probably run more epochs since 100% data is like 66% data with additional data augmentations and data augmentation allows for more epochs but i play it safe and resultantly using 100% at 2/3 epochs versus 66% is always better).
", "rawMarkdown": "My submissions to LB train on 100% train data. And for each NN, I ensemble 6 of the same NN trained with different seeds. I compute the CV using 3 GroupKFold (by video_id). After finding the optimal number of epochs, I then train with 100% using 2/3 of this number epochs. Because the data is enlarging from 66% to 100%. (We could probably run more epochs since 100% data is like 66% data with additional data augmentations and data augmentation allows for more epochs but i play it safe and resultantly using 100% at 2/3 epochs versus 66% is always better).", "votes": null }, { "id": "3378298", "postDate": "12/17/2025 21:31:37", "content": "Congrats! Bro for getting 7th place.
", "rawMarkdown": "Congrats! Bro for getting 7th place.", "votes": null }, { "id": "3378431", "postDate": "12/18/2025 06:58:32", "content": "A lot to be learnt in this write up, thanks!
", "rawMarkdown": "A lot to be learnt in this write up, thanks!", "votes": null }, { "id": "3379307", "postDate": "12/19/2025 15:00:06", "content": "That makes total sense! There are many underrepresented classes in the dataset. Thanks for the explanation!
\nBtw, did you get a chance to try FocalLoss vs. just adding the positive weights to BCE?
", "rawMarkdown": "That makes total sense! There are many underrepresented classes in the dataset. Thanks for the explanation!\n\nBtw, did you get a chance to try FocalLoss vs. just adding the positive weights to BCE?", "votes": null }, { "id": "3380248", "postDate": "12/22/2025 03:49:11", "content": "Thanks a lot for the details @cdeotte
", "rawMarkdown": "Thanks a lot for the details @cdeotte", "votes": null }, { "id": "3380777", "postDate": "12/23/2025 02:49:19", "content": "That's awesome. Thank you for the great work!
", "rawMarkdown": "That's awesome. Thank you for the great work!", "votes": null }, { "id": "3381449", "postDate": "12/24/2025 16:05:56", "content": "congrats bro 7 place just amazing
", "rawMarkdown": "congrats bro 7 place just amazing", "votes": null }, { "id": "3383306", "postDate": "12/29/2025 20:35:48", "content": "Thanks Sheraz
", "rawMarkdown": "Thanks Sheraz", "votes": null }, { "id": "3383327", "postDate": "12/29/2025 21:53:33", "content": "Very very Congrats! for getting 7th place.✨🥳
", "rawMarkdown": "Very very Congrats! for getting 7th place.✨🥳", "votes": null }, { "id": "3385712", "postDate": "01/03/2026 18:45:16", "content": "Amazing write-up and solution, thanks for breaking everything down so clearly. How much of the final gain do you think came from invariance + augmentation vs. the NN architecture itself? @cdeotte
", "rawMarkdown": "Amazing write-up and solution, thanks for breaking everything down so clearly. How much of the final gain do you think came from invariance + augmentation vs. the NN architecture itself? @cdeotte", "votes": null }, { "id": "3385731", "postDate": "01/03/2026 19:28:08", "content": "Thanks! The NN architecture itself was similar to XGB performance CV 0.475. Using invariance and augmentation was the key to boost (single model NN) performance to CV 0.520 ! Then using a blend of 10x NN each with different features and time scales boosted to CV 0.546! (And finally blending 10xNN with XGB achieved CV 0.559)
", "rawMarkdown": "Thanks! The NN architecture itself was similar to XGB performance CV 0.475. Using invariance and augmentation was the key to boost (single model NN) performance to CV 0.520 ! Then using a blend of 10x NN each with different features and time scales boosted to CV 0.546! (And finally blending 10xNN with XGB achieved CV 0.559)", "votes": null }, { "id": "3385908", "postDate": "01/04/2026 08:17:27", "content": "Really appreciate the breakdown. Definitely a good reminder to focus more on feature space than just architecture.
", "rawMarkdown": "Really appreciate the breakdown. Definitely a good reminder to focus more on feature space than just architecture.", "votes": null } ], "comments": [ { "id": 3377230, "author_name": "evanarlenhandy", "author_url": "", "post_date": "12/16/2025 00:13:09", "content": "Great work!
", "votes": null, "replies": [] }, { "id": 3377235, "author_name": "aizealotwu", "author_url": "", "post_date": "12/16/2025 00:18:45", "content": "can not agree more
", "votes": null, "replies": [] }, { "id": 3377244, "author_name": "arc144", "author_url": "", "post_date": "12/16/2025 00:35:39", "content": "Congratulations Chris! And thanks a lot for the writeup!
\nYour NN-only score is super impressive! If you don't mind a few questions..
\nThanks!
", "votes": null, "replies": [ { "id": 3377250, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 00:47:28", "content": "Hi, congratulations winning 4th! I train a single NN on all 847 labeled videos. The secret is invariant features and data augmentations.
\nMy NN can train on one lab and successfully infer a good CV score on different lab because all the labs look the same to my NN. Furthermore, I have one NN that only uses nose, ear_left, ear_right, and tail_base and scores close to gold by itself (using different time scales).
\nSo already those 4 features exist for every video and perform well. Then i either add all the other body parts and mark them as missing if a lab doesn't have them. Or you can just call \"hips\" and \"lateral\" both \"sides\". Then every video has \"sides\". And with data augmentation all \"sides\" look alike too. So all these extra body parts improve upon only using nose, ear, tail_base.
", "votes": null, "replies": [ { "id": 3378128, "author_name": "dankrstev", "author_url": "", "post_date": "12/17/2025 15:25:59", "content": "Is this single NN a CV-fold training or a single run? How did you prevent overfitting/know when to stop when you trained on the full 847labeled videos?
", "votes": null, "replies": [ { "id": 3378144, "author_name": "cdeotte", "author_url": "", "post_date": "12/17/2025 15:48:00", "content": "My submissions to LB train on 100% train data. And for each NN, I ensemble 6 of the same NN trained with different seeds. I compute the CV using 3 GroupKFold (by video_id). After finding the optimal number of epochs, I then train with 100% using 2/3 of this number epochs. Because the data is enlarging from 66% to 100%. (We could probably run more epochs since 100% data is like 66% data with additional data augmentations and data augmentation allows for more epochs but i play it safe and resultantly using 100% at 2/3 epochs versus 66% is always better).
", "votes": null, "replies": [] } ] } ] }, { "id": 3377601, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 16:02:53", "content": "@arc144 One other thing I did was add positive weights to rare actions in loss calculation to boost them. And I used 37 multi-label loss instead of 38 multi-class loss. Even though this is a 38 multi-class (i.e. only 1 class per frame which includes \"no action\"), there are so many \"no action\" frames that multi-class gets too many updates from the 38th \"no action\" class.
def masked_bce_with_logits(logits, targets, mask, pos_weight=None):\n \"\"\"\n logits, targets, mask: [B,T,C]\n pos_weight: [C] or None (scales positive examples only)\n \"\"\"\n loss = F.binary_cross_entropy_with_logits(logits, targets, reduction='none') # [B,T,C]\n if pos_weight is not None:\n # scale positives by pos_weight (like BCEWithLogitsLoss(pos_weight))\n scale = 1 + (pos_weight.view(1,1,-1) - 1) * targets\n loss = loss * scale\n loss = loss * mask\n denom = mask.sum().clamp_min(1.0)\n return loss.sum() / denom\nThat makes total sense! There are many underrepresented classes in the dataset. Thanks for the explanation!
\nBtw, did you get a chance to try FocalLoss vs. just adding the positive weights to BCE?
", "votes": null, "replies": [] } ] } ] }, { "id": 3377251, "author_name": "hassangasem", "author_url": "", "post_date": "12/16/2025 00:47:53", "content": "Totally agree 👍 👏
", "votes": null, "replies": [] }, { "id": 3377252, "author_name": "cody11null", "author_url": "", "post_date": "12/16/2025 00:48:51", "content": "Hey Chris, could you explain the \"After changing that notebook to build one model per lab_id per action. And training on 100% train data.\" to me? I personally did one model per (section, action) across all labs, and used group‑aware CV to prevent leakage between videos. Using groups_action = meta['lab_id'] + '_' + meta['video_id']. Are you saying for this example the model is a full train (1 fold) with one model per lab_id per action?
", "votes": null, "replies": [ { "id": 3377273, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 02:02:12", "content": "Hi. There is no need to use group aware CV (if your only goal is to submit to LB) because i didn't tune anything. I kept the XGB parameters as they were. Instead of training on 9 sections like the public notebook, I train on 20 sections. Each lab_id is its own section. And AdaptableSnail is 2 sections because I actually train on lab_id + body parts.
\nFor for each lab_id, we train one XGB model for 275 iterations on 100% train data (with XGB parameters same as public notebook). That's it. Then during inference, we use these models to infer each test lab_id action pair and get public LB 0.477
\nTo compute the CV score and OOF for ensemble, then I did use GroupKFold based on video_id. I created my own 8 folds and trained 8 fold models and inferred the 8 validation subsets. The CV score was 0.475.
", "votes": null, "replies": [ { "id": 3377276, "author_name": "cody11null", "author_url": "", "post_date": "12/16/2025 02:09:16", "content": "Thank you! That one is going to hurt for a bit. My plan since about month 1 of the competition was to make tabular ensemble, add tabm, and then move into CNN. I knew I had something missing with cv but was trying things far less straight forward thinking people had found some magic. We actually had a model with only .45 LB that we managed to boost with post process and some ensemble tweaks. Thanks for your walkthrough, the CNN dive is super insightful!
", "votes": null, "replies": [ { "id": 3377286, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 02:50:11", "content": "@cody11null There is one more important detail I forgot to say. We need to use the thresholds.pkl file that was generated by the original 3 KFold public notebook because when we train 100% the new thresholds.pkl will be meaningless.
\nOr if you do separately compute an authentic CV OOF set, you can find better thresholds using that.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3377256, "author_name": "aerdem4", "author_url": "", "post_date": "12/16/2025 01:02:46", "content": "Congrats Chris! Solo gold and 7x GM, well deserved! Your NN performance seems great. It was especially challenging in this competition due to sparsity of some labels and differences between the labs.
", "votes": null, "replies": [ { "id": 3377275, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 02:07:15", "content": "Thanks @aerdem4 Congratulations winning 4th place Cash Gold! I remember how you stayed same or greater on private LB in CIBMTR comp. I am impressed once again how your team stays the same or improves on private LB! Great job!
\nI actually removed one of my post process here because i was scared of shake down 😀 If I didn't remove that PP, I would finish with same private LB 0.527 like your team 😀 (It was PP to decide ties when model predicts 2+ actions per frame).
", "votes": null, "replies": [] } ] }, { "id": 3377265, "author_name": "forgakki", "author_url": "", "post_date": "12/16/2025 01:34:06", "content": "Congratulations ！ Your solution is truly impressive，I learned a lot. \nif I might ask a few questions regarding the training setup：
\nDuring training, for self actions, is the input still structured as an agent–target pair?\nIn that case, are the agent and target features identical？
If there are 4 mice in a video, then within a single time window there are theoretically 4×4 possible agent–target combinations.\nDo you actually enumerate all of these pairs during training, or do you apply some form of filtering?
For time windows with no annotated behavior, do you include them as a negative class during training, or is training restricted only to windows that contain at least one labeled behavior?\nThanks again for the excellent work!
Thanks.
\nYes, the model always receives agent and target features. For self actions, it receives duplicates of all these features (and the relationship differences are all zero).
I only train on the pairs that are identified in the train.csv behaviors_labeled column. Whenever we see mouseX,mouseY in behaviors_labeled for a specific video, then we train for the full length of the video on that pair.
We train with all windows (both with and without 1 label). However in family three of my NN, for each video I duplicate all the windows with at least 1 labeled behavior. (i.e. upsample positives by 2x)
=======
\nRegarding #3, i did try a trick from object detection but it didn't work better. I built two models. The first model is only trained on windows with at least 1 positive label. Then then second model predicts when a label exists or not. This trick works great for rare image object detection.
\nSince model 1 always see positive targets, it learns very well to pick the right class. But of course it will always predict a class all the time. So the second model's purpose is to remove a prediction if the second model believes the window doesn't have an action. This has won object detection competitions, but i didn't get it to work here.
", "votes": null, "replies": [] } ] }, { "id": 3377266, "author_name": "mahoganybuttstrings", "author_url": "", "post_date": "12/16/2025 01:37:31", "content": "Congrats! I knew NNs would be in the winning solutions but as I joined late I didn't have time to explore them
\n\n\nAfter changing that notebook to build one model per lab_id per action. And training on 100% train data. The notebook achieved CV = 0.475, Public LB = 0.477, Private LB 0.450, (88th place) top 100 rank, wow!
\n
I have 2 questions about this part:
\nHi @mahoganybuttstrings , to submit to LB we don't need to compute the CV. We just train the notebook as is with 100% train. To force the notebook to only put 1 lab in each section, we just do this. We add the lab as a \"body part\":
\nLABS = list( train.lab_id.unique() )\ntrain[\"body_parts_tracked\"] = train[\"body_parts_tracked\"].str.replace(']','')\ntrain[\"body_parts_tracked\"] = train[\"body_parts_tracked\"] + ',\"' + train.lab_id + '\"]'\n\ntest[\"body_parts_tracked\"] = test[\"body_parts_tracked\"].str.replace(']','')\ntest[\"body_parts_tracked\"] = test[\"body_parts_tracked\"] + ',\"' + test.lab_id + '\"]'\nand then later we remove these labs from body parts with
\ndrop_body_parts += LABS\nThat's it. Afterward, the notebook will train 20 sections instead of 9 sections. But the notebook trains with 3 KFold, so each model is only trained with 66% data. We need to write our own KFold function to pass to function Trainer in library koolbox. So I wrote a KFold function that just yields 100% data for each of the 3 folds with different shuffled order each time.
", "votes": null, "replies": [ { "id": 3377279, "author_name": "mahoganybuttstrings", "author_url": "", "post_date": "12/16/2025 02:22:58", "content": "Ah, ok, thanks for the explanation!
", "votes": null, "replies": [ { "id": 3377285, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 02:48:52", "content": "@mahoganybuttstrings One other comment. We need to use the thresholds.pkl file that was generated by the original public notebook because when we train 100% the new thresholds.pkl will be meaningless.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3377278, "author_name": "tom99763", "author_url": "", "post_date": "12/16/2025 02:18:32", "content": "Congrats Chris, your engineering on the target building and data transformation developments are really impressed. Also really thanks for the write-up I learn a new lesson for tabular competition.
", "votes": null, "replies": [] }, { "id": 3377342, "author_name": "musapeker", "author_url": "", "post_date": "12/16/2025 06:29:52", "content": "Congrats on the 7th place!
\nYour approach to handling lab variations using invariant features (especially the agent-centric coordinate system) is brilliant. The architecture combining CNN-Transformer with a multi-scale ensemble strategy looks very robust. Also, stacking it with XGBoost clearly gave a great boost.
\nThanks for the detailed write-up and sharing your insights!
", "votes": null, "replies": [ { "id": 3377456, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 11:15:26", "content": "Thank you Musa
", "votes": null, "replies": [] } ] }, { "id": 3377633, "author_name": "samithsachidanandan", "author_url": "", "post_date": "12/16/2025 17:26:40", "content": "Congratulations @cdeotte, Thanks a lot for the detailed write-up!! There are a lot of things to learn from this write-up and the discussions.
", "votes": null, "replies": [ { "id": 3377712, "author_name": "cdeotte", "author_url": "", "post_date": "12/16/2025 20:25:42", "content": "Thank you
", "votes": null, "replies": [] } ] }, { "id": 3377783, "author_name": "tanazsetayesh", "author_url": "", "post_date": "12/16/2025 23:57:58", "content": "very good plots.
", "votes": null, "replies": [] }, { "id": 3378298, "author_name": "craftycode", "author_url": "", "post_date": "12/17/2025 21:31:37", "content": "Congrats! Bro for getting 7th place.
", "votes": null, "replies": [] }, { "id": 3378431, "author_name": "irakozekelly", "author_url": "", "post_date": "12/18/2025 06:58:32", "content": "A lot to be learnt in this write up, thanks!
", "votes": null, "replies": [] }, { "id": 3380248, "author_name": "navneetbende", "author_url": "", "post_date": "12/22/2025 03:49:11", "content": "Thanks a lot for the details @cdeotte
", "votes": null, "replies": [] }, { "id": 3380777, "author_name": "eddiebhl", "author_url": "", "post_date": "12/23/2025 02:49:19", "content": "That's awesome. Thank you for the great work!
", "votes": null, "replies": [] }, { "id": 3381449, "author_name": "strsheraz", "author_url": "", "post_date": "12/24/2025 16:05:56", "content": "congrats bro 7 place just amazing
", "votes": null, "replies": [ { "id": 3383306, "author_name": "cdeotte", "author_url": "", "post_date": "12/29/2025 20:35:48", "content": "Thanks Sheraz
", "votes": null, "replies": [] } ] }, { "id": 3383327, "author_name": "khushikyad001", "author_url": "", "post_date": "12/29/2025 21:53:33", "content": "Very very Congrats! for getting 7th place.✨🥳
", "votes": null, "replies": [] }, { "id": 3385712, "author_name": "sudhanshu3112", "author_url": "", "post_date": "01/03/2026 18:45:16", "content": "Amazing write-up and solution, thanks for breaking everything down so clearly. How much of the final gain do you think came from invariance + augmentation vs. the NN architecture itself? @cdeotte
", "votes": null, "replies": [ { "id": 3385731, "author_name": "cdeotte", "author_url": "", "post_date": "01/03/2026 19:28:08", "content": "Thanks! The NN architecture itself was similar to XGB performance CV 0.475. Using invariance and augmentation was the key to boost (single model NN) performance to CV 0.520 ! Then using a blend of 10x NN each with different features and time scales boosted to CV 0.546! (And finally blending 10xNN with XGB achieved CV 0.559)
", "votes": null, "replies": [ { "id": 3385908, "author_name": "sudhanshu3112", "author_url": "", "post_date": "01/04/2026 08:17:27", "content": "Really appreciate the breakdown. Definitely a good reminder to focus more on feature space than just architecture.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3377227": "I really enjoyed participating in this competition. Two months ago, I was immediately attracted when I saw that there were no public deep learning public notebooks because this challenge is exactly what deep learning loves! I joined this competition with the simple goal of beating the best public XGBoost notebook with an NN.\n\n# XGBoost Public Notebook - Top 100!\nThank you Mahdi Ravaghi for the strong XGBoost baseline [here][1] (and thank you other Kagglers whose ideas are contained within). After changing that notebook to build one model per `lab_id` per action. And training on 100% train data. The notebook achieved `CV = 0.475, Public LB = 0.477, Private LB 0.450, (88th place)` top 100 rank, wow!\n\n# Local Validation Score\nIn this competition, we observe a near perfect CV LB relationship when using the correct validation score. (i.e. when XGB CV was 0.475 then XGB Public LB was 0.477). The correct CV is computed using only 15 `lab_id` which excludes labs `MABe22`, `CalMS21`, and `CRIM13` because the host said these labs did not exist in test data. So all CV scores discussed in this write up, use the correct local validation score. (UPDATE: relationship with Private LB isn't as good)\n\n# How To Build an NN\nThe power of NN is transfer learning from one lab to another! To make NN shine, we need to preprocess the data in some invariant way because the labs are different. Also we need to define the target labels correctly. Lastly we use data augmentation to unlock NN's full potential 🔥\n\nWe train our NN on all labs together. From seeing one lab, the NN will learn to make better predictions on another lab. To accomplish this, we need to make input features `lab_id` invariant. We can further encourage transfer learning by using data augmentation to minimize `lab_id` feature differences. (i.e. if mice are difference sizes between labs, then we data augment randomly scale sizes of mice up and down).\n\n# => Positive, Negative, and Masked Targets\nOur NN has 37 output neurons to predict the probability of each of the 37 possible actions. However not all videos nor labs contain all 37 actions as seen below:\n\n\n\nWhen training our NN, we need to define the targets correctly. For every video frame, the `target=1` if `train_annotation` says 1. Otherwise it is either negative or masked. We set `target=0` when `train.csv` says `behavior_labeled` but the target is missing in `train_annotation`. And we set `target=MASK` if an action is both missing from `train_annotation` and not included in the `behavior_labeled` list. \"Mask\" means that we do not include loss for this action in this video frame. (And t=0 and t=1 applies loss).\n\n# => Invariant Mouse Positions\nIf we input the mice body parts x,y as they are in the train_tracking files, the NN will get confused. Because the arena sizes and shapes are different for each lab. The position `x,y = 10,10` in one lab will not mean the same thing as position `x,y = 10,10` in another lab. Below we see the variation among 3 lab arenas by making a scatter plot of 1 mouse's nose over 5 videos. We see that the labs are different shapes and sizes:\n\n\n\nThere are at least 3 ways to make invariant mouse body part positions for our NN:\n* Agent centric coordinate system - Shift the agent mouse to the origin with midpoint between `ears` at `(0,0)` and rotate `tail_base` pointed down negative y axis. (All other mice in the video get shifted and rotated with the same transformation).\n* Min max scale (using 5% and 95% quantiles) all the `x,y` so that left side of arena is 0 and right side of arena is 1. And bottom is 0 and top is 1. (Also provide NN the `pixels_per_cm` feature).\n* Use distances instead of absolute `x,y`. Instead of saying nose is at `(10,10)`, we use the distance between `nose` and other body parts such as `tail_base`. We can compute distance as either a vector or a magnitude.\n\nThe above bullet points make positions invariant. Regarding velocity, we note that velocity vector and velocity magnitude are already invariant. And we note that all distances (as vector or magnitude) between agent mouse body part and target mouse body part are already invariant. \n\nBelow is an example of agent mouse nose position scatterplot after we transform with agent centric coordinate system. We observe in the scatter plot that all noses positions are now around `(0,2)` and when the mouse turns it head left it goes to `(-2,2)` and when it turns right it goes to `(2,2)`:\n\n\n\n# => Invariant Mouse Body Parts\nAnother challenge for transfer learning between labs is body parts. We observe that different labs track different body parts:\n\n\n\nWe observe that all labs have `[\"nose\", \"ear_left\", \"ear_right\", \"tail_base\"]`. Then about half of the labs use `[\"hip_left\", \"hip_right\"]` and the other half use `[\"lateral_left\", \"lateral_right\"]`. It turns out that `hip` and `lateral` are similar enough that we can rename them both `[\"side_left\", \"side_right\"]` to maximize transfer learning!\n\n# Build CNN Transformer\nWe train our NN using all labs together! Using the 8 body parts listed above and the method of positive, negative, masked loss. We build a CNN Transformer using invariant position, velocity, and distance features. The CNN layers come first and convolute in the time direction. They learn to make features based on how everything is changing. Next the NN feeds these features into layers of attention. Using cross (mouse) attention and self (mouse) attention, the transformer layers learn to compare agent and target mice and compare body parts within a single mouse. Finally the head layer outputs 37 outputs giving a probability for each of the 37 possible actions!\n\n\n\n# Different Time Scales\nWe train the same CNN Transformer multiple times and use a different length sliding window each time (i.e. vary from 2 seconds to 16 seconds). We also add more dilated convolution layers to the longer time windows to capture the larger time scale. Afterward we have four different trained NN and we can ensemble them:\n* window = 64 frames, stride = 16 (approx 2 second window), CNN layers = 6\n* window = 128 frames, stride = 32 (approx 4 second), CNN layers = 8\n* window = 256 frames, stride = 64 (approx 8 second), CNN layers = 10\n* window = 512 frames, stride = 128 (approx 16 second), CNN layers = 12\n\n# Data Augmentation\nUsing data augmentation gives a huge boost in performance. Data augmentation helps for two reasons. First is makes it harder for the NN to overfit (i.e. memorize instead of generalize the data) and we can train for more epochs. Second, it helps transfer learning between labs because all the labs start looking the same. For example if one lab had larger mice than another lab, after random scaling up and down, the model receives all sizes mice from every lab. We use\n* time augmentation - (randomly change 30 fps to 15 or 60)\n* rotation augmentation \n* x scale augmentation\n* y scale augmentation\n* horizontal flip\n* relabel right with left body parts\n* body part drop out\n\n# Inference\nWe predict using the same slide window scheme that we train with. Then each video frame predicts 37 probabilities. Next we smooth probs in time and filter actions that were not in `behaviors_labeled` (per video_id, agent_id, target_id triplet). Lastly we apply a dictionary of action thresholds per lab_id. Each action under threshold gets set to prob=0. Lastly we take `argmax` to pick the frame action. (UPDATE: I have an unselected submission which achieves +0.004 private LB (rank 4th or 5th) that uses better tie breaking logic to decide what happens when multiple actions exceed threshold).\n\n# Ensemble of NN - CV 0.546 - Public LB 0.551 - Private LB 0.518 (8th Place)\nWe take an equal weighted average of all our different time scale NN and achieve `CV = 0.546` and `Public LB = 0.551` and `Private LB = 0.518` (rank 8th). In this ensemble we include 3 families of NN. Each family uses a different set of invariant features and a range of time windows. In total our final ensemble use 2xNN from family one, 5xNN from family two, and 3xNN from family three (for a total of 10 NN).\n\n# Stack XGB over NN - CV 0.559 - Public LB 0.562 - Private LB 0.523 (7th Place)\nTo boost our NN score, we train an XGB using one OOF from each of the 3 families of NN and all the features from Mahdi Ravaghi public notebook [here][1]. We train one XGB per lab per action. This XGB's CV achieves `CV = 0.528` and `Public LB = 0.536` and `Private LB = 0.508` (rank 12th). And when we blend it with our Ensemble NN, the result is `CV = 0.559` and `Public LB = 0.562` and `Private LB = 0.523` (rank 7th), wow!\n\n[1]: https://www.kaggle.com/code/ravaghi/social-action-recognition-in-mice-xgboost", "3377230": "Great work!", "3377235": "can not agree more", "3377244": "Congratulations Chris! And thanks a lot for the writeup!\n\nYour NN-only score is super impressive! If you don't mind a few questions..\n\n1. How did you deal with the sparsity in the data? Did you interpolate the sensor keypoints or applied any other preproc?\n2. After making the input features lab invariant, did you train directly on all labs? Or did you first pre-train on all labs to later finetune on each individual lab?\n\nThanks!", "3377250": "Hi, congratulations winning 4th! I train a single NN on all 847 labeled videos. The secret is invariant features and data augmentations. \n\nMy NN can train on one lab and successfully infer a good CV score on different lab because all the labs look the same to my NN. Furthermore, I have one NN that only uses nose, ear_left, ear_right, and tail_base and scores close to gold by itself (using different time scales). \n\nSo already those 4 features exist for every video and perform well. Then i either add all the other body parts and mark them as missing if a lab doesn't have them. Or you can just call \"hips\" and \"lateral\" both \"sides\". Then every video has \"sides\". And with data augmentation all \"sides\" look alike too. So all these extra body parts improve upon only using nose, ear, tail_base.", "3377251": "Totally agree 👍 👏", "3377252": "Hey Chris, could you explain the \"After changing that notebook to build one model per lab_id per action. And training on 100% train data.\" to me? I personally did one model per (section, action) across all labs, and used group‑aware CV to prevent leakage between videos. Using groups_action = meta['lab_id'] + '_' + meta['video_id']. Are you saying for this example the model is a full train (1 fold) with one model per lab_id per action?", "3377256": "Congrats Chris! Solo gold and 7x GM, well deserved! Your NN performance seems great. It was especially challenging in this competition due to sparsity of some labels and differences between the labs.", "3377265": "Congratulations ！ Your solution is truly impressive，I learned a lot. \nif I might ask a few questions regarding the training setup：\n\n1. During training, for self actions, is the input still structured as an agent–target pair?\nIn that case, are the agent and target features identical？\n\n2. If there are 4 mice in a video, then within a single time window there are theoretically 4×4 possible agent–target combinations.\nDo you actually enumerate all of these pairs during training, or do you apply some form of filtering?\n\n3. For time windows with no annotated behavior, do you include them as a negative class during training, or is training restricted only to windows that contain at least one labeled behavior?\nThanks again for the excellent work!", "3377266": "Congrats! I knew NNs would be in the winning solutions but as I joined late I didn't have time to explore them\n\n>After changing that notebook to build one model per lab_id per action. And training on 100% train data. The notebook achieved CV = 0.475, Public LB = 0.477, Private LB 0.450, (88th place) top 100 rank, wow!\n\nI have 2 questions about this part:\n1. If I'm understanding you correctly, you first calculate CV score using KFold splits then you retrain on full data?\n2. What parts would you change in his notebook specifically? Could you share it if you have time? EDIT: I read your response to @cody11null's question and I understand now :D", "3377270": "Thanks.\n1. Yes, the model always receives agent and target features. For self actions, it receives duplicates of all these features (and the relationship differences are all zero).\n\n2. I only train on the pairs that are identified in the `train.csv` behaviors_labeled column. Whenever we see `mouseX,mouseY` in behaviors_labeled for a specific video, then we train for the full length of the video on that pair.\n\n3. We train with all windows (both with and without 1 label). However in family three of my NN, for each video I duplicate all the windows with at least 1 labeled behavior. (i.e. upsample positives by 2x)\n\n=======\n\nRegarding #3, i did try a trick from object detection but it didn't work better. I built two models. The first model is only trained on windows with at least 1 positive label. Then then second model predicts when a label exists or not. This trick works great for rare image object detection. \n\nSince model 1 always see positive targets, it learns very well to pick the right class. But of course it will always predict a class all the time. So the second model's purpose is to remove a prediction if the second model believes the window doesn't have an action. This has won object detection competitions, but i didn't get it to work here.", "3377273": "Hi. There is no need to use group aware CV (if your only goal is to submit to LB) because i didn't tune anything. I kept the XGB parameters as they were. Instead of training on 9 sections like the public notebook, I train on 20 sections. Each lab_id is its own section. And AdaptableSnail is 2 sections because I actually train on lab_id + body parts.\n\nFor for each lab_id, we train one XGB model for 275 iterations on 100% train data (with XGB parameters same as public notebook). That's it. Then during inference, we use these models to infer each test lab_id action pair and get public LB 0.477\n\nTo compute the CV score and OOF for ensemble, then I did use GroupKFold based on video_id. I created my own 8 folds and trained 8 fold models and inferred the 8 validation subsets. The CV score was 0.475.", "3377275": "Thanks @aerdem4 Congratulations winning 4th place Cash Gold! I remember how you stayed same or greater on private LB in CIBMTR comp. I am impressed once again how your team stays the same or improves on private LB! Great job!\n\nI actually removed one of my post process here because i was scared of shake down 😀 If I didn't remove that PP, I would finish with same private LB 0.527 like your team 😀 (It was PP to decide ties when model predicts 2+ actions per frame).", "3377276": "Thank you! That one is going to hurt for a bit. My plan since about month 1 of the competition was to make tabular ensemble, add tabm, and then move into CNN. I knew I had something missing with cv but was trying things far less straight forward thinking people had found some magic. We actually had a model with only .45 LB that we managed to boost with post process and some ensemble tweaks. Thanks for your walkthrough, the CNN dive is super insightful!", "3377277": "Hi @mahoganybuttstrings , to submit to LB we don't need to compute the CV. We just train the notebook as is with 100% train. To force the notebook to only put 1 lab in each section, we just do this. We add the lab as a \"body part\":\n\n LABS = list( train.lab_id.unique() )\n train[\"body_parts_tracked\"] = train[\"body_parts_tracked\"].str.replace(']','')\n train[\"body_parts_tracked\"] = train[\"body_parts_tracked\"] + ',\"' + train.lab_id + '\"]'\n\n test[\"body_parts_tracked\"] = test[\"body_parts_tracked\"].str.replace(']','')\n test[\"body_parts_tracked\"] = test[\"body_parts_tracked\"] + ',\"' + test.lab_id + '\"]'\n\nand then later we remove these labs from body parts with\n\n drop_body_parts += LABS\n\nThat's it. Afterward, the notebook will train 20 sections instead of 9 sections. But the notebook trains with 3 KFold, so each model is only trained with 66% data. We need to write our own KFold function to pass to function Trainer in library koolbox. So I wrote a KFold function that just yields 100% data for each of the 3 folds with different shuffled order each time.", "3377278": "Congrats Chris, your engineering on the target building and data transformation developments are really impressed. Also really thanks for the write-up I learn a new lesson for tabular competition.", "3377279": "Ah, ok, thanks for the explanation!", "3377285": "mahoganybuttstrings One other comment. We need to use the thresholds.pkl file that was generated by the original public notebook because when we train 100% the new thresholds.pkl will be meaningless.", "3377286": "cody11null There is one more important detail I forgot to say. We need to use the thresholds.pkl file that was generated by the original 3 KFold public notebook because when we train 100% the new thresholds.pkl will be meaningless.\n\nOr if you do separately compute an authentic CV OOF set, you can find better thresholds using that.", "3377342": "Congrats on the 7th place!\n\nYour approach to handling lab variations using invariant features (especially the agent-centric coordinate system) is brilliant. The architecture combining CNN-Transformer with a multi-scale ensemble strategy looks very robust. Also, stacking it with XGBoost clearly gave a great boost.\n\nThanks for the detailed write-up and sharing your insights!", "3377456": "Thank you Musa", "3377601": "arc144 One other thing I did was add positive weights to rare actions in loss calculation to boost them. And I used `37 multi-label loss` instead of `38 multi-class loss`. Even though this is a `38 multi-class` (i.e. only 1 class per frame which includes \"no action\"), there are so many \"no action\" frames that `multi-class` gets too many updates from the 38th \"no action\" class. \n\n def masked_bce_with_logits(logits, targets, mask, pos_weight=None):\n \"\"\"\n logits, targets, mask: [B,T,C]\n pos_weight: [C] or None (scales positive examples only)\n \"\"\"\n loss = F.binary_cross_entropy_with_logits(logits, targets, reduction='none') # [B,T,C]\n if pos_weight is not None:\n # scale positives by pos_weight (like BCEWithLogitsLoss(pos_weight))\n scale = 1 + (pos_weight.view(1,1,-1) - 1) * targets\n loss = loss * scale\n loss = loss * mask\n denom = mask.sum().clamp_min(1.0)\n return loss.sum() / denom", "3377633": "Congratulations @cdeotte, Thanks a lot for the detailed write-up!! There are a lot of things to learn from this write-up and the discussions.", "3377712": "Thank you", "3377783": "very good plots.", "3378128": "Is this single NN a CV-fold training or a single run? How did you prevent overfitting/know when to stop when you trained on the full 847labeled videos?", "3378144": "My submissions to LB train on 100% train data. And for each NN, I ensemble 6 of the same NN trained with different seeds. I compute the CV using 3 GroupKFold (by video_id). After finding the optimal number of epochs, I then train with 100% using 2/3 of this number epochs. Because the data is enlarging from 66% to 100%. (We could probably run more epochs since 100% data is like 66% data with additional data augmentations and data augmentation allows for more epochs but i play it safe and resultantly using 100% at 2/3 epochs versus 66% is always better).", "3378298": "Congrats! Bro for getting 7th place.", "3378431": "A lot to be learnt in this write up, thanks!", "3379307": "That makes total sense! There are many underrepresented classes in the dataset. Thanks for the explanation!\n\nBtw, did you get a chance to try FocalLoss vs. just adding the positive weights to BCE?", "3380248": "Thanks a lot for the details @cdeotte", "3380777": "That's awesome. Thank you for the great work!", "3381449": "congrats bro 7 place just amazing", "3383306": "Thanks Sheraz", "3383327": "Very very Congrats! for getting 7th place.✨🥳", "3385712": "Amazing write-up and solution, thanks for breaking everything down so clearly. How much of the final gain do you think came from invariance + augmentation vs. the NN architecture itself? @cdeotte", "3385731": "Thanks! The NN architecture itself was similar to XGB performance CV 0.475. Using invariance and augmentation was the key to boost (single model NN) performance to CV 0.520 ! Then using a blend of 10x NN each with different features and time scales boosted to CV 0.546! (And finally blending 10xNN with XGB achieved CV 0.559)", "3385908": "Really appreciate the breakdown. Definitely a good reminder to focus more on feature space than just architecture." }, "source": "meta" }