\nBackbone is EfficientNet with TSM[1], and head is simple 1D UNet.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1554318%2F0059bfdb6ffbaee44c3c3ac0a20e8030%2FUntitled%20Diagram.drawio.png?generation=1665851779731895&alt=media\")
\nBut there are many tricks in both training and inference.
Congrats to all the winners, especially for Team Hydrogen( @philippsinger, @ybabakhin, @ilu000) and @kmat2019, your scores are really impressive!
\nI’ll write up a summary of my solution.
\nIt ends up being a very simple end2end 2.5d model.
\nBackbone is EfficientNet with TSM[1], and head is simple 1D UNet.
\n
\nBut there are many tricks in both training and inference.
Even though I used ImageNet pretrained model for efficientnet, inserting TSM module changes model structure drastically, so it was hard to control training. (ex. easy to overfit or diverge.) And intuitively it seems important to teach a model to focus around the ball, so I used ball annotation in the SoccerNet tracking dataset[2] to pretrain the model.
\nPretrain set up is following:
\nAt first I was not sure whether we can use SoccerNet or not, so I decided to annotate balls for the video clips by myself, it ended up being more than 20,000 frames. But after confirming we can use SoccerNet, I switched to it. Pretrain performance is better than mine, but the final score is not much different.
\nPrediction is like this. I tried to use it as additional input to the next stage, but no successful result.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1554318%2F9c95367ddb5ea5437ec652ff2a9612d0%2Fepoch7_1606b0e6_0_5724_5786_ball.gif?generation=1665852280112233&alt=media\")
1d gaussian heat map +- 5 frames around the event. Here is an example for challenge at frame 13. (num_frames=16)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1554318%2Fb89a208458dc230637b4beaf60000c53%2FUntitled%20Diagram-Page-3.drawio.png?generation=1665852495588359&alt=media\")
\nAlso create a binary mask to represent whether the frame is in eval interval or not.
Half pos and half neg. Pos means at least 1 event in the label and neg means randomly sampled from eval interval.
\nBasic ones like RandomHorizontalFlip, RandomRotation, ColorJitter.
\nAlso manifold mixup[3] works well. I applied mixup on image-wise features.
Base loss is focal loss[6].
\nThe tasks for play, challenge and throwin are actually independent. But challenge and throwin are difficult to learn as there are much fewer samples than play. So I adjusted the alpha factor in focal loss based on the number of samples in each class. It’s inspired by class balanced loss[7], but the formulation is modified. This stabilizes training, especially before using the ball pretrained model.
def _calc_weight(self, samples_per_class, beta=0.9999):\n effective_num = 1.0 - beta**samples_per_class\n weights = (1.0 - beta) / effective_num\n weights = weights / torch.sum(weights)\n weights = weights.unsqueeze(0).float()\n return weights.cuda()\nAfter training with a relatively short clip like 32 frames, I fine tune the model with a long sequence like 128 frames, but with freezing backbone so that it fits in gpu memory.
\ncv2.VideoCapture + multi worker pytorch dataloader doesn't work well. Video decoding can be easily bottleneck, so I brought imutils[8] library’s threading implementation and modified it a bit.
\nAlso, I used multi image sizes [(540, 960), (480, 800), (360, 640)]. It’s faster to create small images by resizing (540, 960) image after transferring it to the GPU.
My model predicts 64 frames at the same time. But I realized that predictions around edges are relatively inaccurate, so I decided to drop them and take only center 48 frames out of 64 frames. Below diagram shows how it works.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1554318%2F11ee7e5b0bc00a2d178a0ed1f50af8a1%2FUntitled%20Diagram-Page-2.drawio.png?generation=1665852445476883&alt=media\")
Simply apply nms. Thresholds are [12, 6, 6] for [challenge, play, throwin] respectively.
\nTo maximize the effect of the ensemble, I prepared various models with different folds.
\nSingle best model scores 0.768.
\n((360,640), efnet-b2, seq_len=128, all data)
\nEnsemble 6 models with various combination reaches 0.827.
Too many to list up.
\n[1] https://github.com/MIT-HAN-LAB/temporal-shift-module
\n[2] https://github.com/SoccerNet/sn-tracking
\n[3] https://arxiv.org/abs/1806.05236
\n[4] https://github.com/rwightman/pytorch-image-models/blob/a2727c1bf78ba0d7b5727f5f95e37fb7f8866b1f/timm/utils/model_ema.py
\n[5] https://github.com/cybertronai/gradient-checkpointing
\n[6] https://arxiv.org/abs/1708.02002
\n[7] https://arxiv.org/abs/1901.05555
\n[8] https://github.com/PyImageSearch/imutils
Congrats on your great solution @bamps53 and thank you very much for your write-up! I learned some new stuff like TSM and using 1D UNet for temporal information.
\nI wanted to ask you about your label design. In one of our first trials, we tried the same approach based on the competition metric tolerance intervals. This also allowed us to have multi-label predictions for each window. However, due to the great number of zeros in the target tensor, many of the models converged to predict all zeros. Did you see anything like this in your training? Did the class weights solve it? We used sklearn.compute_class_weight with no success.
\nFinally, you mention that your model is 2.5D. How many frames do you stack for each frame?
", "votes": 2, "replies": [ { "id": 1995881, "author_name": "Camaro", "author_url": "", "post_date": "2022-10-19T22:13:34.710000", "content": "I experienced similar things like training diverges or never learn anything especially when I haven't pretrained the weights. I used focal loss as it's designed for imblanced problem so it might helped, but other winner's still use BCE so it may not be necessary.
\nI stacked 32, 64 or 128 frames depending on the model configuration .
", "votes": 1, "replies": [] } ] }, { "id": 2002815, "author_name": "arutema47", "author_url": "", "post_date": "2022-10-25T05:20:44.947000", "content": "I'm quite confused on the use of TSM, it would be great if you can give some comments.
\n\n\nEfnet + TSM (+ shallow conv2d head)
\n
Do you insert the TSM at the Efnet's layer before GAP?
\nSo its like Efnet -> TSM -> conv2d -> GAP?
@kyoshioka47
\nNo, I inserted TSM every after blocks.
Like this;
\nmodel_name = 'tf_efficientnet_b2_ns'\nnum_frames = 32\nn_div = 8\n\nmodel = timm.create_model(model_name)\nfor i, block in enumerate(model.blocks):\n model.blocks[i] = TemporalShift(block, n_segment=num_frames, n_div=n_div)\nThank you for the sample codes!
\nThis is really helpful, will try to replicate your design :)
Good work @bamps53 and thank you so much for your great share.
", "votes": 1, "replies": [] }, { "id": 1991142, "author_name": "makiji", "author_url": "", "post_date": "2022-10-17T02:36:36", "content": "What didn’t work
\nToo many to list up.
What didn’t work
\nToo many to list up.
Crop around ball area
\nCrop out no player area
This is brilliant! And a big thanks for mentioning the references. I don't see a lot of authors posting the references which makes it difficult to understand the entire essence of the solution. Great stuff. @bamps53
", "votes": 0, "replies": [] }, { "id": 2004370, "author_name": "Y-Haneji", "author_url": "", "post_date": "2022-10-26T08:13:55.400000", "content": "@bamps53
\nThanks for sharing great solution!
\nI'm interested in your label design. How did you compute the soft label? (I understand the basics of gaussian distribution.)
It's not easy to understand code, but I created the label like this;
\ndef gaussian_kernel(length, sigma=3):\n x = np.ogrid[-length:length+1]\n h = np.exp(-(x**2) / (2 * sigma * sigma))\n h[h < np.finfo(h.dtype).eps * h.max()] = 0\n return h\n\noffset = 5\nsigma = 3\n# labels has 1 in exact event frame and 0 otherwise. shape is (frame_len, 3).\nfor c, event in enumerate(EVENT_CLASSES):\n labels[:, c] = np.convolve(labels[:, c], gaussian_kernel(offset, sigma), mode='same')\nThank you for sharing great solution @bamps53 !
\nI'd like to ask you one thing. What 2.5D model implementation would be like? If you don't mind, would you give me some hints or references to input multiple images into CNN backbone mentioned above?
Great! Thanks for sharing
", "votes": 1, "replies": [] }, { "id": 2207795, "author_name": "Maruf Hossain", "author_url": "", "post_date": "2023-04-03T16:26:36.517000", "content": "Thanks for sharing, its great!
", "votes": 0, "replies": [] }, { "id": 2002974, "author_name": "tlx123", "author_url": "", "post_date": "2022-10-25T07:41:42.837000", "content": "Thank you For Sharing!!
", "votes": 0, "replies": [] }, { "id": 1996901, "author_name": "Preferrrrr", "author_url": "", "post_date": "2022-10-20T13:40:13.777000", "content": "Thank you For Sharing!!
", "votes": 0, "replies": [] }, { "id": 1995891, "author_name": "Max Wei", "author_url": "", "post_date": "2022-10-19T22:28:11.807000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1993321, "author_name": "nhuerp00", "author_url": "", "post_date": "2022-10-18T09:31:23.567000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1990583, "author_name": "Teja Kumar behves", "author_url": "", "post_date": "2022-10-16T16:40:22.057000", "content": "Thanks for sharing dude!
", "votes": 0, "replies": [] }, { "id": 1990242, "author_name": "Harshit N Panchal", "author_url": "", "post_date": "2022-10-16T12:15:47.950000", "content": "Thanks for sharing dude!
", "votes": 0, "replies": [] } ], "comment_count": 22, "messages": [ { "id": 1988998, "postDate": "2022-10-15T17:04:33.433Z", "content": "Congrats to all the winners, especially for Team Hydrogen( @philippsinger, @ybabakhin, @ilu000) and @kmat2019, your scores are really impressive!
\nI’ll write up a summary of my solution.
\nIt ends up being a very simple end2end 2.5d model.
\nBackbone is EfficientNet with TSM[1], and head is simple 1D UNet.
\n
\nBut there are many tricks in both training and inference.
Even though I used ImageNet pretrained model for efficientnet, inserting TSM module changes model structure drastically, so it was hard to control training. (ex. easy to overfit or diverge.) And intuitively it seems important to teach a model to focus around the ball, so I used ball annotation in the SoccerNet tracking dataset[2] to pretrain the model.
\nPretrain set up is following:
\nAt first I was not sure whether we can use SoccerNet or not, so I decided to annotate balls for the video clips by myself, it ended up being more than 20,000 frames. But after confirming we can use SoccerNet, I switched to it. Pretrain performance is better than mine, but the final score is not much different.
\nPrediction is like this. I tried to use it as additional input to the next stage, but no successful result.
\n
1d gaussian heat map +- 5 frames around the event. Here is an example for challenge at frame 13. (num_frames=16)
\n
\nAlso create a binary mask to represent whether the frame is in eval interval or not.
Half pos and half neg. Pos means at least 1 event in the label and neg means randomly sampled from eval interval.
\nBasic ones like RandomHorizontalFlip, RandomRotation, ColorJitter.
\nAlso manifold mixup[3] works well. I applied mixup on image-wise features.
Base loss is focal loss[6].
\nThe tasks for play, challenge and throwin are actually independent. But challenge and throwin are difficult to learn as there are much fewer samples than play. So I adjusted the alpha factor in focal loss based on the number of samples in each class. It’s inspired by class balanced loss[7], but the formulation is modified. This stabilizes training, especially before using the ball pretrained model.
def _calc_weight(self, samples_per_class, beta=0.9999):\n effective_num = 1.0 - beta**samples_per_class\n weights = (1.0 - beta) / effective_num\n weights = weights / torch.sum(weights)\n weights = weights.unsqueeze(0).float()\n return weights.cuda()\nAfter training with a relatively short clip like 32 frames, I fine tune the model with a long sequence like 128 frames, but with freezing backbone so that it fits in gpu memory.
\ncv2.VideoCapture + multi worker pytorch dataloader doesn't work well. Video decoding can be easily bottleneck, so I brought imutils[8] library’s threading implementation and modified it a bit.
\nAlso, I used multi image sizes [(540, 960), (480, 800), (360, 640)]. It’s faster to create small images by resizing (540, 960) image after transferring it to the GPU.
My model predicts 64 frames at the same time. But I realized that predictions around edges are relatively inaccurate, so I decided to drop them and take only center 48 frames out of 64 frames. Below diagram shows how it works.
\n
Simply apply nms. Thresholds are [12, 6, 6] for [challenge, play, throwin] respectively.
\nTo maximize the effect of the ensemble, I prepared various models with different folds.
\nSingle best model scores 0.768.
\n((360,640), efnet-b2, seq_len=128, all data)
\nEnsemble 6 models with various combination reaches 0.827.
Too many to list up.
\n[1] https://github.com/MIT-HAN-LAB/temporal-shift-module
\n[2] https://github.com/SoccerNet/sn-tracking
\n[3] https://arxiv.org/abs/1806.05236
\n[4] https://github.com/rwightman/pytorch-image-models/blob/a2727c1bf78ba0d7b5727f5f95e37fb7f8866b1f/timm/utils/model_ema.py
\n[5] https://github.com/cybertronai/gradient-checkpointing
\n[6] https://arxiv.org/abs/1708.02002
\n[7] https://arxiv.org/abs/1901.05555
\n[8] https://github.com/PyImageSearch/imutils
Congrats on your great solution @bamps53 and thank you very much for your write-up! I learned some new stuff like TSM and using 1D UNet for temporal information.
\nI wanted to ask you about your label design. In one of our first trials, we tried the same approach based on the competition metric tolerance intervals. This also allowed us to have multi-label predictions for each window. However, due to the great number of zeros in the target tensor, many of the models converged to predict all zeros. Did you see anything like this in your training? Did the class weights solve it? We used sklearn.compute_class_weight with no success.
\nFinally, you mention that your model is 2.5D. How many frames do you stack for each frame?
", "rawMarkdown": "Congrats on your great solution @bamps53 and thank you very much for your write-up! I learned some new stuff like TSM and using 1D UNet for temporal information.\n\nI wanted to ask you about your label design. In one of our first trials, we tried the same approach based on the competition metric tolerance intervals. This also allowed us to have multi-label predictions for each window. However, due to the great number of zeros in the target tensor, many of the models converged to predict all zeros. Did you see anything like this in your training? Did the class weights solve it? We used [sklearn.compute_class_weight](https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html) with no success.\n\nFinally, you mention that your model is 2.5D. How many frames do you stack for each frame?", "votes": 2, "replies": [ { "id": 1995881, "postDate": "2022-10-19T22:13:34.710Z", "content": "I experienced similar things like training diverges or never learn anything especially when I haven't pretrained the weights. I used focal loss as it's designed for imblanced problem so it might helped, but other winner's still use BCE so it may not be necessary.
\nI stacked 32, 64 or 128 frames depending on the model configuration .
", "rawMarkdown": "I experienced similar things like training diverges or never learn anything especially when I haven't pretrained the weights. I used focal loss as it's designed for imblanced problem so it might helped, but other winner's still use BCE so it may not be necessary.\n\nI stacked 32, 64 or 128 frames depending on the model configuration .", "votes": 1 } ] }, { "id": 2002815, "postDate": "2022-10-25T05:20:44.947Z", "content": "I'm quite confused on the use of TSM, it would be great if you can give some comments.
\n\n\nEfnet + TSM (+ shallow conv2d head)
\n
Do you insert the TSM at the Efnet's layer before GAP?
\nSo its like Efnet -> TSM -> conv2d -> GAP?
@kyoshioka47
\nNo, I inserted TSM every after blocks.
Like this;
\nmodel_name = 'tf_efficientnet_b2_ns'\nnum_frames = 32\nn_div = 8\n\nmodel = timm.create_model(model_name)\nfor i, block in enumerate(model.blocks):\n model.blocks[i] = TemporalShift(block, n_segment=num_frames, n_div=n_div)\nThank you for the sample codes!
\nThis is really helpful, will try to replicate your design :)
Good work @bamps53 and thank you so much for your great share.
", "rawMarkdown": "Good work @bamps53 and thank you so much for your great share.", "votes": 1 }, { "id": 1991142, "postDate": "2022-10-17T02:36:36Z", "content": "What didn’t work
\nToo many to list up.
What didn’t work
\nToo many to list up.
Crop around ball area
\nCrop out no player area
This is brilliant! And a big thanks for mentioning the references. I don't see a lot of authors posting the references which makes it difficult to understand the entire essence of the solution. Great stuff. @bamps53
", "rawMarkdown": "This is brilliant! And a big thanks for mentioning the references. I don't see a lot of authors posting the references which makes it difficult to understand the entire essence of the solution. Great stuff. @bamps53 " }, { "id": 2004370, "postDate": "2022-10-26T08:13:55.400Z", "content": "@bamps53
\nThanks for sharing great solution!
\nI'm interested in your label design. How did you compute the soft label? (I understand the basics of gaussian distribution.)
It's not easy to understand code, but I created the label like this;
\ndef gaussian_kernel(length, sigma=3):\n x = np.ogrid[-length:length+1]\n h = np.exp(-(x**2) / (2 * sigma * sigma))\n h[h < np.finfo(h.dtype).eps * h.max()] = 0\n return h\n\noffset = 5\nsigma = 3\n# labels has 1 in exact event frame and 0 otherwise. shape is (frame_len, 3).\nfor c, event in enumerate(EVENT_CLASSES):\n labels[:, c] = np.convolve(labels[:, c], gaussian_kernel(offset, sigma), mode='same')\nThank you for sharing great solution @bamps53 !
\nI'd like to ask you one thing. What 2.5D model implementation would be like? If you don't mind, would you give me some hints or references to input multiple images into CNN backbone mentioned above?
Great! Thanks for sharing
", "rawMarkdown": "Great! Thanks for sharing", "votes": 1 }, { "id": 2207795, "postDate": "2023-04-03T16:26:36.517Z", "content": "Thanks for sharing, its great!
", "rawMarkdown": "Thanks for sharing, its great!" }, { "id": 2002974, "postDate": "2022-10-25T07:41:42.837Z", "content": "Thank you For Sharing!!
", "rawMarkdown": "Thank you For Sharing!!" }, { "id": 1996901, "postDate": "2022-10-20T13:40:13.777Z", "content": "Thank you For Sharing!!
", "rawMarkdown": "Thank you For Sharing!!" }, { "id": 1995891, "postDate": "2022-10-19T22:28:11.807Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1993321, "postDate": "2022-10-18T09:31:23.567Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1990583, "postDate": "2022-10-16T16:40:22.057Z", "content": "Thanks for sharing dude!
", "rawMarkdown": "Thanks for sharing dude!\n" }, { "id": 1990242, "postDate": "2022-10-16T12:15:47.950Z", "content": "Thanks for sharing dude!
", "rawMarkdown": "Thanks for sharing dude!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T08:54:03Z", "competition_id": 37244, "competition_slug": "dfl-bundesliga-data-shootout", "team_id": 8957247, "solution_url": "/competitions/dfl-bundesliga-data-shootout/writeups/camaro-public-3rd-place-solution" }