{ "id": 359855, "title": "Our (frustrated) approach", "content": "

Although it is the norm to share successful solutions, our team thought that our (frustrated) approach could be interesting and thus should be shared.

\n

From the beginning, we thought there was a lot of useless noise in the videos that may prevent the model from learning the right features. Based on this, we put a great effort into the preprocessing side, trying to extract only what we considered essential information. The result is the following:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
originalpreprocessed
\"original\"\"preprocessed\"
\n

In summary, as can be seen in the video, we did the following:

\n\n

It was a great challenge to put all these pieces together. Moreover, it was even harder to make the pipeline efficient enough to comply with the time constraints. Nonetheless, after a great effort by the team to optimize the source code of the libraries we used (specially mmpose), we were able to make the preprocessing pipeline run in half of the video duration. This took a great amount of time, but we were very happy with the result. At least until we started training and validating the model… 😅

\n

The odyssey began when, no matter which architecture we tried, we could not get reasonable results. General action recognition models (SwinTransformer3D, SlowFast, …), soccer-specific models (NetVLAD++, …), custom architectures based on temporal information, … We tried them all, but none of them worked. Even with the original videos (without preprocessing), we could not get a good result. Even after augmenting the training data with the annotated videos of SoccerNet-v2, the boost was not enough.

\n

We have checked and rechecked for bugs, but we could not find any issues. We are very curious to see what top teams tried and check if we missed something critical.

\n

Although the outcome was not the one we expected (we are hungry for gold to become Kaggle Masters), it was a great learning experience for the team. Thanks @alejandrobravoserna @miguelgonzalez2 @danielguzmanolivares @alvarozaera for the effort! 🥇will come …

\n

If you have any doubts about our approach, do not hesitate to ask!

", "comments": [ { "id": 1987066, "author_name": "moth", "author_url": "", "post_date": "2022-10-14T13:38:16.043000", "content": "

Wow, I tried a very similar thing and also didnt work! @atmguille

\n

I went a little more extreme and just surrounded the players and the ball with their bounding boxes from yolov7 and the rest is just black. I did this hoping that only the essential information should be kept.\"\"

", "votes": 2, "replies": [ { "id": 1988369, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-15T09:26:28.427000", "content": "

Interesting approach @alejopaullier, maybe a bit extreme as you said. Did you try coloring the ball with a different color?

", "votes": 2, "replies": [] }, { "id": 1990823, "author_name": "moth", "author_url": "", "post_date": "2022-10-16T18:47:32.400000", "content": "

Yes, I guess this image I shared wasnt the best example since the ball is covered by a player. The ball was colored differently though.

", "votes": 1, "replies": [] } ] }, { "id": 1986848, "author_name": "delai50", "author_url": "", "post_date": "2022-10-14T10:44:07.990000", "content": "

Thanks for the summary! Happy to see more kagglers from Madrid over here 😃

", "votes": 2, "replies": [ { "id": 1986912, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-14T11:48:32.310000", "content": "

Thanks for your comment @delai50! Are you going to Barcelona for the Kaggle Days by any chance? It would be great to meet you there!

", "votes": 1, "replies": [] }, { "id": 1987058, "author_name": "delai50", "author_url": "", "post_date": "2022-10-14T13:28:42.687000", "content": "

I couldn't classify to participate in the competitiom but I am thinking of going as a volunteer, do you participate? Where I will go 100% sure is to Kaggle Days Paris!

", "votes": 1, "replies": [] }, { "id": 1987085, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-14T13:45:26.043000", "content": "

@delai50 I am going as a volunteer! If you happen to be there, let me know!

", "votes": 0, "replies": [] } ] }, { "id": 1986549, "author_name": "Zhu.YX7", "author_url": "", "post_date": "2022-10-14T06:48:45.393000", "content": "

It seems like your preprocessing quality is veeeery good. It's sad that such good work didn't turn into good results….. Our team gave up on preprocessing after several attempts, and adopted end-to-end model training at last

", "votes": 2, "replies": [ { "id": 1986925, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-14T11:58:22.663000", "content": "

Thank you for your supportive comment @zhuyx7! It is interesting that preprocessing didn't work either for your team. We bet too much on it. I am very curious to see if preprocessing helped any of the top teams.

", "votes": 1, "replies": [] } ] }, { "id": 1986330, "author_name": "Leon", "author_url": "", "post_date": "2022-10-14T02:50:57.850000", "content": "

Our team also failed on preprocessing and could not detect the ball perfectly. Therefore, we only use raw RGB input and 2.5D resnet to achieve 0.7+.

", "votes": 2, "replies": [ { "id": 1987091, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-14T13:52:17.957000", "content": "

We also had 2.5D approach in mind, given the good results that we achieved in UW-Madison with it. However, I thought that more temporal context was required here and didn't give it a try. In fact, given your results, it seems like the opposite: we were adding too much. Once again: always try the simplest baseline first 😅

\n

Thanks for providing info about your approach. Good luck with the final evaluation in December!

", "votes": 1, "replies": [] } ] }, { "id": 1986689, "author_name": "Miguel González González", "author_url": "", "post_date": "2022-10-14T08:11:36.320000", "content": "

Thanks for the summary post about our team work! We learned a lot indeed

", "votes": 1, "replies": [ { "id": 1986910, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-14T11:46:04.493000", "content": "

It's always a pleasure to work together and learn from you. More to come!

", "votes": 1, "replies": [] } ] }, { "id": 1986383, "author_name": "uzo dike", "author_url": "", "post_date": "2022-10-14T04:42:38.153000", "content": "

did you try optical flow on ball motion detection. i think a pass to the opponent is still a pass so not much work to be done trying to determine who it was passed to. i am also having issues with throwing detection. if the model could be trained knowing what action and at what time event occurred it will do a fairly good job. Congrats on the encouraging post.

", "votes": 1, "replies": [ { "id": 1987080, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-14T13:43:25.443000", "content": "

We discarded optical flow based on its excessive time requirements. Nonetheless, it seems that @kmat2019 used it as an input to detect the ball (https://twitter.com/Kmat67916008/status/1580767429558693888).

\n

In terms of identifying opponents, you are right that it may not be useful for the pass or the throwins, but we thought that it could play an important role in the challenges.

\n

We trained the model with short clips, trying to classify the action and doing regression on the frame where the action happens. This and other similar approaches did not work for us. It seems that top teams input far less information than us (we used a window of 1.6 seconds), and maybe the key was there. Thank you for the comment @uzodike!

", "votes": 2, "replies": [] } ] }, { "id": 1992825, "author_name": "Muhammad Ammar Jamshed", "author_url": "", "post_date": "2022-10-18T00:57:27.140000", "content": "

I really like the methodology you used even though I don't use Computer vision that much other than CNN-based algorithms for image analysis! Tell me though how did you manage to fit this size of data of videos in your Kaggle notebook server !!🤒

", "votes": 0, "replies": [ { "id": 1993571, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-18T13:40:30.653000", "content": "

By decreasing batch size and frame resolution, we had no problems with memory

", "votes": 1, "replies": [] } ] }, { "id": 1991643, "author_name": "Sankalp Kumar Singh", "author_url": "", "post_date": "2022-10-17T10:02:35.710000", "content": "

This is cool

", "votes": 0, "replies": [] }, { "id": 1991006, "author_name": "Gustavo Mourao", "author_url": "", "post_date": "2022-10-16T22:06:26.673000", "content": "

Thank you for the post, Guillermo!

\n

Have you thought in apply Kalman filter in order to reduce some noise after tracking?

", "votes": 0, "replies": [ { "id": 1993579, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-18T13:42:42.440000", "content": "

Thank you for your suggestion. I am not very familiar with the Kalman filter. How would you use it here?

", "votes": 1, "replies": [] } ] }, { "id": 1989761, "author_name": "SaiKumar Tamminana", "author_url": "", "post_date": "2022-10-16T06:55:48.393000", "content": "

Interesting Bro. Can you get the actual body measurements by taking the photo of the person??
\nIf Yes How Can You achieve it? @atmguille

", "votes": 0, "replies": [ { "id": 1993549, "author_name": "Guillermo García Cobo", "author_url": "", "post_date": "2022-10-18T13:09:54.470000", "content": "

Not sure if I understand your question. Could you rephrase it @saikumartamminana?

", "votes": 1, "replies": [] } ] }, { "id": 1989609, "author_name": "Varun Sanai", "author_url": "", "post_date": "2022-10-16T05:18:35.823000", "content": "

osm your work

", "votes": 0, "replies": [] }, { "id": 1989606, "author_name": "Shreya Mishra 0307", "author_url": "", "post_date": "2022-10-16T05:15:57.720000", "content": "

Thanks for sharing extremely useful!!

", "votes": 1, "replies": [] }, { "id": 2019132, "author_name": "Priyanshu S. Prajapati", "author_url": "", "post_date": "2022-11-06T11:15:06.593000", "content": "

Thanks for @atmguille , really helpful.

", "votes": 0, "replies": [] } ], "comment_count": 25, "messages": [ { "id": 1986255, "postDate": "2022-10-14T00:33:08.877Z", "content": "

Although it is the norm to share successful solutions, our team thought that our (frustrated) approach could be interesting and thus should be shared.

\n

From the beginning, we thought there was a lot of useless noise in the videos that may prevent the model from learning the right features. Based on this, we put a great effort into the preprocessing side, trying to extract only what we considered essential information. The result is the following:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
originalpreprocessed
\"original\"\"preprocessed\"
\n

In summary, as can be seen in the video, we did the following:

\n\n

It was a great challenge to put all these pieces together. Moreover, it was even harder to make the pipeline efficient enough to comply with the time constraints. Nonetheless, after a great effort by the team to optimize the source code of the libraries we used (specially mmpose), we were able to make the preprocessing pipeline run in half of the video duration. This took a great amount of time, but we were very happy with the result. At least until we started training and validating the model… 😅

\n

The odyssey began when, no matter which architecture we tried, we could not get reasonable results. General action recognition models (SwinTransformer3D, SlowFast, …), soccer-specific models (NetVLAD++, …), custom architectures based on temporal information, … We tried them all, but none of them worked. Even with the original videos (without preprocessing), we could not get a good result. Even after augmenting the training data with the annotated videos of SoccerNet-v2, the boost was not enough.

\n

We have checked and rechecked for bugs, but we could not find any issues. We are very curious to see what top teams tried and check if we missed something critical.

\n

Although the outcome was not the one we expected (we are hungry for gold to become Kaggle Masters), it was a great learning experience for the team. Thanks @alejandrobravoserna @miguelgonzalez2 @danielguzmanolivares @alvarozaera for the effort! 🥇will come …

\n

If you have any doubts about our approach, do not hesitate to ask!

", "rawMarkdown": "Although it is the norm to share successful solutions, our team thought that our (frustrated) approach could be interesting and thus should be shared.\n\nFrom the beginning, we thought there was a lot of useless noise in the videos that may prevent the model from learning the right features. Based on this, we put a great effort into the preprocessing side, trying to extract only what we considered essential information. The result is the following:\n\n| original | preprocessed |\n| :---: | :---: |\n| [![original](https://i.ibb.co/3kC6nsc/embed.png)](https://www.youtube.com/watch?v=GlUgM2TUPCw) | [![preprocessed](https://i.ibb.co/kgpLVp2/embed-prerpocessed.png)](https://www.youtube.com/watch?v=VoAcptNeF1k) |\n\nIn summary, as can be seen in the video, we did the following:\n\n* Detect the lines of the football field, via a Top-hat filter.\n* Detect the players and the ball, via a fine-tuned YOLOv7 first and a custom-trained RetinaNet after doubts about the GPL license arouse.\n* Detect each player's team using K-means with the color of the player's jerseys.\n* Detect the skeletons of the five players closer to the ball, via mmpose.\n* We rendered all this information together over a black background, hoping to aid the model in the learning process. We even cropped each frame centered in the ball to further reduce the noise (the result is available in [this video](https://www.youtube.com/watch?v=IZjJMa4u-Ss)).\n\nIt was a great challenge to put all these pieces together. Moreover, it was even harder to make the pipeline efficient enough to comply with the time constraints. Nonetheless, after a great effort by the team to optimize the source code of the libraries we used (specially mmpose), we were able to make the preprocessing pipeline run in half of the video duration. This took a great amount of time, but we were very happy with the result. At least until we started training and validating the model... 😅\n\nThe odyssey began when, no matter which architecture we tried, we could not get reasonable results. General action recognition models (SwinTransformer3D, SlowFast, ...), soccer-specific models (NetVLAD++, ...), custom architectures based on temporal information, ... We tried them all, but none of them worked. Even with the original videos (without preprocessing), we could not get a good result. Even after augmenting the training data with the annotated videos of SoccerNet-v2, the boost was not enough.\n\nWe have checked and rechecked for bugs, but we could not find any issues. We are very curious to see what top teams tried and check if we missed something critical.\n\nAlthough the outcome was not the one we expected (we are hungry for gold to become Kaggle Masters), it was a great learning experience for the team. Thanks @alejandrobravoserna @miguelgonzalez2 @danielguzmanolivares @alvarozaera for the effort! 🥇will come ...\n\nIf you have any doubts about our approach, do not hesitate to ask!", "votes": 42 }, { "id": 1987066, "postDate": "2022-10-14T13:38:16.043Z", "content": "

Wow, I tried a very similar thing and also didnt work! @atmguille

\n

I went a little more extreme and just surrounded the players and the ball with their bounding boxes from yolov7 and the rest is just black. I did this hoping that only the essential information should be kept.\"\"

", "rawMarkdown": "Wow, I tried a very similar thing and also didnt work! @atmguille \n\nI went a little more extreme and just surrounded the players and the ball with their bounding boxes from yolov7 and the rest is just black. I did this hoping that only the essential information should be kept.![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3197853%2F0fa9a3da2b53050c12066e91a2a1f98b%2Fextreme.png?generation=1665754631646220&alt=media)", "votes": 2, "replies": [ { "id": 1988369, "postDate": "2022-10-15T09:26:28.427Z", "content": "

Interesting approach @alejopaullier, maybe a bit extreme as you said. Did you try coloring the ball with a different color?

", "rawMarkdown": "Interesting approach @alejopaullier, maybe a bit extreme as you said. Did you try coloring the ball with a different color?", "votes": 2 }, { "id": 1990823, "postDate": "2022-10-16T18:47:32.400Z", "content": "

Yes, I guess this image I shared wasnt the best example since the ball is covered by a player. The ball was colored differently though.

", "rawMarkdown": "Yes, I guess this image I shared wasnt the best example since the ball is covered by a player. The ball was colored differently though.", "votes": 1 } ] }, { "id": 1986848, "postDate": "2022-10-14T10:44:07.990Z", "content": "

Thanks for the summary! Happy to see more kagglers from Madrid over here 😃

", "rawMarkdown": "Thanks for the summary! Happy to see more kagglers from Madrid over here 😃", "votes": 2, "replies": [ { "id": 1986912, "postDate": "2022-10-14T11:48:32.310Z", "content": "

Thanks for your comment @delai50! Are you going to Barcelona for the Kaggle Days by any chance? It would be great to meet you there!

", "rawMarkdown": "Thanks for your comment @delai50! Are you going to Barcelona for the Kaggle Days by any chance? It would be great to meet you there!", "votes": 1 }, { "id": 1987058, "postDate": "2022-10-14T13:28:42.687Z", "content": "

I couldn't classify to participate in the competitiom but I am thinking of going as a volunteer, do you participate? Where I will go 100% sure is to Kaggle Days Paris!

", "rawMarkdown": "I couldn't classify to participate in the competitiom but I am thinking of going as a volunteer, do you participate? Where I will go 100% sure is to Kaggle Days Paris!", "votes": 1 }, { "id": 1987085, "postDate": "2022-10-14T13:45:26.043Z", "content": "

@delai50 I am going as a volunteer! If you happen to be there, let me know!

", "rawMarkdown": "@delai50 I am going as a volunteer! If you happen to be there, let me know!" } ] }, { "id": 1986549, "postDate": "2022-10-14T06:48:45.393Z", "content": "

It seems like your preprocessing quality is veeeery good. It's sad that such good work didn't turn into good results….. Our team gave up on preprocessing after several attempts, and adopted end-to-end model training at last

", "rawMarkdown": "It seems like your preprocessing quality is veeeery good. It's sad that such good work didn't turn into good results..... Our team gave up on preprocessing after several attempts, and adopted end-to-end model training at last", "votes": 2, "replies": [ { "id": 1986925, "postDate": "2022-10-14T11:58:22.663Z", "content": "

Thank you for your supportive comment @zhuyx7! It is interesting that preprocessing didn't work either for your team. We bet too much on it. I am very curious to see if preprocessing helped any of the top teams.

", "rawMarkdown": "Thank you for your supportive comment @zhuyx7! It is interesting that preprocessing didn't work either for your team. We bet too much on it. I am very curious to see if preprocessing helped any of the top teams.", "votes": 1 } ] }, { "id": 1986330, "postDate": "2022-10-14T02:50:57.850Z", "content": "

Our team also failed on preprocessing and could not detect the ball perfectly. Therefore, we only use raw RGB input and 2.5D resnet to achieve 0.7+.

", "rawMarkdown": "Our team also failed on preprocessing and could not detect the ball perfectly. Therefore, we only use raw RGB input and 2.5D resnet to achieve 0.7+.", "votes": 2, "replies": [ { "id": 1987091, "postDate": "2022-10-14T13:52:17.957Z", "content": "

We also had 2.5D approach in mind, given the good results that we achieved in UW-Madison with it. However, I thought that more temporal context was required here and didn't give it a try. In fact, given your results, it seems like the opposite: we were adding too much. Once again: always try the simplest baseline first 😅

\n

Thanks for providing info about your approach. Good luck with the final evaluation in December!

", "rawMarkdown": "We also had 2.5D approach in mind, given the good results that we achieved in [UW-Madison](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation) with it. However, I thought that more temporal context was required here and didn't give it a try. In fact, given your results, it seems like the opposite: we were adding too much. Once again: always try the simplest baseline first 😅\n\nThanks for providing info about your approach. Good luck with the final evaluation in December!", "votes": 1 } ] }, { "id": 1986689, "postDate": "2022-10-14T08:11:36.320Z", "content": "

Thanks for the summary post about our team work! We learned a lot indeed

", "rawMarkdown": "Thanks for the summary post about our team work! We learned a lot indeed", "votes": 1, "replies": [ { "id": 1986910, "postDate": "2022-10-14T11:46:04.493Z", "content": "

It's always a pleasure to work together and learn from you. More to come!

", "rawMarkdown": "It's always a pleasure to work together and learn from you. More to come!", "votes": 1 } ] }, { "id": 1986383, "postDate": "2022-10-14T04:42:38.153Z", "content": "

did you try optical flow on ball motion detection. i think a pass to the opponent is still a pass so not much work to be done trying to determine who it was passed to. i am also having issues with throwing detection. if the model could be trained knowing what action and at what time event occurred it will do a fairly good job. Congrats on the encouraging post.

", "rawMarkdown": "did you try optical flow on ball motion detection. i think a pass to the opponent is still a pass so not much work to be done trying to determine who it was passed to. i am also having issues with throwing detection. if the model could be trained knowing what action and at what time event occurred it will do a fairly good job. Congrats on the encouraging post.", "votes": 1, "replies": [ { "id": 1987080, "postDate": "2022-10-14T13:43:25.443Z", "content": "

We discarded optical flow based on its excessive time requirements. Nonetheless, it seems that @kmat2019 used it as an input to detect the ball (https://twitter.com/Kmat67916008/status/1580767429558693888).

\n

In terms of identifying opponents, you are right that it may not be useful for the pass or the throwins, but we thought that it could play an important role in the challenges.

\n

We trained the model with short clips, trying to classify the action and doing regression on the frame where the action happens. This and other similar approaches did not work for us. It seems that top teams input far less information than us (we used a window of 1.6 seconds), and maybe the key was there. Thank you for the comment @uzodike!

", "rawMarkdown": "We discarded optical flow based on its excessive time requirements. Nonetheless, it seems that @kmat2019 used it as an input to detect the ball ([https://twitter.com/Kmat67916008/status/1580767429558693888](https://twitter.com/Kmat67916008/status/1580767429558693888)).\n\nIn terms of identifying opponents, you are right that it may not be useful for the pass or the throwins, but we thought that it could play an important role in the challenges.\n\nWe trained the model with short clips, trying to classify the action and doing regression on the frame where the action happens. This and other similar approaches did not work for us. It seems that top teams input far less information than us (we used a window of 1.6 seconds), and maybe the key was there. Thank you for the comment @uzodike!", "votes": 2 } ] }, { "id": 1992825, "postDate": "2022-10-18T00:57:27.140Z", "content": "

I really like the methodology you used even though I don't use Computer vision that much other than CNN-based algorithms for image analysis! Tell me though how did you manage to fit this size of data of videos in your Kaggle notebook server !!🤒

", "rawMarkdown": "I really like the methodology you used even though I don't use Computer vision that much other than CNN-based algorithms for image analysis! Tell me though how did you manage to fit this size of data of videos in your Kaggle notebook server !!🤒", "replies": [ { "id": 1993571, "postDate": "2022-10-18T13:40:30.653Z", "content": "

By decreasing batch size and frame resolution, we had no problems with memory

", "rawMarkdown": "By decreasing batch size and frame resolution, we had no problems with memory", "votes": 1 } ] }, { "id": 1991643, "postDate": "2022-10-17T10:02:35.710Z", "content": "

This is cool

", "rawMarkdown": "This is cool" }, { "id": 1991006, "postDate": "2022-10-16T22:06:26.673Z", "content": "

Thank you for the post, Guillermo!

\n

Have you thought in apply Kalman filter in order to reduce some noise after tracking?

", "rawMarkdown": "Thank you for the post, Guillermo!\n\nHave you thought in apply Kalman filter in order to reduce some noise after tracking? ", "replies": [ { "id": 1993579, "postDate": "2022-10-18T13:42:42.440Z", "content": "

Thank you for your suggestion. I am not very familiar with the Kalman filter. How would you use it here?

", "rawMarkdown": "Thank you for your suggestion. I am not very familiar with the Kalman filter. How would you use it here?", "votes": 1 } ] }, { "id": 1989761, "postDate": "2022-10-16T06:55:48.393Z", "content": "

Interesting Bro. Can you get the actual body measurements by taking the photo of the person??
\nIf Yes How Can You achieve it? @atmguille

", "rawMarkdown": "Interesting Bro. Can you get the actual body measurements by taking the photo of the person??\nIf Yes How Can You achieve it? @atmguille", "replies": [ { "id": 1993549, "postDate": "2022-10-18T13:09:54.470Z", "content": "

Not sure if I understand your question. Could you rephrase it @saikumartamminana?

", "rawMarkdown": "Not sure if I understand your question. Could you rephrase it @saikumartamminana?", "votes": 1 } ] }, { "id": 1989609, "postDate": "2022-10-16T05:18:35.823Z", "content": "

osm your work

", "rawMarkdown": "osm your work" }, { "id": 1989606, "postDate": "2022-10-16T05:15:57.720Z", "content": "

Thanks for sharing extremely useful!!

", "rawMarkdown": "Thanks for sharing extremely useful!!", "votes": 1 }, { "id": 2019132, "postDate": "2022-11-06T11:15:06.593Z", "content": "

Thanks for @atmguille , really helpful.

", "rawMarkdown": "Thanks for @atmguille , really helpful." } ], "complete": true, "source": "live", "fetched_at": "2026-10-24T13:28:33Z", "competition_id": 37244, "competition_slug": "dfl-bundesliga-data-shootout", "team_id": 9045902, "solution_url": "/competitions/dfl-bundesliga-data-shootout/writeups/exactly-viva-ronaldo-our-frustrated-approach" }