![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18473244%2Fa2ce225f5e1910dd2783438811d20643%2FScreenshot%202025-05-11%20153806.png?generation=1746968718495543&alt=media\")
Hi everyone,
\nIt was really a great comp hosted by @mohammad2012191 from KAUST academy and I improved my skills a lot in nlp from this competition
\nI think that one of the main constraints for me was that I was in the heavy part of the semester so I left the competition for a week which made me fall from rank 3 to 12, to be honest I never though that it would be that competitive, also not having a gpu limits me alot, one of the tricks i used was snatching the L4 x4 from the arc comp which even the 1st place used😄. but I then subscribed to a 10 usd subscribtion in colab pro and i burned all the compute units in the last 2 days of the comp only not gonna lie this was really helpful to snatch the win
\nthere are many ideas that came to my mind but i couldn't benefit from it maybe it could work in another places
\nfinetuning only on psuedo test labels, save the model and use it for train as 2nd round of training
\nthe idea was that i wanted the model to learn the test set without overfitting to the mistakes in pseudo labels but didn't work
I noticed that all the preprocessing techniques that were even used in the public notebooks hurting my performance, ex: tokenizing all numbers as a single token, adding unknown tokens, removing latex
\npushing to the 0.88s with large deberta and other bert models was the farthest i could get not sure if someone else managed to outperform this score significantly with berts and i think that bert models are harder to optimize and tune compared to finetuning llms with LoRa
\ndirty ensembling of qwen models was the way to go for me here is a rough uml for the final ensemble
\n
in case you are asking from where I get the pseudo labels for the first model it is an ensemble of multiple bert notebooks which had the best public lb for me at that moment which was 0.881
\nhttps://www.kaggle.com/code/tahaalshatiri/notebookbc935e82db
and before training you need to change the csv to conversations format like the ones in hugging face here is a notebook for changing it in case you are interested
\nhttps://www.kaggle.com/code/tahaalshatiri/notebook8c99a739c2
\nalso there where some stupid mistakes I did that I didn't have enough time to fix it for example i was using a linear schedular for all my qwen models I only was able to fix it to cosine for the last qwen 2.5 model.
\nall the qwen models where trained for a single epoch only not sure how would it perform for higher epochs
\nhere is the colab notebook you just need to generate the final qwen 2.5 14b predictions the same notebook was used for all other models i just changed schedular to cosine for the last model
\nhttps://colab.research.google.com/drive/1EzVMFq4cl7DCMCDltZPmomN7NzzSC47H#scrollTo=y5CzHSgc2ID5
\nand here is the ensemble final solution
\nhttps://www.kaggle.com/code/tahaalshatiri/notebookd3e3a58e5d?scriptVersionId=238957648
\nand here is the augmentation data am refering to this was the only extra data that i used
\nhttps://www.kaggle.com/datasets/prudhvirajuchekuri/math-problem-classification-data
", "comments": [ { "id": 3201436, "author_name": "Mohammed Salem Lanqes", "author_url": "", "post_date": "2025-05-13T21:36:46.387000", "content": "Congratulations, thanks a lot for sharing the notebooks for your solution
", "votes": 1, "replies": [] }, { "id": 3200782, "author_name": "Denver Magtibay", "author_url": "", "post_date": "2025-05-13T03:57:00.530000", "content": "Congrats! Thank you for sharing your idea!
", "votes": 1, "replies": [] }, { "id": 3199797, "author_name": "A.Abbas", "author_url": "", "post_date": "2025-05-11T13:32:25.053000", "content": "Congrats on 2nd place!!
\nLooking forward to learn more from your solutions
thank you for sharing your idea , congratulations
", "votes": 1, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 3199792, "postDate": "2025-05-11T13:23:04.413Z", "content": "Hi everyone,
\nIt was really a great comp hosted by @mohammad2012191 from KAUST academy and I improved my skills a lot in nlp from this competition
\nI think that one of the main constraints for me was that I was in the heavy part of the semester so I left the competition for a week which made me fall from rank 3 to 12, to be honest I never though that it would be that competitive, also not having a gpu limits me alot, one of the tricks i used was snatching the L4 x4 from the arc comp which even the 1st place used😄. but I then subscribed to a 10 usd subscribtion in colab pro and i burned all the compute units in the last 2 days of the comp only not gonna lie this was really helpful to snatch the win
\nthere are many ideas that came to my mind but i couldn't benefit from it maybe it could work in another places
\nfinetuning only on psuedo test labels, save the model and use it for train as 2nd round of training
\nthe idea was that i wanted the model to learn the test set without overfitting to the mistakes in pseudo labels but didn't work
I noticed that all the preprocessing techniques that were even used in the public notebooks hurting my performance, ex: tokenizing all numbers as a single token, adding unknown tokens, removing latex
\npushing to the 0.88s with large deberta and other bert models was the farthest i could get not sure if someone else managed to outperform this score significantly with berts and i think that bert models are harder to optimize and tune compared to finetuning llms with LoRa
\ndirty ensembling of qwen models was the way to go for me here is a rough uml for the final ensemble
\n
in case you are asking from where I get the pseudo labels for the first model it is an ensemble of multiple bert notebooks which had the best public lb for me at that moment which was 0.881
\nhttps://www.kaggle.com/code/tahaalshatiri/notebookbc935e82db
and before training you need to change the csv to conversations format like the ones in hugging face here is a notebook for changing it in case you are interested
\nhttps://www.kaggle.com/code/tahaalshatiri/notebook8c99a739c2
\nalso there where some stupid mistakes I did that I didn't have enough time to fix it for example i was using a linear schedular for all my qwen models I only was able to fix it to cosine for the last qwen 2.5 model.
\nall the qwen models where trained for a single epoch only not sure how would it perform for higher epochs
\nhere is the colab notebook you just need to generate the final qwen 2.5 14b predictions the same notebook was used for all other models i just changed schedular to cosine for the last model
\nhttps://colab.research.google.com/drive/1EzVMFq4cl7DCMCDltZPmomN7NzzSC47H#scrollTo=y5CzHSgc2ID5
\nand here is the ensemble final solution
\nhttps://www.kaggle.com/code/tahaalshatiri/notebookd3e3a58e5d?scriptVersionId=238957648
\nand here is the augmentation data am refering to this was the only extra data that i used
\nhttps://www.kaggle.com/datasets/prudhvirajuchekuri/math-problem-classification-data
", "rawMarkdown": "Hi everyone,\n\nIt was really a great comp hosted by @mohammad2012191 from KAUST academy and I improved my skills a lot in nlp from this competition\n\nI think that one of the main constraints for me was that I was in the heavy part of the semester so I left the competition for a week which made me fall from rank 3 to 12, to be honest I never though that it would be that competitive, also not having a gpu limits me alot, one of the tricks i used was snatching the L4 x4 from the arc comp which even the 1st place used😄. but I then subscribed to a 10 usd subscribtion in colab pro and i burned all the compute units in the last 2 days of the comp only not gonna lie this was really helpful to snatch the win\n\n\n\n# What didn't work?\n\nthere are many ideas that came to my mind but i couldn't benefit from it maybe it could work in another places\n\n##### 1)NO leak pseudo labels\nfinetuning only on psuedo test labels, save the model and use it for train as 2nd round of training\nthe idea was that i wanted the model to learn the test set without overfitting to the mistakes in pseudo labels but didn't work\n\n##### 2) manipulating embeddings and tokens\nI noticed that all the preprocessing techniques that were even used in the public notebooks hurting my performance, ex: tokenizing all numbers as a single token, adding unknown tokens, removing latex\n\n\n##### 3) Bert models in general\npushing to the 0.88s with large deberta and other bert models was the farthest i could get not sure if someone else managed to outperform this score significantly with berts and i think that bert models are harder to optimize and tune compared to finetuning llms with LoRa\n\n\n# What worked?\n\ndirty ensembling of qwen models was the way to go for me here is a rough uml for the final ensemble \n\n\n\nin case you are asking from where I get the pseudo labels for the first model it is an ensemble of multiple bert notebooks which had the best public lb for me at that moment which was 0.881\nhttps://www.kaggle.com/code/tahaalshatiri/notebookbc935e82db\n\nand before training you need to change the csv to conversations format like the ones in hugging face here is a notebook for changing it in case you are interested\n\nhttps://www.kaggle.com/code/tahaalshatiri/notebook8c99a739c2\n\n\nalso there where some stupid mistakes I did that I didn't have enough time to fix it for example i was using a linear schedular for all my qwen models I only was able to fix it to cosine for the last qwen 2.5 model.\n\nall the qwen models where trained for a single epoch only not sure how would it perform for higher epochs\n\n\nhere is the colab notebook you just need to generate the final qwen 2.5 14b predictions the same notebook was used for all other models i just changed schedular to cosine for the last model\n\nhttps://colab.research.google.com/drive/1EzVMFq4cl7DCMCDltZPmomN7NzzSC47H#scrollTo=y5CzHSgc2ID5\n\n\nand here is the ensemble final solution\n\nhttps://www.kaggle.com/code/tahaalshatiri/notebookd3e3a58e5d?scriptVersionId=238957648\n\n\n\nand here is the augmentation data am refering to this was the only extra data that i used\n\nhttps://www.kaggle.com/datasets/prudhvirajuchekuri/math-problem-classification-data", "votes": 7 }, { "id": 3201436, "postDate": "2025-05-13T21:36:46.387Z", "content": "Congratulations, thanks a lot for sharing the notebooks for your solution
", "rawMarkdown": "Congratulations, thanks a lot for sharing the notebooks for your solution ", "votes": 1 }, { "id": 3200782, "postDate": "2025-05-13T03:57:00.530Z", "content": "Congrats! Thank you for sharing your idea!
", "rawMarkdown": "Congrats! Thank you for sharing your idea!", "votes": 1 }, { "id": 3199797, "postDate": "2025-05-11T13:32:25.053Z", "content": "Congrats on 2nd place!!
\nLooking forward to learn more from your solutions
thank you for sharing your idea , congratulations
", "rawMarkdown": "thank you for sharing your idea , congratulations\n \n", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:53:28Z", "competition_id": 97669, "competition_slug": "classification-of-math-problems-by-kasut-academy", "team_id": 13644525, "solution_url": "/competitions/classification-of-math-problems-by-kasut-academy/writeups/taha-no-gpu-no-validation-2nd-place-solution" }