# 2nd place solution

Hi everyone,
It was really a great comp hosted by
[@mohammad2012191](https://www.kaggle.com/mohammad2012191)
from KAUST academy and I improved my skills a lot in nlp from this competition
I think that one of the main constraints for me was that I was in the heavy part of the semester so I left the competition for a week which made me fall from rank 3 to 12, to be honest I never though that it would be that competitive, also  not having a gpu limits me alot, one of the tricks i used was snatching the L4 x4 from the arc comp which  even the 1st place used😄. but I then subscribed to a 10 usd subscribtion in colab pro and i burned all the compute units in the last 2 days of the comp only not gonna lie this was really helpful to snatch the win
What didn't work?
there are many ideas that came to my mind but i couldn't benefit from it maybe it could work in another places
1)NO leak pseudo labels
finetuning only on psuedo test labels, save the model and use it for train as 2nd round of training
the idea was that i wanted the model to learn the test set without overfitting to the mistakes in pseudo labels but didn't work
2) manipulating embeddings and tokens
I noticed that all the preprocessing  techniques that were even used in the public notebooks hurting my performance, ex: tokenizing all numbers as a single token, adding unknown tokens, removing latex
3) Bert models in  general
pushing to the 0.88s with large deberta and other bert models was the farthest i could get not sure if someone else managed to outperform this score significantly with berts and i think that bert models are harder to optimize and tune compared to finetuning llms with LoRa
What worked?
dirty ensembling of  qwen models was the way to go for me here is a rough uml for the final ensemble
![](assets/e63d3394919c7cfaa15e6a13.png)
in case you are asking from where I get the pseudo labels for the first model it is  an ensemble of multiple bert notebooks which had the best public lb for me at that moment which was 0.881
[https://www.kaggle.com/code/tahaalshatiri/notebookbc935e82db](https://www.kaggle.com/code/tahaalshatiri/notebookbc935e82db)
and before training you need to change the csv to conversations format like the ones in hugging face here is a notebook for changing it in  case you are interested
[https://www.kaggle.com/code/tahaalshatiri/notebook8c99a739c2](https://www.kaggle.com/code/tahaalshatiri/notebook8c99a739c2)
also there where some stupid mistakes I did that I didn't have enough time to fix it for example i was using a linear schedular for all my qwen models I only was able to fix it  to cosine for the last qwen 2.5 model.
all the qwen models where trained for a single epoch only not sure how would it perform for higher epochs
here is the colab notebook you just need to generate the final qwen 2.5 14b predictions  the same notebook was used for all other models i just changed schedular to cosine for the last model
[https://colab.research.google.com/drive/1EzVMFq4cl7DCMCDltZPmomN7NzzSC47H#scrollTo=y5CzHSgc2ID5](https://colab.research.google.com/drive/1EzVMFq4cl7DCMCDltZPmomN7NzzSC47H#scrollTo=y5CzHSgc2ID5)
and here is the ensemble final solution
[https://www.kaggle.com/code/tahaalshatiri/notebookd3e3a58e5d?scriptVersionId=238957648](https://www.kaggle.com/code/tahaalshatiri/notebookd3e3a58e5d?scriptVersionId=238957648)
and here is the augmentation data am refering to this was the only extra data that i used
[https://www.kaggle.com/datasets/prudhvirajuchekuri/math-problem-classification-data](https://www.kaggle.com/datasets/prudhvirajuchekuri/math-problem-classification-data)