# 4th place solution

This was our very first NLP task. Thanks to the Kaust Academy team for this opportunity.
Summery
Our top score is an ensemble of Qwen3 (0.6b and 1.7b) and Qwen2.5Math (1.5b and 7b), each fine-tuned using a slightly different approach, but all sharing Layer-wise learning rate.
Data
We only cleaned the data from links and HTML elements, no augmentation or oversampling.
Models and training
The models were trained on kaggle's P100, RTX A40, and RTX4060 8gigs.
We ended up with 6 models to ensemble, most with this setup: Cosine lr Scheduling, 5 Folds, 5 epochs per fold, LLRD, lr=1e-4, LoRA targeting attention and mlp projections, CV using OOF predictions and predicting on the test set by averaging predictions from each fold.
The models are :
Qwen3-1.7b.
Qwen3-0.6b: We trained it two times, one with Weighted Cross entropy loss.
Qwen2.5Math-7b: 4 folds for this one.
Qwen2.5Math-1.5b: also trained 2 times, one with 3 epochs, no Stratified-Kfolds.
All of these models were ensembles by averaging their probs, and the result was used to train Qwen2.5Math-1.5b on a pseudo-labeled dataset (confidence score is 0.85). This model also utilized AMP, gradient accumulation, and starting each fold with a different seed.
Then all of these models were ensembled again.
Code
:
[Qwen2.5Math/pseudo-labeled data](https://www.kaggle.com/code/binabbas/qwen2-5math-pseudo-labeled-data-lb-9054)
[Qwen2.5Math-1.5b/ 3epochs](https://www.kaggle.com/code/binabbas/qwen2-5math-1-5b-3epochs-lb-0-9)
[Qwen2.5Math-7b](https://www.kaggle.com/code/binabbas/qwen2-5math-7b-lb-9049)
[Qwen3-0.6b/WECloss](https://www.kaggle.com/code/binabbas/qwen3-0-6b-llrd-wceloss-lb-0-8970)
[Qwen3-1.7b](https://www.kaggle.com/code/binabbas/qwen3-1-7b-llrd-lb-0-8965)
[Qwen3-0.6b](https://www.kaggle.com/code/binabbas/qwen3-0-6b-llrd-lb-0-8980)
[Models ensembling](https://www.kaggle.com/code/binabbas/model-ensembling-lb-0-9154)