{ "id": 578478, "title": "4th place solution", "content": "

This was our very first NLP task. Thanks to the Kaust Academy team for this opportunity.

\n

Summery
\nOur top score is an ensemble of Qwen3 (0.6b and 1.7b) and Qwen2.5Math (1.5b and 7b), each fine-tuned using a slightly different approach, but all sharing Layer-wise learning rate.

\n

Data
\nWe only cleaned the data from links and HTML elements, no augmentation or oversampling.

\n

Models and training
\nThe models were trained on kaggle's P100, RTX A40, and RTX4060 8gigs.
\nWe ended up with 6 models to ensemble, most with this setup: Cosine lr Scheduling, 5 Folds, 5 epochs per fold, LLRD, lr=1e-4, LoRA targeting attention and mlp projections, CV using OOF predictions and predicting on the test set by averaging predictions from each fold.
\nThe models are :

\n\n

Then all of these models were ensembled again.

\n

Code:

\n", "comments": [ { "id": 3199868, "author_name": "Denver Magtibay", "author_url": "", "post_date": "2025-05-11T15:28:21.317000", "content": "

Congrats sir! Great work! I learned from it!

", "votes": 1, "replies": [] }, { "id": 3199623, "author_name": "Ayyaz Khan", "author_url": "", "post_date": "2025-05-11T08:45:25.173000", "content": "

Impressive work! I appreciate how clearly you explained your methodology—it makes it easy for others to learn from it.

", "votes": 1, "replies": [] }, { "id": 3199618, "author_name": "Mfahimhossen", "author_url": "", "post_date": "2025-05-11T08:33:20.753000", "content": "

Thank you for sharing idea

", "votes": 1, "replies": [] } ], "comment_count": 3, "messages": [ { "id": 3199602, "postDate": "2025-05-11T08:08:41.743Z", "content": "

This was our very first NLP task. Thanks to the Kaust Academy team for this opportunity.

\n

Summery
\nOur top score is an ensemble of Qwen3 (0.6b and 1.7b) and Qwen2.5Math (1.5b and 7b), each fine-tuned using a slightly different approach, but all sharing Layer-wise learning rate.

\n

Data
\nWe only cleaned the data from links and HTML elements, no augmentation or oversampling.

\n

Models and training
\nThe models were trained on kaggle's P100, RTX A40, and RTX4060 8gigs.
\nWe ended up with 6 models to ensemble, most with this setup: Cosine lr Scheduling, 5 Folds, 5 epochs per fold, LLRD, lr=1e-4, LoRA targeting attention and mlp projections, CV using OOF predictions and predicting on the test set by averaging predictions from each fold.
\nThe models are :

\n\n

Then all of these models were ensembled again.

\n

Code:

\n", "rawMarkdown": "This was our very first NLP task. Thanks to the Kaust Academy team for this opportunity.\n\n**Summery**\nOur top score is an ensemble of Qwen3 (0.6b and 1.7b) and Qwen2.5Math (1.5b and 7b), each fine-tuned using a slightly different approach, but all sharing Layer-wise learning rate.\n\n**Data**\nWe only cleaned the data from links and HTML elements, no augmentation or oversampling.\n\n**Models and training**\nThe models were trained on kaggle's P100, RTX A40, and RTX4060 8gigs.\nWe ended up with 6 models to ensemble, most with this setup: Cosine lr Scheduling, 5 Folds, 5 epochs per fold, LLRD, lr=1e-4, LoRA targeting attention and mlp projections, CV using OOF predictions and predicting on the test set by averaging predictions from each fold. \nThe models are :\n- Qwen3-1.7b.\n- Qwen3-0.6b: We trained it two times, one with Weighted Cross entropy loss.\n- Qwen2.5Math-7b: 4 folds for this one.\n- Qwen2.5Math-1.5b: also trained 2 times, one with 3 epochs, no Stratified-Kfolds.\nAll of these models were ensembles by averaging their probs, and the result was used to train Qwen2.5Math-1.5b on a pseudo-labeled dataset (confidence score is 0.85). This model also utilized AMP, gradient accumulation, and starting each fold with a different seed.\n\nThen all of these models were ensembled again.\n\n\n**Code**:\n- [Qwen2.5Math/pseudo-labeled data](https://www.kaggle.com/code/binabbas/qwen2-5math-pseudo-labeled-data-lb-9054)\n- [Qwen2.5Math-1.5b/ 3epochs](https://www.kaggle.com/code/binabbas/qwen2-5math-1-5b-3epochs-lb-0-9)\n- [Qwen2.5Math-7b](https://www.kaggle.com/code/binabbas/qwen2-5math-7b-lb-9049)\n- [Qwen3-0.6b/WECloss](https://www.kaggle.com/code/binabbas/qwen3-0-6b-llrd-wceloss-lb-0-8970)\n- [Qwen3-1.7b](https://www.kaggle.com/code/binabbas/qwen3-1-7b-llrd-lb-0-8965)\n- [Qwen3-0.6b](https://www.kaggle.com/code/binabbas/qwen3-0-6b-llrd-lb-0-8980)\n- [Models ensembling](https://www.kaggle.com/code/binabbas/model-ensembling-lb-0-9154)\n", "votes": 7 }, { "id": 3199868, "postDate": "2025-05-11T15:28:21.317Z", "content": "

Congrats sir! Great work! I learned from it!

", "rawMarkdown": "Congrats sir! Great work! I learned from it!", "votes": 1 }, { "id": 3199623, "postDate": "2025-05-11T08:45:25.173Z", "content": "

Impressive work! I appreciate how clearly you explained your methodology—it makes it easy for others to learn from it.

", "rawMarkdown": "Impressive work! I appreciate how clearly you explained your methodology—it makes it easy for others to learn from it.", "votes": 1 }, { "id": 3199618, "postDate": "2025-05-11T08:33:20.753Z", "content": "

Thank you for sharing idea

", "rawMarkdown": "Thank you for sharing idea", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:53:53Z", "competition_id": 97669, "competition_slug": "classification-of-math-problems-by-kasut-academy", "team_id": 13650251, "solution_url": "/competitions/classification-of-math-problems-by-kasut-academy/writeups/we-are-suffering-4th-place-solution" }