# 1st Place Solution 

Hi everyone,
First, I’d like to thank the host for organizing such a great LLM competition.
Summary
For this competition, I used the
Qwen2.5-14B
and
Qwen3-14B
models as my base models and fine-tuned them specifically for text classification. To make the solution more robust and stable, for each model, I ensembled models that were trained across K-fold splits. On top of that, I did a weighted ensemble for Qwen2.5 and Qwen3.
Inputs
I used only the Math Question field and formatted it into the following prompt template:
prompt = f"""<|im_start|>user
Your task is to classify each Math problem into one of these eight topics using a machine learning or NLP-based approach.
0: Algebra
1: Geometry and Trigonometry
2: Calculus and Analysis
3: Probability and Statistics
4: Number Theory
5: Combinatorics and Discrete Math
6: Linear Algebra
7: Abstract Algebra and Topology
Your answer should be an integer that assigns the most appropriate topic category to the given Math problem based on its content and required reasoning.
Math Problem: {MATH_QUESTION_HERE}
Answer:"""
Model
Based on my recent experience in Kaggle competitions, 14B models strike a good balance for achieving competitive results, validating ideas, and aiming for a win. That’s why, when I joined this competition six days ago, I decided to stick with
Qwen2.5-14B
and the recently released
Qwen3-14B
as the base model and fine-tune them for text classification.
I replaced the base Qwen3-14B model's
lm_head
with a new classification head that has the same size with embedding dim.
Given the causal nature of decoder models, I used the last hidden state of the final token,
Answer:
, and fed it into a linear head for classification.
Cross-validation
Due to the class imbalance in the training set, I used StratifiedKFold (3 folds) to ensure that each fold preserved the overall class distribution. This helped maintain representative splits and allowed the model to be evaluated more reliably across all classes.
Training
To utlize the 4×L4 GPUs on Kaggle, I used Distributed Data Parallel (DDP) for faster training and rapid experimentation.
For parameter-efficient fine-tuning, I applied vanilla LoRA.
A key factor in this competition was tuning the learning rate. I used a Qwen2.5-0.5B model to efficiently search for optimal rates, with each fold taking only 6 minutes for training and validation on 4 GPUs. Interestingly, the settings from the smaller model transferred well to the larger 14B model.
Differential learning rates didn’t work well in this competition, so I ended up using a quite large learning rate of 2e-4 for both the backbone and classification head, with a cosine decay schedule starting from 2e-4 and decaying to zero, without warm-up.
To help the model focus on rare classes, I computed class weights and incorporated them into the cross-entropy loss. This, along with a good learning rate, was crucial for model performance.
The training was performed using 3-fold cross-validation, with each fold trained for 3 epochs. Each epoch took 30 minutes for both training and validation.
Model Performance and Ensemble Results
Model
CV Score
Public LB
Private LB
Qwen3-14B
0.909
-
-
Qwen2.5-14B
0.911
-
-
Ensemble
-
0.9168
0.9253
Inference
Multi-GPU inference was used to reduce inference time.
During training, I noticed that 99% of the math questions were under 300 tokens, so I dynamically truncated inputs longer than 300 tokens. However, at inference time, I used the full input without truncation to fully leverage the math question content and maximize test-time compute.
Ensemble
For each model, I first averaged the K-fold models. Then, I performed a weighted average, giving
0.6
weight to
Qwen3-14B
and
0.4
weight to
Qwen2.5-14B
for the final predictions.
Code
Training Code for:
Qwen3-14B
:
[Qwen3-14B Fine-tuning](https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen3-14b)
Qwen2.5-14B
:
[Qwen2.5-14B Fine-tuning](https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen2-5-14b)
Offline Inference:
[Offline Inference Code](https://www.kaggle.com/code/ducnh279/kacs1-offline-inference?scriptVersionId=238990333)