# 7th place solution 

Firstly, I'd like to thank Mohamed Eltayeb, Ali Habibullah, and KAChallenges for organizing this insightful competition. It provided an excellent opportunity to explore NLP models in a practical and interesting context.
Thoughts on Evaluation Metric
Using the micro F1 score for evaluation effectively simplified the challenge since it becomes equivalent to accuracy, particularly beneficial given the dataset's relative imbalance. It reduced the need for extensive class balancing strategies, allowing participants to focus more directly on model performance.
My Approach and Final Submission
My solution involved fine-tuning several Large Language Models (LLMs), specifically:
LLaMA  3.2 (8B parameters)
Qwen 2.5 (7B parameters)
Phi-4 Mini Reasoning (3.8B parameters)
After several experiments and parameter searches, the Phi-4 Mini Reasoning model emerged as the top performer. My final submission was an ensemble based solely on multiple runs of this single model, which slightly increased robustness and final accuracy.
For those interested, here is my inference notebook detailing the final approach:
[Inference notebook](https://www.kaggle.com/code/ferasoo/7th-solution)
.
Lessons Learned and Mistakes
After looking into the top submissions, I'd like to thank the leading participants for sharing their insights. From analyzing their methods, I realized several areas for improvement in my own approach:
Single-fold Evaluation
: I evaluated my models using a single train-validation split, which limited the reliability of my results. Implementing cross-validation or multiple folds would have provided a more accurate estimation of model performance.
Parameter Search Strategy
: Initially, I performed hyperparameter tuning directly on larger models, consuming substantial computational resources and time. In retrospect, a more efficient approach would have been to first optimize hyperparameters on smaller, simpler models and then transfer insights to larger models.
Model Diversity in Ensembles
: My ensemble consisted exclusively of variations of the same model (Phi-4 Mini Reasoning). Including predictions from different models, even those with slightly lower individual accuracy, could have improved ensemble performance by capitalizing on diverse mistake patterns.
Closing Remarks
This competition was an enriching learning experience. Congratulations to all participants, and once again, thank you to the organizers for this challenging and educational event!