Big thanks to the host for organizing such an interesting competition! Learned quite a lot: from multi-GPU training, to utilizing LLMs for many tasks.
\nThis competition mainly consists of 2 parts, retrieval and reranking. I tried to fine-tune my own retrieval model with synthetic data, but it performs worse than the top openly available retrieval models. In the end, this solution does not require fine-tuning (from my part) at all.
\nMultipleChoiceLogitsProcessor and gave the model 9 choices during inference (numbered 1-9), as Qwen tokenizer seems to split numbers character-by-character, so I avoided double digits.Some things that don't work for me:
\nCode:
\n", "comments": [], "comment_count": 0, "messages": [ { "id": 3071962, "postDate": "2024-12-14T15:09:34.093Z", "content": "Big thanks to the host for organizing such an interesting competition! Learned quite a lot: from multi-GPU training, to utilizing LLMs for many tasks.
\nThis competition mainly consists of 2 parts, retrieval and reranking. I tried to fine-tune my own retrieval model with synthetic data, but it performs worse than the top openly available retrieval models. In the end, this solution does not require fine-tuning (from my part) at all.
\nMultipleChoiceLogitsProcessor and gave the model 9 choices during inference (numbered 1-9), as Qwen tokenizer seems to split numbers character-by-character, so I avoided double digits.Some things that don't work for me:
\nCode:
\n", "rawMarkdown": "Big thanks to the host for organizing such an interesting competition! Learned quite a lot: from multi-GPU training, to utilizing LLMs for many tasks.\n\n# Solution\nThis competition mainly consists of 2 parts, **retrieval** and **reranking**. I tried to fine-tune my own retrieval model with synthetic data, but it performs worse than the top openly available retrieval models. In the end, this solution does not require fine-tuning (from my part) at all.\n## Retrieval\n* I didn't realize before that LLMs can be used as embedding models using last token pool.\n* Ensembled 3 retrieval models based on these great notebooks:\n\t* [Anh Vo's notebook](https://www.kaggle.com/code/anhvth226/eedi-11-21-14b)\n\t* [Okinawa's notebook](https://www.kaggle.com/code/zuoyouzuo/eedi-qwen2-5-14b-it-a-simple-infer-lb-0-422)\n\t* [Ming Sheng Choo's notebook](https://www.kaggle.com/code/mschoo/flag-qwen14b-test)\n* The ensemble is just a simple embedding concatenation.\n## Reranking\n* Based on [Ahmet Erdem's notebook](https://www.kaggle.com/code/aerdem4/eedi-qwen32b-vllm-with-logits-processor-zoo), using Qwen 32B AWQ with vLLM. I did not fine-tune this model.\n* Similar to above notebook, I used `MultipleChoiceLogitsProcessor` and gave the model 9 choices during inference (numbered 1-9), as Qwen tokenizer seems to split numbers character-by-character, so I avoided double digits.\n* Implemented partial reranking based on logits. The first 9 choices were reordered based on logits, and then added 16 more based on retrieval model. In total we have 25 choices for submission.\n# Misc\nSome things that don't work for me:\n* Finetuned Salesforce/SFR-Embedding-2_R for retrieval. [HF repo](https://huggingface.co/evanarlian/sfr-lambda-8-a100_2024-12-05__00.25.39). Please tell me what's wrong 😂\n* Finetuned 4 different Qwen 1.5B models (for concat embeddings). HF repos: [1](https://huggingface.co/evanarlian/synthetic-qwen-instruct_2024-12-09__23.06.21), [2](https://huggingface.co/evanarlian/synthetic-qwen_2024-12-10__02.16.19), [3](https://huggingface.co/evanarlian/synthetic-qwen-math-instruct_2024-12-10__03.32.51), [4](https://huggingface.co/evanarlian/synthetic-qwen-math_2024-12-10__05.07.08).\n* Dataset generation: [synthetic](https://www.kaggle.com/datasets/evanarlian/eedi-synthetic) and [paraphrased](https://www.kaggle.com/datasets/evanarlian/eedi-paraphrased).\n\nCode:\n* [Inference notebook](https://www.kaggle.com/code/evanarlian/eedi-retrieval-rerank)\n* [Training code](https://github.com/evanarlian/kaggle-eedi) ", "votes": 9 } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:44:34Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12637153, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/evan-arlian-86th-place-solution-just-using-public-" }