# 86th Place Solution (just using public models)

Big thanks to the host for organizing such an interesting competition! Learned quite a lot: from multi-GPU training, to utilizing LLMs for many tasks.
Solution
This competition mainly consists of 2 parts,
retrieval
and
reranking
. I tried to fine-tune my own retrieval model with synthetic data, but it performs worse than the top openly available retrieval models. In the end, this solution does not require fine-tuning (from my part) at all.
Retrieval
I didn't realize before that LLMs can be used as embedding models using last token pool.
Ensembled 3 retrieval models based on these great notebooks:
[Anh Vo's notebook](https://www.kaggle.com/code/anhvth226/eedi-11-21-14b)
[Okinawa's notebook](https://www.kaggle.com/code/zuoyouzuo/eedi-qwen2-5-14b-it-a-simple-infer-lb-0-422)
[Ming Sheng Choo's notebook](https://www.kaggle.com/code/mschoo/flag-qwen14b-test)
The ensemble is just a simple embedding concatenation.
Reranking
Based on
[Ahmet Erdem's notebook](https://www.kaggle.com/code/aerdem4/eedi-qwen32b-vllm-with-logits-processor-zoo)
, using Qwen 32B AWQ with vLLM. I did not fine-tune this model.
Similar to above notebook, I used
MultipleChoiceLogitsProcessor
and gave the model 9 choices during inference (numbered 1-9), as Qwen tokenizer seems to split numbers character-by-character, so I avoided double digits.
Implemented partial reranking based on logits. The first 9 choices were reordered based on logits, and then added 16 more based on retrieval model. In total we have 25 choices for submission.
Misc
Some things that don't work for me:
Finetuned Salesforce/SFR-Embedding-2_R for retrieval.
[HF repo](https://huggingface.co/evanarlian/sfr-lambda-8-a100_2024-12-05__00.25.39)
. Please tell me what's wrong 😂
Finetuned 4 different Qwen 1.5B models (for concat embeddings). HF repos:
[1](https://huggingface.co/evanarlian/synthetic-qwen-instruct_2024-12-09__23.06.21)
,
[2](https://huggingface.co/evanarlian/synthetic-qwen_2024-12-10__02.16.19)
,
[3](https://huggingface.co/evanarlian/synthetic-qwen-math-instruct_2024-12-10__03.32.51)
,
[4](https://huggingface.co/evanarlian/synthetic-qwen-math_2024-12-10__05.07.08)
.
Dataset generation:
[synthetic](https://www.kaggle.com/datasets/evanarlian/eedi-synthetic)
and
[paraphrased](https://www.kaggle.com/datasets/evanarlian/eedi-paraphrased)
.
Code:
[Inference notebook](https://www.kaggle.com/code/evanarlian/eedi-retrieval-rerank)
[Training code](https://github.com/evanarlian/kaggle-eedi)