![\"Inference](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4526980%2Fb29b262492a8e319f0520626f0ae3a35%2Feedi_inference_pipeline.png?generation=1734098354476413&alt=media\")
First of all, I would like to thank the organisers for hosting the competition, kaggle team, and my team mates from AIRI, Skoltech and VeinCV
\nrolf110
\ndanyaivanov
\nShout-out to pipmos!
Our solution:
\nQwen2.5-14B retriever to obtain top-25 misconceptionsQwen2.5-32B reranker used to score each of the misconceptions received by retriever. We do it iteratively, by passing one misconception at a time to obtain a probability of it being the one that leads to the incorrect answer. The whole notebook runs approximately 7 hours. We didn't use any upper-level frameworks such as vllm to run the inference pipeline.
\nAll models were fine-tuned using the competition data, consisting of 4370 (question, construct, correct answer, incorrect answer) quadruplets, + 13921 synthetic (question, incorrect answer, correct answer, incorrect answer) samples generated using gpt-4o.
\nMisconceptionId in the given database, we generated new (QuestionText, ConstructName, Correct Answer, Incorrect Answer) samples, ensuring at least 7 examples is presented for each MisconceptionId in the final real+synthetic dataset. We then filtered those examples that have a full-match in the QuestionText (approximately 500 samples removed).SFR-Embedding-2_R embeddings of MisconceptionName to obtain \"closest\" misconceptions. We ensured that each reference is different between different generations for a particular MisconceptionId.prompt we passed to gpt-4o:\nWe manually checked some of the generated examples, and they were pretty good and close to the real data.
\nOther staff we found important for generation:
ConstructName) field; but we found that such cases were also observed in the competition data, so we decided to keep it as is.We used 20% of the competition data for validation, which we additionally split on 2 subsets:
\nMisconceptionId and stratification based on the number of misconceptions per question.QuestionId and stratification based on the number of misconceptions per question.We used the ID test to account for how the model works on the misconceptions it knows, while the OOD test was used to account for how the model generalizes to the unseen misconceptions. However, we mainly tracked the OOD data because it was well-aligned with the lb and accounted for the most complex scenario.
\nTo obtain fine-grained hard negatives, we first fine-tuned Qwen2.5-14B on real+synthetic data. Then we repeat the training with the exact same setup, but using the hard-negatives extracted using our trained Qwen2.5-14B. After each epoch, we collect new hard negatives using the latest checkpoint.
We also utilized in-batch negative sampling during training. To mitigate the problem of treating negatives which appear to be the same misconceptions as the positive ones, we applied dynamic masking by setting the similarities of such misconceptions to -inf before calculating the loss for each sample in the batch.
| Data | \nRecall@25 | \nMAP@25 | \n
|---|---|---|
| CV OOD | \n0.88 | \n0.42 | \n
| Public LB | \nNone | \n0.438 | \n
| Private LB | \nNone | \n0.434 | \n
Judging from the experiments provided in the paper Novice Learner and Expert Tutor and some research done by other participants during the competition, we had come up with the idea to pass misconceptions list as an additional input to better guide the reranker model.
\nWe didn't want the model to output the misconception prediction as a text because it was not very reliable as we thought, and the multi-classification problem didn't fit here either. Finally, we have decided to solve a binary classification problem for each misconception separately:
\npast_key_values. This helps to save a lot of runtime because we only need to compute it once for the given sample.past_key_values from the previous step as cache. We use a batch size of 5, so 5 iterations is required for the whole list of misconceptions.Given the (question, incorrect answer) pair, we create 24 label = 0 and 1 label = 1 samples for training. The main difference between them is the way we construct the \"list of possible misconceptions\":
label = 0 samples per pair.label = 1), we include 29 randomly selected negatives from the top 50 to create a set of 30 misconceptions we pass to the model.To mitigate the positional bias caused by the retriever, we randomly shuffle the list for every input sample.
\nExample of the input prompt:
\nQuestion: {Question}\nBrief description of the concept skill being tested the Question: {ConstructName}\nCorrect Answer: {Correct Answer}\nIncorrect Answer: {Incorrect Answer}\n of possible misconceptions:\n {Misconception_1}\n...\n {Misconception_30}\nMisconception: {n}. {Misconception_n}\nDoes this Misconception lead to the Incorrect Answer? \nTo account for high imbalance between 0/1 classes, we oversampled positive class 4 times.
\nWe used h2o-studio framework to train our model.
\n| Data | \nMAP@25 | \n
|---|---|
| CV OOD | \n0.571 | \n
| Public LB | \n0.589 | \n
| Private LB | \n0.558 | \n
reranker adapter and classification head weights: https://www.kaggle.com/models/andreygalichin/qwen2.5-32b-r32-a16-synth-merge-smooth-full/
", "comments": [ { "id": 3071267, "author_name": "S.Osuga", "author_url": "", "post_date": "2024-12-13T14:57:38.173000", "content": "Thank you for sharing such a brilliant solution! I didn't know about h2o-studio framework. I'd like to try it myself.
", "votes": 1, "replies": [ { "id": 3071291, "author_name": "Andrey Galichin", "author_url": "", "post_date": "2024-12-13T15:32:32.720000", "content": "Thank you! Yes, it is actually a perfect framework for those who are not very familiar with the LLM fine-tuning or don't want to waste much time to write their own code. It has a pretty nice GUI too.
", "votes": 1, "replies": [] } ] }, { "id": 3071359, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2024-12-13T17:09:04.117000", "content": "Congratulations on your strong finish and thank your for the detailed write up.
", "votes": 0, "replies": [] }, { "id": 3071269, "author_name": "Pakhnenko Ivan", "author_url": "", "post_date": "2024-12-13T15:00:35.810000", "content": "We also tried to implement two ideas but they didn’t work out:
\nInstead of ranking all 25 misconceptions at once we considered feeding the rerank model 10 misconceptions at a time, selecting the top 5, then comparing these top 5 with the next batch of 5, and so on until processing all misconceptions. This approach worked well with qwen-14, but we couldn’t adapt the code for qwen-32 within time constraints, despite experimenting with various batch processing strategies.
\nBased on the top 7 private LB team's solution, our main mistake was generating a single number per model output and querying the reranker for each misconception individually. This took significantly more time than asking about all misconceptions in a single query.
Assuming the reranker is robust, we hypothesized that final misconception should appear in the top 15 results. Based on that, we tried replacing the 16-25 range with ones closer in cosine similarity to the top 15 to boost recall in the retrieval step. It worked in some tests, but for the final submission with the best models, we decided it was too risky and didn’t go for it.
First of all, I would like to thank the organisers for hosting the competition, kaggle team, and my team mates from AIRI, Skoltech and VeinCV
\nrolf110
\ndanyaivanov
\nShout-out to pipmos!
Our solution:
\nQwen2.5-14B retriever to obtain top-25 misconceptionsQwen2.5-32B reranker used to score each of the misconceptions received by retriever. We do it iteratively, by passing one misconception at a time to obtain a probability of it being the one that leads to the incorrect answer. The whole notebook runs approximately 7 hours. We didn't use any upper-level frameworks such as vllm to run the inference pipeline.
\nAll models were fine-tuned using the competition data, consisting of 4370 (question, construct, correct answer, incorrect answer) quadruplets, + 13921 synthetic (question, incorrect answer, correct answer, incorrect answer) samples generated using gpt-4o.
\nMisconceptionId in the given database, we generated new (QuestionText, ConstructName, Correct Answer, Incorrect Answer) samples, ensuring at least 7 examples is presented for each MisconceptionId in the final real+synthetic dataset. We then filtered those examples that have a full-match in the QuestionText (approximately 500 samples removed).SFR-Embedding-2_R embeddings of MisconceptionName to obtain \"closest\" misconceptions. We ensured that each reference is different between different generations for a particular MisconceptionId.prompt we passed to gpt-4o:\nWe manually checked some of the generated examples, and they were pretty good and close to the real data.
\nOther staff we found important for generation:
ConstructName) field; but we found that such cases were also observed in the competition data, so we decided to keep it as is.We used 20% of the competition data for validation, which we additionally split on 2 subsets:
\nMisconceptionId and stratification based on the number of misconceptions per question.QuestionId and stratification based on the number of misconceptions per question.We used the ID test to account for how the model works on the misconceptions it knows, while the OOD test was used to account for how the model generalizes to the unseen misconceptions. However, we mainly tracked the OOD data because it was well-aligned with the lb and accounted for the most complex scenario.
\nTo obtain fine-grained hard negatives, we first fine-tuned Qwen2.5-14B on real+synthetic data. Then we repeat the training with the exact same setup, but using the hard-negatives extracted using our trained Qwen2.5-14B. After each epoch, we collect new hard negatives using the latest checkpoint.
We also utilized in-batch negative sampling during training. To mitigate the problem of treating negatives which appear to be the same misconceptions as the positive ones, we applied dynamic masking by setting the similarities of such misconceptions to -inf before calculating the loss for each sample in the batch.
| Data | \nRecall@25 | \nMAP@25 | \n
|---|---|---|
| CV OOD | \n0.88 | \n0.42 | \n
| Public LB | \nNone | \n0.438 | \n
| Private LB | \nNone | \n0.434 | \n
Judging from the experiments provided in the paper Novice Learner and Expert Tutor and some research done by other participants during the competition, we had come up with the idea to pass misconceptions list as an additional input to better guide the reranker model.
\nWe didn't want the model to output the misconception prediction as a text because it was not very reliable as we thought, and the multi-classification problem didn't fit here either. Finally, we have decided to solve a binary classification problem for each misconception separately:
\npast_key_values. This helps to save a lot of runtime because we only need to compute it once for the given sample.past_key_values from the previous step as cache. We use a batch size of 5, so 5 iterations is required for the whole list of misconceptions.Given the (question, incorrect answer) pair, we create 24 label = 0 and 1 label = 1 samples for training. The main difference between them is the way we construct the \"list of possible misconceptions\":
label = 0 samples per pair.label = 1), we include 29 randomly selected negatives from the top 50 to create a set of 30 misconceptions we pass to the model.To mitigate the positional bias caused by the retriever, we randomly shuffle the list for every input sample.
\nExample of the input prompt:
\nQuestion: {Question}\nBrief description of the concept skill being tested the Question: {ConstructName}\nCorrect Answer: {Correct Answer}\nIncorrect Answer: {Incorrect Answer}\n of possible misconceptions:\n {Misconception_1}\n...\n {Misconception_30}\nMisconception: {n}. {Misconception_n}\nDoes this Misconception lead to the Incorrect Answer? \nTo account for high imbalance between 0/1 classes, we oversampled positive class 4 times.
\nWe used h2o-studio framework to train our model.
\n| Data | \nMAP@25 | \n
|---|---|
| CV OOD | \n0.571 | \n
| Public LB | \n0.589 | \n
| Private LB | \n0.558 | \n
reranker adapter and classification head weights: https://www.kaggle.com/models/andreygalichin/qwen2.5-32b-r32-a16-synth-merge-smooth-full/
", "rawMarkdown": "First of all, I would like to thank the organisers for hosting the competition, kaggle team, and my team mates from AIRI, Skoltech and VeinCV\n[rolf110](https://www.kaggle.com/rolf110)\n[danyaivanov](https://www.kaggle.com/danyaivanov)\nShout-out to [pipmos](https://www.kaggle.com/pipmos)!\n\n## Summary\n\n### Overall Inference Pipeline\n\n\nOur solution:\n- `Qwen2.5-14B` retriever to obtain top-25 misconceptions\n- `Qwen2.5-32B` reranker used to score each of the misconceptions received by retriever. We do it iteratively, by passing one misconception at a time to obtain a probability of it being the one that leads to the incorrect answer. \n\nThe whole notebook runs approximately 7 hours. We didn't use any upper-level frameworks such as vllm to run the inference pipeline.\n\nAll models were fine-tuned using the competition data, consisting of 4370 (question, construct, correct answer, incorrect answer) quadruplets, + 13921 synthetic (question, incorrect answer, correct answer, incorrect answer) samples generated using gpt-4o.\n\n## Description\n\n### Synthetic Data\n\n- For each `MisconceptionId` in the given database, we generated new (`QuestionText`, `ConstructName`, `Correct Answer`, `Incorrect Answer`) samples, ensuring at least 7 examples is presented for each `MisconceptionId` in the final real+synthetic dataset. We then filtered those examples that have a full-match in the `QuestionText` (approximately 500 samples removed).\n- To provide some guidance to the LLM, we passed a reference sample from the train dataset corresponding to the closest misconceptions for the one considered for generation. We used `SFR-Embedding-2_R` embeddings of `MisconceptionName` to obtain \"closest\" misconceptions. We ensured that each reference is different between different generations for a particular `MisconceptionId`.\n- To construct the prompt, we followed the ideas from [here](https://arxiv.org/pdf/2403.04706). The template of the `prompt` we passed to gpt-4o:\n\n```python\n'''Please act as a professional math tutor.\nYour goal is to create high quality math problems to help students learn math.\nYou will be given a misconception. Please create a multiple choice math question with two options: one correct answer and one distractor (incorrect answer). The distractor should be based on the Misconception. \nFollow the instructions below.\nTo achieve the goal, you have four tasks:\n1. Please generate a brief and formal description of the concept or skill being tested in the question. DO NOT mention or refer to the Misconception in the description.\n2. Create a realistic and contextually appropriate math question that aligns with the description. Provide two options: the correct answer and a distractor (incorrect answer). The distractor should be a plausible answer that a student might choose if they hold the Misconception.\n3. Check the question by solving it step-by-step to find out if it adheres to all principles.\n4. Modify the created question and options according to your checking comment to ensure it is of high quality.\nYou have the following principles to guide you:\n1. Ensure the question is realistic, natural, and contextually appropriate, adhering to common sense and fundamental mathematical principles.\n2. Ensure the question asks for only one specific thing and is clearly stated.\n3. Ensure your student can answer the question correctly by using only the information provided in the question. If visual data is needed (e.g. plot, histogram, bar, etc.), describe it's content explicitly in text.\n4. Ensure the distractor is directly related to the question, stems from the Misconception, and is a plausible answer that a student might select if they have that Misconception.\n5. If the created question already follows these principles upon verification, keep it without modification.\n\nYour output should be in the following format:\nDESCRIPTION:Thank you for sharing such a brilliant solution! I didn't know about h2o-studio framework. I'd like to try it myself.
", "rawMarkdown": "Thank you for sharing such a brilliant solution! I didn't know about h2o-studio framework. I'd like to try it myself.", "votes": 1, "replies": [ { "id": 3071291, "postDate": "2024-12-13T15:32:32.720Z", "content": "Thank you! Yes, it is actually a perfect framework for those who are not very familiar with the LLM fine-tuning or don't want to waste much time to write their own code. It has a pretty nice GUI too.
", "rawMarkdown": "Thank you! Yes, it is actually a perfect framework for those who are not very familiar with the LLM fine-tuning or don't want to waste much time to write their own code. It has a pretty nice GUI too.", "votes": 1 } ] }, { "id": 3071359, "postDate": "2024-12-13T17:09:04.117Z", "content": "Congratulations on your strong finish and thank your for the detailed write up.
", "rawMarkdown": "Congratulations on your strong finish and thank your for the detailed write up." }, { "id": 3071269, "postDate": "2024-12-13T15:00:35.810Z", "content": "We also tried to implement two ideas but they didn’t work out:
\nInstead of ranking all 25 misconceptions at once we considered feeding the rerank model 10 misconceptions at a time, selecting the top 5, then comparing these top 5 with the next batch of 5, and so on until processing all misconceptions. This approach worked well with qwen-14, but we couldn’t adapt the code for qwen-32 within time constraints, despite experimenting with various batch processing strategies.
\nBased on the top 7 private LB team's solution, our main mistake was generating a single number per model output and querying the reranker for each misconception individually. This took significantly more time than asking about all misconceptions in a single query.
Assuming the reranker is robust, we hypothesized that final misconception should appear in the top 15 results. Based on that, we tried replacing the 16-25 range with ones closer in cosine similarity to the top 15 to boost recall in the retrieval step. It worked in some tests, but for the final submission with the best models, we decided it was too risky and didn’t go for it.