![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fa140b9d748aade03b5bda478668627fe%2Feedi-architecture.png?generation=1734073366567506&alt=media\")
I sincerely thank the Kaggle team and competition hosts for providing this valuable opportunity. Through this competition, I explored the latest advancements in large language models (LLMs) and deepened my understanding of zero-shot inference techniques.
\nA key feature of my solution is its low computational cost for training. Training light-weight adapter modules takes only a few minutes, with the majority of computational resources allocated to inference.
\nAlthough the solution ultimately proved less competitive than fine-tuned approaches, I would like to highlight the potential of zero-shot inference. I hope this document inspires participants with limited computational resources to explore similar approaches.
The system consists of three main components, as illustrated in Fig.1:
\nQwen2.5-72B-Instruct-AWQ, equipped with logit processors guiding to output specific choice tokens.Details of each component are explained in the following sections.
\n
Fig.1: System Architecture
\nOne of the key challenges of this competition was the fact that 38% of the misconceptions in the test set were not present in the training dataset [1]. This design aligns with the competition's goal to develop models that can generalize to unseen misconceptions:
\n\n\n\"The goal is to create a model that not only aligns with known misconceptions but also generalizes to new, emerging misconceptions.\"
\n
To address this challenge, I implemented a custom cross-validation (CV) split that simulates the real-world training and test conditions. The dataset was divided into two subsets:
\ninner-test: 20% of the misconceptions (and their corresponding question-answer pairs) were randomly selected.inner-train: The remaining pairs that did not overlap with the selected misconceptions.After excluding samples without misconception labels, the inner-test set contained 1,719 question-answer pairs, while the inner-train set contained 2,651 pairs. This new split significantly improved the correlation between CV and LB scores.
This approach is similar to publicly shared ones[2], but the public approach were limited to the query domain. In contrast, my goal was to build models capable of aligning two distinct domains -- query and misconception. To achieve this, I applied zero-shot inference to both domains as follows:
SubjectName, ConstructName, QuestionText, AnswerText, and CorrectAnswerText.I used Qwen2.5-32B-Instruct-GPTQ-INT4 as the text generation model to implement this approach. The specific prompts used for text generation are detailed in Appendix A1.
I hypothesized that the text features already encapsulate specific details of both domains (query and misconception). Therefore, pre-computed embeddings from a general text embedding model, combined with lightweight MLP adapters, are sufficient to build high-quality retrieval models.
\nFor this purpose, I used gte-Qwen2-7B-Instruct to generate the pre-computed embeddings, which were subsequently fed into 2-3 layer MLPs for classification.
\nTraining was conducted in a manner similar to few-shot text classification (Fig.2), utilizing only in-batch negative samples to train a classifier for 2,587 misconceptions.
\nPre-computing the embeddings significantly reduces computational costs by eliminating repeated forward passes through the large embedding model. This optimization allows 5-fold models to be trained in just a few minutes.
\nDetails of the training parameters are provided in Appendix A2.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fd59836d45da7a0230f36c6f8c69e6f00%2Feedi-train-retriever.png?generation=1734073392563757&alt=media\")
Fig.2: Training Scheme of MLP Adapters: Fig.2 illustrates the MLP adapter training scheme, where blue squares denote positive pairs, white squares denote negative pairs, and gray squares represent masked regions excluded from loss calculation.
\nThe reranker module complements the retriever by incorporating cross-document features. While the retriever module processes only single-document features, the reranker refines the results by evaluating the top-25 documents retrieved and selecting the top-K most relevant ones using an LLM.
\nThe inference scheme follows approaches discussed in public discussions [3-5], with key differences outlined below:
\nlogprobs for up to the top 20 candidate tokens, providing more accurate probability estimates for each candidate.id=a<|im_end|>, which are split into [id, =a, <|im_end|>]. To ensure accurate token guidance, the reranker targets =a instead of a.I adopted the Qwen2.5-72B-Instruct-AWQ model for reranker. On the test dataset, inference for the reranker module takes approximately 2 hours, which balances computational efficiency and performance. Details about the specific prompts used are provided in Appendix A3.
The competition labels are divided into two groups: the 62% that appear in the training data and the 38% that do not [1]. I refer to these groups as seen and unseen, respectively.
\nEarly results suggested that the test set contained a higher proportion of unseen labels compared to seen labels, with an estimated ratio of 3:7 in the inner-test set. Given the observed correlation between CV scores and public leaderboard scores, I hypothesized that the public test set would exhibit a similar ratio.
\nTo address this bias, I applied an affine transformation to the similarity scores as follows:
\n$$
\n\\mathrm{similarity}(q_i, d_j) = \\begin{cases}
\nw \\times s(d_j) + b & \\text{if } d_j \\in \\mathrm{unseen} \\\\
\ns(d_j) & \\text{otherwise}
\n\\end{cases}
\n$$
For the inner-test set, the optimal parameters were (w=1.247 and b=1.772e-04).
| +MLP adapter | \n+increase unseen prob | \n+reranker | \n+ensemble | \nRecall@25-seen | \nRecall@25-unseen | \nRecall@25 | \nCV-seen | \nCV-unseen | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|---|---|---|---|---|---|---|
| \n | \n | \n | \n | \n | \n | 0.754 | \n\n | \n | 0.299 | \n\n | \n |
| ✔️ | \n\n | \n | \n | 0.907 | \n0.859 | \n0.873 | \n0.411 | \n0.388 | \n0.395 | \n\n | \n |
| ✔️ | \n✔️ | \n\n | \n | 0.864 | \n0.900 | \n0.890 | \n0.184 | \n0.518 | \n0.418 | \n\n | \n |
| ✔️ | \n✔️ | \n✔️ | \n\n | 0.831 | \n0.895 | \n0.876 | \n0.427 | \n0.557 | \n0.518 | \n\n | \n |
| ✔️ | \n✔️ | \n✔️ | \n✔️ | \n0.831 | \n0.895 | \n0.876 | \n0.362 | \n0.611 | \n0.537 | \n0.559 | \n0.495 | \n
The following shows a math problem given in elementary school the student\n\n**Subject**: {}\n\n**Construct**: {}\n\n**Question**: {}\n\n**Correct Answer**: {}\n\n**The Student\nThe following passage an interpretation given by a teacher regarding a student\n\n**Misconception**: {}\nEvaluate the student's misconception and rank the candidates likelihood.\nAmong the candidates, is the most likely to be the student's misconception?\nYou should only output the most relevant candidate ID number and other characters are allowed.\n\nOutput :\nid=(the most relevant candidate ID number)\n\n\n\n\n\n\n\n\n\n\n\n\n{}\nExample:
\nEvaluate student's misconception rank candidates likelihood.\nAmong candidates, which one most likely be student's misconception?\nYou should only output most relevant candidate ID no other are allowed.\n\nOutput Format:\n=( most relevant candidate ID )\n\n**Subject**: Multiplying Terms\n\n**Construct**: Multiply algebraic terms, both are linear versions different letters (e.g. a x b)\n\n**Question**: Simplify, possible:\n\\[\n a \\ b\n\\]\n\n**Correct Answer**: \\( a b \\)\n\n**Student's Incorrect Answer**: Does simplify\n\n**Candidates**:\n(=a) Multiplies coefficients letters separately writes an addition\n(=b) Does realise two letters together, a letter together, are multiplying\n(=c) Does understand how multiply algebraic terms\n(=d) Believes you add terms together when multiplying like terms.\n(=e) Does recognise distributive \n(=f) Adds multiplies when multiplying algebraic terms\n(=g) Thinks two variables next each other represent addition\n(=h) Does realise multiplying a variable itself results a power variable\n(=i) Confuses collecting like terms multiplication\n(=j) Thinks multiplying a variable a same raising variable power \n(=k) Does correctly apply commutative multiplication\n(=l) Does understand when like terms can be collected\n(=m) Multiplies all terms together when simplifying an expression\n(=n) When squaring a variable, believes they also need square coefficient\n(=o) Believes they can ignore coefficient when expanding double brackets\n(=p) Does understand concept inverse operations algebra\n(=q) Thinks we can remove a division symbol just variables next each other\n(=r) Does include brackets when attempting multiply an expression\n(=s) Thinks any values multiplied together can be denoted a squared symbol\n(=t) Thinks two letters next each other a division\n(=u) Adds rather than multiplies when asked a product\n(=v) Does know meaning \n(=w) Only multiplies numerical terms when expanding a bracket\n(=x) Does correctly apply distributive multiplication\n(=y) Mixes up squaring multiplying \n(=z) Thinks you apply one inverse operation one term other inverse operation other term\nThanks for being a source of inspiration.
", "votes": -1, "replies": [] }, { "id": 3073310, "author_name": "xianZ_waikato", "author_url": "", "post_date": "2024-12-16T10:24:58.960000", "content": "Impression on Text Generation part, thanks for share~
", "votes": 0, "replies": [] }, { "id": 3072876, "author_name": "MrSimple", "author_url": "", "post_date": "2024-12-15T17:56:01.633000", "content": "Thanks for being a source of inspiration.
", "votes": 0, "replies": [] }, { "id": 3071106, "author_name": "Digital-Munchikin", "author_url": "", "post_date": "2024-12-13T10:47:22.743000", "content": "thank you for sharing yout work! wonderful write-up as well, I can surely learn a lot from it.
", "votes": 0, "replies": [] }, { "id": 3071055, "author_name": "Octavi Grau", "author_url": "", "post_date": "2024-12-13T09:33:25.400000", "content": "Congrats for the solution and for an excellent write-up @tatamikenn !
", "votes": 0, "replies": [] }, { "id": 3071033, "author_name": "Vladimir Demidov", "author_url": "", "post_date": "2024-12-13T09:02:27.380000", "content": "What a professional, well-structured, literate write-up! Regarding the Reranker Module, I saw in the public notebooks a fixed value for the top 25 documents retrieved (performed by NearestNeighbors). Is its value tunable, or is it determined by other constraints?
I sincerely thank the Kaggle team and competition hosts for providing this valuable opportunity. Through this competition, I explored the latest advancements in large language models (LLMs) and deepened my understanding of zero-shot inference techniques.
\nA key feature of my solution is its low computational cost for training. Training light-weight adapter modules takes only a few minutes, with the majority of computational resources allocated to inference.
\nAlthough the solution ultimately proved less competitive than fine-tuned approaches, I would like to highlight the potential of zero-shot inference. I hope this document inspires participants with limited computational resources to explore similar approaches.
The system consists of three main components, as illustrated in Fig.1:
\nQwen2.5-72B-Instruct-AWQ, equipped with logit processors guiding to output specific choice tokens.Details of each component are explained in the following sections.
\n
Fig.1: System Architecture
\nOne of the key challenges of this competition was the fact that 38% of the misconceptions in the test set were not present in the training dataset [1]. This design aligns with the competition's goal to develop models that can generalize to unseen misconceptions:
\n\n\n\"The goal is to create a model that not only aligns with known misconceptions but also generalizes to new, emerging misconceptions.\"
\n
To address this challenge, I implemented a custom cross-validation (CV) split that simulates the real-world training and test conditions. The dataset was divided into two subsets:
\ninner-test: 20% of the misconceptions (and their corresponding question-answer pairs) were randomly selected.inner-train: The remaining pairs that did not overlap with the selected misconceptions.After excluding samples without misconception labels, the inner-test set contained 1,719 question-answer pairs, while the inner-train set contained 2,651 pairs. This new split significantly improved the correlation between CV and LB scores.
This approach is similar to publicly shared ones[2], but the public approach were limited to the query domain. In contrast, my goal was to build models capable of aligning two distinct domains -- query and misconception. To achieve this, I applied zero-shot inference to both domains as follows:
SubjectName, ConstructName, QuestionText, AnswerText, and CorrectAnswerText.I used Qwen2.5-32B-Instruct-GPTQ-INT4 as the text generation model to implement this approach. The specific prompts used for text generation are detailed in Appendix A1.
I hypothesized that the text features already encapsulate specific details of both domains (query and misconception). Therefore, pre-computed embeddings from a general text embedding model, combined with lightweight MLP adapters, are sufficient to build high-quality retrieval models.
\nFor this purpose, I used gte-Qwen2-7B-Instruct to generate the pre-computed embeddings, which were subsequently fed into 2-3 layer MLPs for classification.
\nTraining was conducted in a manner similar to few-shot text classification (Fig.2), utilizing only in-batch negative samples to train a classifier for 2,587 misconceptions.
\nPre-computing the embeddings significantly reduces computational costs by eliminating repeated forward passes through the large embedding model. This optimization allows 5-fold models to be trained in just a few minutes.
\nDetails of the training parameters are provided in Appendix A2.
\n
Fig.2: Training Scheme of MLP Adapters: Fig.2 illustrates the MLP adapter training scheme, where blue squares denote positive pairs, white squares denote negative pairs, and gray squares represent masked regions excluded from loss calculation.
\nThe reranker module complements the retriever by incorporating cross-document features. While the retriever module processes only single-document features, the reranker refines the results by evaluating the top-25 documents retrieved and selecting the top-K most relevant ones using an LLM.
\nThe inference scheme follows approaches discussed in public discussions [3-5], with key differences outlined below:
\nlogprobs for up to the top 20 candidate tokens, providing more accurate probability estimates for each candidate.id=a<|im_end|>, which are split into [id, =a, <|im_end|>]. To ensure accurate token guidance, the reranker targets =a instead of a.I adopted the Qwen2.5-72B-Instruct-AWQ model for reranker. On the test dataset, inference for the reranker module takes approximately 2 hours, which balances computational efficiency and performance. Details about the specific prompts used are provided in Appendix A3.
The competition labels are divided into two groups: the 62% that appear in the training data and the 38% that do not [1]. I refer to these groups as seen and unseen, respectively.
\nEarly results suggested that the test set contained a higher proportion of unseen labels compared to seen labels, with an estimated ratio of 3:7 in the inner-test set. Given the observed correlation between CV scores and public leaderboard scores, I hypothesized that the public test set would exhibit a similar ratio.
\nTo address this bias, I applied an affine transformation to the similarity scores as follows:
\n$$
\n\\mathrm{similarity}(q_i, d_j) = \\begin{cases}
\nw \\times s(d_j) + b & \\text{if } d_j \\in \\mathrm{unseen} \\\\
\ns(d_j) & \\text{otherwise}
\n\\end{cases}
\n$$
For the inner-test set, the optimal parameters were (w=1.247 and b=1.772e-04).
| +MLP adapter | \n+increase unseen prob | \n+reranker | \n+ensemble | \nRecall@25-seen | \nRecall@25-unseen | \nRecall@25 | \nCV-seen | \nCV-unseen | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|---|---|---|---|---|---|---|
| \n | \n | \n | \n | \n | \n | 0.754 | \n\n | \n | 0.299 | \n\n | \n |
| ✔️ | \n\n | \n | \n | 0.907 | \n0.859 | \n0.873 | \n0.411 | \n0.388 | \n0.395 | \n\n | \n |
| ✔️ | \n✔️ | \n\n | \n | 0.864 | \n0.900 | \n0.890 | \n0.184 | \n0.518 | \n0.418 | \n\n | \n |
| ✔️ | \n✔️ | \n✔️ | \n\n | 0.831 | \n0.895 | \n0.876 | \n0.427 | \n0.557 | \n0.518 | \n\n | \n |
| ✔️ | \n✔️ | \n✔️ | \n✔️ | \n0.831 | \n0.895 | \n0.876 | \n0.362 | \n0.611 | \n0.537 | \n0.559 | \n0.495 | \n
The following shows a math problem given in elementary school the student\n\n**Subject**: {}\n\n**Construct**: {}\n\n**Question**: {}\n\n**Correct Answer**: {}\n\n**The Student\nThe following passage an interpretation given by a teacher regarding a student\n\n**Misconception**: {}\nEvaluate the student's misconception and rank the candidates likelihood.\nAmong the candidates, is the most likely to be the student's misconception?\nYou should only output the most relevant candidate ID number and other characters are allowed.\n\nOutput :\nid=(the most relevant candidate ID number)\n\n\n\n\n\n\n\n\n\n\n\n\n{}\nExample:
\nEvaluate student's misconception rank candidates likelihood.\nAmong candidates, which one most likely be student's misconception?\nYou should only output most relevant candidate ID no other are allowed.\n\nOutput Format:\n=( most relevant candidate ID )\n\n**Subject**: Multiplying Terms\n\n**Construct**: Multiply algebraic terms, both are linear versions different letters (e.g. a x b)\n\n**Question**: Simplify, possible:\n\\[\n a \\ b\n\\]\n\n**Correct Answer**: \\( a b \\)\n\n**Student's Incorrect Answer**: Does simplify\n\n**Candidates**:\n(=a) Multiplies coefficients letters separately writes an addition\n(=b) Does realise two letters together, a letter together, are multiplying\n(=c) Does understand how multiply algebraic terms\n(=d) Believes you add terms together when multiplying like terms.\n(=e) Does recognise distributive \n(=f) Adds multiplies when multiplying algebraic terms\n(=g) Thinks two variables next each other represent addition\n(=h) Does realise multiplying a variable itself results a power variable\n(=i) Confuses collecting like terms multiplication\n(=j) Thinks multiplying a variable a same raising variable power \n(=k) Does correctly apply commutative multiplication\n(=l) Does understand when like terms can be collected\n(=m) Multiplies all terms together when simplifying an expression\n(=n) When squaring a variable, believes they also need square coefficient\n(=o) Believes they can ignore coefficient when expanding double brackets\n(=p) Does understand concept inverse operations algebra\n(=q) Thinks we can remove a division symbol just variables next each other\n(=r) Does include brackets when attempting multiply an expression\n(=s) Thinks any values multiplied together can be denoted a squared symbol\n(=t) Thinks two letters next each other a division\n(=u) Adds rather than multiplies when asked a product\n(=v) Does know meaning \n(=w) Only multiplies numerical terms when expanding a bracket\n(=x) Does correctly apply distributive multiplication\n(=y) Mixes up squaring multiplying \n(=z) Thinks you apply one inverse operation one term other inverse operation other term\nThanks for being a source of inspiration.
", "rawMarkdown": "Thanks for being a source of inspiration.", "votes": -1 }, { "id": 3073310, "postDate": "2024-12-16T10:24:58.960Z", "content": "Impression on Text Generation part, thanks for share~
", "rawMarkdown": "Impression on Text Generation part, thanks for share~" }, { "id": 3072876, "postDate": "2024-12-15T17:56:01.633Z", "content": "Thanks for being a source of inspiration.
", "rawMarkdown": "Thanks for being a source of inspiration.\n\n\n" }, { "id": 3071106, "postDate": "2024-12-13T10:47:22.743Z", "content": "thank you for sharing yout work! wonderful write-up as well, I can surely learn a lot from it.
", "rawMarkdown": "thank you for sharing yout work! wonderful write-up as well, I can surely learn a lot from it." }, { "id": 3071055, "postDate": "2024-12-13T09:33:25.400Z", "content": "Congrats for the solution and for an excellent write-up @tatamikenn !
", "rawMarkdown": "Congrats for the solution and for an excellent write-up @tatamikenn !" }, { "id": 3071033, "postDate": "2024-12-13T09:02:27.380Z", "content": "What a professional, well-structured, literate write-up! Regarding the Reranker Module, I saw in the public notebooks a fixed value for the top 25 documents retrieved (performed by NearestNeighbors). Is its value tunable, or is it determined by other constraints?