Firstly, thanks to Kaggle organizers and my team mates.
\nI will briefly share our solution.
Our pipeline mainly follows flagembedding https://github.com/FlagOpen/FlagEmbedding. It can be divided into 2 stage: retrieval and reranker.
\n@sayoulala : https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519 Thanks for sharing, that's where a lot of our design of pipeline comes from.
\nIn addition to training data, we also use external datas from the disscussion https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/533764 and https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/541222, thanks for sharing. For convenience, we refer to them as ext1 and ext2.
\nWe divide the training process into 3 stage. First, we train the model using ext1, then we train the model using ext2, finally we train using training data.
\nWe observe that, maybe the ext1 has noise, so when we train the model using ext1+training data, the performance is worse than 2-stage training. And for ext2, because it lacks the construct name and subject name, we continue using the multi-stage training strategy.
\nWe obtain three trained retrieval models from each of the above three stages. And then We use these three models to create hard negative data for ext1, ext2, and training data, respectively. Then we also divide the training process of reranker into 3 stages, in each stage we use corresponding hn data to train.
\nWhen we train the 3rd stage of retrieval, we use negative cross device to enlarge the number of negatives for each positive. When we use it, public score 0.567->0.574.
\nWe use Qwen2.5-32B for retrieval and reranker. Because of the limit, we only rerank top 18 in inference.
\n| Model | \nPublic LB | \n
|---|---|
| retrieval | \n0.582 | \n
| retrieval+reranker | \n0.623 | \n
The main training code is provided in https://github.com/MilchstraB/Eedi
", "comments": [ { "id": 3070961, "author_name": "no fit just luck", "author_url": "", "post_date": "2024-12-13T07:30:51.807000", "content": "It's good to see a simple but effective method. Especially, by using two open source datasets can achieve such a nice LB and PB. May I ask how do you use MalAlgoQA, since there is no misconception in it. It can be observed that synthetic data is import for the competition, especially the extra data should cover all misconceptions. However, it is hard to find this point in your sharing method. May you sharing more about the synthetic data you used, since it may be the main point for get such a great PB.
", "votes": 1, "replies": [ { "id": 3070970, "author_name": "huangxiaojian", "author_url": "", "post_date": "2024-12-13T07:41:19.657000", "content": "For MalAlgoQA, I just treat the rationale of incorrect answer as misconception. In my experiments, the external datas can bring about 0.02~0.03 score improvement in retrieval stage. I think the data of high quality is most important, because the main gain comes from parameter adjustments when training on the training set, like number of negatives or batch size.
", "votes": 0, "replies": [ { "id": 3071001, "author_name": "no fit just luck", "author_url": "", "post_date": "2024-12-13T08:19:35.717000", "content": "Great job with such two dataset can get retrieval model for LB 0.582. We also tried using the first dataset you mentioned, but it does not work. And maybe MalAlgoQA is the salient reason, although it guide model on predicting different misconceptions, which usually is not useful for improving model performance.
", "votes": 1, "replies": [] }, { "id": 3071013, "author_name": "huangxiaojian", "author_url": "", "post_date": "2024-12-13T08:33:51.580000", "content": "We first using ext1 to train retrieval. The model we used for training is Qwen-2.5-7B-math. Then we using the trained model to relabel ext1, which caculates the similarity of ext1 and misconceptions the competition offered.
", "votes": 0, "replies": [] }, { "id": 3071030, "author_name": "no fit just luck", "author_url": "", "post_date": "2024-12-13T08:57:49.097000", "content": "May I ask more clearly. You mean you used ext1 dataset trained Qwen-2.5-7B-math, and then used the model relabel ext1, in order to correct the original misconception in ext1? It's a method we have not considered before, learn a lot from it.
", "votes": 1, "replies": [] }, { "id": 3071032, "author_name": "huangxiaojian", "author_url": "", "post_date": "2024-12-13T09:01:08.677000", "content": "Yes, we follow the 3-stage training strategy to train Qwen-2.5-7B-math and then relabel ext1.
", "votes": 0, "replies": [] }, { "id": 3071035, "author_name": "no fit just luck", "author_url": "", "post_date": "2024-12-13T09:10:26.130000", "content": "Maybe we can give it a try next time, though it does seem a bit strange. The method you used seems like it could replace the pseudo-labeling approach. What you need to do is train a model on the entire dataset first, then relabel the dataset and train it again. It seems like the model is just relearning what it has already learned, including any biases. I am not sure why it works so well, resulting in your retrieval model achieving such a high score, especially given that it is based on two datasets that are not of high quality.
", "votes": 1, "replies": [] }, { "id": 3071151, "author_name": "Yu Chengzhi", "author_url": "", "post_date": "2024-12-13T12:11:12.257000", "content": "In fact, this approach is very effective in correcting noise in the data set, especially for low-quality data sets. At the same time, there will be a lot of work on self-improvement.
", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 8, "messages": [ { "id": 3070833, "postDate": "2024-12-13T05:02:08.237Z", "content": "Firstly, thanks to Kaggle organizers and my team mates.
\nI will briefly share our solution.
Our pipeline mainly follows flagembedding https://github.com/FlagOpen/FlagEmbedding. It can be divided into 2 stage: retrieval and reranker.
\n@sayoulala : https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519 Thanks for sharing, that's where a lot of our design of pipeline comes from.
\nIn addition to training data, we also use external datas from the disscussion https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/533764 and https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/541222, thanks for sharing. For convenience, we refer to them as ext1 and ext2.
\nWe divide the training process into 3 stage. First, we train the model using ext1, then we train the model using ext2, finally we train using training data.
\nWe observe that, maybe the ext1 has noise, so when we train the model using ext1+training data, the performance is worse than 2-stage training. And for ext2, because it lacks the construct name and subject name, we continue using the multi-stage training strategy.
\nWe obtain three trained retrieval models from each of the above three stages. And then We use these three models to create hard negative data for ext1, ext2, and training data, respectively. Then we also divide the training process of reranker into 3 stages, in each stage we use corresponding hn data to train.
\nWhen we train the 3rd stage of retrieval, we use negative cross device to enlarge the number of negatives for each positive. When we use it, public score 0.567->0.574.
\nWe use Qwen2.5-32B for retrieval and reranker. Because of the limit, we only rerank top 18 in inference.
\n| Model | \nPublic LB | \n
|---|---|
| retrieval | \n0.582 | \n
| retrieval+reranker | \n0.623 | \n
The main training code is provided in https://github.com/MilchstraB/Eedi
", "rawMarkdown": "Firstly, thanks to Kaggle organizers and my team mates.\nI will briefly share our solution.\n\n## Overview\nOur pipeline mainly follows flagembedding https://github.com/FlagOpen/FlagEmbedding. It can be divided into 2 stage: retrieval and reranker. \n\n@sayoulala : https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519 Thanks for sharing, that's where a lot of our design of pipeline comes from.\n\n## Retrieval\nIn addition to training data, we also use external datas from the disscussion https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/533764 and https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/541222, thanks for sharing. For convenience, we refer to them as ext1 and ext2.\n\n\nWe divide the training process into 3 stage. First, we train the model using ext1, then we train the model using ext2, finally we train using training data. \n\n\nWe observe that, maybe the ext1 has noise, so when we train the model using ext1+training data, the performance is worse than 2-stage training. And for ext2, because it lacks the construct name and subject name, we continue using the multi-stage training strategy. \n\n\n## Reranker\nWe obtain three trained retrieval models from each of the above three stages. And then We use these three models to create hard negative data for ext1, ext2, and training data, respectively. Then we also divide the training process of reranker into 3 stages, in each stage we use corresponding hn data to train.\n\n## Some Details\nWhen we train the 3rd stage of retrieval, we use negative cross device to enlarge the number of negatives for each positive. When we use it, public score 0.567->0.574.\n\nWe use Qwen2.5-32B for retrieval and reranker. Because of the limit, we only rerank top 18 in inference. \n\n| Model | Public LB |\n|----------|----------|\n| retrieval | 0.582 |\n| retrieval+reranker | 0.623 |\n\n## Code\nThe main training code is provided in https://github.com/MilchstraB/Eedi\n", "votes": 13 }, { "id": 3070961, "postDate": "2024-12-13T07:30:51.807Z", "content": "It's good to see a simple but effective method. Especially, by using two open source datasets can achieve such a nice LB and PB. May I ask how do you use MalAlgoQA, since there is no misconception in it. It can be observed that synthetic data is import for the competition, especially the extra data should cover all misconceptions. However, it is hard to find this point in your sharing method. May you sharing more about the synthetic data you used, since it may be the main point for get such a great PB.
", "rawMarkdown": "It's good to see a simple but effective method. Especially, by using two open source datasets can achieve such a nice LB and PB. May I ask how do you use [MalAlgoQA](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/541222), since there is no misconception in it. It can be observed that synthetic data is import for the competition, especially the extra data should cover all misconceptions. However, it is hard to find this point in your sharing method. May you sharing more about the synthetic data you used, since it may be the main point for get such a great PB. ", "votes": 1, "replies": [ { "id": 3070970, "postDate": "2024-12-13T07:41:19.657Z", "content": "For MalAlgoQA, I just treat the rationale of incorrect answer as misconception. In my experiments, the external datas can bring about 0.02~0.03 score improvement in retrieval stage. I think the data of high quality is most important, because the main gain comes from parameter adjustments when training on the training set, like number of negatives or batch size.
", "rawMarkdown": "For MalAlgoQA, I just treat the rationale of incorrect answer as misconception. In my experiments, the external datas can bring about 0.02~0.03 score improvement in retrieval stage. I think the data of high quality is most important, because the main gain comes from parameter adjustments when training on the training set, like number of negatives or batch size.", "replies": [ { "id": 3071001, "postDate": "2024-12-13T08:19:35.717Z", "content": "Great job with such two dataset can get retrieval model for LB 0.582. We also tried using the first dataset you mentioned, but it does not work. And maybe MalAlgoQA is the salient reason, although it guide model on predicting different misconceptions, which usually is not useful for improving model performance.
", "rawMarkdown": "Great job with such two dataset can get retrieval model for LB 0.582. We also tried using the first dataset you mentioned, but it does not work. And maybe MalAlgoQA is the salient reason, although it guide model on predicting different misconceptions, which usually is not useful for improving model performance. ", "votes": 1 }, { "id": 3071013, "postDate": "2024-12-13T08:33:51.580Z", "content": "We first using ext1 to train retrieval. The model we used for training is Qwen-2.5-7B-math. Then we using the trained model to relabel ext1, which caculates the similarity of ext1 and misconceptions the competition offered.
", "rawMarkdown": "We first using ext1 to train retrieval. The model we used for training is Qwen-2.5-7B-math. Then we using the trained model to relabel ext1, which caculates the similarity of ext1 and misconceptions the competition offered. " }, { "id": 3071030, "postDate": "2024-12-13T08:57:49.097Z", "content": "May I ask more clearly. You mean you used ext1 dataset trained Qwen-2.5-7B-math, and then used the model relabel ext1, in order to correct the original misconception in ext1? It's a method we have not considered before, learn a lot from it.
", "rawMarkdown": "May I ask more clearly. You mean you used ext1 dataset trained Qwen-2.5-7B-math, and then used the model relabel ext1, in order to correct the original misconception in ext1? It's a method we have not considered before, learn a lot from it.", "votes": 1 }, { "id": 3071032, "postDate": "2024-12-13T09:01:08.677Z", "content": "Yes, we follow the 3-stage training strategy to train Qwen-2.5-7B-math and then relabel ext1.
", "rawMarkdown": "Yes, we follow the 3-stage training strategy to train Qwen-2.5-7B-math and then relabel ext1." }, { "id": 3071035, "postDate": "2024-12-13T09:10:26.130Z", "content": "Maybe we can give it a try next time, though it does seem a bit strange. The method you used seems like it could replace the pseudo-labeling approach. What you need to do is train a model on the entire dataset first, then relabel the dataset and train it again. It seems like the model is just relearning what it has already learned, including any biases. I am not sure why it works so well, resulting in your retrieval model achieving such a high score, especially given that it is based on two datasets that are not of high quality.
", "rawMarkdown": "Maybe we can give it a try next time, though it does seem a bit strange. The method you used seems like it could replace the pseudo-labeling approach. What you need to do is train a model on the entire dataset first, then relabel the dataset and train it again. It seems like the model is just relearning what it has already learned, including any biases. I am not sure why it works so well, resulting in your retrieval model achieving such a high score, especially given that it is based on two datasets that are not of high quality.", "votes": 1 }, { "id": 3071151, "postDate": "2024-12-13T12:11:12.257Z", "content": "In fact, this approach is very effective in correcting noise in the data set, especially for low-quality data sets. At the same time, there will be a lot of work on self-improvement.
", "rawMarkdown": "In fact, this approach is very effective in correcting noise in the data set, especially for low-quality data sets. At the same time, there will be a lot of work on self-improvement." } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:47:09Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12610193, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/h-y-l-private-11th-public-9th-place-solution-summa" }