The notebook of the solution can be found here
\n\n\nWe fine-tuned two DeepSeek-Math-7B-RL models, one is used as policy model (to generate solutions), one as reward model (to score the solutions for weighted majority voting). The training infra is adopted from (Sun et al., 2024), bibtex is in the end.
\nOur dataset is a combination of AMC, AIME, and Odyssey-Math. We only select those problems of integer answers since the answer of the test problems will be integers. We also delete the choices of the multiple choice problem in AMC.
\nWe prompt GPT4 to sample code to solutions to the problems in our dataset using few-shot examples, and select the correct solutions as our training dataset for our policy model.
\nThe datasets can be found at our github AIMO-CMU-MATH.
\nWe finetune the model for 3 epochs with learning rate 2e-5.
\nWe use the problems in MATH, AIME, AMC and Odyssey-Math with nonnegative integer answers as our problem set for training the reward model.
\nOn MATH dataset, if we sample solutions with DeepSeek-MATH-7B RL, we will only get correct solutions for some problems, and those solutions can be similar. On the other hand, sampling solutions with DeepSeek-MATH-7B Base fails to output any correct solutions on some problems.
\nOn AMC, AIME and Odyssey, the observation is similar. When using DeepSeek-Math-7B-RL to sample solutions, most of the results are wrong; When using our fine-tuned policy model, the accuracy is high.
\nWe believe that in order to train a good reward model, the dataset should satisfies:
\nDiversity: The solutions for each question are quite different.
Label balance: The dataset contains both correct solutions (positive examples) and incorrect ones (negative examples) for each question.
Therefore, we adopt the following sample strategies.
\nFor MATH dataset, we interpolate the model parameters of DeepSeek-Math-7B RL and DeepSeek-Math-7B Base to get models with different level of capabilities. Denote the model parameters of DeepSeek-Math-7B RL and DeepSeek-Math-7B Base as $M_{RL}$ and $M_{Base}$, respectively. The interpolated model parameter $M_{\\alpha}$ is a linear combination of $M_{RL}$ and $M_{Base}$.
\n$$
\nM_{\\alpha}=\\alpha M_{RL} + (1-\\alpha)M_{Base}
\n$$
\nWe choose 8 different models with $\\alpha=0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0$. Each model generates two solutions for each question. The larger $\\alpha$ is, the more correct answers the model samples.
By interpolation, we can get different models which yield diverse solutions, and the proportion of positive labels and negative labels are much more balanced compare to sampling with only one model.
\nFor AMC, AIME, Odyssey, we use the training dataset we collect for the policy model, and train the DeepSeek-Math-RL 7B with 0, 1, 2, and 3 epoch(s) to get 4 different models. Each model generates 12 solutions for each problem.
\nIn this way, we obtain different solutions and each problems are more likely to have both correct solutions and wrong solutions.
\nOnce we collected all the solutions with labels, we do the filtering below:
\nWe find that the above methods don't result in models that perform better on our validation dataset. Therefore we use the methods described before.
\nFor the policy model, we use vLLM to speed up sampling and huggingface transformer for reward model. Scoring a solution is just a forward pass and doesn't take much time. Each model uses one GPU (T4).
\nWe sample 42 solutions for each problem, the policy model is asked to write code. Prompt is simply problem + tool_instruction.
\ntool_instruction = '\\nPlease integrate natural language reasoning with programs to solve the problem above, and put your final answer within \\\\boxed{}.'
If the result of code execution is not an integer, we give feedback and ask policy model to do it again, the code is below:
\n ():\n prompt = problem + tool_instruction\n texts = [prompt _ (num_sequences)]\n results = []\n result_texts = []\n i (rounds):\n sampling_params = SamplingParams(temperature=T, top_p=, n=, max_tokens=, stop=)\n generated_texts = llm.generate(texts, sampling_params)\n next_round_texts = []\n completed = []\n ((generated_texts))\n j, (text, generated_text) ((texts, generated_texts)):\n new_text = generated_text.outputs[].text\n completed = \n new_text:\n :\n code_text = new_text.split()[-].split()[]\n code_result, CODE_STATUS = process_code(code_text, return_shell_output=)\n :\n result = ((code_result.strip()))\n result >= :\n results.append(result % )\n result_texts.append((prompt + + new_text.split()[-]).split()[])\n completed = \n Exception e:\n code_result += \n Exception e:\n code_result = \n \n completed == :\n next_round_texts.append(text + new_text + + code_result + )\n\n texts = next_round_texts\n (, (texts))\n texts == []:\n \n results, result_texts\nWhere num_sequences was set to be 42, T=0.9, rounds=2.
\nThe outcome reward model (ORM) assigns a score to each solution. The input of the ORM is problem + code instead of problem + solution. We let the ORM to score the code generated by the policy model since the final candidate answers are simply execution results of the code.
\ndef (reward_model, tokenizer, text):\n input_id = torch.([tokenizer.(text)]).(reward_model.device)\n with torch.():\n logits = (input_id).logits\n score = logits[].(dim=-).().().()[-]\n return score\nTo calculate the total weight of one answer, we calculate it as the geometric mean times $N$, i.e. if $N$ solutions lead to the same answer $a$, the $i$-th solution has a score of $S_i$, the weight of answer a is:
\n$$
\nN(\\prod_{i=1}^n S_i)^{\\frac{1}{N}}.
\n$$
\nWe choose geometric mean times $N$ instead of the sum the weights because it prevents over-optimization. For example, in our experiment, the answer of the question \"There exists a unique increasing geometric sequence of five 2-digit positive integers. What is their sum?\" is 211. ORM gives score 0.75 to one solution with 211, but 0.85, 0.03, 0.15 to answer 50. The 0.85 can be a false positive of reward model and geometric mean prevents us from choosing such answer.
We also note that there is a tendency for the policy model to generate solutions with wrong answer 0, so we add some penalty for answer 0. Below is the code of weighted majority voting.
\n ():\n math\n max_weight = -\n number_weight_dict = {}\n max_weight_answer = \n answer, score (answers, scores):\n answer == :\n score /= \n answer number_weight_dict:\n number_weight_dict[answer] = [score]\n :\n number_weight_dict[answer].append(score)\n answer number_weight_dict.keys():\n score_list = number_weight_dict[answer]\n weight = (score_list) * math.prod(score_list) ** ( / (score_list))\n weight > max_weight:\n max_weight = weight\n max_weight_answer = answer\n max_weight_answer\nWe attribute the high performance of our notebook to the two points below:
\nIn the AIMO train set of 10 problems, with the fine-tuned policy model, the accuracy improves from 1/10 to 2/10, and the reward model further improves it from 2/10 to 4/10.
\nThe fine-tuning code base is adopted from the code of the paper below:
\n@article{sun2024easy,\n title={Easy-to-Hard Generalization: Alignment Human Supervision},\n author={Sun, Zhiqing Yu, Longhui Yikang Liu, Weiyang Yang, Yiming Welleck, Sean Gan, Chuang},\n preprint arXiv:.},\n year={}\n}\nUpdate: We are pleased to share the code and datasets used in our solution, including scripts for fine-tuning, dataset collection, and the complete training and validation datasets. With this repository, you can fully reproduce our solution. We hope it is helpful for your research and for the next round of AIMO.
\nPlease check the materials in our GitHub repository.
\nThanks to Kaggle and XTX Markets for organizing this wonderful competition!
", "votes": 13, "replies": [] }, { "id": 2913001, "author_name": "JM", "author_url": "", "post_date": "2024-07-09T07:00:12.110000", "content": "I had a similar ORM solution too but did not select it, what was your public LB score with this?
\nCongratulations on 2nd place!
", "votes": 3, "replies": [] }, { "id": 2914134, "author_name": "Gcdelgado", "author_url": "", "post_date": "2024-07-09T18:27:23.593000", "content": "Nice! Very good job my friend, thanks for sharing!
", "votes": 4, "replies": [] }, { "id": 2913074, "author_name": "Gusti Winata", "author_url": "", "post_date": "2024-07-09T08:22:57.927000", "content": "Can we have the scripts to train the reward and policy model?
", "votes": 0, "replies": [] }, { "id": 2919247, "author_name": "Ahmed Arham", "author_url": "", "post_date": "2024-07-12T18:53:19.480000", "content": "Nice! Very good job my friend, thanks for sharing!
\nAlso checkout my notebook!!
", "votes": 1, "replies": [] }, { "id": 2913049, "author_name": "Sparsh Tewatia", "author_url": "", "post_date": "2024-07-09T07:44:12.993000", "content": "Training scripts for reward model and validation sets if can be made public. Also did you find local cv and leaderboard correlation in your testing.
", "votes": 0, "replies": [ { "id": 2913586, "author_name": "Sparsh Tewatia", "author_url": "", "post_date": "2024-07-09T15:05:01.013000", "content": "Don't know the reason for downvote , it was stated in the competition that all details should be open sourced as to replicate the solution and which can help in research for next round of competition.
", "votes": 1, "replies": [ { "id": 2913628, "author_name": "Zhiqing SUN", "author_url": "", "post_date": "2024-07-09T15:29:38.953000", "content": "Thank you for your interest in our work! It was not downvoted by me. You can see the codebase of fine-tuning from the link in the citation part.
", "votes": 2, "replies": [] }, { "id": 2913884, "author_name": "Sparsh Tewatia", "author_url": "", "post_date": "2024-07-09T17:00:47.013000", "content": "Thanks , also if the validation sets can be released.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3107197, "author_name": "harrywh", "author_url": "", "post_date": "2025-01-26T06:03:04.797000", "content": "Nice sharing, follow you here from the bilibili webniar and it helpe me on the prize 2
", "votes": 0, "replies": [] }, { "id": 3025009, "author_name": "SegFault", "author_url": "", "post_date": "2024-10-22T08:58:16.957000", "content": "Can I run the NoteBook using Kaggle GPU T4x2? I am facing out of memory error
", "votes": 0, "replies": [] }, { "id": 2937578, "author_name": "Lady Gaga", "author_url": "", "post_date": "2024-07-27T08:10:32.587000", "content": "I want to ask why the labels during fine-tuning are the code of the solutions, rather than just the text of the solutions.
", "votes": 0, "replies": [] }, { "id": 2913739, "author_name": "Chew Kok Wah", "author_url": "", "post_date": "2024-07-09T16:08:14.727000", "content": "Please share the full working finetuning code for this competition, you had not yet share your Full finetuning code for both the policy and reward model .
", "votes": 0, "replies": [ { "id": 2913790, "author_name": "Zhiqing SUN", "author_url": "", "post_date": "2024-07-09T16:24:01.153000", "content": "Thanks for your interest. Please be patient, we are still working on cleaning the code, we will push the code to our github soon.
", "votes": 6, "replies": [ { "id": 2913910, "author_name": "Chew Kok Wah", "author_url": "", "post_date": "2024-07-09T17:22:58.553000", "content": "Thanks, please also share:
\nThe notebook of the solution can be found here
\n\n\nWe fine-tuned two DeepSeek-Math-7B-RL models, one is used as policy model (to generate solutions), one as reward model (to score the solutions for weighted majority voting). The training infra is adopted from (Sun et al., 2024), bibtex is in the end.
\nOur dataset is a combination of AMC, AIME, and Odyssey-Math. We only select those problems of integer answers since the answer of the test problems will be integers. We also delete the choices of the multiple choice problem in AMC.
\nWe prompt GPT4 to sample code to solutions to the problems in our dataset using few-shot examples, and select the correct solutions as our training dataset for our policy model.
\nThe datasets can be found at our github AIMO-CMU-MATH.
\nWe finetune the model for 3 epochs with learning rate 2e-5.
\nWe use the problems in MATH, AIME, AMC and Odyssey-Math with nonnegative integer answers as our problem set for training the reward model.
\nOn MATH dataset, if we sample solutions with DeepSeek-MATH-7B RL, we will only get correct solutions for some problems, and those solutions can be similar. On the other hand, sampling solutions with DeepSeek-MATH-7B Base fails to output any correct solutions on some problems.
\nOn AMC, AIME and Odyssey, the observation is similar. When using DeepSeek-Math-7B-RL to sample solutions, most of the results are wrong; When using our fine-tuned policy model, the accuracy is high.
\nWe believe that in order to train a good reward model, the dataset should satisfies:
\nDiversity: The solutions for each question are quite different.
Label balance: The dataset contains both correct solutions (positive examples) and incorrect ones (negative examples) for each question.
Therefore, we adopt the following sample strategies.
\nFor MATH dataset, we interpolate the model parameters of DeepSeek-Math-7B RL and DeepSeek-Math-7B Base to get models with different level of capabilities. Denote the model parameters of DeepSeek-Math-7B RL and DeepSeek-Math-7B Base as $M_{RL}$ and $M_{Base}$, respectively. The interpolated model parameter $M_{\\alpha}$ is a linear combination of $M_{RL}$ and $M_{Base}$.
\n$$
\nM_{\\alpha}=\\alpha M_{RL} + (1-\\alpha)M_{Base}
\n$$
\nWe choose 8 different models with $\\alpha=0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0$. Each model generates two solutions for each question. The larger $\\alpha$ is, the more correct answers the model samples.
By interpolation, we can get different models which yield diverse solutions, and the proportion of positive labels and negative labels are much more balanced compare to sampling with only one model.
\nFor AMC, AIME, Odyssey, we use the training dataset we collect for the policy model, and train the DeepSeek-Math-RL 7B with 0, 1, 2, and 3 epoch(s) to get 4 different models. Each model generates 12 solutions for each problem.
\nIn this way, we obtain different solutions and each problems are more likely to have both correct solutions and wrong solutions.
\nOnce we collected all the solutions with labels, we do the filtering below:
\nWe find that the above methods don't result in models that perform better on our validation dataset. Therefore we use the methods described before.
\nFor the policy model, we use vLLM to speed up sampling and huggingface transformer for reward model. Scoring a solution is just a forward pass and doesn't take much time. Each model uses one GPU (T4).
\nWe sample 42 solutions for each problem, the policy model is asked to write code. Prompt is simply problem + tool_instruction.
\ntool_instruction = '\\nPlease integrate natural language reasoning with programs to solve the problem above, and put your final answer within \\\\boxed{}.'
If the result of code execution is not an integer, we give feedback and ask policy model to do it again, the code is below:
\n ():\n prompt = problem + tool_instruction\n texts = [prompt _ (num_sequences)]\n results = []\n result_texts = []\n i (rounds):\n sampling_params = SamplingParams(temperature=T, top_p=, n=, max_tokens=, stop=)\n generated_texts = llm.generate(texts, sampling_params)\n next_round_texts = []\n completed = []\n ((generated_texts))\n j, (text, generated_text) ((texts, generated_texts)):\n new_text = generated_text.outputs[].text\n completed = \n new_text:\n :\n code_text = new_text.split()[-].split()[]\n code_result, CODE_STATUS = process_code(code_text, return_shell_output=)\n :\n result = ((code_result.strip()))\n result >= :\n results.append(result % )\n result_texts.append((prompt + + new_text.split()[-]).split()[])\n completed = \n Exception e:\n code_result += \n Exception e:\n code_result = \n \n completed == :\n next_round_texts.append(text + new_text + + code_result + )\n\n texts = next_round_texts\n (, (texts))\n texts == []:\n \n results, result_texts\nWhere num_sequences was set to be 42, T=0.9, rounds=2.
\nThe outcome reward model (ORM) assigns a score to each solution. The input of the ORM is problem + code instead of problem + solution. We let the ORM to score the code generated by the policy model since the final candidate answers are simply execution results of the code.
\ndef (reward_model, tokenizer, text):\n input_id = torch.([tokenizer.(text)]).(reward_model.device)\n with torch.():\n logits = (input_id).logits\n score = logits[].(dim=-).().().()[-]\n return score\nTo calculate the total weight of one answer, we calculate it as the geometric mean times $N$, i.e. if $N$ solutions lead to the same answer $a$, the $i$-th solution has a score of $S_i$, the weight of answer a is:
\n$$
\nN(\\prod_{i=1}^n S_i)^{\\frac{1}{N}}.
\n$$
\nWe choose geometric mean times $N$ instead of the sum the weights because it prevents over-optimization. For example, in our experiment, the answer of the question \"There exists a unique increasing geometric sequence of five 2-digit positive integers. What is their sum?\" is 211. ORM gives score 0.75 to one solution with 211, but 0.85, 0.03, 0.15 to answer 50. The 0.85 can be a false positive of reward model and geometric mean prevents us from choosing such answer.
We also note that there is a tendency for the policy model to generate solutions with wrong answer 0, so we add some penalty for answer 0. Below is the code of weighted majority voting.
\n ():\n math\n max_weight = -\n number_weight_dict = {}\n max_weight_answer = \n answer, score (answers, scores):\n answer == :\n score /= \n answer number_weight_dict:\n number_weight_dict[answer] = [score]\n :\n number_weight_dict[answer].append(score)\n answer number_weight_dict.keys():\n score_list = number_weight_dict[answer]\n weight = (score_list) * math.prod(score_list) ** ( / (score_list))\n weight > max_weight:\n max_weight = weight\n max_weight_answer = answer\n max_weight_answer\nWe attribute the high performance of our notebook to the two points below:
\nIn the AIMO train set of 10 problems, with the fine-tuned policy model, the accuracy improves from 1/10 to 2/10, and the reward model further improves it from 2/10 to 4/10.
\nThe fine-tuning code base is adopted from the code of the paper below:
\n@article{sun2024easy,\n title={Easy-to-Hard Generalization: Alignment Human Supervision},\n author={Sun, Zhiqing Yu, Longhui Yikang Liu, Weiyang Yang, Yiming Welleck, Sean Gan, Chuang},\n preprint arXiv:.},\n year={}\n}\nUpdate: We are pleased to share the code and datasets used in our solution, including scripts for fine-tuning, dataset collection, and the complete training and validation datasets. With this repository, you can fully reproduce our solution. We hope it is helpful for your research and for the next round of AIMO.
\nPlease check the materials in our GitHub repository.
\nThanks to Kaggle and XTX Markets for organizing this wonderful competition!
", "rawMarkdown": "Update: We are pleased to share the code and datasets used in our solution, including scripts for fine-tuning, dataset collection, and the complete training and validation datasets. With this repository, you can fully reproduce our solution. We hope it is helpful for your research and for the next round of AIMO.\n\nPlease check the materials in our [**GitHub repository**](https://github.com/AIMO-CMU-MATH/CMU_MATH-AIMO).\n\nThanks to Kaggle and XTX Markets for organizing this wonderful competition!", "votes": 13 }, { "id": 2913001, "postDate": "2024-07-09T07:00:12.110Z", "content": "I had a similar ORM solution too but did not select it, what was your public LB score with this?
\nCongratulations on 2nd place!
", "rawMarkdown": "I had a similar ORM solution too but did not select it, what was your public LB score with this?\n\nCongratulations on 2nd place!", "votes": 3 }, { "id": 2914134, "postDate": "2024-07-09T18:27:23.593Z", "content": "Nice! Very good job my friend, thanks for sharing!
", "rawMarkdown": "Nice! Very good job my friend, thanks for sharing!", "votes": 4 }, { "id": 2913074, "postDate": "2024-07-09T08:22:57.927Z", "content": "Can we have the scripts to train the reward and policy model?
", "rawMarkdown": "Can we have the scripts to train the reward and policy model?" }, { "id": 2919247, "postDate": "2024-07-12T18:53:19.480Z", "content": "Nice! Very good job my friend, thanks for sharing!
\nAlso checkout my notebook!!
", "rawMarkdown": "Nice! Very good job my friend, thanks for sharing!\n\n\nAlso checkout my notebook!!", "votes": 1 }, { "id": 2913049, "postDate": "2024-07-09T07:44:12.993Z", "content": "Training scripts for reward model and validation sets if can be made public. Also did you find local cv and leaderboard correlation in your testing.
", "rawMarkdown": "Training scripts for reward model and validation sets if can be made public. Also did you find local cv and leaderboard correlation in your testing.", "replies": [ { "id": 2913586, "postDate": "2024-07-09T15:05:01.013Z", "content": "Don't know the reason for downvote , it was stated in the competition that all details should be open sourced as to replicate the solution and which can help in research for next round of competition.
", "rawMarkdown": "Don't know the reason for downvote , it was stated in the competition that all details should be open sourced as to replicate the solution and which can help in research for next round of competition.", "votes": 1, "replies": [ { "id": 2913628, "postDate": "2024-07-09T15:29:38.953Z", "content": "Thank you for your interest in our work! It was not downvoted by me. You can see the codebase of fine-tuning from the link in the citation part.
", "rawMarkdown": "Thank you for your interest in our work! It was not downvoted by me. You can see the codebase of fine-tuning from the link in the citation part.", "votes": 2 }, { "id": 2913884, "postDate": "2024-07-09T17:00:47.013Z", "content": "Thanks , also if the validation sets can be released.
", "rawMarkdown": "Thanks , also if the validation sets can be released." } ] } ] }, { "id": 3107197, "postDate": "2025-01-26T06:03:04.797Z", "content": "Nice sharing, follow you here from the bilibili webniar and it helpe me on the prize 2
", "rawMarkdown": "Nice sharing, follow you here from the bilibili webniar and it helpe me on the prize 2" }, { "id": 3025009, "postDate": "2024-10-22T08:58:16.957Z", "content": "Can I run the NoteBook using Kaggle GPU T4x2? I am facing out of memory error
", "rawMarkdown": "Can I run the NoteBook using Kaggle GPU T4x2? I am facing out of memory error " }, { "id": 2937578, "postDate": "2024-07-27T08:10:32.587Z", "content": "I want to ask why the labels during fine-tuning are the code of the solutions, rather than just the text of the solutions.
", "rawMarkdown": "I want to ask why the labels during fine-tuning are the code of the solutions, rather than just the text of the solutions." }, { "id": 2913739, "postDate": "2024-07-09T16:08:14.727Z", "content": "Please share the full working finetuning code for this competition, you had not yet share your Full finetuning code for both the policy and reward model .
", "rawMarkdown": "Please share the full working finetuning code for this competition, you had not yet share your Full finetuning code for both the policy and reward model .", "replies": [ { "id": 2913790, "postDate": "2024-07-09T16:24:01.153Z", "content": "Thanks for your interest. Please be patient, we are still working on cleaning the code, we will push the code to our github soon.
", "rawMarkdown": "Thanks for your interest. Please be patient, we are still working on cleaning the code, we will push the code to our github soon.", "votes": 6, "replies": [ { "id": 2913910, "postDate": "2024-07-09T17:22:58.553Z", "content": "Thanks, please also share:
\n