Code for our final solution can be found here.
\nOur final selected solution was an adaptation of the notebook of AbdurRafae. We used DeepSeek-Math-7B-RL without any finetuning and did majority voting with a scoring rule we made.
\nWe decided to go with vLLM due to the better speed when compared to the Huggingface Transformers.
\nWe generate the solutions in one batch using iterations. An iteration is made for each code that has to be run.
\nRegarding the scoring rule we found that at least on our validation, the model often had two types of errors as the final result:
\nWe thus penalized these results as a percentage of the total number of generations we attempted. We have observed a significant boost in score by penalizing numbers from the problem statement.
\nFor the final submission we chose this old version due to the possibility of having ties in the final leaderboard.
\nWe have also created a number of different versions.
We observed that running the model in BF16 improved performance and thus we tried to make our solution work with BF16. The problem is that vLLM does not support this by default on the T4s. One can however perform the computations in FP32 and cast back to BF16.
\nWe have thus slightly modified the vLLM library code to remove the check for BF16 support and wrapped the attention module so we convert back and forth between fp32 and bf16.
\nWe thus have two versions of the libraries:
\nWe found that the making the MLP be in bf16 made the model very slow and thus we also created the version that keeps everything but the MLP in bf16.
\nModels with these libraries performed well but were not stable enough for our liking. They gave us the highest score we have on our validation but did not go over 25 on the public leaderboard.
We observed that making multiple calls to vllm took a significant amount of time, further one cannot early exit reliably with intermediate results of solutions. We have thus made a new version of the notebook we initially scored 26-27 with that feeds the result of the code generation directly back to the model without needing two vllm generate calls.
\nWe do this by using the logit processors and forcing the model to output the tokens of the code output. We also enforce all early breaking and pruning rules directly in the logit processor function.
\nCode for this can be found here
\nWe personally liked this submission more than our final one but due to time constraints we could not make it score as high as the old one
Numbers which are small (<), this often happened because errors code.\n Numbers which were part problem statement. We found this be especially pronounced times this happened outweighed chances problem statement actually contain answer.\nNice observation! We also did something very similar for postprocessing ๐ on lucky runs it improved CV by up till 6-7 points!
\nAIME 2022-24 we observed from a trial run that 25 out of 61 problems generated a single digit answer as the most frequent vote itself! Its not code errors but a model behaviour LOL
", "votes": 4, "replies": [ { "id": 2906161, "author_name": "David Dinucu-Jianu", "author_url": "", "post_date": "2024-07-05T11:54:23.987000", "content": "Interestingly at least from our tests most of the wrong small number solutions were also part of the problem statement.
\nOur other submission which only penalized 0 and applied the second rule got 20 on private instead of 21.
We are using transformers.generate not vLLM so probably made a larger impact lol since the number of iterations is lower.
\nWe tried vLLM but didnโt get more than 13 (let alone the mysterious timeouts which we got from vLLM), even for runs where the CV is high (>25/50 for old AIME on Kaggle kernel) - donโt know what actually went wrong ๐ 2 subs per day is just not enough to trial and error by shooting in the dark. I looked at other implementations of vLLM that scored 20+ on public after the comp - still not sure how is it much different from ours that fixes the โbugโ ๐ it still confuses me why it works well
", "votes": 4, "replies": [] } ] } ] }, { "id": 2909770, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2024-07-07T09:46:51.457000", "content": "Congratulations! Great ML engineering. \"We found that the making the MLP be in bf16 made the model very slow\" --- What profiling tools have you used to detect and troubleshoot performance issues?
", "votes": 2, "replies": [ { "id": 2909795, "author_name": "David Dinucu-Jianu", "author_url": "", "post_date": "2024-07-07T10:15:45.880000", "content": "We just measured how many generations it could do on average on our validation set on Kaggle.
\nThe process was a bit slow since we did not use T4s while making the modification so we had to upload it to kaggle to test it.
Congratulations on achieving 3rd position in this competition. Thanks for sharing details of your solution.
\nVery interesting and innovative indeed.
Seems like you batch run the inferences, and then batch run the codes, and then repeat until the time is up.
\nIf my interpretation is correct, the LLM is idle when the code is running.
\nHave your tried building a system that runs another batch of inferences while the code is running?
", "votes": 0, "replies": [ { "id": 2906420, "author_name": "David Dinucu-Jianu", "author_url": "", "post_date": "2024-07-05T15:13:44.647000", "content": "Yes indeed the GPUs are idle during code execution. We did not try to do them asynchronously because the speed was already sufficient (on our validation set more generations did not improve the score).
\nAlthough while trying to make the BF16 submission work we did experiment with the AsyncLLM thing from vLLM but abandoned it cause it was close to the competition deadline.
", "votes": 1, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 2906109, "postDate": "2024-07-05T11:01:11.183Z", "content": "Code for our final solution can be found here.
\nOur final selected solution was an adaptation of the notebook of AbdurRafae. We used DeepSeek-Math-7B-RL without any finetuning and did majority voting with a scoring rule we made.
\nWe decided to go with vLLM due to the better speed when compared to the Huggingface Transformers.
\nWe generate the solutions in one batch using iterations. An iteration is made for each code that has to be run.
\nRegarding the scoring rule we found that at least on our validation, the model often had two types of errors as the final result:
\nWe thus penalized these results as a percentage of the total number of generations we attempted. We have observed a significant boost in score by penalizing numbers from the problem statement.
\nFor the final submission we chose this old version due to the possibility of having ties in the final leaderboard.
\nWe have also created a number of different versions.
We observed that running the model in BF16 improved performance and thus we tried to make our solution work with BF16. The problem is that vLLM does not support this by default on the T4s. One can however perform the computations in FP32 and cast back to BF16.
\nWe have thus slightly modified the vLLM library code to remove the check for BF16 support and wrapped the attention module so we convert back and forth between fp32 and bf16.
\nWe thus have two versions of the libraries:
\nWe found that the making the MLP be in bf16 made the model very slow and thus we also created the version that keeps everything but the MLP in bf16.
\nModels with these libraries performed well but were not stable enough for our liking. They gave us the highest score we have on our validation but did not go over 25 on the public leaderboard.
We observed that making multiple calls to vllm took a significant amount of time, further one cannot early exit reliably with intermediate results of solutions. We have thus made a new version of the notebook we initially scored 26-27 with that feeds the result of the code generation directly back to the model without needing two vllm generate calls.
\nWe do this by using the logit processors and forcing the model to output the tokens of the code output. We also enforce all early breaking and pruning rules directly in the logit processor function.
\nCode for this can be found here
\nWe personally liked this submission more than our final one but due to time constraints we could not make it score as high as the old one
Numbers which are small (<), this often happened because errors code.\n Numbers which were part problem statement. We found this be especially pronounced times this happened outweighed chances problem statement actually contain answer.\nNice observation! We also did something very similar for postprocessing ๐ on lucky runs it improved CV by up till 6-7 points!
\nAIME 2022-24 we observed from a trial run that 25 out of 61 problems generated a single digit answer as the most frequent vote itself! Its not code errors but a model behaviour LOL
", "rawMarkdown": "```\n1. Numbers which are small (<10), this often happened because of errors in the code.\n2. Numbers which were part of the problem statement. We found this to be especially pronounced and the number of times this happened outweighed the chances for the problem statement to actually contain the answer.\n```\n\nNice observation! We also did something very similar for postprocessing ๐ on lucky runs it improved CV by up till 6-7 points!\n\nAIME 2022-24 we observed from a trial run that 25 out of 61 problems generated a single digit answer as the most frequent vote itself! Its not code errors but a model behaviour LOL", "votes": 4, "replies": [ { "id": 2906161, "postDate": "2024-07-05T11:54:23.987Z", "content": "Interestingly at least from our tests most of the wrong small number solutions were also part of the problem statement.
\nOur other submission which only penalized 0 and applied the second rule got 20 on private instead of 21.
We are using transformers.generate not vLLM so probably made a larger impact lol since the number of iterations is lower.
\nWe tried vLLM but didnโt get more than 13 (let alone the mysterious timeouts which we got from vLLM), even for runs where the CV is high (>25/50 for old AIME on Kaggle kernel) - donโt know what actually went wrong ๐ 2 subs per day is just not enough to trial and error by shooting in the dark. I looked at other implementations of vLLM that scored 20+ on public after the comp - still not sure how is it much different from ours that fixes the โbugโ ๐ it still confuses me why it works well
", "rawMarkdown": "We are using transformers.generate not vLLM so probably made a larger impact lol since the number of iterations is lower. \n\nWe tried vLLM but didnโt get more than 13 (let alone the mysterious timeouts which we got from vLLM), even for runs where the CV is high (>25/50 for old AIME on Kaggle kernel) - donโt know what actually went wrong ๐ 2 subs per day is just not enough to trial and error by shooting in the dark. I looked at other implementations of vLLM that scored 20+ on public after the comp - still not sure how is it much different from ours that fixes the โbugโ ๐ it still confuses me why it works well", "votes": 4 } ] } ] }, { "id": 2909770, "postDate": "2024-07-07T09:46:51.457Z", "content": "Congratulations! Great ML engineering. \"We found that the making the MLP be in bf16 made the model very slow\" --- What profiling tools have you used to detect and troubleshoot performance issues?
", "rawMarkdown": "Congratulations! Great ML engineering. \"We found that the making the MLP be in bf16 made the model very slow\" --- What profiling tools have you used to detect and troubleshoot performance issues?", "votes": 2, "replies": [ { "id": 2909795, "postDate": "2024-07-07T10:15:45.880Z", "content": "We just measured how many generations it could do on average on our validation set on Kaggle.
\nThe process was a bit slow since we did not use T4s while making the modification so we had to upload it to kaggle to test it.
Congratulations on achieving 3rd position in this competition. Thanks for sharing details of your solution.
\nVery interesting and innovative indeed.
Seems like you batch run the inferences, and then batch run the codes, and then repeat until the time is up.
\nIf my interpretation is correct, the LLM is idle when the code is running.
\nHave your tried building a system that runs another batch of inferences while the code is running?
", "rawMarkdown": "Seems like you batch run the inferences, and then batch run the codes, and then repeat until the time is up.\n\nIf my interpretation is correct, the LLM is idle when the code is running.\n\nHave your tried building a system that runs another batch of inferences while the code is running?", "replies": [ { "id": 2906420, "postDate": "2024-07-05T15:13:44.647Z", "content": "Yes indeed the GPUs are idle during code execution. We did not try to do them asynchronously because the speed was already sufficient (on our validation set more generations did not improve the score).
\nAlthough while trying to make the BF16 submission work we did experiment with the AsyncLLM thing from vLLM but abandoned it cause it was close to the competition deadline.
", "rawMarkdown": "Yes indeed the GPUs are idle during code execution. We did not try to do them asynchronously because the speed was already sufficient (on our validation set more generations did not improve the score).\n\nAlthough while trying to make the BF16 submission work we did experiment with the AsyncLLM thing from vLLM but abandoned it cause it was close to the competition deadline.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T06:54:20Z", "competition_id": 73231, "competition_slug": "ai-mathematical-olympiad-prize", "team_id": 11919070, "solution_url": "/competitions/ai-mathematical-olympiad-prize/writeups/after-exams-3rd-place-solution" }