Similar to others, I'll share my road to public 28, while awaiting the final results.
\nOne of the first things I did was collect another 40 problems from the OpenR1-Math-220k. I selected hard problems with longer reasoning traces. These, plus the reference problems became my evaluation set for local eval.
\nLocal evaluation was done based on the percentage of sequences that completed and yielded a correct answer. After generating many solutions, I ran some bootstrapping and determined that the 14B Deepseek model seemed to be the optimal balance between majority voting and quality. Standard deviation seemed to be in the range of $\\pm 2-3$ problems on a 50 set with the majority vote, so evaluating models for me was based on the percentage that would be correct.
\nThat in hand, it was clear the best thing to do was to devote more time to longer problems. I had a similar solution to @optimo and smart stopping https://www.kaggle.com/code/optimo/fork-of-smart-stopping-14b-clean. I couldn't get the later vLLMs to properly abort, but I see he used 0.8.2 so maybe that was fixed in the later versions. I used 0.7.2. Once some number of solutions agreed, it would terminate the run. Similarly if we had a solution and things were taking a long time, it would stop early.
\nFinally, it seemed AWQ quantization ran the fastest, so I stuck with that method. On average it would be able to complete 6-7 answers in 12288 tokens.
\nSimilar to the Light-R1 paper, I selected a subset of hard problems from the OpenR1-Math-220 set and did an additional few epochs of training. This boosted the per sequence correctness fraction from 0.29 to 0.35, a 20% increase. Nothing fancy, just additional distillation.
\nAgain using the OpenR1-Math-220, I attempted to decrease the average number of tokens per response with a similar method to https://novasky-ai.github.io/posts/reduce-overthinking/. In the end, this did not increase the number of sequences terminated, nor improve the accuracy of the results. This was discarded.
\nI additionally tried to follow paper TokenSkip: Controllable Chain-of-Thought Compression in LLMs. If we could compress the output by 0.5-0.7, this should make the reasoning tighter (less attention to worry about) and faster. After compressing some of the responses from OpenR1-Math-220k, we ran it through some fine tuning rounds. Unfortunately, there was a catastrophic forgetting phase in the training that wiped out some of the knowledge, and while the number of tokens went down as expected, so did the accuracy.
\nI tried training on different subtasks (Algebra, Geometry, etc) and merging, but again, there was a decrease in accuracy as a result.
\nI think the TokenSkip idea is elegant. Math notation is very compact, and since we are resource bound any decrease in the number of tokens to generate would really help in a competition like this, where efficiency is valued. Some study of the vocabulary and more reward shaping could convert some of the extra blathering that humans find nice in chat responses to math notation and allow better solutions, or a translation method from the CoT produced in the OpenR1 set to compact notation.
\nLate in the competition ByteDance had and interesting paper DAPO. I think it would be a better than the GRPO used now, however I didn't have the time or resources to follow up on it. It would also be incremental gains over the current distilled models. Their paper is training from a baseline with RL, rather than distilling a much larger model.
\nWe'll see how things go with the results!
", "comments": [ { "id": 3172163, "author_name": "AI-Cat", "author_url": "", "post_date": "2025-04-06T14:27:00", "content": "All of my top five notebooks with LB=29 all use vLLM=0.7.2.
", "votes": 1, "replies": [] }, { "id": 3170642, "author_name": "ducnh279", "author_url": "", "post_date": "2025-04-04T21:27:19.047000", "content": "Thanks for sharing this, @danielphalen! Bootstrapping is actually a great practice I learned from you.
\nRegarding the TokenSkip paper - while I really enjoy novel ideas like that, it reminded me of when I read it about a month ago, and it didn’t fully make sense to me at the time.
\nI used to think that every token generated by a reasoning model plays a crucial role in reaching the final answer. Especially in reinforcement learning setups like DeepSeek’s, where sparse rewards (e.g., based only on the final answer) are used, it feels natural to assume the model is incentivized to make each token contribute meaningfully to the overall return.
\nThat’s why methods like TokenSkip initially felt counterintuitive to me 😅
\nBut on second thought, it could make sense: not every token may be equally useful for reward propagation, and skipping low-impact or redundant tokens might actually help the model focus its capacity on the parts that matter most.
\nIn the end, I think using techniques like adding based-length rewards during RL training, as some teams/papers have done, offers a good balance - encouraging concise yet effective reasoning without skipping too aggressively. By considering long-term rewards, the model can prioritize tokens that contribute to the overall success of the reasoning process, while still ensuring that the final answer is reached efficiently.
", "votes": 1, "replies": [] }, { "id": 3170641, "author_name": "M4XD", "author_url": "", "post_date": "2025-04-04T21:26:34.183000", "content": "Reducing overthinking is an actual problem for humans, too 😄
", "votes": 1, "replies": [] }, { "id": 3231646, "author_name": "Anastasiya Kotelnikova", "author_url": "", "post_date": "2025-06-24T17:18:37.310000", "content": "Thanks for sharing such a detailed breakdown! It's incredibly valuable to read about both what worked and what didn’t — especially your reflections on TokenSkip, smart stopping, and distillation trade-offs. The compact notation and AWQ quantization insights were great takeaways. Looking forward to seeing how your ideas evolve in future competitions!
", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 3170592, "postDate": "2025-04-04T20:33:13.063Z", "content": "Similar to others, I'll share my road to public 28, while awaiting the final results.
\nOne of the first things I did was collect another 40 problems from the OpenR1-Math-220k. I selected hard problems with longer reasoning traces. These, plus the reference problems became my evaluation set for local eval.
\nLocal evaluation was done based on the percentage of sequences that completed and yielded a correct answer. After generating many solutions, I ran some bootstrapping and determined that the 14B Deepseek model seemed to be the optimal balance between majority voting and quality. Standard deviation seemed to be in the range of $\\pm 2-3$ problems on a 50 set with the majority vote, so evaluating models for me was based on the percentage that would be correct.
\nThat in hand, it was clear the best thing to do was to devote more time to longer problems. I had a similar solution to @optimo and smart stopping https://www.kaggle.com/code/optimo/fork-of-smart-stopping-14b-clean. I couldn't get the later vLLMs to properly abort, but I see he used 0.8.2 so maybe that was fixed in the later versions. I used 0.7.2. Once some number of solutions agreed, it would terminate the run. Similarly if we had a solution and things were taking a long time, it would stop early.
\nFinally, it seemed AWQ quantization ran the fastest, so I stuck with that method. On average it would be able to complete 6-7 answers in 12288 tokens.
\nSimilar to the Light-R1 paper, I selected a subset of hard problems from the OpenR1-Math-220 set and did an additional few epochs of training. This boosted the per sequence correctness fraction from 0.29 to 0.35, a 20% increase. Nothing fancy, just additional distillation.
\nAgain using the OpenR1-Math-220, I attempted to decrease the average number of tokens per response with a similar method to https://novasky-ai.github.io/posts/reduce-overthinking/. In the end, this did not increase the number of sequences terminated, nor improve the accuracy of the results. This was discarded.
\nI additionally tried to follow paper TokenSkip: Controllable Chain-of-Thought Compression in LLMs. If we could compress the output by 0.5-0.7, this should make the reasoning tighter (less attention to worry about) and faster. After compressing some of the responses from OpenR1-Math-220k, we ran it through some fine tuning rounds. Unfortunately, there was a catastrophic forgetting phase in the training that wiped out some of the knowledge, and while the number of tokens went down as expected, so did the accuracy.
\nI tried training on different subtasks (Algebra, Geometry, etc) and merging, but again, there was a decrease in accuracy as a result.
\nI think the TokenSkip idea is elegant. Math notation is very compact, and since we are resource bound any decrease in the number of tokens to generate would really help in a competition like this, where efficiency is valued. Some study of the vocabulary and more reward shaping could convert some of the extra blathering that humans find nice in chat responses to math notation and allow better solutions, or a translation method from the CoT produced in the OpenR1 set to compact notation.
\nLate in the competition ByteDance had and interesting paper DAPO. I think it would be a better than the GRPO used now, however I didn't have the time or resources to follow up on it. It would also be incremental gains over the current distilled models. Their paper is training from a baseline with RL, rather than distilling a much larger model.
\nWe'll see how things go with the results!
", "rawMarkdown": "Similar to others, I'll share my road to public 28, while awaiting the final results.\n\n## Early Analysis\n\nOne of the first things I did was collect another 40 problems from the [OpenR1-Math-220k](https://www.kaggle.com/datasets/alejopaullier/openr1-math-220k). I selected hard problems with longer reasoning traces. These, plus the reference problems became my evaluation set for local eval.\n\nLocal evaluation was done based on the percentage of sequences that completed and yielded a correct answer. After generating many solutions, I ran some bootstrapping and determined that the 14B Deepseek model seemed to be the optimal balance between majority voting and quality. Standard deviation seemed to be in the range of $\\pm 2-3$ problems on a 50 set with the majority vote, so evaluating models for me was based on the percentage that would be correct.\n\nThat in hand, it was clear the best thing to do was to devote more time to longer problems. I had a similar solution to @optimo and smart stopping [https://www.kaggle.com/code/optimo/fork-of-smart-stopping-14b-clean](https://www.kaggle.com/code/optimo/fork-of-smart-stopping-14b-clean). I couldn't get the later vLLMs to properly abort, but I see he used 0.8.2 so maybe that was fixed in the later versions. I used 0.7.2. Once some number of solutions agreed, it would terminate the run. Similarly if we had a solution and things were taking a long time, it would stop early.\n\nFinally, it seemed AWQ quantization ran the fastest, so I stuck with that method. On average it would be able to complete 6-7 answers in 12288 tokens.\n\n## Fine tuning\n\nSimilar to the Light-R1 paper, I selected a subset of hard problems from the OpenR1-Math-220 set and did an additional few epochs of training. This boosted the per sequence correctness fraction from 0.29 to 0.35, a 20% increase. Nothing fancy, just additional distillation.\n\n## Things that didn't work\n\n### Preference optimization on lengths\n\nAgain using the OpenR1-Math-220, I attempted to decrease the average number of tokens per response with a similar method to [https://novasky-ai.github.io/posts/reduce-overthinking/](https://novasky-ai.github.io/posts/reduce-overthinking/). In the end, this did not increase the number of sequences terminated, nor improve the accuracy of the results. This was discarded.\n\n### TokenSkip\n\nI additionally tried to follow paper [TokenSkip: Controllable Chain-of-Thought Compression in LLMs](https://arxiv.org/abs/2502.12067). If we could compress the output by 0.5-0.7, this should make the reasoning tighter (less attention to worry about) and faster. After compressing some of the responses from OpenR1-Math-220k, we ran it through some fine tuning rounds. Unfortunately, there was a catastrophic forgetting phase in the training that wiped out some of the knowledge, and while the number of tokens went down as expected, so did the accuracy.\n\n### Merging\n\nI tried training on different subtasks (Algebra, Geometry, etc) and merging, but again, there was a decrease in accuracy as a result.\n\n## Things that I wanted to try\n\n### Retrain with math symbols. \n\nI think the TokenSkip idea is elegant. Math notation is very compact, and since we are resource bound any decrease in the number of tokens to generate would really help in a competition like this, where efficiency is valued. Some study of the vocabulary and more reward shaping could convert some of the extra blathering that humans find nice in chat responses to math notation and allow better solutions, or a translation method from the CoT produced in the OpenR1 set to compact notation. \n\n### DAPO\n\nLate in the competition ByteDance had and interesting paper [DAPO](https://arxiv.org/abs/2502.12067). I think it would be a better than the GRPO used now, however I didn't have the time or resources to follow up on it. It would also be incremental gains over the current distilled models. Their paper is training from a baseline with RL, rather than distilling a much larger model.\n\nWe'll see how things go with the results!", "votes": 14 }, { "id": 3172163, "postDate": "2025-04-06T14:27:00Z", "content": "All of my top five notebooks with LB=29 all use vLLM=0.7.2.
", "rawMarkdown": "All of my top five notebooks with LB=29 all use vLLM=0.7.2.", "votes": 1 }, { "id": 3170642, "postDate": "2025-04-04T21:27:19.047Z", "content": "Thanks for sharing this, @danielphalen! Bootstrapping is actually a great practice I learned from you.
\nRegarding the TokenSkip paper - while I really enjoy novel ideas like that, it reminded me of when I read it about a month ago, and it didn’t fully make sense to me at the time.
\nI used to think that every token generated by a reasoning model plays a crucial role in reaching the final answer. Especially in reinforcement learning setups like DeepSeek’s, where sparse rewards (e.g., based only on the final answer) are used, it feels natural to assume the model is incentivized to make each token contribute meaningfully to the overall return.
\nThat’s why methods like TokenSkip initially felt counterintuitive to me 😅
\nBut on second thought, it could make sense: not every token may be equally useful for reward propagation, and skipping low-impact or redundant tokens might actually help the model focus its capacity on the parts that matter most.
\nIn the end, I think using techniques like adding based-length rewards during RL training, as some teams/papers have done, offers a good balance - encouraging concise yet effective reasoning without skipping too aggressively. By considering long-term rewards, the model can prioritize tokens that contribute to the overall success of the reasoning process, while still ensuring that the final answer is reached efficiently.
", "rawMarkdown": "Thanks for sharing this, @danielphalen! Bootstrapping is actually a great practice I learned from you.\n\nRegarding the TokenSkip paper - while I really enjoy novel ideas like that, it reminded me of when I read it about a month ago, and it didn’t fully make sense to me at the time.\n\nI used to think that every token generated by a reasoning model plays a crucial role in reaching the final answer. Especially in reinforcement learning setups like DeepSeek’s, where sparse rewards (e.g., based only on the final answer) are used, it feels natural to assume the model is incentivized to make each token contribute meaningfully to the overall return.\n\nThat’s why methods like TokenSkip initially felt counterintuitive to me 😅\n\nBut on second thought, it could make sense: not every token may be equally useful for reward propagation, and skipping low-impact or redundant tokens might actually help the model focus its capacity on the parts that matter most.\n\nIn the end, I think using techniques like adding based-length rewards during RL training, as some teams/papers have done, offers a good balance - encouraging concise yet effective reasoning without skipping too aggressively. By considering long-term rewards, the model can prioritize tokens that contribute to the overall success of the reasoning process, while still ensuring that the final answer is reached efficiently.", "votes": 1 }, { "id": 3170641, "postDate": "2025-04-04T21:26:34.183Z", "content": "Reducing overthinking is an actual problem for humans, too 😄
", "rawMarkdown": "Reducing overthinking is an actual problem for humans, too 😄", "votes": 1 }, { "id": 3231646, "postDate": "2025-06-24T17:18:37.310Z", "content": "Thanks for sharing such a detailed breakdown! It's incredibly valuable to read about both what worked and what didn’t — especially your reflections on TokenSkip, smart stopping, and distillation trade-offs. The compact notation and AWQ quantization insights were great takeaways. Looking forward to seeing how your ideas evolve in future competitions!
", "rawMarkdown": "Thanks for sharing such a detailed breakdown! It's incredibly valuable to read about both what worked and what didn’t — especially your reflections on TokenSkip, smart stopping, and distillation trade-offs. The compact notation and AWQ quantization insights were great takeaways. Looking forward to seeing how your ideas evolve in future competitions!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T06:42:32Z", "competition_id": 86023, "competition_slug": "ai-mathematical-olympiad-progress-prize-2", "team_id": 13244442, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-2/writeups/daniel-phalen-public-28-more-distillation-and-thin" }