{ "id": 574262, "title": "5th place solution", "content": "

First, I would like to express my sincere gratitude to XTX Markets and Kaggle for hosting this excellent competition. Organizing an event of this quality and scale is a significant undertaking. I also extend my appreciation to all the participants for their dedication and contributions.
\nSolution Details (luck + 14b-awq + lmdeploy):

\n\n

Overview

\n

Compared to Vladimir Demidov's public notebook, I utilized the 14B-AWQ version of the DeepSeek-R1 Distill-Qwen model and implemented inference using lmdeploy, resulting in a 28% throughput gain over vllm. I learned these from the public model and discussion shared by the imagination-research team – whose insights were greatly appreciated.
\nI attribute my 5th place finish largely to fortunate circumstances, as a near identical submission achieved a public score of only 23. I share this outcome with those who may have more contributions or developed stronger models, yet did not experience the same favorable results. I hope everyone has the opportunity to benefit from a similar stroke of luck in future events.
\nFinally, I am also sharing a simulation technique applied during this competition, which reduces evaluation compute from O(N) to O(1). To avoid potential submission waste or a slight performance downgrade, I directly adopted answer extraction/voting codes and prompts from Vladimir Demidov's notebook (though I had my own version). I am very grateful for the original author(s) for their work.

\n

Throughput

\n

Compared to vllm, lmdeploy increases the throughput by 28% in the 14b-awq setting.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
lmdeploysglangvllm 0.7.2
14b-awq bs 9 seqlen 13500Question Time400s462s511s
Throughput304 token/s263 tokens/s238 tokens/s
32b-awq bs 9 seqlen 9000Question Time355s374s396s
Throughput228 token/s193 tokens/s183 tokens/s
7b-awq bs 32 seqlen 13500Question Time403s391s505s
Throughput1073 token/s1103 tokens/s856 tokens/s
\n\n

Details and Strategies

\n\n

Running details:

\n\n

Strategies:

\n\n

Simulations

\n

Overview:

\n

Doing simulations can help saving compute resource from O(N) to O(1) by alternating the evaluation steps. It helped me to save a lot of submissions and compute resources during the competition.
\nVanilla Evaluation:

\n
For each new scheme:\n   Use llm to generate sufficient samples of this scheme\n   Obtain evaluation result\n
\n

Simulation:

\n
Use llm to generate sufficient samples as the 'groundtruth' \nFor each new scheme:\n   Sample from the 'groundtruth' the outputs of this scheme (until the evaluation result converges)\n   Obtain evaluation result\n
\n

Note that step 1 of simulation only needs to be conducted once, and there is no need to generate new outputs when evaluating new schemes. Also, more samples lead to a more accurate 'groundtruth'.
\ncode: https://www.kaggle.com/code/stuveee/aimo2-simulation
\nI conducted simulations for the purposes including:

\n\n

Grid Search - Batchsize & Seqlen:

\n

I empirically used max_question_time = 410s to estimate the seqlen and batchsize that ensures the entire run to finish in 4.5~4.7h, resulting in:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Batchsize5678910111213141516
seqlen200001700015500145001380013300128001220011600111001060010200
\n\n

Grid Search - Temperature:

\n\n

Estimate Early Stop Benefits and Losses:

\n\n

Failed Attempts

\n

AWQ enhancement

\n

groupsize 128 -> 32: not supported by lmdeploy
\ncustomized AWQ:

\n
    \n
  1. calib_data:
    2. \n
  2. max_calib_data_seqlen:
    3. \n
  3. increasing sample num
    \n 
    \nNote: though the original AWQ paper shows that AWQ is robust to and don't need a large scale of calibration data, I still made an attempt.
    \n\"\"
    \nThe AIME25 result shows that 14b-awqv2 (i.e., customized AWQ) with 8 calibration samples performs better than 14b-awq-casperhansen. However, it couldn't outperform 14b-awq-casperhansen in my few submission attempts.
    4. \n
\n

SFT

\n

I failed to conduct SFT because the sequence lengths for reasoning tasks were too large, and there was not enough GPU memory to fit, causing GPU OOMs even with lora.

\n

failure of reproducing the public notebook

\n

I could not reproduce the score of Vladimir Demidov's notebook in my own code. I spent 20 submissions, bisected the code differences, and ended up finding that either change below leads to a score downgrade from 23~26 to 18~21:

\n
    \n
  1. merely changing vllm engine from v0 to v1
    2. \n
  2. merely using vllm-0.6.3.post1 engine v0 without seed
    3. \n
\n

Appreciation

\n

Finally, I would like to once again thank XTX Markets and Kaggle for hosting this wonderful competition, and to all the participants for their incredible effort and dedication. I am also deeply grateful to everyone who supported me throughout this journey. Their encouragement was truly invaluable.

", "comments": [ { "id": 3183960, "author_name": "Arindam Singh", "author_url": "", "post_date": "2025-04-21T13:25:47.487000", "content": "

Good Work By the way.
\nJust wanted to ask if you find any good solution to this, please add it in the op.

\n
\n

I failed to conduct SFT because the sequence lengths for reasoning tasks were too large, and there was not enough GPU memory to fit, causing GPU OOMs even with lora.

\n
", "votes": 0, "replies": [ { "id": 3184095, "author_name": "usernam", "author_url": "", "post_date": "2025-04-21T16:32:39.303000", "content": "
\n

Update:

\n\n\n

 

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
seqlen0 (before forward)200050001000020000
memory (GB)30.431.033.137.1OOM
\n

 
\nAlso see the bold part below for the updated approach.

\n
\n

Thank you for your comment!

\n

For 14B with LoRA, I just did a theoretical VRAM calculation:

\n\n

Total is theoretically around 39.4G, which is not likely to fit into a 40GB A100 considering other overheads omitted in the compute. However, I believe using MLA can reduce the activation memory to < 3G. Another approach is to offload activations to RAM, and I guess if pin memory is used smoothly, the offload process might have 0 overhead.

\n

Kaggle's L4x4 VRAM should be sufficient in aggregate, but successfully utilizing it requires a training framework that supports LoRA combined with model parallelism. I guess NeMo-Megatron or DeepSpeed might work?

\n

Furthermore, I haven't encountered reports of LoRA alone achieving strong performance on reasoning tasks requiring such long sequences.

\n

For 14B full SFT, the parameter, gradient, and optimizer states alone sum to roughly 200GB, making it infeasible without a larger server.

", "votes": 3, "replies": [ { "id": 3184925, "author_name": "Arindam Singh", "author_url": "", "post_date": "2025-04-22T16:34:03.823000", "content": "

DeepSpeed is a strong candidate, as it handles LoRA and model parallelism well, especially with ZeRO Stage 3 for memory optimization. NeMo-Megatron could also work but might require more configuration for LoRA-specific tasks. Hugging Face’s Accelerate with FSDP is another option to explore for distributed training.On LoRA’s performance for long-sequence reasoning tasks, your skepticism is warranted. LoRA tends to underperform on complex reasoning with extended contexts compared to full fine-tuning, as it only updates low-rank adapters. There’s limited evidence of LoRA excelling in such scenarios, especially at this scale, so you might need to experiment or combine it with techniques like prompt tuning or sparse attention.

", "votes": 0, "replies": [] }, { "id": 3185277, "author_name": "", "author_url": "", "post_date": "2025-04-23T04:47:05.390000", "content": "", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 4, "messages": [ { "id": 3183564, "postDate": "2025-04-21T02:19:00.730Z", "content": "

First, I would like to express my sincere gratitude to XTX Markets and Kaggle for hosting this excellent competition. Organizing an event of this quality and scale is a significant undertaking. I also extend my appreciation to all the participants for their dedication and contributions.
\nSolution Details (luck + 14b-awq + lmdeploy):

\n\n

Overview

\n

Compared to Vladimir Demidov's public notebook, I utilized the 14B-AWQ version of the DeepSeek-R1 Distill-Qwen model and implemented inference using lmdeploy, resulting in a 28% throughput gain over vllm. I learned these from the public model and discussion shared by the imagination-research team – whose insights were greatly appreciated.
\nI attribute my 5th place finish largely to fortunate circumstances, as a near identical submission achieved a public score of only 23. I share this outcome with those who may have more contributions or developed stronger models, yet did not experience the same favorable results. I hope everyone has the opportunity to benefit from a similar stroke of luck in future events.
\nFinally, I am also sharing a simulation technique applied during this competition, which reduces evaluation compute from O(N) to O(1). To avoid potential submission waste or a slight performance downgrade, I directly adopted answer extraction/voting codes and prompts from Vladimir Demidov's notebook (though I had my own version). I am very grateful for the original author(s) for their work.

\n

Throughput

\n

Compared to vllm, lmdeploy increases the throughput by 28% in the 14b-awq setting.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
lmdeploysglangvllm 0.7.2
14b-awq bs 9 seqlen 13500Question Time400s462s511s
Throughput304 token/s263 tokens/s238 tokens/s
32b-awq bs 9 seqlen 9000Question Time355s374s396s
Throughput228 token/s193 tokens/s183 tokens/s
7b-awq bs 32 seqlen 13500Question Time403s391s505s
Throughput1073 token/s1103 tokens/s856 tokens/s
\n\n

Details and Strategies

\n\n

Running details:

\n\n

Strategies:

\n\n

Simulations

\n

Overview:

\n

Doing simulations can help saving compute resource from O(N) to O(1) by alternating the evaluation steps. It helped me to save a lot of submissions and compute resources during the competition.
\nVanilla Evaluation:

\n
For each new scheme:\n   Use llm to generate sufficient samples of this scheme\n   Obtain evaluation result\n
\n

Simulation:

\n
Use llm to generate sufficient samples as the 'groundtruth' \nFor each new scheme:\n   Sample from the 'groundtruth' the outputs of this scheme (until the evaluation result converges)\n   Obtain evaluation result\n
\n

Note that step 1 of simulation only needs to be conducted once, and there is no need to generate new outputs when evaluating new schemes. Also, more samples lead to a more accurate 'groundtruth'.
\ncode: https://www.kaggle.com/code/stuveee/aimo2-simulation
\nI conducted simulations for the purposes including:

\n\n

Grid Search - Batchsize & Seqlen:

\n

I empirically used max_question_time = 410s to estimate the seqlen and batchsize that ensures the entire run to finish in 4.5~4.7h, resulting in:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Batchsize5678910111213141516
seqlen200001700015500145001380013300128001220011600111001060010200
\n\n

Grid Search - Temperature:

\n\n

Estimate Early Stop Benefits and Losses:

\n\n

Failed Attempts

\n

AWQ enhancement

\n

groupsize 128 -> 32: not supported by lmdeploy
\ncustomized AWQ:

\n
    \n
  1. calib_data:
    2. \n
  2. max_calib_data_seqlen:
    3. \n
  3. increasing sample num
    \n 
    \nNote: though the original AWQ paper shows that AWQ is robust to and don't need a large scale of calibration data, I still made an attempt.
    \n\"\"
    \nThe AIME25 result shows that 14b-awqv2 (i.e., customized AWQ) with 8 calibration samples performs better than 14b-awq-casperhansen. However, it couldn't outperform 14b-awq-casperhansen in my few submission attempts.
    4. \n
\n

SFT

\n

I failed to conduct SFT because the sequence lengths for reasoning tasks were too large, and there was not enough GPU memory to fit, causing GPU OOMs even with lora.

\n

failure of reproducing the public notebook

\n

I could not reproduce the score of Vladimir Demidov's notebook in my own code. I spent 20 submissions, bisected the code differences, and ended up finding that either change below leads to a score downgrade from 23~26 to 18~21:

\n
    \n
  1. merely changing vllm engine from v0 to v1
    2. \n
  2. merely using vllm-0.6.3.post1 engine v0 without seed
    3. \n
\n

Appreciation

\n

Finally, I would like to once again thank XTX Markets and Kaggle for hosting this wonderful competition, and to all the participants for their incredible effort and dedication. I am also deeply grateful to everyone who supported me throughout this journey. Their encouragement was truly invaluable.

", "rawMarkdown": "\nFirst, I would like to express my sincere gratitude to XTX Markets and Kaggle for hosting this excellent competition. Organizing an event of this quality and scale is a significant undertaking. I also extend my appreciation to all the participants for their dedication and contributions.\n\n**Solution Details (luck + 14b-awq + lmdeploy):**\n\n* **Public Score:** 28\n* **Private Score:** 29\n* **Model:** Deepseek-R1 Distill-Qwen-14B-AWQ ([https://huggingface.co/casperhansen/deepseek-r1-distill-qwen-14b-awq](https://huggingface.co/casperhansen/deepseek-r1-distill-qwen-14b-awq) - MIT License)\n* **Inference Server:** lmdeploy ([https://github.com/InternLM/lmdeploy](https://github.com/InternLM/lmdeploy) - Apache 2.0 License)\n* **Notebook:** [https://www.kaggle.com/code/stuveee/aimo-lmdeploy](https://www.kaggle.com/code/stuveee/aimo-lmdeploy)\n* **Key References:** [https://www.kaggle.com/code/yekenot/aimo-2-deepseek-r1-distill-qwen-7b-awq](https://www.kaggle.com/code/yekenot/aimo-2-deepseek-r1-distill-qwen-7b-awq) (public notebook of Vladimir Demidov)\n* **E2E handler for HF Endpoints:** https://huggingface.co/usernameeeeeeee/r1-distill-14b-awq-casperhansen-e2e-solution-handler\n* **Computational Resources (to reach the winning submission):** Kaggle's L4x4 GPU (15h/week)\n\n## Overview\n\nCompared to Vladimir Demidov's public notebook, I utilized the 14B-AWQ version of the DeepSeek-R1 Distill-Qwen model and implemented inference using lmdeploy, resulting in a 28% throughput gain over vllm. I learned these from the public model and discussion shared by the imagination-research team – whose insights were greatly appreciated.\n\nI attribute my 5th place finish largely to fortunate circumstances, as a near identical submission achieved a public score of only 23. I share this outcome with those who may have more contributions or developed stronger models, yet did not experience the same favorable results. I hope everyone has the opportunity to benefit from a similar stroke of luck in future events.\n\nFinally, I am also sharing a simulation technique applied during this competition, which reduces evaluation compute from O(N) to O(1). To avoid potential submission waste or a slight performance downgrade, I directly adopted answer extraction/voting codes and prompts from Vladimir Demidov's notebook (though I had my own version). I am very grateful for the original author(s) for their work.\n\n\n\n## Throughput\n\n\nCompared to vllm, lmdeploy increases the throughput by 28% in the 14b-awq setting.\n\n\n|||lmdeploy|sglang|vllm 0.7.2|||||||||\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|14b-awq bs 9 seqlen 13500|Question Time| 400s|462s|511s|\n||Throughput|304 token/s | 263 tokens/s|238 tokens/s|\n|32b-awq bs 9 seqlen 9000|Question Time| 355s|374s|396s|\n||Throughput|228 token/s | 193 tokens/s|183 tokens/s|\n|7b-awq bs 32 seqlen 13500|Question Time| 403s|391s|505s|\n||Throughput|1073 token/s | 1103 tokens/s|856 tokens/s|\n\n\n* Throughput testing code: https://www.kaggle.com/code/stuveee/aimo2-throughput-sglang-lmdeploy-vllm\n\n\n* I gave up on vllm engine v1 during the competition because I couldn't reproduce Vladimir Demidov's score on public lb with engine v1 and there were also accuracy downgrade problems reported in the forum. (Later, it seemed that overfitting was the actual cause.)\n\n* note: according to team imagination-research's and tascj's great solution writeups, using int8 kv quantization can further enhance throughput without sacrificing accuracy.\n\n\n\n\n## Details and Strategies\n\n\n\n### Running details:\n\n* temperature: 0.9\n* top_p: 0.9\n* min_p: 0.05\n* stop_words: ['</think>']\n* batchsize: 9\n* seqlen: 13500\n\n### Strategies:\n\n* To avoid complexity, I use the same batchsize and cutoff length for all questions, just to ensure the entire run to complete in about 4.5~4.7h.\n\n* For each question, 9 requests run concurrently, and cutoff at seqlen 13500. Majority vote is applied to get the answer.\n\n* Note: a question-level early stop strategy saves about 4.5~9% tokens without losing accuracy (see next section). However, it might bring bugs and implementing it in lmdeploy turbomind seemed effort consuming during the competition, so I didn't apply it.\n\n\n## Simulations\n\n### Overview:\n\nDoing simulations can help saving compute resource from O(N) to O(1) by alternating the evaluation steps. It helped me to save a lot of submissions and compute resources during the competition.\n\nVanilla Evaluation:\n``` markdown\nFor each new scheme:\n Use llm to generate sufficient samples of this scheme\n Obtain evaluation result\n```\n\nSimulation:\n``` markdown\nUse llm to generate sufficient samples as the 'groundtruth' \nFor each new scheme:\n Sample from the 'groundtruth' the outputs of this scheme (until the evaluation result converges)\n Obtain evaluation result\n```\n\nNote that step 1 of simulation only needs to be conducted once, and there is no need to generate new outputs when evaluating new schemes. Also, more samples lead to a more accurate 'groundtruth'.\n\n\n\ncode: https://www.kaggle.com/code/stuveee/aimo2-simulation\n\n\nI conducted simulations for the purposes including:\n* grid search - batchsize & seqlen\n* grid search - temperature\n* estimate early stop benefits and losses\n\n\n## Grid Search - Batchsize & Seqlen:\n\nI empirically used max_question_time = 410s to estimate the seqlen and batchsize that ensures the entire run to finish in 4.5~4.7h, resulting in:\n|Batchsize|5|6|7|8|9|10|11|12|13|14|15|16|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|seqlen|20000|17000|15500|14500|13800|13300|12800|12200|11600|11100|10600|10200|\n\n\n\n* The simulation runs show that the optimal (batchsize, seqlen) is at (10, 13300) or (9, 13800).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2988556%2F9089026a7616f70127aa745c9c533bbb%2Fimage-14.png?generation=1745164795971663&alt=media)\n\n\n### Grid Search - Temperature:\n\n* Temperature 0.9 seems to have an observable advantage over temperature 0.8 for the 14b model:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2988556%2Fc34fb7d42207f0867510bc34f098ad69%2Fimage-15.png?generation=1745164761238215&alt=media)\n\n* In the meantime, temperature 0.85/0.95 do not seem better than 0.9:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2988556%2F0ef08ba151431a6433aedd6206417a40%2Fimage-16.png?generation=1745164883466768&alt=media)\n\n\n\n\n### Estimate Early Stop Benefits and Losses:\n\n\n* According to the simulation, at batchsize 9, a radical early stop strategy saves about 9% tokens, while a conservative strategy saves about 4.5%. Both the radical strategy and the conservative strategy have no accuracy loss when batchsize < 12.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2988556%2Faea29449ae85ce167a6f3ecc6393392f%2Fimage-11.png?generation=1745164981705719&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2988556%2F8ecc3aa403e71d259834161dbe44269f%2Fimage-17.png?generation=1745164842429171&alt=media)\n\nDetails of the strategies are as follows:\n* conservative\n * stop when: running_requests + 2nd_most_answer <= 1st_most_answer\n* radical\n * In addition, stop when: 1st_most_answer >= current_answers * (80% - (current_answers - 5) / (batchsize - 5) * 30%)\n * i.e., linearly interpolate between:\n * 4 consistent answers when 5 are obtained\n * 50% consistent answers when all obtained\n\n\n## Failed Attempts\n\n### AWQ enhancement\n\n**groupsize 128 -> 32:** not supported by lmdeploy\n\n**customized AWQ:**\n1. calib_data: \n * mit-han-lab/pile-val-backup -> selective samples from lightr1-stage2.json (I appreciate LightR1 team for their work), question + solution\n\n2. max_calib_data_seqlen: \n * 512 -> 20000\n\n3. increasing sample num\n\n \n Note: though the original AWQ paper shows that AWQ is robust to and don't need a large scale of calibration data, I still made an attempt.\n\n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2988556%2F58aee13b07eb3405e32fdb082952597b%2Fimage-19.png?generation=1745164814021926&alt=media)\n\n The AIME25 result shows that 14b-awqv2 (i.e., customized AWQ) with 8 calibration samples performs better than 14b-awq-casperhansen. However, it couldn't outperform 14b-awq-casperhansen in my few submission attempts.\n\n\n\n### SFT\nI failed to conduct SFT because the sequence lengths for reasoning tasks were too large, and there was not enough GPU memory to fit, causing GPU OOMs even with lora.\n\n\n\n### failure of reproducing the public notebook\nI could not reproduce the score of Vladimir Demidov's notebook in my own code. I spent 20 submissions, bisected the code differences, and ended up finding that either change below leads to a score downgrade from 23~26 to 18~21:\n\n1. merely changing vllm engine from v0 to v1\n2. merely using vllm-0.6.3.post1 engine v0 without seed\n\n\n\n\n## Appreciation\n\nFinally, I would like to once again thank XTX Markets and Kaggle for hosting this wonderful competition, and to all the participants for their incredible effort and dedication. I am also deeply grateful to everyone who supported me throughout this journey. Their encouragement was truly invaluable.", "votes": 17 }, { "id": 3183960, "postDate": "2025-04-21T13:25:47.487Z", "content": "

Good Work By the way.
\nJust wanted to ask if you find any good solution to this, please add it in the op.

\n
\n

I failed to conduct SFT because the sequence lengths for reasoning tasks were too large, and there was not enough GPU memory to fit, causing GPU OOMs even with lora.

\n
", "rawMarkdown": "Good Work By the way. \nJust wanted to ask if you find any good solution to this, please add it in the op. \n\n>I failed to conduct SFT because the sequence lengths for reasoning tasks were too large, and there was not enough GPU memory to fit, causing GPU OOMs even with lora.", "replies": [ { "id": 3184095, "postDate": "2025-04-21T16:32:39.303Z", "content": "
\n

Update:

\n\n\n

 

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
seqlen0 (before forward)200050001000020000
memory (GB)30.431.033.137.1OOM
\n

 
\nAlso see the bold part below for the updated approach.

\n
\n

Thank you for your comment!

\n

For 14B with LoRA, I just did a theoretical VRAM calculation:

\n\n

Total is theoretically around 39.4G, which is not likely to fit into a 40GB A100 considering other overheads omitted in the compute. However, I believe using MLA can reduce the activation memory to < 3G. Another approach is to offload activations to RAM, and I guess if pin memory is used smoothly, the offload process might have 0 overhead.

\n

Kaggle's L4x4 VRAM should be sufficient in aggregate, but successfully utilizing it requires a training framework that supports LoRA combined with model parallelism. I guess NeMo-Megatron or DeepSpeed might work?

\n

Furthermore, I haven't encountered reports of LoRA alone achieving strong performance on reasoning tasks requiring such long sequences.

\n

For 14B full SFT, the parameter, gradient, and optimizer states alone sum to roughly 200GB, making it infeasible without a larger server.

", "rawMarkdown": "---\nUpdate: \n* **TLDR**:\n\n * **Solutions to reduce activation memory for long sequence training**:\n * (1) MLA + recompute activation\n * (2) activation offloading to RAM, may reduce offload overhead to 0 by pin memory.\n\n * **14B full SFT**: Model + Gradient + Optimizer States > 200G, a large server is needed. \n * **14B LoRA**: (no guaranteed result)\n * A100 40G:\n * implement either or both of solution (1) and (2)\n * L4x4: \n * Try a training framework supports LoRA + Model Parallelism. Possible options:\n * NeMo-Megatron\n * DeepSpeed\n\n \n\n* I tried 14B LoRA with Llamafactory and 40GB VRAM:\n\n|seqlen|0 (before forward)|2000|5000|10000|20000|\n|---|---|---|---|---|---|\n|memory (GB)|30.4|31.0|33.1|37.1|OOM|\n\n \nAlso see the bold part below for the updated approach.\n\n---\n\n\nThank you for your comment!\n\nFor 14B with LoRA, I just did a theoretical VRAM calculation:\n\n* Parameters (16-bit): ~28GB\n* Optimizer/Gradient States: ~1GB\n* Activations (with gradient checkpointing, seqlen=20000, batchsize=1):~~ ~3.8GB~~ ~10.4GB\n~~(seqlen * intermidiate_size * 16/8 * sqrt(num_layers) / 1,000,000,000 = 20000 * 13824 * 16/8 * sqrt(48) / 1,000,000,000 = 3.8)~~\n(seqlen * 16/8 * (intermidiate_size + hidden_size * num_layers) / 1,000,000,000 = 20000 * 16 / 8 * (13824 + 5120 * 48) / 1,000,000,000 = 10.4)\n\nTotal is theoretically around ~~~33GB (not sure if there is any mistake)~~ 39.4G, ~~which might fit on a 40GB A100.~~ which is not likely to fit into a 40GB A100 considering other overheads omitted in the compute. **However, I believe using MLA can reduce the activation memory to < 3G. Another approach is to offload activations to RAM, and I guess if pin memory is used smoothly, the offload process might have 0 overhead.**\n\n\n\nKaggle's L4x4 VRAM should be sufficient in aggregate, but successfully utilizing it requires a training framework that supports LoRA combined with model parallelism. I guess NeMo-Megatron or DeepSpeed might work?\n\nFurthermore, I haven't encountered reports of LoRA alone achieving strong performance on reasoning tasks requiring such long sequences.\n\nFor 14B full SFT, the parameter, gradient, and optimizer states alone sum to roughly 200GB, making it infeasible without a larger server.", "votes": 3, "replies": [ { "id": 3184925, "postDate": "2025-04-22T16:34:03.823Z", "content": "

DeepSpeed is a strong candidate, as it handles LoRA and model parallelism well, especially with ZeRO Stage 3 for memory optimization. NeMo-Megatron could also work but might require more configuration for LoRA-specific tasks. Hugging Face’s Accelerate with FSDP is another option to explore for distributed training.On LoRA’s performance for long-sequence reasoning tasks, your skepticism is warranted. LoRA tends to underperform on complex reasoning with extended contexts compared to full fine-tuning, as it only updates low-rank adapters. There’s limited evidence of LoRA excelling in such scenarios, especially at this scale, so you might need to experiment or combine it with techniques like prompt tuning or sparse attention.

", "rawMarkdown": "DeepSpeed is a strong candidate, as it handles LoRA and model parallelism well, especially with ZeRO Stage 3 for memory optimization. NeMo-Megatron could also work but might require more configuration for LoRA-specific tasks. Hugging Face’s Accelerate with FSDP is another option to explore for distributed training.On LoRA’s performance for long-sequence reasoning tasks, your skepticism is warranted. LoRA tends to underperform on complex reasoning with extended contexts compared to full fine-tuning, as it only updates low-rank adapters. There’s limited evidence of LoRA excelling in such scenarios, especially at this scale, so you might need to experiment or combine it with techniques like prompt tuning or sparse attention." }, { "id": 3185277, "postDate": "2025-04-23T04:47:05.390Z", "rawMarkdown": "", "isDeleted": true } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T06:40:46Z", "competition_id": 86023, "competition_slug": "ai-mathematical-olympiad-progress-prize-2", "team_id": 12781901, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-2/writeups/usernam-5th-place-solution" }