Thanks to all participants, especially those who created awesome notebooks, models and engaging discussions. I learned a lot. Thanks to Kaggle and the Host for their continued support throughout this long, challenging but exciting competition.
\nI developed many approaches some of which drew inspiration from discussions and public notebooks. Below I mainly share my successful approach so far with public score 28.
\nSince the idea is to learn how Qwen-QWQ 32B solves problems, I needed problems that are relevant for the AIMO2 competition. Hence, I included the reference set as well, see below
\n1. Initialize three models on shared GPUs as follows
\n2. Generate text, form inputs, forward and compute loss
\nSYSTEM_PROMPT = \n\nMAX_SEQ_LEN = \n\nsampling_params = SamplingParams(max_tokens=MAX_SEQ_LEN, temperature=, skip_special_tokens=)\n\ngenerated_text = teacher_model.generate(\n prompts=teacher_prompt,\n sampling_params=sampling_params\n )\n\n\n\nteacher_logits = teacher_logits_model(input_ids=teacher_input_ids, attention_mask=teacher_attention_mask).logits\n\nstudent_logits = model(input_ids=student_input_ids, attention_mask=student_attention_mask).logits\n\nstudent_log_logits = torch.nn.functional.log_softmax(student_logits, dim=-).to()\nteacher_log_logits = torch.nn.functional.log_softmax(teacher_logits, dim=-).to()\n\nloss = torch.nn.functional.kl_div(student_log_logits, teacher_log_logits, log_target=, reduction=)\n\ntraining_args = TrainingArguments(\n output_dir=,\n per_device_train_batch_size=,\n num_train_epochs=,\n logging_dir=,\n logging_steps=,\n learning_rate=,\n weight_decay=,\n save_strategy=,\n save_steps=,\n remove_unused_columns=,\n report_to=,\n)\nAIME2024/2025: 43/60
Reference Set: 7/10
Public LB: 28 (with max_seq_len=8192*3//2 + 1024, top_p=0.9, min_p=0.05, temperature=1.0). I also used vLLM Engine class (not LLM) and could cancel requests when a given answer reaches 33% (frequency) w.r.t. max_num_seqs=12
Did distillation and linear weighting improve stability? We can also achieve 27 points using the early stopping strategy of vLLM Engine.
", "votes": 1, "replies": [ { "id": 3168542, "author_name": "白夜", "author_url": "", "post_date": "2025-04-02T14:41:32.160000", "content": "Based on my understanding, the distillation here seems to be a method for controlling the generation length in low-resource scenarios. What inspired you to use distillation in this competition?
", "votes": 0, "replies": [ { "id": 3168595, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-04-02T15:29:46.023000", "content": "Not only generation length in my settings. Also the \"approach to problem solving\"
", "votes": 0, "replies": [] } ] }, { "id": 3168599, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-04-02T15:30:54.900000", "content": "Unfortunately I only had 2 trials with my model. Training completed in the last days of the competition. So stability has not been fully tested
", "votes": 0, "replies": [ { "id": 3168600, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-04-02T15:33:20.793000", "content": "First trial scored 28. Then in second trial I used top_k=20 and got a score of 25. I believe distillation worked. This is why:
\n1- I was not able to achieve 28 before training, with my evaluation settings.
\n2- The loss decreased during training. Loss started around 40k and decreased to 20k-25k depending on running batch
Very good idea, would like to learn from you, may I ask how the Linear Weight Ensembling part code is implemented.
", "votes": 0, "replies": [] }, { "id": 3168585, "author_name": "ducnh279", "author_url": "", "post_date": "2025-04-02T15:19:20.573000", "content": "\n\nAveraged checkpoints are: 0.1ckpt-20 + 0.1ckpt-180 + 0.1ckpt-260 + 0.1ckpt-280 + 0.1ckpt-360 + 0.1ckpt-640 + 0.1ckpt-1840 + 0.1ckpt-2300 + 0.2*last-ckpt
\n
Simple and beautiful solution using knowledge distillation!
\nDo you compare the performance between Averaged checkpoints vs only last-ckpt on the public leaderboard?
Yes. I did that. Last epoch scored 22 though I didn't have time to evaluate further
", "votes": 1, "replies": [ { "id": 3169996, "author_name": "ducnh279", "author_url": "", "post_date": "2025-04-04T06:49:14.727000", "content": "Wow, weighted averaging of checkpoints seems to be quite effective! Thanks for sharing this technique!
", "votes": 0, "replies": [] } ] } ] }, { "id": 3168562, "author_name": "jyf", "author_url": "", "post_date": "2025-04-02T15:04:09.030000", "content": "How did you implement early stopping based on consensus? I tried the asynchronous engine and it was not faster.
", "votes": 0, "replies": [ { "id": 3168610, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-04-02T15:47:32.417000", "content": "So this is how I implemented early stopping:
\n ():\n \n request_id = \n outputs = []\n output_lengths = []\n inner_start_time = time.time()\n killed_req_id = MAX_NUM_SEQS - \n prompts engine.has_unfinished_requests():\n prompts:\n prompt, sampling_params = prompts.pop()\n engine.add_request((request_id),\n tokenizer.apply_chat_template(conversation=prompt, tokenize=, add_generation_prompt=),\n sampling_params)\n request_id += \n\n time.time()-inner_start_time >= :\n (++++)\n killed_req_id >= :\n :\n engine.abort_request((killed_req_id))\n Exception e:\n (+(e))\n \n killed_req_id -= \n time.sleep()\n (++++)\n\n request_outputs = engine.step()\n i, request_output (request_outputs):\n request_output.finished:\n outputs.append(request_output.outputs[].text)\n output_lengths.append((request_output.outputs[].token_ids))\n\n (outputs) > request_id/:\n \n list_of_idx = (((outputs)))\n _, extracted_answers, _ = batch_message_filter(outputs, list_of_idx)\n\n answer_stats = Counter(extracted_answers)\n answer_stats = ([(k,v) k,v answer_stats.items()], key= x: -x[])\n\n answer_stats answer_stats[][] > request_id/:\n (++++)\n trials = \n engine.has_unfinished_requests():\n ()\n req_id (request_id):\n :\n engine.abort_request((req_id))\n Exception e:\n (+(e))\n \n time.sleep()\n trials += \n trials > :\n ()\n \n (++++)\n (, output_lengths)\n extracted_answers\n\n\n list_of_idx = (((outputs)))\n _, extracted_answers, _ = batch_message_filter(outputs, list_of_idx)\n (, output_lengths)\n extracted_answers\nWill share the complete code soon!
", "votes": 2, "replies": [ { "id": 3169244, "author_name": "AI-Cat", "author_url": "", "post_date": "2025-04-03T08:59:44.360000", "content": "Great code!
", "votes": 0, "replies": [] }, { "id": 3169348, "author_name": "JK-Piece", "author_url": "", "post_date": "2025-04-03T11:47:35.060000", "content": "Thanks. Looking forward to your code too
", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 13, "messages": [ { "id": 3168410, "postDate": "2025-04-02T12:00:01.207Z", "content": "Thanks to all participants, especially those who created awesome notebooks, models and engaging discussions. I learned a lot. Thanks to Kaggle and the Host for their continued support throughout this long, challenging but exciting competition.
\nI developed many approaches some of which drew inspiration from discussions and public notebooks. Below I mainly share my successful approach so far with public score 28.
\nSince the idea is to learn how Qwen-QWQ 32B solves problems, I needed problems that are relevant for the AIMO2 competition. Hence, I included the reference set as well, see below
\n1. Initialize three models on shared GPUs as follows
\n2. Generate text, form inputs, forward and compute loss
\nSYSTEM_PROMPT = \n\nMAX_SEQ_LEN = \n\nsampling_params = SamplingParams(max_tokens=MAX_SEQ_LEN, temperature=, skip_special_tokens=)\n\ngenerated_text = teacher_model.generate(\n prompts=teacher_prompt,\n sampling_params=sampling_params\n )\n\n\n\nteacher_logits = teacher_logits_model(input_ids=teacher_input_ids, attention_mask=teacher_attention_mask).logits\n\nstudent_logits = model(input_ids=student_input_ids, attention_mask=student_attention_mask).logits\n\nstudent_log_logits = torch.nn.functional.log_softmax(student_logits, dim=-).to()\nteacher_log_logits = torch.nn.functional.log_softmax(teacher_logits, dim=-).to()\n\nloss = torch.nn.functional.kl_div(student_log_logits, teacher_log_logits, log_target=, reduction=)\n\ntraining_args = TrainingArguments(\n output_dir=,\n per_device_train_batch_size=,\n num_train_epochs=,\n logging_dir=,\n logging_steps=,\n learning_rate=,\n weight_decay=,\n save_strategy=,\n save_steps=,\n remove_unused_columns=,\n report_to=,\n)\nAIME2024/2025: 43/60
Reference Set: 7/10
Public LB: 28 (with max_seq_len=8192*3//2 + 1024, top_p=0.9, min_p=0.05, temperature=1.0). I also used vLLM Engine class (not LLM) and could cancel requests when a given answer reaches 33% (frequency) w.r.t. max_num_seqs=12
Did distillation and linear weighting improve stability? We can also achieve 27 points using the early stopping strategy of vLLM Engine.
", "rawMarkdown": "Did distillation and linear weighting improve stability? We can also achieve 27 points using the early stopping strategy of vLLM Engine.", "votes": 1, "replies": [ { "id": 3168542, "postDate": "2025-04-02T14:41:32.160Z", "content": "Based on my understanding, the distillation here seems to be a method for controlling the generation length in low-resource scenarios. What inspired you to use distillation in this competition?
", "rawMarkdown": "Based on my understanding, the distillation here seems to be a method for controlling the generation length in low-resource scenarios. What inspired you to use distillation in this competition?", "replies": [ { "id": 3168595, "postDate": "2025-04-02T15:29:46.023Z", "content": "Not only generation length in my settings. Also the \"approach to problem solving\"
", "rawMarkdown": "Not only generation length in my settings. Also the \"approach to problem solving\"" } ] }, { "id": 3168599, "postDate": "2025-04-02T15:30:54.900Z", "content": "Unfortunately I only had 2 trials with my model. Training completed in the last days of the competition. So stability has not been fully tested
", "rawMarkdown": "Unfortunately I only had 2 trials with my model. Training completed in the last days of the competition. So stability has not been fully tested ", "replies": [ { "id": 3168600, "postDate": "2025-04-02T15:33:20.793Z", "content": "First trial scored 28. Then in second trial I used top_k=20 and got a score of 25. I believe distillation worked. This is why:
\n1- I was not able to achieve 28 before training, with my evaluation settings.
\n2- The loss decreased during training. Loss started around 40k and decreased to 20k-25k depending on running batch
Very good idea, would like to learn from you, may I ask how the Linear Weight Ensembling part code is implemented.
", "rawMarkdown": "Very good idea, would like to learn from you, may I ask how the Linear Weight Ensembling part code is implemented." }, { "id": 3168585, "postDate": "2025-04-02T15:19:20.573Z", "content": "\n\nAveraged checkpoints are: 0.1ckpt-20 + 0.1ckpt-180 + 0.1ckpt-260 + 0.1ckpt-280 + 0.1ckpt-360 + 0.1ckpt-640 + 0.1ckpt-1840 + 0.1ckpt-2300 + 0.2*last-ckpt
\n
Simple and beautiful solution using knowledge distillation!
\nDo you compare the performance between Averaged checkpoints vs only last-ckpt on the public leaderboard?
Yes. I did that. Last epoch scored 22 though I didn't have time to evaluate further
", "rawMarkdown": "Yes. I did that. Last epoch scored 22 though I didn't have time to evaluate further ", "votes": 1, "replies": [ { "id": 3169996, "postDate": "2025-04-04T06:49:14.727Z", "content": "Wow, weighted averaging of checkpoints seems to be quite effective! Thanks for sharing this technique!
", "rawMarkdown": "Wow, weighted averaging of checkpoints seems to be quite effective! Thanks for sharing this technique!\n\n\n\n\n\n\n\n" } ] } ] }, { "id": 3168562, "postDate": "2025-04-02T15:04:09.030Z", "content": "How did you implement early stopping based on consensus? I tried the asynchronous engine and it was not faster.
", "rawMarkdown": "How did you implement early stopping based on consensus? I tried the asynchronous engine and it was not faster.", "replies": [ { "id": 3168610, "postDate": "2025-04-02T15:47:32.417Z", "content": "So this is how I implemented early stopping:
\n ():\n \n request_id = \n outputs = []\n output_lengths = []\n inner_start_time = time.time()\n killed_req_id = MAX_NUM_SEQS - \n prompts engine.has_unfinished_requests():\n prompts:\n prompt, sampling_params = prompts.pop()\n engine.add_request((request_id),\n tokenizer.apply_chat_template(conversation=prompt, tokenize=, add_generation_prompt=),\n sampling_params)\n request_id += \n\n time.time()-inner_start_time >= :\n (++++)\n killed_req_id >= :\n :\n engine.abort_request((killed_req_id))\n Exception e:\n (+(e))\n \n killed_req_id -= \n time.sleep()\n (++++)\n\n request_outputs = engine.step()\n i, request_output (request_outputs):\n request_output.finished:\n outputs.append(request_output.outputs[].text)\n output_lengths.append((request_output.outputs[].token_ids))\n\n (outputs) > request_id/:\n \n list_of_idx = (((outputs)))\n _, extracted_answers, _ = batch_message_filter(outputs, list_of_idx)\n\n answer_stats = Counter(extracted_answers)\n answer_stats = ([(k,v) k,v answer_stats.items()], key= x: -x[])\n\n answer_stats answer_stats[][] > request_id/:\n (++++)\n trials = \n engine.has_unfinished_requests():\n ()\n req_id (request_id):\n :\n engine.abort_request((req_id))\n Exception e:\n (+(e))\n \n time.sleep()\n trials += \n trials > :\n ()\n \n (++++)\n (, output_lengths)\n extracted_answers\n\n\n list_of_idx = (((outputs)))\n _, extracted_answers, _ = batch_message_filter(outputs, list_of_idx)\n (, output_lengths)\n extracted_answers\nWill share the complete code soon!
", "rawMarkdown": "So this is how I implemented early stopping:\n\n```python\ndef process_requests(prompts):\n \"\"\"Continuously process a list of prompts and handle the outputs.\"\"\"\n request_id = 0\n outputs = []\n output_lengths = []\n inner_start_time = time.time()\n killed_req_id = MAX_NUM_SEQS - 1\n while prompts or engine.has_unfinished_requests():\n if prompts:\n prompt, sampling_params = prompts.pop(0)\n engine.add_request(str(request_id),\n tokenizer.apply_chat_template(conversation=prompt, tokenize=False, add_generation_prompt=True),\n sampling_params)\n request_id += 1\n\n if time.time()-inner_start_time >= 550:\n print(\"\\n\"+\"\\x1b[0;30;43m\"+\"Aborting all vLLM engine requests!\"+\"\\x1b[0m\"+\"\\n\")\n while killed_req_id >= 0:\n try:\n engine.abort_request(str(killed_req_id))\n except Exception as e:\n print(\"\\n\"+str(e))\n pass\n killed_req_id -= 1\n time.sleep(0.05)\n print(\"\\n\"+\"\\x1b[0;30;43m\"+\"Done.\"+\"\\x1b[0m\"+\"\\n\")\n\n request_outputs = engine.step()\n for i, request_output in enumerate(request_outputs):\n if request_output.finished:\n outputs.append(request_output.outputs[0].text)\n output_lengths.append(len(request_output.outputs[0].token_ids))\n\n if len(outputs) > request_id/3:\n # Get temporary answers\n list_of_idx = list(range(len(outputs)))\n _, extracted_answers, _ = batch_message_filter(outputs, list_of_idx)\n\n answer_stats = Counter(extracted_answers)\n answer_stats = sorted([(k,v) for k,v in answer_stats.items()], key=lambda x: -x[1])\n\n if answer_stats and answer_stats[0][1] > request_id/3:\n print(\"\\n\\n\"+\"\\x1b[0;30;43m\"+\"Starting Early Stopping!\"+\"\\x1b[0m\"+\"\\n\")\n trials = 0\n while engine.has_unfinished_requests():\n print(\"...\")\n for req_id in range(request_id):\n try:\n engine.abort_request(str(req_id))\n except Exception as e:\n print(\"\\n\"+str(e))\n pass\n time.sleep(0.05)\n trials += 1\n if trials > 3:\n print(\"Did not abort successfully, stopping.\\n\")\n break\n print(\"\\n\"+\"\\x1b[0;30;43m\"+\"Early Stopping Complete.\"+\"\\x1b[0m\"+\"\\n\")\n print(\"\\nOutputs lengths: \", output_lengths)\n return extracted_answers\n \n \n list_of_idx = list(range(len(outputs)))\n _, extracted_answers, _ = batch_message_filter(outputs, list_of_idx)\n print(\"\\nOutputs lengths: \", output_lengths)\n return extracted_answers\n```\nWill share the complete code soon!", "votes": 2, "replies": [ { "id": 3169244, "postDate": "2025-04-03T08:59:44.360Z", "content": "Great code!
", "rawMarkdown": "Great code!" }, { "id": 3169348, "postDate": "2025-04-03T11:47:35.060Z", "content": "Thanks. Looking forward to your code too
", "rawMarkdown": "Thanks. Looking forward to your code too" } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T06:41:41Z", "competition_id": 86023, "competition_slug": "ai-mathematical-olympiad-progress-prize-2", "team_id": 12818513, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-2/writeups/jk-piece-21st-place-solution" }