{ "id": 516966, "title": "AIMO: 36th Place Solution", "content": "

Thanks to @abdurrafae 's early sharing, we had a solid foundation already and we started with that. Initially what worked well are:

\n
    \n
  1. Improved the parsing method, so that there are less noise during voting
    2. \n
  2. Early stopping of iterations, for example, we introduced something like the following in the begining of the iteration loop:
    3. \n
\n
 jj >= n_repetitions-: \n     len(occurances)>=:\n        , freq2 = occurances[][], occurances[][]\n         freq1 - freq2>=:\n             = True\n             best_stats[][], cond_iter_increase\n
\n

If, cond_iter_increase = True then we could use an extra round of iteration in the next problem.

\n
    \n
  1. We observed from the log and the validation set that, for easier problems, if the answer is found, it can be found in even 10 or 11 iterations. But there are some problems (1-2 problems generally out of 50 problems in the validation set) where it helps with more iterations. After some experiments, we realised that, we can afford around 20/ 21 iterations for 50% of the problems and 10/11 iterations for the rest. So, we had the following setup:
    2. \n
\n
prob_count = \nincrease = False\n\n PRIVATE:\n     test, sample_submission  iter_test:\n        prob_count = prob_count + \n         prob_count>:\n            n_repetitions =   increase  \n        :\n            n_repetitions =   increase  \n        answer, increase = (test, n_repetitions)\n        sample_submission = answer\n        env(sample_submission)\n        (test)\n        (sample_submission)\n
\n

The submission notebook is here: https://www.kaggle.com/code/indranilbhattacharya/aimo-2024-private-36th-place-solution/notebook

\n

While this was done a month ago, we picked this strategically as we reckoned that there could be many teams with same score, so we picked an older submission. Later we also tried with few other ideas. While, we still think some of them has potential, we could not get a steady result from validation and public LB.

\n
    \n
  1. Student- Teacher setup (Student generating the initial response and teacher validating that response)
    2. \n
  2. Identify difficult problems earlier.
    3. \n
  3. MCQ rounds for medium and difficult problems after early stopping. This gave good results in validation but without stability, so skipped this idea after a while. The idea is briefly as follows:
    4. \n
\n
def final_pred(problem, n_repetitions, n_repetitions_mcq):\n    occurances, best_stats, cond_iter_increase, hard = predict(problem, n_repetitions)\n    (occurances)\n    ()\n    (best_stats)\n\n      occurances   best_stats:\n        return 24,  ## This happens when time limit exceeds  all other problems are predicted as 24\n\n     len(occurances)<2:\n        (, best_stats[0][0])\n        return best_stats[0][0], cond_iter_increase\n\n    option1, option2 = occurances[0][0], occurances[1][0]\n    freq1, freq2 = occurances[0][1], occurances[1][1]\n\n     freq1>=5:\n        (, best_stats[0][0])\n        return best_stats[0][0], cond_iter_increase\n\n    all_options = [opt[0]  opt  occurances]\n\n    ans_dict = {ans:freq  ans, freq  occurances}\n\n    (, all_options)\n\n    global model,tokenizer,USE_PAST_KEY\n\n    final_message = f\n    prompt = f\n    (prompt)\n\n    final_iterations = []\n\n     m  tqdm(range(n_repetitions_mcq)):\n\n        torch.cuda.empty_cache()\n        gc.collect()\n\n        model_inputs = tokenizer(prompt, =).to(model.device)\n        input_len = len(model_inputs[][0])\n        generation_output = model.generate(**model_inputs,\n                                            =TOTAL_TOKENS,\n                                            =USE_PAST_KEY,\n                                            =,\n                                            temperature = 0.7,\n                                            top_p = 1.0,\n                                            =1, =stopping_criteria)\n\n        output_ids = generation_output.sequences[0]\n        raw_output = tokenizer.decode(output_ids[input_len:], =)\n        (f)\n        result_output = process_text_output(raw_output)\n\n        (, m,  , result_output)\n         result_output!=-1:\n            final_iterations.append(result_output)\n             result_output  ans_dict:\n                ans_dict[result_output] = ans_dict[result_output] + 1\n                 ans_dict[result_output]>=5:\n                    ()\n                    return result_output, cond_iter_increase\n            :\n                ans_dict[result_output] = 1\n\n    sorted_answers_combined = sorted(ans_dict.items(), =lambda x: x[1], =)\n\n    combined_ans, combined_freq = sorted_answers_combined[0][0], sorted_answers_combined[0][1]\n     combined_freq>=4:\n        return combined_ans, cond_iter_increase\n\n    counter = Counter(final_iterations)\n    sorted_answers_mcq = sorted(counter.items(), =lambda x: x[1], =)\n\n    ()\n    (sorted_answers_mcq)\n\n     sorted_answers_mcq  sorted_answers_mcq[0][1]>=2:\n        final_try = sorted_answers_mcq[0][0]\n         final_try  all_options:\n            (, final_try)\n            return final_try, cond_iter_increase \n         final_try   all_options:\n            (,final_try)\n            return final_try, cond_iter_increase \n        :\n            ()\n            return option1, cond_iter_increase\n\n    :\n        (, option1)\n        return option1, cond_iter_increase\n
", "comments": [ { "id": 2905652, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2024-07-05T04:59:09.927000", "content": "

Congratulations on securing 36th place in this competition. Thanks for sharing details of your solution.
\nI think MCQ round may require multiple options. How to generate these options ?

", "votes": 1, "replies": [ { "id": 2905932, "author_name": "Indranil Bhattacharya", "author_url": "", "post_date": "2024-07-05T08:50:04.257000", "content": "

Thank you. For the MCQ set-up, we start with the n_repetitions with the original predict function. Then we collect all the unique answers and use them in the prompt for the MCQ rounds for n_repetitions_mcq number of iterations. Note that we can consider two options, one where the voting from the original predict functions are kept and then we update the counter from the MCQ along with the original counter. Another option is to discard the original counter and only consider the voting from the MCQ rounds. Both were promising in the validation. However, they both had instability.

\n
def final_pred(problem, n_repetitions, n_repetitions_mcq):\n    occurances, best_stats, cond_iter_increase, hard = predict(problem, n_repetitions)\n    ...\n    ...\n    all_options = [opt[] for opt in occurances]\n
", "votes": 0, "replies": [] } ] } ], "comment_count": 2, "messages": [ { "id": 2904350, "postDate": "2024-07-04T10:05:40.157Z", "content": "

Thanks to @abdurrafae 's early sharing, we had a solid foundation already and we started with that. Initially what worked well are:

\n
    \n
  1. Improved the parsing method, so that there are less noise during voting
    2. \n
  2. Early stopping of iterations, for example, we introduced something like the following in the begining of the iteration loop:
    3. \n
\n
 jj >= n_repetitions-: \n     len(occurances)>=:\n        , freq2 = occurances[][], occurances[][]\n         freq1 - freq2>=:\n             = True\n             best_stats[][], cond_iter_increase\n
\n

If, cond_iter_increase = True then we could use an extra round of iteration in the next problem.

\n
    \n
  1. We observed from the log and the validation set that, for easier problems, if the answer is found, it can be found in even 10 or 11 iterations. But there are some problems (1-2 problems generally out of 50 problems in the validation set) where it helps with more iterations. After some experiments, we realised that, we can afford around 20/ 21 iterations for 50% of the problems and 10/11 iterations for the rest. So, we had the following setup:
    2. \n
\n
prob_count = \nincrease = False\n\n PRIVATE:\n     test, sample_submission  iter_test:\n        prob_count = prob_count + \n         prob_count>:\n            n_repetitions =   increase  \n        :\n            n_repetitions =   increase  \n        answer, increase = (test, n_repetitions)\n        sample_submission = answer\n        env(sample_submission)\n        (test)\n        (sample_submission)\n
\n

The submission notebook is here: https://www.kaggle.com/code/indranilbhattacharya/aimo-2024-private-36th-place-solution/notebook

\n

While this was done a month ago, we picked this strategically as we reckoned that there could be many teams with same score, so we picked an older submission. Later we also tried with few other ideas. While, we still think some of them has potential, we could not get a steady result from validation and public LB.

\n
    \n
  1. Student- Teacher setup (Student generating the initial response and teacher validating that response)
    2. \n
  2. Identify difficult problems earlier.
    3. \n
  3. MCQ rounds for medium and difficult problems after early stopping. This gave good results in validation but without stability, so skipped this idea after a while. The idea is briefly as follows:
    4. \n
\n
def final_pred(problem, n_repetitions, n_repetitions_mcq):\n    occurances, best_stats, cond_iter_increase, hard = predict(problem, n_repetitions)\n    (occurances)\n    ()\n    (best_stats)\n\n      occurances   best_stats:\n        return 24,  ## This happens when time limit exceeds  all other problems are predicted as 24\n\n     len(occurances)<2:\n        (, best_stats[0][0])\n        return best_stats[0][0], cond_iter_increase\n\n    option1, option2 = occurances[0][0], occurances[1][0]\n    freq1, freq2 = occurances[0][1], occurances[1][1]\n\n     freq1>=5:\n        (, best_stats[0][0])\n        return best_stats[0][0], cond_iter_increase\n\n    all_options = [opt[0]  opt  occurances]\n\n    ans_dict = {ans:freq  ans, freq  occurances}\n\n    (, all_options)\n\n    global model,tokenizer,USE_PAST_KEY\n\n    final_message = f\n    prompt = f\n    (prompt)\n\n    final_iterations = []\n\n     m  tqdm(range(n_repetitions_mcq)):\n\n        torch.cuda.empty_cache()\n        gc.collect()\n\n        model_inputs = tokenizer(prompt, =).to(model.device)\n        input_len = len(model_inputs[][0])\n        generation_output = model.generate(**model_inputs,\n                                            =TOTAL_TOKENS,\n                                            =USE_PAST_KEY,\n                                            =,\n                                            temperature = 0.7,\n                                            top_p = 1.0,\n                                            =1, =stopping_criteria)\n\n        output_ids = generation_output.sequences[0]\n        raw_output = tokenizer.decode(output_ids[input_len:], =)\n        (f)\n        result_output = process_text_output(raw_output)\n\n        (, m,  , result_output)\n         result_output!=-1:\n            final_iterations.append(result_output)\n             result_output  ans_dict:\n                ans_dict[result_output] = ans_dict[result_output] + 1\n                 ans_dict[result_output]>=5:\n                    ()\n                    return result_output, cond_iter_increase\n            :\n                ans_dict[result_output] = 1\n\n    sorted_answers_combined = sorted(ans_dict.items(), =lambda x: x[1], =)\n\n    combined_ans, combined_freq = sorted_answers_combined[0][0], sorted_answers_combined[0][1]\n     combined_freq>=4:\n        return combined_ans, cond_iter_increase\n\n    counter = Counter(final_iterations)\n    sorted_answers_mcq = sorted(counter.items(), =lambda x: x[1], =)\n\n    ()\n    (sorted_answers_mcq)\n\n     sorted_answers_mcq  sorted_answers_mcq[0][1]>=2:\n        final_try = sorted_answers_mcq[0][0]\n         final_try  all_options:\n            (, final_try)\n            return final_try, cond_iter_increase \n         final_try   all_options:\n            (,final_try)\n            return final_try, cond_iter_increase \n        :\n            ()\n            return option1, cond_iter_increase\n\n    :\n        (, option1)\n        return option1, cond_iter_increase\n
", "rawMarkdown": "Thanks to @abdurrafae 's early sharing, we had a solid foundation already and we started with that. Initially what worked well are:\n1. Improved the parsing method, so that there are less noise during voting\n2. Early stopping of iterations, for example, we introduced something like the following in the begining of the iteration loop:\n```\nif jj >= n_repetitions-1: \n if len(occurances)>=2:\n freq1, freq2 = occurances[0][1], occurances[1][1]\n if freq1 - freq2>=1:\n cond_iter_increase = True\n return best_stats[0][0], cond_iter_increase\n```\nIf, `cond_iter_increase = True` then we could use an extra round of iteration in the next problem.\n3. We observed from the log and the validation set that, for easier problems, if the answer is found, it can be found in even 10 or 11 iterations. But there are some problems (1-2 problems generally out of 50 problems in the validation set) where it helps with more iterations. After some experiments, we realised that, we can afford around 20/ 21 iterations for 50% of the problems and 10/11 iterations for the rest. So, we had the following setup:\n\n```\nprob_count = 0\nincrease = False\n\nif PRIVATE:\n for test, sample_submission in iter_test:\n prob_count = prob_count + 1\n if prob_count>25:\n n_repetitions = 21 if increase else 22\n else:\n n_repetitions = 11 if increase else 12\n answer, increase = predict(test['problem'].values[0], n_repetitions)\n sample_submission['answer'] = answer\n env.predict(sample_submission)\n print(test)\n print(sample_submission)\n```\n The submission notebook is here: https://www.kaggle.com/code/indranilbhattacharya/aimo-2024-private-36th-place-solution/notebook\n\nWhile this was done a month ago, we picked this strategically as we reckoned that there could be many teams with same score, so we picked an older submission. Later we also tried with few other ideas. While, we still think some of them has potential, we could not get a steady result from validation and public LB.\n\n1. Student- Teacher setup (Student generating the initial response and teacher validating that response)\n2. Identify difficult problems earlier. \n3. MCQ rounds for medium and difficult problems after early stopping. This gave good results in validation but without stability, so skipped this idea after a while. The idea is briefly as follows:\n\n```\ndef final_pred(problem, n_repetitions, n_repetitions_mcq):\n occurances, best_stats, cond_iter_increase, hard = predict(problem, n_repetitions)\n print(occurances)\n print(\"--\"*10)\n print(best_stats)\n \n if not occurances and not best_stats:\n return 24, False ## This happens when time limit exceeds and all other problems are predicted as 24\n \n if len(occurances)<2:\n print(\"The question was easy and the asnwer is: \", best_stats[0][0])\n return best_stats[0][0], cond_iter_increase\n \n option1, option2 = occurances[0][0], occurances[1][0]\n freq1, freq2 = occurances[0][1], occurances[1][1]\n \n if freq1>=5:\n print(\"The question was easy and the asnwer is: \", best_stats[0][0])\n return best_stats[0][0], cond_iter_increase\n \n all_options = [opt[0] for opt in occurances]\n \n ans_dict = {ans:freq for ans, freq in occurances}\n \n print(\"all options are: \", all_options)\n \n global model,tokenizer,USE_PAST_KEY\n \n final_message = f\"Please solve the following problem: {problem}. Hint: The final answer is a non-negative integer modulo 1000 and is one of the following integers: {all_options}. Please reason step by step and check which one is correct and put final numerical answer within \\\\boxed{{}}.\\Solution:\"\n prompt = f\"User: {final_message}\"\n print(prompt)\n \n final_iterations = []\n \n for m in tqdm(range(n_repetitions_mcq)):\n \n torch.cuda.empty_cache()\n gc.collect()\n\n model_inputs = tokenizer(prompt, return_tensors='pt').to(model.device)\n input_len = len(model_inputs['input_ids'][0])\n generation_output = model.generate(**model_inputs,\n max_new_tokens=TOTAL_TOKENS,\n return_dict_in_generate=USE_PAST_KEY,\n do_sample=True,\n temperature = 0.7,\n top_p = 1.0,\n num_return_sequences=1, stopping_criteria=stopping_criteria)\n \n output_ids = generation_output.sequences[0]\n raw_output = tokenizer.decode(output_ids[input_len:], skip_special_tokens=True)\n print(f\"\\FINAL ITERATION:\\n{raw_output}\\n\")\n result_output = process_text_output(raw_output)\n \n print(\"Answer from final iteration MCQ round \", m, \":\" , result_output)\n if result_output!=-1:\n final_iterations.append(result_output)\n if result_output in ans_dict:\n ans_dict[result_output] = ans_dict[result_output] + 1\n if ans_dict[result_output]>=5:\n print(\"MCQ round found originally most frequnet solution\")\n return result_output, cond_iter_increase\n else:\n ans_dict[result_output] = 1\n \n sorted_answers_combined = sorted(ans_dict.items(), key=lambda x: x[1], reverse=True)\n \n combined_ans, combined_freq = sorted_answers_combined[0][0], sorted_answers_combined[0][1]\n if combined_freq>=4:\n return combined_ans, cond_iter_increase\n \n counter = Counter(final_iterations)\n sorted_answers_mcq = sorted(counter.items(), key=lambda x: x[1], reverse=True)\n \n print(\"From MCQ Rounds\")\n print(sorted_answers_mcq)\n \n if sorted_answers_mcq and sorted_answers_mcq[0][1]>=2:\n final_try = sorted_answers_mcq[0][0]\n if final_try in all_options:\n print(\"Final Answer from MCQ is: \", final_try)\n return final_try, cond_iter_increase \n if final_try not in all_options:\n print(\"Final Answer from MCQ is not from option and is: \",final_try)\n return final_try, cond_iter_increase \n else:\n print(\"MCQ answer is not from options\")\n return option1, cond_iter_increase\n \n else:\n print(\"MCQ round did not help and the answer is: \", option1)\n return option1, cond_iter_increase\n```\n\n\n\n\n\n", "votes": 13 }, { "id": 2905652, "postDate": "2024-07-05T04:59:09.927Z", "content": "

Congratulations on securing 36th place in this competition. Thanks for sharing details of your solution.
\nI think MCQ round may require multiple options. How to generate these options ?

", "rawMarkdown": "Congratulations on securing 36th place in this competition. Thanks for sharing details of your solution. \nI think MCQ round may require multiple options. How to generate these options ? ", "votes": 1, "replies": [ { "id": 2905932, "postDate": "2024-07-05T08:50:04.257Z", "content": "

Thank you. For the MCQ set-up, we start with the n_repetitions with the original predict function. Then we collect all the unique answers and use them in the prompt for the MCQ rounds for n_repetitions_mcq number of iterations. Note that we can consider two options, one where the voting from the original predict functions are kept and then we update the counter from the MCQ along with the original counter. Another option is to discard the original counter and only consider the voting from the MCQ rounds. Both were promising in the validation. However, they both had instability.

\n
def final_pred(problem, n_repetitions, n_repetitions_mcq):\n    occurances, best_stats, cond_iter_increase, hard = predict(problem, n_repetitions)\n    ...\n    ...\n    all_options = [opt[] for opt in occurances]\n
", "rawMarkdown": "Thank you. For the MCQ set-up, we start with the `n_repetitions` with the original predict function. Then we collect all the unique answers and use them in the prompt for the MCQ rounds for `n_repetitions_mcq` number of iterations. Note that we can consider two options, one where the voting from the original predict functions are kept and then we update the counter from the MCQ along with the original counter. Another option is to discard the original counter and only consider the voting from the MCQ rounds. Both were promising in the validation. However, they both had instability. \n\n```\ndef final_pred(problem, n_repetitions, n_repetitions_mcq):\n occurances, best_stats, cond_iter_increase, hard = predict(problem, n_repetitions)\n ...\n ...\n all_options = [opt[0] for opt in occurances]\n```" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T06:54:07Z", "competition_id": 73231, "competition_slug": "ai-mathematical-olympiad-prize", "team_id": 11993094, "solution_url": "/competitions/ai-mathematical-olympiad-prize/writeups/the-ai-of-pi-aimo-36th-place-solution" }