{ "id": 550336, "title": "5th place solution", "content": "

Our code is here.

\n

Basically, our solution consists of 3 ideas:

\n

1) Ensembling different solutions.

\n

\n

2) Applying different postprocessing filters.

\n

\n

We noticed that an algorithm can make typical mistakes. For instance, a genetic algorithm tends to produce redundant vertical or horizontal lines.

\n

\n

\n

\n

Also, another typical mistake is a wrong shape.

\n

\n

So, you can implement as many postprocessing filters as you can to cover the most common mistakes of algorithms in your ensemble.

\n

3) Brute-force. Tasks can be sorted by their identifiers, and the order remains the same across different submissions.

\n

\n

To identify the 26 tasks solved by the well-known 26 notebook, you need no more than 100 submissions. Once the 26 tasks are identified, you can try new algorithms one by one (checking whether they are able to solve at least one new task, that is one of the 74 remaining tasks). Once a new task is found, you can identify its ordinal number with the help of binary search. Also, you can leave all other tasks to the strongest algorithm.

\n

Our summary:

\n\n

Congratulations to all the winners, and thanks to the host team for organizing the competition! Looking forward to ARC Prize 2025!

", "comments": [ { "id": 3065711, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-12-07T07:31:23.487000", "content": "
\n

To identify the 26 tasks solved by the well-known 26 notebook, you need no more than 100 submissions. Once the 26 tasks are identified, you can try new algorithms one by one (checking whether they are able to solve at least one new task, that is one of the 74 remaining tasks). Once a new task is found, you can identify its ordinal number with the help of binary search. Also, you can leave all other tasks to the strongest algorithm.

\n
\n

Leaderboard probing is totally against the spirit of the competition.

\n

@gregkamradt probably next year edition should try to prevent this leaderboard probing without limiting legit approaches. On the AIMO competition, they created a python library that gave the tasks one by one, and you could not receive one task without answering the current one. That removed the option of leaderboard probing, but also removed the option of dynamically assigning resources to the different tasks. I don't know what is the best solution, but hopefully there is enough time to find one.

", "votes": 1, "replies": [ { "id": 3065946, "author_name": "Oleg Trott", "author_url": "", "post_date": "2024-12-07T13:35:43.493000", "content": "
\n

On the AIMO competition, they created a python library that gave the tasks one by one, and you could not receive one task without answering the current one.

\n
\n

I'm not finding anything probing-related in the AIMO2 discussions or rules. Could you post a link?

", "votes": 0, "replies": [ { "id": 3066013, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-12-07T14:42:24.357000", "content": "

I was talking about the first edition:

\n", "votes": 2, "replies": [] }, { "id": 3066016, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-12-07T14:43:09.433000", "content": "

And also this link: https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/497123

", "votes": 2, "replies": [] } ] }, { "id": 3068686, "author_name": "", "author_url": "", "post_date": "2024-12-10T14:55:30.923000", "content": "

I admit, the zero-feedback-except-the-final-score situation leaves one hungry for such information. If @gromml were really bent on breaking that rule, they probably would not have published their intent in their discussion entry. This odd level of honesty is probably a simple disconnect from the world of malicious behavior. I suffer from that, and I'm proud! But rules are rules… so that.

\n

Also, the simple act of introducing 100 (or 200) brand new challenges for the next contest would totally quash any gains from gaming 2024. The challenges must be orthogonally skewed, but seeing as almost every evaluation task handles that requirement with aplomb, it shouldn't be a too-extremely-high bar.

", "votes": 0, "replies": [] } ] } ], "comment_count": 5, "messages": [ { "id": 3065486, "postDate": "2024-12-06T21:56:25.173Z", "content": "

Our code is here.

\n

Basically, our solution consists of 3 ideas:

\n

1) Ensembling different solutions.

\n

\n

2) Applying different postprocessing filters.

\n

\n

We noticed that an algorithm can make typical mistakes. For instance, a genetic algorithm tends to produce redundant vertical or horizontal lines.

\n

\n

\n

\n

Also, another typical mistake is a wrong shape.

\n

\n

So, you can implement as many postprocessing filters as you can to cover the most common mistakes of algorithms in your ensemble.

\n

3) Brute-force. Tasks can be sorted by their identifiers, and the order remains the same across different submissions.

\n

\n

To identify the 26 tasks solved by the well-known 26 notebook, you need no more than 100 submissions. Once the 26 tasks are identified, you can try new algorithms one by one (checking whether they are able to solve at least one new task, that is one of the 74 remaining tasks). Once a new task is found, you can identify its ordinal number with the help of binary search. Also, you can leave all other tasks to the strongest algorithm.

\n

Our summary:

\n\n

Congratulations to all the winners, and thanks to the host team for organizing the competition! Looking forward to ARC Prize 2025!

", "rawMarkdown": "Our code is [here](https://www.kaggle.com/code/gromml/arc-prize-2024-poohai-solution).\n\nBasically, our solution consists of 3 ideas:\n\n1) **Ensembling different solutions.**\n\n\n\n2) **Applying different postprocessing filters.**\n\n\n\nWe noticed that an algorithm can make typical mistakes. For instance, a genetic algorithm tends to produce redundant vertical or horizontal lines.\n\n\n\n\n\n\n\nAlso, another typical mistake is a wrong shape.\n\n\n\nSo, you can implement as many postprocessing filters as you can to cover the most common mistakes of algorithms in your ensemble.\n\n3) **Brute-force.** Tasks can be sorted by their identifiers, and the order remains the same across different submissions.\n\n\n\nTo identify the 26 tasks solved by the [well-known 26 notebook](https://www.kaggle.com/code/mehrankazeminia/3-arc24-developed-2020-winning-solutions), you need no more than 100 submissions. Once the 26 tasks are identified, you can try new algorithms one by one (checking whether they are able to solve at least one new task, that is one of the 74 remaining tasks). Once a new task is found, you can identify its ordinal number with the help of binary search. Also, you can leave all other tasks to the strongest algorithm.\n\n**Our summary:**\n\n* A very efficient way to improve an ensemble was to identify tasks solved by the ensemble, and then try to solve other tasks by other algorithms\n* It seems that LLMs could be added to our ensemble, but fine-tuning is needed\n* Selecting the right attempt (the correct answer) is difficult, while removing wrong attempts is easier\n* An algorithm can make typical mistakes (like the genetic algorithm often produces redundant lines, and you can create the corresponding postprocessing filters)\n\nCongratulations to all the winners, and thanks to the host team for organizing the competition! Looking forward to ARC Prize 2025!", "votes": 11 }, { "id": 3065711, "postDate": "2024-12-07T07:31:23.487Z", "content": "
\n

To identify the 26 tasks solved by the well-known 26 notebook, you need no more than 100 submissions. Once the 26 tasks are identified, you can try new algorithms one by one (checking whether they are able to solve at least one new task, that is one of the 74 remaining tasks). Once a new task is found, you can identify its ordinal number with the help of binary search. Also, you can leave all other tasks to the strongest algorithm.

\n
\n

Leaderboard probing is totally against the spirit of the competition.

\n

@gregkamradt probably next year edition should try to prevent this leaderboard probing without limiting legit approaches. On the AIMO competition, they created a python library that gave the tasks one by one, and you could not receive one task without answering the current one. That removed the option of leaderboard probing, but also removed the option of dynamically assigning resources to the different tasks. I don't know what is the best solution, but hopefully there is enough time to find one.

", "rawMarkdown": "> To identify the 26 tasks solved by the well-known 26 notebook, you need no more than 100 submissions. Once the 26 tasks are identified, you can try new algorithms one by one (checking whether they are able to solve at least one new task, that is one of the 74 remaining tasks). Once a new task is found, you can identify its ordinal number with the help of binary search. Also, you can leave all other tasks to the strongest algorithm.\n\nLeaderboard probing is totally against the spirit of the competition.\n\n@gregkamradt probably next year edition should try to prevent this leaderboard probing without limiting legit approaches. On the AIMO competition, they created a python library that gave the tasks one by one, and you could not receive one task without answering the current one. That removed the option of leaderboard probing, but also removed the option of dynamically assigning resources to the different tasks. I don't know what is the best solution, but hopefully there is enough time to find one.", "votes": 1, "replies": [ { "id": 3065946, "postDate": "2024-12-07T13:35:43.493Z", "content": "
\n

On the AIMO competition, they created a python library that gave the tasks one by one, and you could not receive one task without answering the current one.

\n
\n

I'm not finding anything probing-related in the AIMO2 discussions or rules. Could you post a link?

", "rawMarkdown": "> On the AIMO competition, they created a python library that gave the tasks one by one, and you could not receive one task without answering the current one. \n\nI'm not finding anything probing-related in the AIMO2 discussions or rules. Could you post a link?", "replies": [ { "id": 3066013, "postDate": "2024-12-07T14:42:24.357Z", "content": "

I was talking about the first edition:

\n", "rawMarkdown": "I was talking about the first edition: \n\n- https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize\n- https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/495133\n- https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/495362", "votes": 2 }, { "id": 3066016, "postDate": "2024-12-07T14:43:09.433Z", "content": "

And also this link: https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/497123

", "rawMarkdown": "And also this link: https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/497123", "votes": 2 } ] }, { "id": 3068686, "postDate": "2024-12-10T14:55:30.923Z", "content": "

I admit, the zero-feedback-except-the-final-score situation leaves one hungry for such information. If @gromml were really bent on breaking that rule, they probably would not have published their intent in their discussion entry. This odd level of honesty is probably a simple disconnect from the world of malicious behavior. I suffer from that, and I'm proud! But rules are rules… so that.

\n

Also, the simple act of introducing 100 (or 200) brand new challenges for the next contest would totally quash any gains from gaming 2024. The challenges must be orthogonally skewed, but seeing as almost every evaluation task handles that requirement with aplomb, it shouldn't be a too-extremely-high bar.

", "rawMarkdown": "I admit, the zero-feedback-except-the-final-score situation leaves one hungry for such information. If @gromml were really bent on breaking that rule, they probably would not have published their intent in their discussion entry. This odd level of honesty is probably a simple disconnect from the world of malicious behavior. I suffer from that, and I'm proud! But rules are rules... so that.\n\nAlso, the simple act of introducing 100 (or 200) brand new challenges for the next contest would totally quash any gains from gaming 2024. The challenges must be orthogonally skewed, but seeing as almost every evaluation task handles that requirement with aplomb, it shouldn't be a too-extremely-high bar.", "isDeleted": true } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:23:25Z", "competition_id": 67357, "competition_slug": "arc-prize-2024", "team_id": 12333075, "solution_url": "/competitions/arc-prize-2024/writeups/poohai-5th-place-solution" }