![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6388%2Fcfa0463f4c5add35e5bbb29706700c9a%2F2025-11-01%2017.37.03.png?generation=1761986797391835&alt=media\")
First off, thank you to all the participants and the organizers for this fantastic competition!
\nI spent a ton of time on code golf for the first time in about 20 years. Back in the day, as a code golf enthusiast, I enjoyed golfing various ICPC problems in C++. This time, I participated as an enthusiast of AI Agents and test-time inference, and I thoroughly enjoyed every moment of it. I found myself endlessly spending time trying to push the boundaries of what AI can do. I'm actually relieved the competition is finally over☺️.
\nFor my contributions, at least, I'd say my solutions were 98% generated by an LLM and only about 2% by human effort. To be honest, I don't even understand most of the final golfed code. Here, I'll describe my part of our team's approach.
\nMy core strategy was to have an LLM (gpt5-codex) perform code golf using tools in parallel. gpt5-codex has very high tool utilization capabilities, and was able to analyze JSON test cases in a REPL and do various things on its own. For each task, we would run a single conversation session consisting of 10-30 turns. This entire process was repeated many times every day. The duration of a single turn varied depending on the problem's difficulty, ranging from 5 minutes to over an hour.
The figure below illustrates one \"turn\" in a session. It begins with a prompt and generated 4 parallel versions. Then, each generated code is evaluated, and the shortest valid code is selected. Code golfing continues with a follow-up prompt for the next turn.
\n
Through experimentation, I found that providing the LLM with specific, concrete ideas for improvement was effective. The base follow-up prompt was simple:
\n\nOn top of this, I generated additional rule-based prompts using the code's AST. For example:
\nI co-developed the AST-based prompt generation script with grok4-code-fast-1 and gpt5-codex. If I had embedded more domain-specific code golf knowledge into these rules, the speed of improvement could have been even faster.
I terminated a conversation session if the score failed to improve for $s$ consecutive turns. If there was no improvement for $t$ consecutive turns (where $t$ < $s$), I would prompt the model to be more exploratory with instructions like:
\n\n\n\nI stopped using the third, commented-out prompt as I found it wasn't very effective.
\nWith the exception of the first two weeks of August, I primarily used Codex Cloud, following the framework described above. In Codex Cloud, you can't explicitly select a model, but I believe gpt5 was used for the first month and gpt5-codex for the latter two. I ran about 15 conversation sessions in parallel, with each turn in every session executing 4 parallel versions (N=4).\nAll of this was done within the limits of a ChatGPT Pro subscription.
\nThis workflow was essentially a human-in-the-loop algorithm: I manually operated Codex Cloud based on a set of rules. Even when I was away from my PC—on a bus, on a train, or waiting for a flight—I was constantly pasting prompts into my smartphone and hitting submit every single day for three months. It felt like being a worker in a dystopian world.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6388%2Fcf9302e509ea35fa05a8bfb6f6c9824b%2F2025-11-01%2017.52.53.png?generation=1761987192978230&alt=media\")
The daily work of copying and pasting in Codex Cloud was painful, so I attempted to automate and scale up the process using the codex exec (CLI) and API credits. Due to cost constraints, I haven't run it very often. I ran it while sleeping or during work.
To create a safe environment for code improvement, I set up a Docker container and ran the codex exec (CLI) inside it. The results of these experiments were logged to an SQLite database. Using a managed sandbox environment like katakate might be a better approach.
I also experimented with ensembling different models like Grok4, but I didn't spend enough time for prompt engineering and gave up before getting good results. For example, when I used grok4-code-fast-1 with codex, I recall it trying to download ARC testdata with curl, failing to use tools correctly, and taking a roundabout thought process. This could likely have been improved with better prompt tuning. grok4 performed relatively well but was burning through my API credits at an alarming rate, so I abandoned it quickly.
When using Codex Cloud (Web), the model had access to the repository containing the previously generated code. This often caused it to get stuck in a local optimum, unable to break away from the existing solution.
\nTo solve this, when using codex exec (CLI), I created a clean, empty sandbox environment for each new session. This forced the model to start from a blank slate. I remember breaking out of several difficult local optima on a few tasks using this method.
\nI also tried to achieve this in Codex Cloud (Web) by explicitly instructing the model to start from scratch. It wasn't perfect, but I think it worked. The prompt included a line like:
\n\nIn the last 2 weeks, I managed to significantly improve our scores by having the LLM optimize for \"compression-friendly\" code. The idea was to generate code that would be highly compressible with zip(zopfli). I used the same Parallel Sampling framework but changed the evaluation function to score based on the post-compression file size. The prompt for this was:
\n\nI thought this was a unique improvement that other teams might not have tried, but in the end, their code was often much shorter, which was a bit disappointing. I did have one 💎 solution where this technique worked exceptionally well: Task 324. You can see our compressed solution here: https://gcgc.pages.dev/task324.
\n ():\n n=(a,[]);c=n.count;o=n[c(n[])<];f,s={},{()}\n i,e (a):\n j,t (e):\n c(t)<:f[o e o n[j::(e)]]=t;s|={i+j,(i-j,)}\n i,e (a):\n j,t (e):\n {i+j,(i-j,)}&s:e[j]=f[o e o n[j::(e)]]\n a\nI might add more details about other minor tricks later.
\nFinally, I want to thank my teammate Jokrasa, who dedicated so much time to this competition despite his busy university schedule. I was initially very skeptical about regex-based improvements, but he completely changed my mind.
\nI also want to thank Thunder Thunder, who brought a sophisticated golfer's perspective to tasks where our AI approach had completely stalled. His manual improvements were crucial for breaking out of local optima, as you can see in the Solution History on our dashboard. Congratulations on becoming a Kaggle Master!
", "comments": [ { "id": 3309959, "author_name": "jazivxt", "author_url": "", "post_date": "2025-11-01T19:15:55.563000", "content": "Love the honesty on \" I don't even understand most of the final golfed code\", I would echo that sentiment. But on the comment of about 2% by human effort, I'd say we are downplaying the ingenuity and effort it took to just get the initial solutions to kick start the LLMs and all the human iterations and testing necessary to achieve these low scores was a huge undertaking. Our team was complimented with @radek1 tech stack and ingenuity along with complementing hand coding and brute force that kept the LLMs sweeping through new instances but it was a huge amount of human effort, that if we had only focused on the coding would have probably yielded similar results possibly. The only difference the LLMs will be smarter now with all the wealth of back and fourth rich discussion generated to be better for the next round. Great writeup, loved it!
", "votes": 4, "replies": [ { "id": 3310061, "author_name": "Kohei", "author_url": "", "post_date": "2025-11-02T01:47:59.663000", "content": "Thanks for putting that into perspective! You're absolutely right to call me out on the \"2%\" figure. It was more of a rough metric and not meant to diminish the huge amount of human work that went into this. It all depends on how you measure it—whether it's the final code size or the crucial thinking time.
", "votes": 1, "replies": [] } ] }, { "id": 3309829, "author_name": "bizy-coder", "author_url": "", "post_date": "2025-11-01T14:51:19.070000", "content": "Some people still golf some ICPC problems (mostly just in Python) https://open.kattis.com/problem-sources/2025%20ICPC%20North%20America%20Championship\nWish more people did! They’re often very fun problems. I’m likely in a minority but I think the easier half of these sets would be great problems for a future code golf contest.
\nI’m curious how easy you found it to get AI to use regex? From just prompting on the web app, my experience was that it didn’t understand the idea. At best it would try to remove all list parts which would be very long, and for non-obvious regex it failed to find the approach at all.
\nAlso, how did you handle the problems which had one bad test case? I think there were a few where people decided to hard code a condition for a single case. This seems like something AI would be very hesitant to do - in general I imagine they don’t love brute forcing for hard codes?
", "votes": 4, "replies": [ { "id": 3310059, "author_name": "Kohei", "author_url": "", "post_date": "2025-11-02T01:18:26.097000", "content": "Hi, thank you for the link and information! It's great to learn about a scene I wasn't aware of.
\nRegarding your question about regex, my experience was very similar to yours. I also found it difficult to get the model to generate regex-based code. I believe one key reason for my eventual success was the number of sampling attempts. I saw it generate code that would import re but then never actually use the module many times…
Here are some examples. One successful example is here: https://gcgc.pages.dev/task332#raw-88c00bd6. As I wrote in the article, this was achieved simply by instructing it to try regex in a follow-up prompt. My hypothesis is that codex (gpt5-codex) was able to read the ARCGEN code included in the repository, verbalize the task, correctly infer the necessary regex from that description, and then verify its hypothesis in the REPL. I need to review the CoT logs to confirm this, which I plan to do. I also tried adding a regex template to the prompt, but it didn't seem effective, so I quickly abandoned that approach.
In several cases, our AI Agent's main contribution was shortening an existing regex solution originally created by THUNDER THUNDER. Like in https://gcgc.pages.dev/task077#raw-311d84fe and https://gcgc.pages.dev/task379#raw-311d84fe. It rarely generated regex code from scratch. This highlights the need for a benchmark to evaluate the regex capabilities of AI Agents.
\n\n\nhow did you handle the problems which had one bad test case?
\n
That's a great question! I didn't implement any special handling for that. However, I was closely monitoring the gpt5-codex's CoT logs and noticed something interesting. Even without any specific instructions, it would execute Python code in its REPL to load the test case JSON data and then count or identify the specific cases that were failing. Since this analysis becomes part of the LLM's context, I believe it's possible in principle for it to hard-code a solution for a single bad test case.
\nIf you have the taskIDs for those problems, I'd love to take a look at the logs! Let me know.
", "votes": 2, "replies": [] } ] }, { "id": 3310019, "author_name": "Chris Deotte", "author_url": "", "post_date": "2025-11-01T21:56:15.093000", "content": "Congratulations achieving 4th place! We used similar LLM agents and combined them with human solutions.
\nDo you think LLM could achieve gold without beginning with human created solutions? i.e. do you think in 3 months, LLM could just read all the 400 problem descriptions (with no prior human solutions) and finish in gold LB?
", "votes": 1, "replies": [ { "id": 3310068, "author_name": "Kohei", "author_url": "", "post_date": "2025-11-02T02:36:24.717000", "content": "I believe it is possible under certain conditions.
\nWow! Inspirational work and write-up. Thank you.
", "votes": 1, "replies": [] }, { "id": 3310525, "author_name": "Aaditya Khanal", "author_url": "", "post_date": "2025-11-03T06:13:40.547000", "content": "Your approach with parallel sampling and dynamic prompts is brilliant! Love the creativity in escaping local optima. Great job on 4th place!
", "votes": 0, "replies": [] }, { "id": 3310062, "author_name": "James Leung2123", "author_url": "", "post_date": "2025-11-02T02:13:14.560000", "content": "Congrats on 4th! I actually tried the exact same parallel sampling framework in the first few weeks of the competition, but unfortunately none of us had enough compute credits so we ended up resorting to golfing by hand 😂. (P.S. we did the compression golfing by hand too but it's very cool that you managed to get a diamond using gpt for this!)
", "votes": 0, "replies": [] }, { "id": 3309780, "author_name": "", "author_url": "", "post_date": "2025-11-01T11:37:39.570000", "content": "", "votes": 0, "replies": [] }, { "id": 3310129, "author_name": "Navneet", "author_url": "", "post_date": "2025-11-02T06:38:45.487000", "content": "Thank you for compression-friendly code
", "votes": 0, "replies": [] } ], "comment_count": 11, "messages": [ { "id": 3309737, "postDate": "2025-11-01T09:03:12.103Z", "content": "First off, thank you to all the participants and the organizers for this fantastic competition!
\nI spent a ton of time on code golf for the first time in about 20 years. Back in the day, as a code golf enthusiast, I enjoyed golfing various ICPC problems in C++. This time, I participated as an enthusiast of AI Agents and test-time inference, and I thoroughly enjoyed every moment of it. I found myself endlessly spending time trying to push the boundaries of what AI can do. I'm actually relieved the competition is finally over☺️.
\nFor my contributions, at least, I'd say my solutions were 98% generated by an LLM and only about 2% by human effort. To be honest, I don't even understand most of the final golfed code. Here, I'll describe my part of our team's approach.
\nMy core strategy was to have an LLM (gpt5-codex) perform code golf using tools in parallel. gpt5-codex has very high tool utilization capabilities, and was able to analyze JSON test cases in a REPL and do various things on its own. For each task, we would run a single conversation session consisting of 10-30 turns. This entire process was repeated many times every day. The duration of a single turn varied depending on the problem's difficulty, ranging from 5 minutes to over an hour.
The figure below illustrates one \"turn\" in a session. It begins with a prompt and generated 4 parallel versions. Then, each generated code is evaluated, and the shortest valid code is selected. Code golfing continues with a follow-up prompt for the next turn.
\n
Through experimentation, I found that providing the LLM with specific, concrete ideas for improvement was effective. The base follow-up prompt was simple:
\n\nOn top of this, I generated additional rule-based prompts using the code's AST. For example:
\nI co-developed the AST-based prompt generation script with grok4-code-fast-1 and gpt5-codex. If I had embedded more domain-specific code golf knowledge into these rules, the speed of improvement could have been even faster.
I terminated a conversation session if the score failed to improve for $s$ consecutive turns. If there was no improvement for $t$ consecutive turns (where $t$ < $s$), I would prompt the model to be more exploratory with instructions like:
\n\n\n\nI stopped using the third, commented-out prompt as I found it wasn't very effective.
\nWith the exception of the first two weeks of August, I primarily used Codex Cloud, following the framework described above. In Codex Cloud, you can't explicitly select a model, but I believe gpt5 was used for the first month and gpt5-codex for the latter two. I ran about 15 conversation sessions in parallel, with each turn in every session executing 4 parallel versions (N=4).\nAll of this was done within the limits of a ChatGPT Pro subscription.
\nThis workflow was essentially a human-in-the-loop algorithm: I manually operated Codex Cloud based on a set of rules. Even when I was away from my PC—on a bus, on a train, or waiting for a flight—I was constantly pasting prompts into my smartphone and hitting submit every single day for three months. It felt like being a worker in a dystopian world.
\n
The daily work of copying and pasting in Codex Cloud was painful, so I attempted to automate and scale up the process using the codex exec (CLI) and API credits. Due to cost constraints, I haven't run it very often. I ran it while sleeping or during work.
To create a safe environment for code improvement, I set up a Docker container and ran the codex exec (CLI) inside it. The results of these experiments were logged to an SQLite database. Using a managed sandbox environment like katakate might be a better approach.
I also experimented with ensembling different models like Grok4, but I didn't spend enough time for prompt engineering and gave up before getting good results. For example, when I used grok4-code-fast-1 with codex, I recall it trying to download ARC testdata with curl, failing to use tools correctly, and taking a roundabout thought process. This could likely have been improved with better prompt tuning. grok4 performed relatively well but was burning through my API credits at an alarming rate, so I abandoned it quickly.
When using Codex Cloud (Web), the model had access to the repository containing the previously generated code. This often caused it to get stuck in a local optimum, unable to break away from the existing solution.
\nTo solve this, when using codex exec (CLI), I created a clean, empty sandbox environment for each new session. This forced the model to start from a blank slate. I remember breaking out of several difficult local optima on a few tasks using this method.
\nI also tried to achieve this in Codex Cloud (Web) by explicitly instructing the model to start from scratch. It wasn't perfect, but I think it worked. The prompt included a line like:
\n\nIn the last 2 weeks, I managed to significantly improve our scores by having the LLM optimize for \"compression-friendly\" code. The idea was to generate code that would be highly compressible with zip(zopfli). I used the same Parallel Sampling framework but changed the evaluation function to score based on the post-compression file size. The prompt for this was:
\n\nI thought this was a unique improvement that other teams might not have tried, but in the end, their code was often much shorter, which was a bit disappointing. I did have one 💎 solution where this technique worked exceptionally well: Task 324. You can see our compressed solution here: https://gcgc.pages.dev/task324.
\n ():\n n=(a,[]);c=n.count;o=n[c(n[])<];f,s={},{()}\n i,e (a):\n j,t (e):\n c(t)<:f[o e o n[j::(e)]]=t;s|={i+j,(i-j,)}\n i,e (a):\n j,t (e):\n {i+j,(i-j,)}&s:e[j]=f[o e o n[j::(e)]]\n a\nI might add more details about other minor tricks later.
\nFinally, I want to thank my teammate Jokrasa, who dedicated so much time to this competition despite his busy university schedule. I was initially very skeptical about regex-based improvements, but he completely changed my mind.
\nI also want to thank Thunder Thunder, who brought a sophisticated golfer's perspective to tasks where our AI approach had completely stalled. His manual improvements were crucial for breaking out of local optima, as you can see in the Solution History on our dashboard. Congratulations on becoming a Kaggle Master!
", "rawMarkdown": "First off, thank you to all the participants and the organizers for this fantastic competition!\n\nI spent a ton of time on code golf for the first time in about 20 years. Back in the day, as a code golf enthusiast, I enjoyed golfing various ICPC problems in C++. This time, I participated as an enthusiast of AI Agents and test-time inference, and I thoroughly enjoyed every moment of it. I found myself endlessly spending time trying to push the boundaries of what AI can do. I'm actually relieved the competition is finally over☺️.\n\nFor my contributions, at least, I'd say my solutions were 98% generated by an LLM and only about 2% by human effort. To be honest, I don't even understand most of the final golfed code. Here, I'll describe my part of our team's approach.\n\n# Parallel Sampling\n\nMy core strategy was to have an LLM (`gpt5-codex`) perform code golf using tools in parallel. gpt5-codex has very high tool utilization capabilities, and was able to analyze JSON test cases in a REPL and do various things on its own. For each task, we would run a single conversation session consisting of 10-30 turns. This entire process was repeated many times every day. The duration of a single turn varied depending on the problem's difficulty, ranging from 5 minutes to over an hour.\n\nThe figure below illustrates one \"turn\" in a session. It begins with a prompt and generated 4 parallel versions. Then, each generated code is evaluated, and the shortest valid code is selected. Code golfing continues with a follow-up prompt for the next turn.\n\n\n\n# Rule-based Prompt Generation\n\nThrough experimentation, I found that providing the LLM with specific, concrete ideas for improvement was effective. The base follow-up prompt was simple:\n\n```\n\"Please make it as short as possible in bytes for Code Golf and tell me the file size.\"\n```\n\nOn top of this, I generated additional rule-based prompts using the code's AST. For example:\n\n- If the code didn't use regular expressions, I added: \"Try to use the regex re package to code golf. It might be a key to a breakthrough.\"\n- If the code used loops, I instructed it to try converting them into recursive functions.\n- If a function was defined with def, I prompted it to use lambda. While not always optimal, this was often the better choice.\n\nI co-developed the AST-based prompt generation script with `grok4-code-fast-1` and `gpt5-codex`. If I had embedded more domain-specific code golf knowledge into these rules, the speed of improvement could have been even faster.\n\n# Stop Criterion & Exploration\n\nI terminated a conversation session if the score failed to improve for $s$ consecutive turns. If there was no improvement for $t$ consecutive turns (where $t$ < $s$), I would prompt the model to be more exploratory with instructions like:\n\n```python\n\"Analyze the test cases generation code.\"\n\"Rethink a more compact algorithm.\"\n# \"Focus on reducing the number of bytes, even if it means sacrificing some functionality.\"\n```\n\nI stopped using the third, commented-out prompt as I found it wasn't very effective.\n\n# Codex Cloud (Web)\n\nWith the exception of the first two weeks of August, I primarily used Codex Cloud, following the framework described above. In Codex Cloud, you can't explicitly select a model, but I believe gpt5 was used for the first month and gpt5-codex for the latter two. I ran about 15 conversation sessions in parallel, with each turn in every session executing 4 parallel versions (N=4).\nAll of this was done within the limits of a ChatGPT Pro subscription.\n\nThis workflow was essentially a human-in-the-loop algorithm: I manually operated Codex Cloud based on a set of rules. Even when I was away from my PC—on a bus, on a train, or waiting for a flight—I was constantly pasting prompts into my smartphone and hitting submit every single day for three months. It felt like being a worker in a dystopian world.\n\n\n\n# codex exec (CLI)\n\nThe daily work of copying and pasting in Codex Cloud was painful, so I attempted to automate and scale up the process using the `codex exec` (CLI) and API credits. Due to cost constraints, I haven't run it very often. I ran it while sleeping or during work.\n\nTo create a safe environment for code improvement, I set up a Docker container and ran the `codex exec` (CLI) inside it. The results of these experiments were logged to an SQLite database. Using a managed sandbox environment like [katakate](https://katakate.org/) might be a better approach.\n\nI also experimented with ensembling different models like Grok4, but I didn't spend enough time for prompt engineering and gave up before getting good results. For example, when I used `grok4-code-fast-1` with codex, I recall it trying to download ARC testdata with curl, failing to use tools correctly, and taking a roundabout thought process. This could likely have been improved with better prompt tuning. `grok4` performed relatively well but was burning through my API credits at an alarming rate, so I abandoned it quickly.\n\n# Escaping Local Optima\n\nWhen using Codex Cloud (Web), the model had access to the repository containing the previously generated code. This often caused it to get stuck in a local optimum, unable to break away from the existing solution.\n\nTo solve this, when using codex exec (CLI), I created a clean, empty sandbox environment for each new session. This forced the model to start from a blank slate. I remember breaking out of several difficult local optima on a few tasks using this method.\n\nI also tried to achieve this in Codex Cloud (Web) by explicitly instructing the model to start from scratch. It wasn't perfect, but I think it worked. The prompt included a line like:\n\n```\n\"Ignore the code that's already implemented and write it from scratch.\"\n```\n\n# LLM optimize for \"compression-friendly\" code\n\nIn the last 2 weeks, I managed to significantly improve our scores by having the LLM optimize for \"compression-friendly\" code. The idea was to generate code that would be highly compressible with zip(zopfli). I used the same Parallel Sampling framework but changed the evaluation function to score based on the post-compression file size. The prompt for this was:\n\n```python\n\"This must be achieved with minimal Kolmogorov complexity and is evaluated based on the file size after zip compression.\"\n```\n\nI thought this was a unique improvement that other teams might not have tried, but in the end, their code was often much shorter, which was a bit disappointing. I did have one 💎 solution where this technique worked exceptionally well: Task 324. You can see our compressed solution here: https://gcgc.pages.dev/task324.\n\n```python\ndef p(a):\n n=sum(a,[]);c=n.count;o=n[c(n[0])<4];f,s={},{()}\n for i,e in enumerate(a):\n for j,t in enumerate(e):\n if c(t)<4:f[o in e and o in n[j::len(e)]]=t;s|={i+j,(i-j,)}\n for i,e in enumerate(a):\n for j,t in enumerate(e):\n if{i+j,(i-j,)}&s:e[j]=f[o in e and o in n[j::len(e)]]\n return a\n```\n\n# Other things\n\n* Sometimes, LLMs seem to recognize their own limitations and stop trying to improve. To encourage them to continue with CoT, I added a prompt, \"Shorter code is known to exist.\"\n* I converted the code defined in the ARCGEN repository, made it executable in a single file, and placed it in the repository for agent use. This proved useful for verbalizing the task. The converted code can be viewed under the “Generator Code” tab at https://gcgc.pages.dev/task003.\n* The origin of \"Fuun\": https://web.archive.org/web/20110724172015/http://save-endo.cs.uu.nl/Endo.pdf\n* ...\n\nI might add more details about other minor tricks later.\n\n# Acknowledgments\n\nFinally, I want to thank my teammate Jokrasa, who dedicated so much time to this competition despite his busy university schedule. I was initially very skeptical about regex-based improvements, but he completely changed my mind.\n\nI also want to thank Thunder Thunder, who brought a sophisticated golfer's perspective to tasks where our AI approach had completely stalled. His manual improvements were crucial for breaking out of local optima, as you can see in the Solution History on our dashboard. Congratulations on becoming a Kaggle Master!", "votes": 52 }, { "id": 3309959, "postDate": "2025-11-01T19:15:55.563Z", "content": "Love the honesty on \" I don't even understand most of the final golfed code\", I would echo that sentiment. But on the comment of about 2% by human effort, I'd say we are downplaying the ingenuity and effort it took to just get the initial solutions to kick start the LLMs and all the human iterations and testing necessary to achieve these low scores was a huge undertaking. Our team was complimented with @radek1 tech stack and ingenuity along with complementing hand coding and brute force that kept the LLMs sweeping through new instances but it was a huge amount of human effort, that if we had only focused on the coding would have probably yielded similar results possibly. The only difference the LLMs will be smarter now with all the wealth of back and fourth rich discussion generated to be better for the next round. Great writeup, loved it!
", "rawMarkdown": "Love the honesty on \" I don't even understand most of the final golfed code\", I would echo that sentiment. But on the comment of about 2% by human effort, I'd say we are downplaying the ingenuity and effort it took to just get the initial solutions to kick start the LLMs and all the human iterations and testing necessary to achieve these low scores was a huge undertaking. Our team was complimented with @radek1 tech stack and ingenuity along with complementing hand coding and brute force that kept the LLMs sweeping through new instances but it was a huge amount of human effort, that if we had only focused on the coding would have probably yielded similar results possibly. The only difference the LLMs will be smarter now with all the wealth of back and fourth rich discussion generated to be better for the next round. Great writeup, loved it!", "votes": 4, "replies": [ { "id": 3310061, "postDate": "2025-11-02T01:47:59.663Z", "content": "Thanks for putting that into perspective! You're absolutely right to call me out on the \"2%\" figure. It was more of a rough metric and not meant to diminish the huge amount of human work that went into this. It all depends on how you measure it—whether it's the final code size or the crucial thinking time.
", "rawMarkdown": "Thanks for putting that into perspective! You're absolutely right to call me out on the \"2%\" figure. It was more of a rough metric and not meant to diminish the huge amount of human work that went into this. It all depends on how you measure it—whether it's the final code size or the crucial thinking time.", "votes": 1 } ] }, { "id": 3309829, "postDate": "2025-11-01T14:51:19.070Z", "content": "Some people still golf some ICPC problems (mostly just in Python) https://open.kattis.com/problem-sources/2025%20ICPC%20North%20America%20Championship\nWish more people did! They’re often very fun problems. I’m likely in a minority but I think the easier half of these sets would be great problems for a future code golf contest.
\nI’m curious how easy you found it to get AI to use regex? From just prompting on the web app, my experience was that it didn’t understand the idea. At best it would try to remove all list parts which would be very long, and for non-obvious regex it failed to find the approach at all.
\nAlso, how did you handle the problems which had one bad test case? I think there were a few where people decided to hard code a condition for a single case. This seems like something AI would be very hesitant to do - in general I imagine they don’t love brute forcing for hard codes?
", "rawMarkdown": "Some people still golf some ICPC problems (mostly just in Python) https://open.kattis.com/problem-sources/2025%20ICPC%20North%20America%20Championship\nWish more people did! They’re often very fun problems. I’m likely in a minority but I think the easier half of these sets would be great problems for a future code golf contest.\n\n\nI’m curious how easy you found it to get AI to use regex? From just prompting on the web app, my experience was that it didn’t understand the idea. At best it would try to remove all list parts which would be very long, and for non-obvious regex it failed to find the approach at all.\n\nAlso, how did you handle the problems which had one bad test case? I think there were a few where people decided to hard code a condition for a single case. This seems like something AI would be very hesitant to do - in general I imagine they don’t love brute forcing for hard codes?", "votes": 4, "replies": [ { "id": 3310059, "postDate": "2025-11-02T01:18:26.097Z", "content": "Hi, thank you for the link and information! It's great to learn about a scene I wasn't aware of.
\nRegarding your question about regex, my experience was very similar to yours. I also found it difficult to get the model to generate regex-based code. I believe one key reason for my eventual success was the number of sampling attempts. I saw it generate code that would import re but then never actually use the module many times…
Here are some examples. One successful example is here: https://gcgc.pages.dev/task332#raw-88c00bd6. As I wrote in the article, this was achieved simply by instructing it to try regex in a follow-up prompt. My hypothesis is that codex (gpt5-codex) was able to read the ARCGEN code included in the repository, verbalize the task, correctly infer the necessary regex from that description, and then verify its hypothesis in the REPL. I need to review the CoT logs to confirm this, which I plan to do. I also tried adding a regex template to the prompt, but it didn't seem effective, so I quickly abandoned that approach.
In several cases, our AI Agent's main contribution was shortening an existing regex solution originally created by THUNDER THUNDER. Like in https://gcgc.pages.dev/task077#raw-311d84fe and https://gcgc.pages.dev/task379#raw-311d84fe. It rarely generated regex code from scratch. This highlights the need for a benchmark to evaluate the regex capabilities of AI Agents.
\n\n\nhow did you handle the problems which had one bad test case?
\n
That's a great question! I didn't implement any special handling for that. However, I was closely monitoring the gpt5-codex's CoT logs and noticed something interesting. Even without any specific instructions, it would execute Python code in its REPL to load the test case JSON data and then count or identify the specific cases that were failing. Since this analysis becomes part of the LLM's context, I believe it's possible in principle for it to hard-code a solution for a single bad test case.
\nIf you have the taskIDs for those problems, I'd love to take a look at the logs! Let me know.
", "rawMarkdown": "Hi, thank you for the link and information! It's great to learn about a scene I wasn't aware of.\n\nRegarding your question about regex, my experience was very similar to yours. I also found it difficult to get the model to generate regex-based code. I believe one key reason for my eventual success was the number of sampling attempts. I saw it generate code that would import `re` but then never actually use the module many times...\n\nHere are some examples. One successful example is here: https://gcgc.pages.dev/task332#raw-88c00bd6. As I wrote in the article, this was achieved simply by instructing it to try regex in a follow-up prompt. My hypothesis is that codex (`gpt5-codex`) was able to read the ARCGEN code included in the repository, verbalize the task, correctly infer the necessary regex from that description, and then verify its hypothesis in the REPL. I need to review the CoT logs to confirm this, which I plan to do. I also tried adding a regex template to the prompt, but it didn't seem effective, so I quickly abandoned that approach.\n\nIn several cases, our AI Agent's main contribution was shortening an existing regex solution originally created by THUNDER THUNDER. Like in https://gcgc.pages.dev/task077#raw-311d84fe and https://gcgc.pages.dev/task379#raw-311d84fe. It rarely generated regex code from scratch. This highlights the need for a benchmark to evaluate the regex capabilities of AI Agents.\n\n>how did you handle the problems which had one bad test case?\n\nThat's a great question! I didn't implement any special handling for that. However, I was closely monitoring the gpt5-codex's CoT logs and noticed something interesting. Even without any specific instructions, it would execute Python code in its REPL to load the test case JSON data and then count or identify the specific cases that were failing. Since this analysis becomes part of the LLM's context, I believe it's possible in principle for it to hard-code a solution for a single bad test case.\n\nIf you have the taskIDs for those problems, I'd love to take a look at the logs! Let me know.", "votes": 2 } ] }, { "id": 3310019, "postDate": "2025-11-01T21:56:15.093Z", "content": "Congratulations achieving 4th place! We used similar LLM agents and combined them with human solutions.
\nDo you think LLM could achieve gold without beginning with human created solutions? i.e. do you think in 3 months, LLM could just read all the 400 problem descriptions (with no prior human solutions) and finish in gold LB?
", "rawMarkdown": "Congratulations achieving 4th place! We used similar LLM agents and combined them with human solutions.\n\nDo you think LLM could achieve gold without beginning with human created solutions? i.e. do you think in 3 months, LLM could just read all the 400 problem descriptions (with no prior human solutions) and finish in gold LB? ", "votes": 1, "replies": [ { "id": 3310068, "postDate": "2025-11-02T02:36:24.717Z", "content": "I believe it is possible under certain conditions.
\nWow! Inspirational work and write-up. Thank you.
", "rawMarkdown": "Wow! Inspirational work and write-up. Thank you.", "votes": 1 }, { "id": 3310525, "postDate": "2025-11-03T06:13:40.547Z", "content": "Your approach with parallel sampling and dynamic prompts is brilliant! Love the creativity in escaping local optima. Great job on 4th place!
", "rawMarkdown": "Your approach with parallel sampling and dynamic prompts is brilliant! Love the creativity in escaping local optima. Great job on 4th place!" }, { "id": 3310062, "postDate": "2025-11-02T02:13:14.560Z", "content": "Congrats on 4th! I actually tried the exact same parallel sampling framework in the first few weeks of the competition, but unfortunately none of us had enough compute credits so we ended up resorting to golfing by hand 😂. (P.S. we did the compression golfing by hand too but it's very cool that you managed to get a diamond using gpt for this!)
", "rawMarkdown": "Congrats on 4th! I actually tried the exact same parallel sampling framework in the first few weeks of the competition, but unfortunately none of us had enough compute credits so we ended up resorting to golfing by hand 😂. (P.S. we did the compression golfing by hand too but it's very cool that you managed to get a diamond using gpt for this!)" }, { "id": 3309780, "postDate": "2025-11-01T11:37:39.570Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3310129, "postDate": "2025-11-02T06:38:45.487Z", "content": "Thank you for compression-friendly code
", "rawMarkdown": "Thank you for compression-friendly code" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:51:40Z", "competition_id": 95282, "competition_slug": "google-code-golf-2025", "team_id": 14194500, "solution_url": "/competitions/google-code-golf-2025/writeups/parallel-sampling-rule-based-prompt-generation" }