{
  "id": 614225,
  "title": "5th-Place writeup (with Better Compression Algorithm and Seed Cracking)",
  "content": "<p>First, we want to thank the organizers for hosting such a competition and all the participants for battling it out with us. The past three months were exciting for us.</p>\n<p>Our team’s background is mainly in competitive programming and CTF. So this competition was a step into unfamiliar territory, since we had little experience with code golf and almost none with Kaggle. In fact, it looked like many teams in this competition were also new to Kaggle. We’re a student team (someone questioned our odd team name, it's an inside pun related to <a href=\"https://www.tsukuba.ac.jp/en/about/public-branding/future/\" target=\"_blank\">our university</a>), and we worked especially hard during our summer break in August and September.</p>\n<p>This writeup explains our approach and highlights a few interesting solutions. The visualization tool we used and our GitHub repository are listed at the end.</p>\n<h1>Overview of Our Solutions</h1>\n<p>Our approach was 90% grinding code golf, 9% tooling, and 1% weird hacks. Since the 90% grind was something many teams did, we’ll just showcase two of our most memorable solutions.</p>\n<h2>1. task 111 (62 bytes, best+2)</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F010fa1d41f9b192b5216869ad868be59%2FScreenshot%20from%202025-11-02%2015-02-06.png?generation=1762063336648581&amp;alt=media\" alt=\"\"></p>\n<pre><code>p= g:[[*((g,g).pop,)][-i:-i:-] i  ]\n</code></pre>\n<p><code>iter(l.pop, n)</code> walks <code>l</code> in reverse until it pops <code>n</code>. This task was the only one where we could exploit that behavior.</p>\n<pre><code>a = [,,,,,,]\n([*(,(a.pop,))]) \n(a) \n</code></pre>\n<h2>2. task 270 (115 bytes, best+2)</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fd59ef4bdb80a7ea4ffc107f84afe9bca%2FScreenshot%20from%202025-11-02%2015-02-29.png?generation=1762063355613187&amp;alt=media\" alt=\"\"></p>\n<pre><code> re;p= g,c=-:c*g [*(*(re.sub(,,(p(g,c+)))))][::-]\n</code></pre>\n<p>This solution was the only one we saw that used the conditional-group syntax <code>(?(id/name)yes-pattern|no-pattern)</code> to change the regex pattern depending on whether a group matched.</p>\n<h1>Tooling and Some Odd Hacks</h1>\n<p>Here are the tools and odd hacks we built along the way.</p>\n<h2>Compression Optimizations</h2>\n<p>Even before the contest started, we suspected there would be cases where compression strategy might be used. That hunch turned out to be right. We built our own compressor and optimizer stronger than Zopfli, which is a better zlib compressor developed by Google, and lifted our score by about 700 points relative to the native zlib.</p>\n<h3>Compression Basics</h3>\n<p>First, a quick overview of the compressed-code setup. Compressed code needs to be decompressed and executed at runtime. A key consideration is how to embed the compressed payload into the source. Python byte literals cannot contain characters ≥ 0x80, so we need to convert from another format.</p>\n<p>We handled this by embedding the payload in a string literal and converting it to bytes. In Python, the file encoding can be specified with a comment like <code># -*- coding: latin-1 -*-</code> (<a href=\"https://peps.python.org/pep-0263/\" target=\"_blank\">PEP 263</a>). The shortest form that matches the regex <code>^[ \\t\\f]*#.*?coding[:=][ \\t]*([-_.a-zA-Z0-9]+)</code> is <code>#coding:L1</code>. <code>L1</code> is an alias of the <code>latin_1</code> encoding (<a href=\"https://docs.python.org/3/library/codecs.html#standard-encodings\" target=\"_blank\">Documentation</a>), which maps every character from <code>ord(0)</code> to <code>ord(255)</code> directly to its byte value. Using this, the decompression stub becomes:</p>\n<pre><code>\n zlib\n(zlib.decompress((,)))\n</code></pre>\n<p>We used zlib because it fit our use case best. Python gives you zlib, lzma, and bzip. zlib and lzma are LZ77-family compressors, and bzip is based on the Burrows–Wheeler Transform. BWT can be weaker on these inputs compared to LZ77, and lzma is strong but its header overhead is large, so it didn’t beat zlib unless the code exceeded about 1000 bytes. The newer zstd will be available in Python 3.14 (<a href=\"https://docs.python.org/3/library/compression.zstd.html\" target=\"_blank\">Documentation</a>), but sadly it was out of reach this time.</p>\n<p>zlib is basically deflate plus a 2-byte header and a 4-byte CRC checksum. <code>zlib.decompress()</code> supports raw deflate when you pass a negative second argument, so <code>zlib.decompress(code,-15)</code> saved 6 bytes at the cost of 4 extra bytes, netting a win. For small zlib streams, <code>zlib.decompress(code,-9)</code> sometimes also worked, shaving another byte. Our final decompression stub looked like this:</p>\n<pre><code>\n zlib\n(zlib.decompress((,),-))\n</code></pre>\n<h3>The Structure of deflate</h3>\n<p>Before talking about deflate-specific optimizations, here’s the bare minimum structure. Deflate encodes data as copies of previous substrings plus single-character additions. For instance, <code>abracadabra</code> becomes <code>[a][b][r][a][c][a][d][copy(distance=7,length=4)]</code>. Think of each piece as a factor. Minimizing the number of factors is easy with a greedy approach. In practice, we must also encode those factors into a bitstream, and that coding step complicates matters. Minimizing factor count alone doesn’t always minimize the final deflate size.</p>\n<p>For coding, deflate uses Huffman codes for literals, and for length/distance pairs. You can either use a preset Huffman tree or embed your own. In our use case, presets were inefficient, so we mostly embedded our own. Switching trees midstream is possible, but embedding a tree costs about 300 bits (~40 bytes), so one tree is typically best here. (Rarely, using the preset tree for the last block is better. But you can brute-force the boundary to easily compensate this.)</p>\n<p>There are two Huffman trees, one for literals/lengths and one for distances. These trees are themselves run-length encoded (RLE) and then encoded by another Huffman tree (the \"cl-code\") and inserted as a header.</p>\n<p>We built a visualization app for deflate(<a href=\"https://key-moon.github.io/deflate-viz/\" target=\"_blank\">deflate-viz</a>, <a href=\"https://github.com/key-moon/deflate-viz\" target=\"_blank\">repository</a>). By attaching a deflate or zlib payload via query parameters, you can inspect the internals.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fe8762c54471c6f6c8d3f9595273d1001%2Fdeflate-viz.png?generation=1762062312506864&amp;alt=media\" alt=\"\"></p>\n<h3>Code-Level Optimizations</h3>\n<p>By exploring the visualization, we got a few key insights.</p>\n<h4>Make the same code fragments appear as much as possible</h4>\n<p>Obvious, but the golden rule of compression golf. Also loop unrolling sometimes produce better results than loops.</p>\n<h4>Avoid uppercase variable names / Reuse names that appear in keywords</h4>\n<p>As mentioned, the Huffman tree is represented by code lengths ordered by byte value and then RLE-compressed. Longer runs help. Python syntax doesn’t use uppercase letters, so those bytes would ideally be all zeros in the code-length stream; introducing uppercase disrupts RLE. Also, to help the literal Huffman distribution, it’s better to reuse letters that appear in common keywords like <code>def</code> and <code>return</code>, or built-ins like <code>zip</code>, and avoid introducing stray characters.</p>\n<h4>Prefer tabs over spaces for indentation / Prefer <code>'</code> over <code>\"</code></h4>\n<p>Again, RLE awareness. Tabs sit next to newline in ASCII and tend to disrupt RLE less than spaces. Single quotes tend to work better because <code>'</code> is adjacent to <code>(</code>.</p>\n<h3>Optimizing the Compressed Payload</h3>\n<p>We first set out to produce an encoding-friendly deflate payload. Since the payload is embedded as a string, some characters require escaping, e.g., the quote characters used for the string delimiter, or <code>\\x00</code>. Each such character costs an extra byte; we want to avoid them. Empirically, roughly 1/100 of bytes tended to need escaping.</p>\n<p>So we tried randomly perturbing parts of the deflate payload to dodge those characters. The most leverage comes from the embedded Huffman tree, which influences many bytes. Under an optimistic assumption that all bytes randomize independently, a ~200-byte payload would avoid escapes with probability about <code>(99/100)^200 ≈ 0.13</code>. Reality is less kind, so it took more trials.</p>\n<p>We implemented a deflate parser and performed hill climbing on top of Zopfli’s output (<a href=\"https://github.com/key-moon/golf/blob/2568f97b8677aa1898c5dee6c4ec240a37f578ae/deflate_optimizer/optimizer.py#L261\" target=\"_blank\">implementation</a>). This worked well in practice and often reduced the number of escapes to zero.</p>\n<h3>Beyond Zopfli</h3>\n<p>Along the way, we generated payloads that beat Zopfli’s size, initially as a side effect of hill climbing to remove escapes. That suggested we might actually build a compressor that outperforms Zopfli.</p>\n<p>The full story is long, so details are in a teammate’s writeup(To Be Written). Here is a broad, high-level overview of the approach.</p>\n<p>We first tried improving the Huffman tree with metaheuristics like genetic algorithms and simulated annealing, but this underperformed. We suspect that this because of the search landscape wasn’t smooth, and the objective (bits) is discrete.</p>\n<p>The big breakthrough was realizing that when <strong>everything except</strong> the Huffman tree is fixed, we can compute the optimal Huffman code lengths by dynamic programming (DP). Concretely, fixing the factorization and the cl-code (the Huffman code for the Huffman codes) allows a DP to find optimal code lengths. Implementing this let us shrink the Huffman trees that Zopfli produced (<a href=\"https://github.com/key-moon/golf/blob/2568f97b8677aa1898c5dee6c4ec240a37f578ae/deflate_optimizer_cpp/optimal_lit_code_lengths.hpp#L14\" target=\"_blank\">implementation</a>). Here’s a comparison:</p>\n<ul>\n<li><a href=\"https://deflate-viz.pages.dev/?deflate=le9FlgIwDAUArNODzGuGFHeHY%2BAOFdz16rg7u8jXSrUGHV7HIINEZM4lKay1eyBBtqBXbNWr3FRb2%2F%2F%2Bqu6uaXsWUdagnpbZtMr%2BOZehZMTFYNyQpgqOIAMoU58kqUizOOG8P2xyLi0rUSz1uRYO%2FDekzqtzu%2BLWIAE7L33xcu%2FNzeWgo3bqI5Lo0ylw%2F%2FJNkufXDrjNLfoMQNagHHYEe9XBsNeCOoPHLLtW8CSQ6O9fO40vOgYTIjL%2FgFvu5apmG%2BpQIHqLjwpH%2BJYSPlcK1tO34Gz6FpqN%2BDc%3D\" target=\"_blank\">Zopfli, 218 bytes</a></li>\n<li><a href=\"https://deflate-viz.pages.dev/?deflate=le9FDkMhEAUArIeDNNBCUne%2FBmHx3ZW6nb3uvht5qhsmxNgiTQSDzhI71CVmlIIDTgipEloG9o1w9z9c3YcrzwtCHBMs7gjuikxx3Rp2SgimtuMbUGgiAI1K6lC3EygzjOU4wNjJbZiiSuyxAsn61L2sxd1KdgYD2Ht5V6%2Fywdy%2FHrxunkpCKJP0HFhev0Na%2BbcD2eVmEgE4JmjtQjM1RuM0BAvBc5Z9K3gRiMnDa6%2FxQ8fmgHWWX3Drg5zh70IdC3Tv8V1WaN9T2pdKTYvfgwW%2Fh4pOfQs%3D\" target=\"_blank\">DP-optimized Huffman tree, 215 bytes</a></li>\n</ul>\n<h4>Automatic Variable-Name Optimization</h4>\n<p>Observation suggested variable-name assignment strongly affects compression. To mechanize this, we statically analyzed code to extract dependencies and then added a routine that reassigns variable names to improve compressibility. This was the single most impactful optimization, netting several hundred bytes overall.</p>\n<h4>Genetic Algorithm</h4>\n<p>We also ran a GA whose state combined variable-name assignments and the deflate payload itself, interleaving factorization and Huffman optimizations. This gained around 100 bytes (<a href=\"https://github.com/key-moon/golf/blob/main/deflate_optimizer_cpp/geneticalgo.cpp\" target=\"_blank\">implementation</a>).</p>\n<h2>Recovering the Random Seeds for Test Cases</h2>\n<p>It was disclosed that test cases were generated using <a href=\"https://github.com/google/ARC-GEN/\" target=\"_blank\">ARC-GEN</a>. The generator is deterministic given a seed. From <code>arc_gen.py</code>:</p>\n<pre><code>\n ():\n  \n  task_info = task_list.task_list()[task_num]\n  _, generator, _ = task_info\n  examples = []\n   example_id  (num_examples):\n    random.seed(task_num + example_id)\n    examples.append(generator())\n  (examples)\n</code></pre>\n<p>However, running arc_gen didn’t reproduce the exact test cases.</p>\n<p>When the anomaly in task100 was fixed and the test cases remains as same in almost all cases, it suggested some seed mechanism was in play. We reasoned that the high quality source to leak randomness is via <code>randint</code>. Skipping the nerdy details, if a generator uses only calls like <code>randint(a, b)</code> where <code>b - a + 1</code> is a power of two, those outputs can leak clean bits of the PRNG state.</p>\n<p>We filtered ARC-GEN and identified three tasks, which is task043, task127, task142, as good candidates. From each ARC-GEN case we extracted the random sequences used.</p>\n<pre><code>LEAK_CASES = \n\n\n\n\n\ncases = get_task()[]\nleak_43 = []\n   cases[:LEAK_CASES]:\n  g = []  \n  leak = [*(==r[-]) r  g[:]] + [*(==c) c  g[][:-]]\n  leak_43.append(leak)\n\n\n\n\ncases = get_task()[]\nleaks_127 = []\n   cases[:LEAK_CASES]:\n  g = []\n  leak = []\n  leak.append([, ][(g)==])\n  leak.extend([v-  v  g[][::]])\n   (g) == :\n    leak.extend([v-  v  g[][::]])\n  leaks_127.append(leak)\n\n\n\n\n\n\n\ncases = get_task()[]\nleaks_142 = []\n   cases[:LEAK_CASES]:\n  g = []\n  leak = []\n   s  g:\n     v  s:\n      leak.append(v)\n  leaks_142.append(leak)\n\n(leaks_127)\n(leaks_142)\n\n\n</code></pre>\n<p>A bit of inspection shows the random sequences share the same prefixes. In other words, across all tasks, the same seed was used per index!</p>\n<p>With that in mind, we searched for seeds that would produce the sequences above.</p>\n<pre><code> i  ():\n  random.seed(i)\n  r = [random.randint(,)  i  ()]\n   r  leak_43:\n    (i, leak_43.index(r))\n\n\n\n</code></pre>\n<p>So the seeds were <code>2025 + i</code>. This hinted we might compress some tasks dramatically by regenerating from the seed. For this to be useful, the following criteria had to be met:</p>\n<ol>\n<li>It’s hard to extract the vital information in Python directly (easier to regenerate from randomness).</li>\n<li>The generation code is simple.</li>\n<li>Both train and test are derivable from a same algorithm and a short seed.</li>\n<li>The seed is recoverable from available information.</li>\n<li>Constructing the solution from the recovered information is straightforward.</li>\n</ol>\n<p>By scouting tasks, only task096 fit these criteria. This tasks has tons of corner cases and was one of the hardest to optimize.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F07f32fcee52b745d1bfaa82d9c4a9397%2FScreenshot%20from%202025-11-02%2014-55-16.png?generation=1762062925269548&amp;alt=media\" alt=\"\"></p>\n<p>In this task, the grid width/height, background color, reticle colors, and each segment length are generated randomly. Here is a simplified version of the algorithm for those:</p>\n<pre><code>\nwidth, height = random.randint(, ), random.randint(, )\nb = random.randint(, ) \ncolors = random.sample((((, ))-{b}), random.randint(, ))\nlengths = []\n i  ((colors)):\n  min_length, max_length = (i + , ), i + (  i &gt;   )\n  lengths.append(random.randint(min_length, max_length))\n</code></pre>\n<p>These are exactly the pieces of information you need to determine the seed and construct the output. Our experiments showed that width, height, the set of colors, and the background color suffice to identify the seed. And, for non-ARC-GEN train and test cases, we also found seeds via exhaustive search.</p>\n<p>Here’s the code we built using this strategy:</p>\n<pre><code> random *\n ():\n seed([-l,,,,][l*(&lt;l)])\n d,t,f=randrange(,)==(r[]),randrange(,)==(r),randrange(,)\n s=sample([*{*(,)}^{f*(l&lt;)}],o:=randrange(,))\n u=[+randrange(&lt;m,(m&lt;)+m) m  (o)]\n n=[(o*-)*[f] m  (o)]\n  m  (o):\n   e  (u[m]):e-=m;n[e-][o+m-]=n[e-][o-m-]=n[-m-][o+e-]=n[-m-][o-e-]=s[m]\n  ([d,t,{*(r,[])}=={*s,f},f  r[]]) n+n[:-][::-] p(r,l-)\n</code></pre>\n<p>After compression this ended up at best+48, which is not quite we are looking for, but we’re proud we had this kind of solution.</p>\n<h2>GX (Golf Experience) / QoL Features</h2>\n<p>Given the marathon nature of the contest, we built QoL tooling: mainly CI/CD and a local Web UI.</p>\n<h3>CI/CD</h3>\n<p>On every push (and on schedules) we ran jobs to do the following.</p>\n<h4>Generate README</h4>\n<p>We generated a stats summary in Markdown as the README. Later the Web UI replaced it, but it was handy on the go.</p>\n<h4>Generate per-file banners</h4>\n<p>We auto-updated banners in files to show size targets like this:</p>\n<pre><code>\n\n</code></pre>\n<h4>Discord Integration</h4>\n<p>We notified our Discord on every teammate push. This made it easy to spot missed shrink opportunities and pick the low-hanging fruit.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F90c30fbaaf30f328ef51bfaf7c12565a%2FScreenshot%20from%202025-11-02%2004-48-08.png?generation=1762063444300991&amp;alt=media\" alt=\"\"></p>\n<h4>Watch Standings / Spreadsheet</h4>\n<p>We automatically updated the best-known sizes referenced by the banners and Web UI.</p>\n<h3>Web UI</h3>\n<p>We've created(99% vibe-coded) local Web UI for daily golfing.</p>\n<h4>Top Page</h4>\n<p>We could track our progress. A histogram highlighted which tasks deserved compression attention.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fc90f52efe70694f4c7f046da468feb80%2FScreenshot%20from%202025-11-02%2014-58-05.png?generation=1762063098220077&amp;alt=media\" alt=\"\"></p>\n<h4>Task List</h4>\n<p>Thumbnail overview, color-coded by delta from best. We added manual tags and search.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F92816db89d2e92b34f4b87b2c796d66d%2FScreenshot%20from%202025-11-02%2014-58-59.png?generation=1762063151089124&amp;alt=media\" alt=\"\"></p>\n<h4>Task Detail</h4>\n<p>As shown: tags and code, all cases, and score trajectories by team. We also added a “Open current best in VSCode” button.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fc0dac6489524be375907ed0ad9b9008a%2FScreenshot%20from%202025-11-02%2014-59-49.png?generation=1762063195601066&amp;alt=media\" alt=\"\"></p>\n<h4>Judge View</h4>\n<p>Integrated with <a href=\"https://github.com/key-moon/golf/blob/main/checker.py\" target=\"_blank\">checker.py</a> to show the latest run results. Passing values through the identity function <a href=\"https://github.com/key-moon/golf/blob/main/checker.py#L83\" target=\"_blank\"><code>DUMP</code></a> printed its argument, which helped debug recursive solutions.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fd2a18ffb82a338e5ac9eb4576d4aaaf6%2FScreenshot%20from%202025-11-02%2015-00-06.png?generation=1762063212521821&amp;alt=media\" alt=\"\"></p>\n<h2>Other Things We Tried</h2>\n<ul>\n<li>Early on, we auto-generated solutions with GPT-o4-mini, which is why we ranked high at the start. Those codes live in <a href=\"https://github.com/key-moon/golf/tree/main/_junkyard/old_base/base_code\" target=\"_blank\">base_code</a>. None of the final surviving solutions relied on AI, though.</li>\n<li>We also discovered the universal solution via <code>__eq__</code> that was published in mid-August. Our short version was:<ul>\n<li><code>class p:__init__=help;__eq__=any</code></li></ul></li>\n<li>Until late August we used minified versions of <a href=\"https://github.com/michaelhodel/arc-dsl\" target=\"_blank\">arc-dsl</a> and <a href=\"https://github.com/michaelhodel/re-arc\" target=\"_blank\">re-arc</a> to chase 400/400. Some compressed to ~2000 bytes. By early September, we removed them all and replaced them with handwritten solutions.</li>\n</ul>\n<h1>Data</h1>\n<p>Our competition repository (very messy; it wasn’t meant to be public) is here:</p>\n<p><a href=\"https://github.com/key-moon/golf\" target=\"_blank\">https://github.com/key-moon/golf</a></p>\n<p>Repository overview:</p>\n<ul>\n<li><code>dist</code>: results</li>\n<li><code>base_[name]</code>: each member’s “golf course”</li>\n</ul>\n<p>If you want to run anything, execute <code>source tools/setup.sh</code> to set up the environment.</p>",
  "comments": [
    {
      "id": 3310160,
      "author_name": "Mukundan",
      "author_url": "",
      "post_date": "2025-11-02T08:23:11.677000",
      "content": "<p>Wow I had completely overlooked trying to optimize the huffman tree since I thought it would only end up saving only a few bits in most cases, but we did write a DP to optimize the LZ77 for minimizing the escaped size rather than the raw size (<a href=\"https://github.com/Seek64/NeurIPS-Code-Golf-2025/blob/main/reencoder.py\" target=\"_blank\">implementation</a>).</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 3310137,
      "author_name": "hellopir2",
      "author_url": "",
      "post_date": "2025-11-02T06:48:15.453000",
      "content": "<p>lol, we contemplated using the seed but didn't bother once the default ones didn't work. cool to see that it's actually possible though. </p>\n<p>looks like that task 270 solution can be used for a 114b diamond</p>\n<pre><code> re;p= g:(*) g\n</code></pre>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 3311688,
      "author_name": "jacekwl",
      "author_url": "",
      "post_date": "2025-11-05T15:14:34.363000",
      "content": "<p>Seed cracking is crazy.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3310529,
      "author_name": "Aaditya Khanal",
      "author_url": "",
      "post_date": "2025-11-03T06:16:02.230000",
      "content": "<p>This is an absolutely incredible writeup! The depth of optimization and creativity with deflate compression is mind-blowing. Thank you for sharing all these insights!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 3310116,
      "postDate": "2025-11-02T06:06:23.613Z",
      "content": "<p>First, we want to thank the organizers for hosting such a competition and all the participants for battling it out with us. The past three months were exciting for us.</p>\n<p>Our team’s background is mainly in competitive programming and CTF. So this competition was a step into unfamiliar territory, since we had little experience with code golf and almost none with Kaggle. In fact, it looked like many teams in this competition were also new to Kaggle. We’re a student team (someone questioned our odd team name, it's an inside pun related to <a href=\"https://www.tsukuba.ac.jp/en/about/public-branding/future/\" target=\"_blank\">our university</a>), and we worked especially hard during our summer break in August and September.</p>\n<p>This writeup explains our approach and highlights a few interesting solutions. The visualization tool we used and our GitHub repository are listed at the end.</p>\n<h1>Overview of Our Solutions</h1>\n<p>Our approach was 90% grinding code golf, 9% tooling, and 1% weird hacks. Since the 90% grind was something many teams did, we’ll just showcase two of our most memorable solutions.</p>\n<h2>1. task 111 (62 bytes, best+2)</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F010fa1d41f9b192b5216869ad868be59%2FScreenshot%20from%202025-11-02%2015-02-06.png?generation=1762063336648581&amp;alt=media\" alt=\"\"></p>\n<pre><code>p= g:[[*((g,g).pop,)][-i:-i:-] i  ]\n</code></pre>\n<p><code>iter(l.pop, n)</code> walks <code>l</code> in reverse until it pops <code>n</code>. This task was the only one where we could exploit that behavior.</p>\n<pre><code>a = [,,,,,,]\n([*(,(a.pop,))]) \n(a) \n</code></pre>\n<h2>2. task 270 (115 bytes, best+2)</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fd59ef4bdb80a7ea4ffc107f84afe9bca%2FScreenshot%20from%202025-11-02%2015-02-29.png?generation=1762063355613187&amp;alt=media\" alt=\"\"></p>\n<pre><code> re;p= g,c=-:c*g [*(*(re.sub(,,(p(g,c+)))))][::-]\n</code></pre>\n<p>This solution was the only one we saw that used the conditional-group syntax <code>(?(id/name)yes-pattern|no-pattern)</code> to change the regex pattern depending on whether a group matched.</p>\n<h1>Tooling and Some Odd Hacks</h1>\n<p>Here are the tools and odd hacks we built along the way.</p>\n<h2>Compression Optimizations</h2>\n<p>Even before the contest started, we suspected there would be cases where compression strategy might be used. That hunch turned out to be right. We built our own compressor and optimizer stronger than Zopfli, which is a better zlib compressor developed by Google, and lifted our score by about 700 points relative to the native zlib.</p>\n<h3>Compression Basics</h3>\n<p>First, a quick overview of the compressed-code setup. Compressed code needs to be decompressed and executed at runtime. A key consideration is how to embed the compressed payload into the source. Python byte literals cannot contain characters ≥ 0x80, so we need to convert from another format.</p>\n<p>We handled this by embedding the payload in a string literal and converting it to bytes. In Python, the file encoding can be specified with a comment like <code># -*- coding: latin-1 -*-</code> (<a href=\"https://peps.python.org/pep-0263/\" target=\"_blank\">PEP 263</a>). The shortest form that matches the regex <code>^[ \\t\\f]*#.*?coding[:=][ \\t]*([-_.a-zA-Z0-9]+)</code> is <code>#coding:L1</code>. <code>L1</code> is an alias of the <code>latin_1</code> encoding (<a href=\"https://docs.python.org/3/library/codecs.html#standard-encodings\" target=\"_blank\">Documentation</a>), which maps every character from <code>ord(0)</code> to <code>ord(255)</code> directly to its byte value. Using this, the decompression stub becomes:</p>\n<pre><code>\n zlib\n(zlib.decompress((,)))\n</code></pre>\n<p>We used zlib because it fit our use case best. Python gives you zlib, lzma, and bzip. zlib and lzma are LZ77-family compressors, and bzip is based on the Burrows–Wheeler Transform. BWT can be weaker on these inputs compared to LZ77, and lzma is strong but its header overhead is large, so it didn’t beat zlib unless the code exceeded about 1000 bytes. The newer zstd will be available in Python 3.14 (<a href=\"https://docs.python.org/3/library/compression.zstd.html\" target=\"_blank\">Documentation</a>), but sadly it was out of reach this time.</p>\n<p>zlib is basically deflate plus a 2-byte header and a 4-byte CRC checksum. <code>zlib.decompress()</code> supports raw deflate when you pass a negative second argument, so <code>zlib.decompress(code,-15)</code> saved 6 bytes at the cost of 4 extra bytes, netting a win. For small zlib streams, <code>zlib.decompress(code,-9)</code> sometimes also worked, shaving another byte. Our final decompression stub looked like this:</p>\n<pre><code>\n zlib\n(zlib.decompress((,),-))\n</code></pre>\n<h3>The Structure of deflate</h3>\n<p>Before talking about deflate-specific optimizations, here’s the bare minimum structure. Deflate encodes data as copies of previous substrings plus single-character additions. For instance, <code>abracadabra</code> becomes <code>[a][b][r][a][c][a][d][copy(distance=7,length=4)]</code>. Think of each piece as a factor. Minimizing the number of factors is easy with a greedy approach. In practice, we must also encode those factors into a bitstream, and that coding step complicates matters. Minimizing factor count alone doesn’t always minimize the final deflate size.</p>\n<p>For coding, deflate uses Huffman codes for literals, and for length/distance pairs. You can either use a preset Huffman tree or embed your own. In our use case, presets were inefficient, so we mostly embedded our own. Switching trees midstream is possible, but embedding a tree costs about 300 bits (~40 bytes), so one tree is typically best here. (Rarely, using the preset tree for the last block is better. But you can brute-force the boundary to easily compensate this.)</p>\n<p>There are two Huffman trees, one for literals/lengths and one for distances. These trees are themselves run-length encoded (RLE) and then encoded by another Huffman tree (the \"cl-code\") and inserted as a header.</p>\n<p>We built a visualization app for deflate(<a href=\"https://key-moon.github.io/deflate-viz/\" target=\"_blank\">deflate-viz</a>, <a href=\"https://github.com/key-moon/deflate-viz\" target=\"_blank\">repository</a>). By attaching a deflate or zlib payload via query parameters, you can inspect the internals.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fe8762c54471c6f6c8d3f9595273d1001%2Fdeflate-viz.png?generation=1762062312506864&amp;alt=media\" alt=\"\"></p>\n<h3>Code-Level Optimizations</h3>\n<p>By exploring the visualization, we got a few key insights.</p>\n<h4>Make the same code fragments appear as much as possible</h4>\n<p>Obvious, but the golden rule of compression golf. Also loop unrolling sometimes produce better results than loops.</p>\n<h4>Avoid uppercase variable names / Reuse names that appear in keywords</h4>\n<p>As mentioned, the Huffman tree is represented by code lengths ordered by byte value and then RLE-compressed. Longer runs help. Python syntax doesn’t use uppercase letters, so those bytes would ideally be all zeros in the code-length stream; introducing uppercase disrupts RLE. Also, to help the literal Huffman distribution, it’s better to reuse letters that appear in common keywords like <code>def</code> and <code>return</code>, or built-ins like <code>zip</code>, and avoid introducing stray characters.</p>\n<h4>Prefer tabs over spaces for indentation / Prefer <code>'</code> over <code>\"</code></h4>\n<p>Again, RLE awareness. Tabs sit next to newline in ASCII and tend to disrupt RLE less than spaces. Single quotes tend to work better because <code>'</code> is adjacent to <code>(</code>.</p>\n<h3>Optimizing the Compressed Payload</h3>\n<p>We first set out to produce an encoding-friendly deflate payload. Since the payload is embedded as a string, some characters require escaping, e.g., the quote characters used for the string delimiter, or <code>\\x00</code>. Each such character costs an extra byte; we want to avoid them. Empirically, roughly 1/100 of bytes tended to need escaping.</p>\n<p>So we tried randomly perturbing parts of the deflate payload to dodge those characters. The most leverage comes from the embedded Huffman tree, which influences many bytes. Under an optimistic assumption that all bytes randomize independently, a ~200-byte payload would avoid escapes with probability about <code>(99/100)^200 ≈ 0.13</code>. Reality is less kind, so it took more trials.</p>\n<p>We implemented a deflate parser and performed hill climbing on top of Zopfli’s output (<a href=\"https://github.com/key-moon/golf/blob/2568f97b8677aa1898c5dee6c4ec240a37f578ae/deflate_optimizer/optimizer.py#L261\" target=\"_blank\">implementation</a>). This worked well in practice and often reduced the number of escapes to zero.</p>\n<h3>Beyond Zopfli</h3>\n<p>Along the way, we generated payloads that beat Zopfli’s size, initially as a side effect of hill climbing to remove escapes. That suggested we might actually build a compressor that outperforms Zopfli.</p>\n<p>The full story is long, so details are in a teammate’s writeup(To Be Written). Here is a broad, high-level overview of the approach.</p>\n<p>We first tried improving the Huffman tree with metaheuristics like genetic algorithms and simulated annealing, but this underperformed. We suspect that this because of the search landscape wasn’t smooth, and the objective (bits) is discrete.</p>\n<p>The big breakthrough was realizing that when <strong>everything except</strong> the Huffman tree is fixed, we can compute the optimal Huffman code lengths by dynamic programming (DP). Concretely, fixing the factorization and the cl-code (the Huffman code for the Huffman codes) allows a DP to find optimal code lengths. Implementing this let us shrink the Huffman trees that Zopfli produced (<a href=\"https://github.com/key-moon/golf/blob/2568f97b8677aa1898c5dee6c4ec240a37f578ae/deflate_optimizer_cpp/optimal_lit_code_lengths.hpp#L14\" target=\"_blank\">implementation</a>). Here’s a comparison:</p>\n<ul>\n<li><a href=\"https://deflate-viz.pages.dev/?deflate=le9FlgIwDAUArNODzGuGFHeHY%2BAOFdz16rg7u8jXSrUGHV7HIINEZM4lKay1eyBBtqBXbNWr3FRb2%2F%2F%2Bqu6uaXsWUdagnpbZtMr%2BOZehZMTFYNyQpgqOIAMoU58kqUizOOG8P2xyLi0rUSz1uRYO%2FDekzqtzu%2BLWIAE7L33xcu%2FNzeWgo3bqI5Lo0ylw%2F%2FJNkufXDrjNLfoMQNagHHYEe9XBsNeCOoPHLLtW8CSQ6O9fO40vOgYTIjL%2FgFvu5apmG%2BpQIHqLjwpH%2BJYSPlcK1tO34Gz6FpqN%2BDc%3D\" target=\"_blank\">Zopfli, 218 bytes</a></li>\n<li><a href=\"https://deflate-viz.pages.dev/?deflate=le9FDkMhEAUArIeDNNBCUne%2FBmHx3ZW6nb3uvht5qhsmxNgiTQSDzhI71CVmlIIDTgipEloG9o1w9z9c3YcrzwtCHBMs7gjuikxx3Rp2SgimtuMbUGgiAI1K6lC3EygzjOU4wNjJbZiiSuyxAsn61L2sxd1KdgYD2Ht5V6%2Fywdy%2FHrxunkpCKJP0HFhev0Na%2BbcD2eVmEgE4JmjtQjM1RuM0BAvBc5Z9K3gRiMnDa6%2FxQ8fmgHWWX3Drg5zh70IdC3Tv8V1WaN9T2pdKTYvfgwW%2Fh4pOfQs%3D\" target=\"_blank\">DP-optimized Huffman tree, 215 bytes</a></li>\n</ul>\n<h4>Automatic Variable-Name Optimization</h4>\n<p>Observation suggested variable-name assignment strongly affects compression. To mechanize this, we statically analyzed code to extract dependencies and then added a routine that reassigns variable names to improve compressibility. This was the single most impactful optimization, netting several hundred bytes overall.</p>\n<h4>Genetic Algorithm</h4>\n<p>We also ran a GA whose state combined variable-name assignments and the deflate payload itself, interleaving factorization and Huffman optimizations. This gained around 100 bytes (<a href=\"https://github.com/key-moon/golf/blob/main/deflate_optimizer_cpp/geneticalgo.cpp\" target=\"_blank\">implementation</a>).</p>\n<h2>Recovering the Random Seeds for Test Cases</h2>\n<p>It was disclosed that test cases were generated using <a href=\"https://github.com/google/ARC-GEN/\" target=\"_blank\">ARC-GEN</a>. The generator is deterministic given a seed. From <code>arc_gen.py</code>:</p>\n<pre><code>\n ():\n  \n  task_info = task_list.task_list()[task_num]\n  _, generator, _ = task_info\n  examples = []\n   example_id  (num_examples):\n    random.seed(task_num + example_id)\n    examples.append(generator())\n  (examples)\n</code></pre>\n<p>However, running arc_gen didn’t reproduce the exact test cases.</p>\n<p>When the anomaly in task100 was fixed and the test cases remains as same in almost all cases, it suggested some seed mechanism was in play. We reasoned that the high quality source to leak randomness is via <code>randint</code>. Skipping the nerdy details, if a generator uses only calls like <code>randint(a, b)</code> where <code>b - a + 1</code> is a power of two, those outputs can leak clean bits of the PRNG state.</p>\n<p>We filtered ARC-GEN and identified three tasks, which is task043, task127, task142, as good candidates. From each ARC-GEN case we extracted the random sequences used.</p>\n<pre><code>LEAK_CASES = \n\n\n\n\n\ncases = get_task()[]\nleak_43 = []\n   cases[:LEAK_CASES]:\n  g = []  \n  leak = [*(==r[-]) r  g[:]] + [*(==c) c  g[][:-]]\n  leak_43.append(leak)\n\n\n\n\ncases = get_task()[]\nleaks_127 = []\n   cases[:LEAK_CASES]:\n  g = []\n  leak = []\n  leak.append([, ][(g)==])\n  leak.extend([v-  v  g[][::]])\n   (g) == :\n    leak.extend([v-  v  g[][::]])\n  leaks_127.append(leak)\n\n\n\n\n\n\n\ncases = get_task()[]\nleaks_142 = []\n   cases[:LEAK_CASES]:\n  g = []\n  leak = []\n   s  g:\n     v  s:\n      leak.append(v)\n  leaks_142.append(leak)\n\n(leaks_127)\n(leaks_142)\n\n\n</code></pre>\n<p>A bit of inspection shows the random sequences share the same prefixes. In other words, across all tasks, the same seed was used per index!</p>\n<p>With that in mind, we searched for seeds that would produce the sequences above.</p>\n<pre><code> i  ():\n  random.seed(i)\n  r = [random.randint(,)  i  ()]\n   r  leak_43:\n    (i, leak_43.index(r))\n\n\n\n</code></pre>\n<p>So the seeds were <code>2025 + i</code>. This hinted we might compress some tasks dramatically by regenerating from the seed. For this to be useful, the following criteria had to be met:</p>\n<ol>\n<li>It’s hard to extract the vital information in Python directly (easier to regenerate from randomness).</li>\n<li>The generation code is simple.</li>\n<li>Both train and test are derivable from a same algorithm and a short seed.</li>\n<li>The seed is recoverable from available information.</li>\n<li>Constructing the solution from the recovered information is straightforward.</li>\n</ol>\n<p>By scouting tasks, only task096 fit these criteria. This tasks has tons of corner cases and was one of the hardest to optimize.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F07f32fcee52b745d1bfaa82d9c4a9397%2FScreenshot%20from%202025-11-02%2014-55-16.png?generation=1762062925269548&amp;alt=media\" alt=\"\"></p>\n<p>In this task, the grid width/height, background color, reticle colors, and each segment length are generated randomly. Here is a simplified version of the algorithm for those:</p>\n<pre><code>\nwidth, height = random.randint(, ), random.randint(, )\nb = random.randint(, ) \ncolors = random.sample((((, ))-{b}), random.randint(, ))\nlengths = []\n i  ((colors)):\n  min_length, max_length = (i + , ), i + (  i &gt;   )\n  lengths.append(random.randint(min_length, max_length))\n</code></pre>\n<p>These are exactly the pieces of information you need to determine the seed and construct the output. Our experiments showed that width, height, the set of colors, and the background color suffice to identify the seed. And, for non-ARC-GEN train and test cases, we also found seeds via exhaustive search.</p>\n<p>Here’s the code we built using this strategy:</p>\n<pre><code> random *\n ():\n seed([-l,,,,][l*(&lt;l)])\n d,t,f=randrange(,)==(r[]),randrange(,)==(r),randrange(,)\n s=sample([*{*(,)}^{f*(l&lt;)}],o:=randrange(,))\n u=[+randrange(&lt;m,(m&lt;)+m) m  (o)]\n n=[(o*-)*[f] m  (o)]\n  m  (o):\n   e  (u[m]):e-=m;n[e-][o+m-]=n[e-][o-m-]=n[-m-][o+e-]=n[-m-][o-e-]=s[m]\n  ([d,t,{*(r,[])}=={*s,f},f  r[]]) n+n[:-][::-] p(r,l-)\n</code></pre>\n<p>After compression this ended up at best+48, which is not quite we are looking for, but we’re proud we had this kind of solution.</p>\n<h2>GX (Golf Experience) / QoL Features</h2>\n<p>Given the marathon nature of the contest, we built QoL tooling: mainly CI/CD and a local Web UI.</p>\n<h3>CI/CD</h3>\n<p>On every push (and on schedules) we ran jobs to do the following.</p>\n<h4>Generate README</h4>\n<p>We generated a stats summary in Markdown as the README. Later the Web UI replaced it, but it was handy on the go.</p>\n<h4>Generate per-file banners</h4>\n<p>We auto-updated banners in files to show size targets like this:</p>\n<pre><code>\n\n</code></pre>\n<h4>Discord Integration</h4>\n<p>We notified our Discord on every teammate push. This made it easy to spot missed shrink opportunities and pick the low-hanging fruit.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F90c30fbaaf30f328ef51bfaf7c12565a%2FScreenshot%20from%202025-11-02%2004-48-08.png?generation=1762063444300991&amp;alt=media\" alt=\"\"></p>\n<h4>Watch Standings / Spreadsheet</h4>\n<p>We automatically updated the best-known sizes referenced by the banners and Web UI.</p>\n<h3>Web UI</h3>\n<p>We've created(99% vibe-coded) local Web UI for daily golfing.</p>\n<h4>Top Page</h4>\n<p>We could track our progress. A histogram highlighted which tasks deserved compression attention.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fc90f52efe70694f4c7f046da468feb80%2FScreenshot%20from%202025-11-02%2014-58-05.png?generation=1762063098220077&amp;alt=media\" alt=\"\"></p>\n<h4>Task List</h4>\n<p>Thumbnail overview, color-coded by delta from best. We added manual tags and search.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F92816db89d2e92b34f4b87b2c796d66d%2FScreenshot%20from%202025-11-02%2014-58-59.png?generation=1762063151089124&amp;alt=media\" alt=\"\"></p>\n<h4>Task Detail</h4>\n<p>As shown: tags and code, all cases, and score trajectories by team. We also added a “Open current best in VSCode” button.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fc0dac6489524be375907ed0ad9b9008a%2FScreenshot%20from%202025-11-02%2014-59-49.png?generation=1762063195601066&amp;alt=media\" alt=\"\"></p>\n<h4>Judge View</h4>\n<p>Integrated with <a href=\"https://github.com/key-moon/golf/blob/main/checker.py\" target=\"_blank\">checker.py</a> to show the latest run results. Passing values through the identity function <a href=\"https://github.com/key-moon/golf/blob/main/checker.py#L83\" target=\"_blank\"><code>DUMP</code></a> printed its argument, which helped debug recursive solutions.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fd2a18ffb82a338e5ac9eb4576d4aaaf6%2FScreenshot%20from%202025-11-02%2015-00-06.png?generation=1762063212521821&amp;alt=media\" alt=\"\"></p>\n<h2>Other Things We Tried</h2>\n<ul>\n<li>Early on, we auto-generated solutions with GPT-o4-mini, which is why we ranked high at the start. Those codes live in <a href=\"https://github.com/key-moon/golf/tree/main/_junkyard/old_base/base_code\" target=\"_blank\">base_code</a>. None of the final surviving solutions relied on AI, though.</li>\n<li>We also discovered the universal solution via <code>__eq__</code> that was published in mid-August. Our short version was:<ul>\n<li><code>class p:__init__=help;__eq__=any</code></li></ul></li>\n<li>Until late August we used minified versions of <a href=\"https://github.com/michaelhodel/arc-dsl\" target=\"_blank\">arc-dsl</a> and <a href=\"https://github.com/michaelhodel/re-arc\" target=\"_blank\">re-arc</a> to chase 400/400. Some compressed to ~2000 bytes. By early September, we removed them all and replaced them with handwritten solutions.</li>\n</ul>\n<h1>Data</h1>\n<p>Our competition repository (very messy; it wasn’t meant to be public) is here:</p>\n<p><a href=\"https://github.com/key-moon/golf\" target=\"_blank\">https://github.com/key-moon/golf</a></p>\n<p>Repository overview:</p>\n<ul>\n<li><code>dist</code>: results</li>\n<li><code>base_[name]</code>: each member’s “golf course”</li>\n</ul>\n<p>If you want to run anything, execute <code>source tools/setup.sh</code> to set up the environment.</p>",
      "rawMarkdown": "First, we want to thank the organizers for hosting such a competition and all the participants for battling it out with us. The past three months were exciting for us.\n\nOur team’s background is mainly in competitive programming and CTF. So this competition was a step into unfamiliar territory, since we had little experience with code golf and almost none with Kaggle. In fact, it looked like many teams in this competition were also new to Kaggle. We’re a student team (someone questioned our odd team name, it's an inside pun related to [our university](https://www.tsukuba.ac.jp/en/about/public-branding/future/)), and we worked especially hard during our summer break in August and September.\n\nThis writeup explains our approach and highlights a few interesting solutions. The visualization tool we used and our GitHub repository are listed at the end.\n\n# Overview of Our Solutions\n\nOur approach was 90% grinding code golf, 9% tooling, and 1% weird hacks. Since the 90% grind was something many teams did, we’ll just showcase two of our most memorable solutions.\n\n## 1. task 111 (62 bytes, best+2)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F010fa1d41f9b192b5216869ad868be59%2FScreenshot%20from%202025-11-02%2015-02-06.png?generation=1762063336648581&alt=media)\n\n```py\np=lambda g:[[*iter(sum(g,g).pop,5)][3-i:-i:-1]for i in b'\\x0c\\x16\\x20']\n```\n\n`iter(l.pop, n)` walks `l` in reverse until it pops `n`. This task was the only one where we could exploit that behavior.\n\n```py\na = [3,1,4,1,5,9,2]\nprint([*map(str,iter(a.pop,4))]) # ['2', '9', '5', '1']\nprint(a) # [3, 1]\n```\n\n## 2. task 270 (115 bytes, best+2)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fd59ef4bdb80a7ea4ffc107f84afe9bca%2FScreenshot%20from%202025-11-02%2015-02-29.png?generation=1762063355613187&alt=media)\n\n```py\nimport re;p=lambda g,c=-7:c*g or[*zip(*eval(re.sub(\"((3)|7)([^[(]+)0(, (?(2)2|1))\",r\"0\\3\\1\\4\",str(p(g,c+1)))))][::-1]\n```\n\nThis solution was the only one we saw that used the conditional-group syntax `(?(id/name)yes-pattern|no-pattern)` to change the regex pattern depending on whether a group matched.\n\n# Tooling and Some Odd Hacks\n\nHere are the tools and odd hacks we built along the way.\n\n## Compression Optimizations\n\nEven before the contest started, we suspected there would be cases where compression strategy might be used. That hunch turned out to be right. We built our own compressor and optimizer stronger than Zopfli, which is a better zlib compressor developed by Google, and lifted our score by about 700 points relative to the native zlib.\n\n### Compression Basics\n\nFirst, a quick overview of the compressed-code setup. Compressed code needs to be decompressed and executed at runtime. A key consideration is how to embed the compressed payload into the source. Python byte literals cannot contain characters ≥ 0x80, so we need to convert from another format.\n\nWe handled this by embedding the payload in a string literal and converting it to bytes. In Python, the file encoding can be specified with a comment like `# -*- coding: latin-1 -*-` ([PEP 263](https://peps.python.org/pep-0263/)). The shortest form that matches the regex `^[ \\t\\f]*#.*?coding[:=][ \\t]*([-_.a-zA-Z0-9]+)` is `#coding:L1`. `L1` is an alias of the `latin_1` encoding ([Documentation](https://docs.python.org/3/library/codecs.html#standard-encodings)), which maps every character from `ord(0)` to `ord(255)` directly to its byte value. Using this, the decompression stub becomes:\n\n```py\n#coding:L1\nimport zlib\nexec(zlib.decompress(bytes(\"<compressed code>\",'L1')))\n```\n\nWe used zlib because it fit our use case best. Python gives you zlib, lzma, and bzip. zlib and lzma are LZ77-family compressors, and bzip is based on the Burrows–Wheeler Transform. BWT can be weaker on these inputs compared to LZ77, and lzma is strong but its header overhead is large, so it didn’t beat zlib unless the code exceeded about 1000 bytes. The newer zstd will be available in Python 3.14 ([Documentation](https://docs.python.org/3/library/compression.zstd.html)), but sadly it was out of reach this time.\n\nzlib is basically deflate plus a 2-byte header and a 4-byte CRC checksum. `zlib.decompress()` supports raw deflate when you pass a negative second argument, so `zlib.decompress(code,-15)` saved 6 bytes at the cost of 4 extra bytes, netting a win. For small zlib streams, `zlib.decompress(code,-9)` sometimes also worked, shaving another byte. Our final decompression stub looked like this:\n\n```py\n#coding:L1\nimport zlib\nexec(zlib.decompress(bytes(\"<compressed code>\",'L1'),-9))\n```\n\n### The Structure of deflate\n\nBefore talking about deflate-specific optimizations, here’s the bare minimum structure. Deflate encodes data as copies of previous substrings plus single-character additions. For instance, `abracadabra` becomes `[a][b][r][a][c][a][d][copy(distance=7,length=4)]`. Think of each piece as a factor. Minimizing the number of factors is easy with a greedy approach. In practice, we must also encode those factors into a bitstream, and that coding step complicates matters. Minimizing factor count alone doesn’t always minimize the final deflate size.\n\nFor coding, deflate uses Huffman codes for literals, and for length/distance pairs. You can either use a preset Huffman tree or embed your own. In our use case, presets were inefficient, so we mostly embedded our own. Switching trees midstream is possible, but embedding a tree costs about 300 bits (~40 bytes), so one tree is typically best here. (Rarely, using the preset tree for the last block is better. But you can brute-force the boundary to easily compensate this.)\n\nThere are two Huffman trees, one for literals/lengths and one for distances. These trees are themselves run-length encoded (RLE) and then encoded by another Huffman tree (the \"cl-code\") and inserted as a header.\n\nWe built a visualization app for deflate([deflate-viz](https://key-moon.github.io/deflate-viz/), [repository](https://github.com/key-moon/deflate-viz)). By attaching a deflate or zlib payload via query parameters, you can inspect the internals.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fe8762c54471c6f6c8d3f9595273d1001%2Fdeflate-viz.png?generation=1762062312506864&alt=media)\n\n### Code-Level Optimizations\n\nBy exploring the visualization, we got a few key insights.\n\n#### Make the same code fragments appear as much as possible\n\nObvious, but the golden rule of compression golf. Also loop unrolling sometimes produce better results than loops.\n\n#### Avoid uppercase variable names / Reuse names that appear in keywords\n\nAs mentioned, the Huffman tree is represented by code lengths ordered by byte value and then RLE-compressed. Longer runs help. Python syntax doesn’t use uppercase letters, so those bytes would ideally be all zeros in the code-length stream; introducing uppercase disrupts RLE. Also, to help the literal Huffman distribution, it’s better to reuse letters that appear in common keywords like `def` and `return`, or built-ins like `zip`, and avoid introducing stray characters.\n\n#### Prefer tabs over spaces for indentation / Prefer `'` over `\"`\n\nAgain, RLE awareness. Tabs sit next to newline in ASCII and tend to disrupt RLE less than spaces. Single quotes tend to work better because `'` is adjacent to `(`.\n\n### Optimizing the Compressed Payload\n\nWe first set out to produce an encoding-friendly deflate payload. Since the payload is embedded as a string, some characters require escaping, e.g., the quote characters used for the string delimiter, or `\\x00`. Each such character costs an extra byte; we want to avoid them. Empirically, roughly 1/100 of bytes tended to need escaping.\n\nSo we tried randomly perturbing parts of the deflate payload to dodge those characters. The most leverage comes from the embedded Huffman tree, which influences many bytes. Under an optimistic assumption that all bytes randomize independently, a ~200-byte payload would avoid escapes with probability about `(99/100)^200 ≈ 0.13`. Reality is less kind, so it took more trials.\n\nWe implemented a deflate parser and performed hill climbing on top of Zopfli’s output ([implementation](https://github.com/key-moon/golf/blob/2568f97b8677aa1898c5dee6c4ec240a37f578ae/deflate_optimizer/optimizer.py#L261)). This worked well in practice and often reduced the number of escapes to zero.\n\n### Beyond Zopfli\n\nAlong the way, we generated payloads that beat Zopfli’s size, initially as a side effect of hill climbing to remove escapes. That suggested we might actually build a compressor that outperforms Zopfli.\n\nThe full story is long, so details are in a teammate’s writeup(To Be Written). Here is a broad, high-level overview of the approach.\n\nWe first tried improving the Huffman tree with metaheuristics like genetic algorithms and simulated annealing, but this underperformed. We suspect that this because of the search landscape wasn’t smooth, and the objective (bits) is discrete.\n\nThe big breakthrough was realizing that when **everything except** the Huffman tree is fixed, we can compute the optimal Huffman code lengths by dynamic programming (DP). Concretely, fixing the factorization and the cl-code (the Huffman code for the Huffman codes) allows a DP to find optimal code lengths. Implementing this let us shrink the Huffman trees that Zopfli produced ([implementation](https://github.com/key-moon/golf/blob/2568f97b8677aa1898c5dee6c4ec240a37f578ae/deflate_optimizer_cpp/optimal_lit_code_lengths.hpp#L14)). Here’s a comparison:\n\n* [Zopfli, 218 bytes](https://deflate-viz.pages.dev/?deflate=le9FlgIwDAUArNODzGuGFHeHY%2BAOFdz16rg7u8jXSrUGHV7HIINEZM4lKay1eyBBtqBXbNWr3FRb2%2F%2F%2Bqu6uaXsWUdagnpbZtMr%2BOZehZMTFYNyQpgqOIAMoU58kqUizOOG8P2xyLi0rUSz1uRYO%2FDekzqtzu%2BLWIAE7L33xcu%2FNzeWgo3bqI5Lo0ylw%2F%2FJNkufXDrjNLfoMQNagHHYEe9XBsNeCOoPHLLtW8CSQ6O9fO40vOgYTIjL%2FgFvu5apmG%2BpQIHqLjwpH%2BJYSPlcK1tO34Gz6FpqN%2BDc%3D)\n* [DP-optimized Huffman tree, 215 bytes](https://deflate-viz.pages.dev/?deflate=le9FDkMhEAUArIeDNNBCUne%2FBmHx3ZW6nb3uvht5qhsmxNgiTQSDzhI71CVmlIIDTgipEloG9o1w9z9c3YcrzwtCHBMs7gjuikxx3Rp2SgimtuMbUGgiAI1K6lC3EygzjOU4wNjJbZiiSuyxAsn61L2sxd1KdgYD2Ht5V6%2Fywdy%2FHrxunkpCKJP0HFhev0Na%2BbcD2eVmEgE4JmjtQjM1RuM0BAvBc5Z9K3gRiMnDa6%2FxQ8fmgHWWX3Drg5zh70IdC3Tv8V1WaN9T2pdKTYvfgwW%2Fh4pOfQs%3D)\n\n#### Automatic Variable-Name Optimization\n\nObservation suggested variable-name assignment strongly affects compression. To mechanize this, we statically analyzed code to extract dependencies and then added a routine that reassigns variable names to improve compressibility. This was the single most impactful optimization, netting several hundred bytes overall.\n\n#### Genetic Algorithm\n\nWe also ran a GA whose state combined variable-name assignments and the deflate payload itself, interleaving factorization and Huffman optimizations. This gained around 100 bytes ([implementation](https://github.com/key-moon/golf/blob/main/deflate_optimizer_cpp/geneticalgo.cpp)).\n\n## Recovering the Random Seeds for Test Cases\n\nIt was disclosed that test cases were generated using [ARC-GEN](https://github.com/google/ARC-GEN/). The generator is deterministic given a seed. From `arc_gen.py`:\n\n```py\n# https://github.com/google/ARC-GEN/blob/main/arc_gen.py\ndef generate_benchmarks(task_num, num_examples):\n  \"\"\"Creates a benchmark suite for a given task.\"\"\"\n  task_info = task_list.task_list()[task_num]\n  _, generator, _ = task_info\n  examples = []\n  for example_id in range(num_examples):\n    random.seed(task_num + example_id)\n    examples.append(generator())\n  print(examples)\n```\n\nHowever, running arc_gen didn’t reproduce the exact test cases.\n\nWhen the anomaly in task100 was fixed and the test cases remains as same in almost all cases, it suggested some seed mechanism was in play. We reasoned that the high quality source to leak randomness is via `randint`. Skipping the nerdy details, if a generator uses only calls like `randint(a, b)` where `b - a + 1` is a power of two, those outputs can leak clean bits of the PRNG state.\n\nWe filtered ARC-GEN and identified three tasks, which is task043, task127, task142, as good candidates. From each ARC-GEN case we extracted the random sequences used.\n\n```py\nLEAK_CASES = 3\n\n# https://github.com/google/ARC-GEN/blob/main/tasks/training/task043.py\n# rows = [item for item in range(1, size) if common.randint(0, 1) == 0]\n# cols = [item for item in range(0, size - 1) if common.randint(0, 1) == 0]\n\ncases = get_task(43)['arc-gen']\nleak_43 = []\nfor case in cases[:LEAK_CASES]:\n  g = case['input']  \n  leak = [1*(0==r[-1])for r in g[1:]] + [1*(0==c)for c in g[0][:-1]]\n  leak_43.append(leak)\n\n# https://github.com/google/ARC-GEN/blob/main/tasks/training/task127.py\n# colors = [common.randint(1, 4) for _ in range(3 * common.randint(1, 2))]\n\ncases = get_task(127)['arc-gen']\nleaks_127 = []\nfor case in cases[:LEAK_CASES]:\n  g = case['input']\n  leak = []\n  leak.append([0, 1][len(g)==7])\n  leak.extend([v-1 for v in g[1][1::4]])\n  if len(g) == 7:\n    leak.extend([v-1 for v in g[5][1::4]])\n  leaks_127.append(leak)\n\n# https://github.com/google/ARC-GEN/blob/main/tasks/training/task142.py\n# colors = [common.randint(0, 3) for _ in range(size * size)]\n# grid, output = common.grid(size, size), common.grid(2 * size, 2 * size)\n# for r in range(size):\n#   for c in range(size):\n#     grid[r][c] = colors[r * size + c]\ncases = get_task(142)['arc-gen']\nleaks_142 = []\nfor case in cases[:LEAK_CASES]:\n  g = case['input']\n  leak = []\n  for s in g:\n    for v in s:\n      leak.append(v)\n  leaks_142.append(leak)\n\nprint(leaks_127)\nprint(leaks_142)\n# [[0, 3, 1, 0], [0, 2, 0, 1], [0, 3, 0, 0]]\n# [[0, 3, 1, 0, 2, 3, 1, 0, 3], [0, 2, 0, 1, 3, 3, 3, 1, 0], [0, 3, 0, 0, 2, 3, 1, 3, 1]]\n```\n\nA bit of inspection shows the random sequences share the same prefixes. In other words, across all tasks, the same seed was used per index!\n\nWith that in mind, we searched for seeds that would produce the sequences above.\n\n```py\nfor i in range(10000):\n  random.seed(i)\n  r = [random.randint(0,1) for i in range(18)]\n  if r in leak_43:\n    print(i, leak_43.index(r))\n# 2025 0\n# 2026 1\n# 2027 2\n```\n\nSo the seeds were `2025 + i`. This hinted we might compress some tasks dramatically by regenerating from the seed. For this to be useful, the following criteria had to be met:\n\n1. It’s hard to extract the vital information in Python directly (easier to regenerate from randomness).\n2. The generation code is simple.\n3. Both train and test are derivable from a same algorithm and a short seed.\n4. The seed is recoverable from available information.\n5. Constructing the solution from the recovered information is straightforward.\n\nBy scouting tasks, only task096 fit these criteria. This tasks has tons of corner cases and was one of the hardest to optimize.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F07f32fcee52b745d1bfaa82d9c4a9397%2FScreenshot%20from%202025-11-02%2014-55-16.png?generation=1762062925269548&alt=media)\n\nIn this task, the grid width/height, background color, reticle colors, and each segment length are generated randomly. Here is a simplified version of the algorithm for those:\n\n```py\n# ref. https://github.com/google/ARC-GEN/blob/main/tasks/training/task096.py\nwidth, height = random.randint(13, 19), random.randint(13, 19)\nb = random.randint(1, 9) # background\ncolors = random.sample(sorted(set(range(1, 10))-{b}), random.randint(4, 6))\nlengths = []\nfor i in range(len(colors)):\n  min_length, max_length = min(i + 1, 2), i + (0 if i > 1 else 1)\n  lengths.append(random.randint(min_length, max_length))\n```\n\nThese are exactly the pieces of information you need to determine the seed and construct the output. Our experiments showed that width, height, the set of colors, and the background color suffice to identify the seed. And, for non-ARC-GEN train and test cases, we also found seeds via exhaustive search.\n\nHere’s the code we built using this strategy:\n\n```py\nfrom random import*\ndef p(r,l=4):\n seed([2024-l,'+u-)+e[','41-efe[','41*l}{','f(3,+e['][l*(0<l)])\n d,t,f=randrange(13,20)==len(r[0]),randrange(13,20)==len(r),randrange(1,10)\n s=sample([*{*range(1,10)}^{f*(l<0)}],o:=randrange(4,7))\n u=[1+randrange(0<m,(m<2)+m)for m in range(o)]\n n=[(o*2-1)*[f]for m in range(o)]\n for m in range(o):\n  for e in range(u[m]):e-=m;n[e-1][o+m-1]=n[e-1][o-m-1]=n[-m-1][o+e-1]=n[-m-1][o-e-1]=s[m]\n return all([d,t,{*sum(r,[])}=={*s,f},f in r[2]])and n+n[:-1][::-1]or p(r,l-1)\n```\n\nAfter compression this ended up at best+48, which is not quite we are looking for, but we’re proud we had this kind of solution.\n\n## GX (Golf Experience) / QoL Features\n\nGiven the marathon nature of the contest, we built QoL tooling: mainly CI/CD and a local Web UI.\n\n### CI/CD\n\nOn every push (and on schedules) we ran jobs to do the following.\n\n#### Generate README\n\nWe generated a stats summary in Markdown as the README. Later the Web UI replaced it, but it was handy on the go.\n\n#### Generate per-file banners\n\nWe auto-updated banners in files to show size targets like this:\n\n```py\n# best: 51(LogicLynx, ALE-Agent) / others: 53(Ravi Annaswamy), 54(jonas ryno kg583 kabutack), 54(cubbus), 54(jacekwl Potatoman nauti), 54(jacekw Potatoman nauti natte)\n# ======================= 51 ======================\n```\n\n#### Discord Integration\n\nWe notified our Discord on every teammate push. This made it easy to spot missed shrink opportunities and pick the low-hanging fruit.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F90c30fbaaf30f328ef51bfaf7c12565a%2FScreenshot%20from%202025-11-02%2004-48-08.png?generation=1762063444300991&alt=media)\n\n#### Watch Standings / Spreadsheet\n\nWe automatically updated the best-known sizes referenced by the banners and Web UI.\n\n### Web UI\n\nWe've created(99% vibe-coded) local Web UI for daily golfing.\n\n#### Top Page\n\nWe could track our progress. A histogram highlighted which tasks deserved compression attention.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fc90f52efe70694f4c7f046da468feb80%2FScreenshot%20from%202025-11-02%2014-58-05.png?generation=1762063098220077&alt=media)\n\n#### Task List\n\nThumbnail overview, color-coded by delta from best. We added manual tags and search.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2F92816db89d2e92b34f4b87b2c796d66d%2FScreenshot%20from%202025-11-02%2014-58-59.png?generation=1762063151089124&alt=media)\n\n#### Task Detail\n\nAs shown: tags and code, all cases, and score trajectories by team. We also added a “Open current best in VSCode” button.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fc0dac6489524be375907ed0ad9b9008a%2FScreenshot%20from%202025-11-02%2014-59-49.png?generation=1762063195601066&alt=media)\n\n#### Judge View\n\nIntegrated with [checker.py](https://github.com/key-moon/golf/blob/main/checker.py) to show the latest run results. Passing values through the identity function [`DUMP`](https://github.com/key-moon/golf/blob/main/checker.py#L83) printed its argument, which helped debug recursive solutions.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1953252%2Fd2a18ffb82a338e5ac9eb4576d4aaaf6%2FScreenshot%20from%202025-11-02%2015-00-06.png?generation=1762063212521821&alt=media)\n\n\n## Other Things We Tried\n\n* Early on, we auto-generated solutions with GPT-o4-mini, which is why we ranked high at the start. Those codes live in [base_code](https://github.com/key-moon/golf/tree/main/_junkyard/old_base/base_code). None of the final surviving solutions relied on AI, though.\n* We also discovered the universal solution via `__eq__` that was published in mid-August. Our short version was:\n  * `class p:__init__=help;__eq__=any`\n* Until late August we used minified versions of [arc-dsl](https://github.com/michaelhodel/arc-dsl) and [re-arc](https://github.com/michaelhodel/re-arc) to chase 400/400. Some compressed to ~2000 bytes. By early September, we removed them all and replaced them with handwritten solutions.\n\n# Data\n\n\nOur competition repository (very messy; it wasn’t meant to be public) is here:\n\n[https://github.com/key-moon/golf](https://github.com/key-moon/golf)\n\nRepository overview:\n\n* `dist`: results\n* `base_[name]`: each member’s “golf course”\n\nIf you want to run anything, execute `source tools/setup.sh` to set up the environment.",
      "votes": 27
    },
    {
      "id": 3310160,
      "postDate": "2025-11-02T08:23:11.677Z",
      "content": "<p>Wow I had completely overlooked trying to optimize the huffman tree since I thought it would only end up saving only a few bits in most cases, but we did write a DP to optimize the LZ77 for minimizing the escaped size rather than the raw size (<a href=\"https://github.com/Seek64/NeurIPS-Code-Golf-2025/blob/main/reencoder.py\" target=\"_blank\">implementation</a>).</p>",
      "rawMarkdown": "Wow I had completely overlooked trying to optimize the huffman tree since I thought it would only end up saving only a few bits in most cases, but we did write a DP to optimize the LZ77 for minimizing the escaped size rather than the raw size ([implementation](https://github.com/Seek64/NeurIPS-Code-Golf-2025/blob/main/reencoder.py)).",
      "votes": 3
    },
    {
      "id": 3310137,
      "postDate": "2025-11-02T06:48:15.453Z",
      "content": "<p>lol, we contemplated using the seed but didn't bother once the default ones didn't work. cool to see that it's actually possible though. </p>\n<p>looks like that task 270 solution can be used for a 114b diamond</p>\n<pre><code> re;p= g:(*) g\n</code></pre>",
      "rawMarkdown": "lol, we contemplated using the seed but didn't bother once the default ones didn't work. cool to see that it's actually possible though. \n\nlooks like that task 270 solution can be used for a 114b diamond\n```py\nimport re;p=lambda g:exec(r'g[::-1]=zip(*eval(re.sub(\"((3)|7)([^[(]+)0(, (?(2)2|1))\",r\"0\\3\\1\\4\",str(g))));'*8)or g\n```",
      "votes": 4
    },
    {
      "id": 3311688,
      "postDate": "2025-11-05T15:14:34.363Z",
      "content": "<p>Seed cracking is crazy.</p>",
      "rawMarkdown": "Seed cracking is crazy."
    },
    {
      "id": 3310529,
      "postDate": "2025-11-03T06:16:02.230Z",
      "content": "<p>This is an absolutely incredible writeup! The depth of optimization and creativity with deflate compression is mind-blowing. Thank you for sharing all these insights!</p>",
      "rawMarkdown": "This is an absolutely incredible writeup! The depth of optimization and creativity with deflate compression is mind-blowing. Thank you for sharing all these insights!"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T12:51:29Z",
  "competition_id": 95282,
  "competition_slug": "google-code-golf-2025",
  "team_id": 14194523,
  "solution_url": "/competitions/google-code-golf-2025/writeups/himagine-the-future-5th-place-writeup"
}