Thanks to @fchollet for making this dataset, and organizing the competition. I think understanding and making intelligence is extremely interesting, and working on this problem and reading the dataset's paper significantly changed my view on AGI. The problem of squeezing prior knowledge and generalization into your model really becomes clear when you need to do it yourself. I had actually read the paper before the contest was announced, but working on the contest gave me a much more detailed view of the practical side of things.
\n\nUnfortunately, I don't feel like my solution itself brings us closer to AGI. The main component is a DSL which applies up to 4 of 142 unary transformations (42 different functions, some have different variants). This DSL is solved by enumeration (exploiting duplicates) + a greedy stacking combiner. Everything is implemented efficiently in C++ (with no dependencies) and running in parallel.
\n\nMy code is currently > 10k lines long, but (I think) most of it is not used. I will publish it after removing the unused / deprecated / failed code.
\n\nEdit: Code and official documentation now available at\nhttps://github.com/top-quarks/ARC-solution and \nhttps://github.com/top-quarks/ARC-solution/blob/master/ARC-solution_documentation.pdf\n
\n\nEdit 2: Public notebook scoring 0.794\nhttps://www.kaggle.com/icecuber/arc-1st-place-solution\n
\n\nIn practice I started out with implementing the first 100 training tasks by hand, extracting and generalizing useful functions from that. I made a brute force combiner of these transformations, choosing to only support transformations from images to a single image. Originally I wanted more types such as positions and colors, but it turned out that it was easier to represent these as images (such as a dot at the position, or a dot of a given color). This decision made it much easier to make an DSL, and got 10 tasks on the LB.
\n\nI then noticed that distribution of training, evaluation and LB were quite different, so I decided the evaluation dataset was better used as training data. I hand-coded 100 evaluation tasks, which I used to add more transformations, and improve my DSL. I noticed that functions taking more than 1 argument were growing my search space super-exponentially, and that they usually just took either the input or output image size as second argument. This led me to only keep unary functions. I also added lists of images as a type, to solve tasks that required looping. I also started representing my DSL in a DAG to exploit duplicates (multiple transformations giving same output). This required me to rewrite most of my code, but gave me 14 tasks on LB.
\n\nOptimizations, multithreading, reducing memory usage, etc. gave me 17 on LB. And augmenting my samples with diagonally flipped tasks, boosted me up to 21.
\n\nNote that the above (and below) is simplified, and there are many, many details I'm forgetting / omitting as I write this.
\n\nI remap colors in each sample input to be consistent across samples. A typical use of this is when each sample has a single color, but it is different for each sample. The recoloring would make all of them equal. I do this through iteratively, greedily recoloring the most extreme color across all images using features such as size, number of objects or position. So if in each sample input there is a single-colored object considerably larger than all others, I make all those objects the same color. This works surprisingly well, even for cases I did not think I accounted for. It even straight up solves some easy tasks from the training set by making all outputs equal.
\n\nI double the sample inputs by adding the same tasks flipped along a diagonal. This makes for 3 full runs (depth 3 for performance reasons): one normal, and one for each diagonal. This simple addition moved me from 17 to 21 tasks solved on the LB (and hence helped more than all the optimizations needed for doing depth 4 search instead of depth 3).
\n\nI first generate pieces as explained in \"Overview of DSL\". I do combinations and transformations of piece sizes to fit the training samples, and pick the one consistent with the most training samples. I break ties by approximately ordering the combinations by complexity (so transforming a piece with size WxH to get (2W)x(H+1) is more likely than (W+2)x(3H). The output size is correct for about 97% of the evaluation tasks.
\n\nDetails in \"Overview of DSL\". Output size is used to give the second argument to functions like embed, which crops / pads images to a given size.
\n\nSee \"Overview of DSL\".
\n\nI order candidates by the following criteria in decreasing order of importance:\n - Fits most training samples\n - Uses fewest transformations (max depth across pieces)\n - Uses fewest pieces in the stack
\n\nAll the nodes in my DSL are either images, or lists of images. An image has a position, a size, and a 2d array of colors (0-9). Black (0) is treated as transparent in most of the functions.
\n\nFirst, sequentially apply up to 3 or 4 unary transformations (see \"List of transformations\") to the input image. Then optionally apply a subset of the following (in order):\n - Stack lists of images\n - Move image to origin\n - Color all non-black pixels some color (for each color in every training output)\n - Resize image (crop / pad) to fit output size
\n\nI call result of the above transformations \"pieces\". Finally, stack some pieces to produce a final output.
\n\nMy solver mainly enumerates all the pieces, and applies a greedy solver to stack them. Since there are 142 image transformations at each depth, plus transformations at the end, we can't naively generate all the pieces. Fortunately, many different paths of transformations lead to the same image. I therefore make a DAG (Directed Acyclic Graph) for each training input, where each node is a image, and edges are transformations. This typically leaves on the order of 10^7 nodes in the DAGs and a few million unique pieces (where pieces are different if they differ for any sample input image or the test image).
\n\nI run multiple tasks in parallel (4 at depth 3, 2 at depth 4) using a script which tries to schedule all the tasks to fit within memory and 9 hour time limit, this runs maybe 50% of the images at depth 4 and the rest at depth 3 (I didn't really benchmark it, except 67% were run at depth 4 in an old version). The memory limit was typically more restrictive than the time limit, so I reduced memory usage through several tricks, such as huffman coding the images in the DAGs and using a custom memory-efficient hashmap.
\n\nMy greedy stacker produces each candidate output as follows:\n - Split pieces into black parts and non-black parts (parts not picked are treated as transparent)\n - Pick a guiding sample\n - Pick some subset of samples to \"care about\"\n - Up to 10 times:\n - Among all the pieces which do not contradict any of the images we care about\n - Add the piece which fits the most new pixels in the guiding image\n - If the test output image contains no holes, add this as a candidate
\n\nThe stacker is optimized using bitsets to be fast and memory efficient.
\n\nGreedily stacking outputs worked surprisingly well, even solving many tasks I didn't think it could solve. It consistently beat all my attempts at alternatives and improvements, so it just stayed mostly the same from the beginning.
\n\nI tried several ideas which didn't make it. One of these was adding general functions of more than one variable. However, none of my many attempts seemed to be able to give a significant improvement.
\n\nAnother idea was a system of deducing transformations from the training outputs, which I really wanted to work. If I could make this in a general way, I could maybe apply meet-in-the-middle to search double the depth! However, after many attempts I found that deductions are way harder to stack and generalize than transformations. This made it much harder to generalize deductions, and ultimately only a single deduction was kept (deducing \"kronecker products\" as last step).
\n\nYet another approach was generating a large number of sample tasks to train ML algorithms and fit hyperparameters (f.ex. different costs for each transformation). However, I concluded that generating new tasks was harder than solving them.
\n\nI always felt ranking candidates was a much easier problem than generating candidates (f.ex. if all inputs are symmetrical, then output should probably be symmetrical). Ideally this could be used to fill in parts of the image, or generate a huge number of candidates and leave most of the work to the ranking algorithm. However, I never found a way to exploit powerful ranking to my advantage. My simple ranking heuristics worked well enough for its purpose.
\n\nDepth 3, without adding flipped images. Faster, but probably worse than a full run.
\n\nTotal: 419 (Some of the 400 tasks have multiple tests)\nSize : 409 (97.6% deduced the correct output size)\nCands: 176 (42% had the correct answer among the generated candidates)\nCorrect: 164 (39% gave the correct answer among top 3 answers)
\n\nMany functions have many variations, but are only listed once (f.ex. there are 14 variations of pickMax for different criteria).\nFill, Move, border, broadcast, center, colShape, composeGrowing, compress, compress2, compress3, connect, count, cut, embed, eraseCol, filterCol, getPos, getSize, getSize0, gravity, half, hull, hull0, insideMarked, interior, interior2, majCol, makeBorder, makeBorder2, mirror, myStack, pickMax, pickMaxes, pickNotMaxes, pickUnique, repeat, rigid, smear, splitAll, splitCols, splitColumns, splitRows, spreadCols, stackLine, toOrigin, wrap.
\n\nEach transformation takes either an image or a list of images, and produces either an image or a list of images. Image -> image transformations are also broadcast across lists of images to produce list -> list transformations. Transformations which take multiple inputs are given the original sample input image, or a black image with the dimensions of the output image, to make them unary.
\n\nTransformations are implemented to be O(#input pixels) to ensure no transformation takes significantly more time than any other, although some exceptions were made when they take negligible time for other reasons.
", "comments": [ { "id": 865995, "author_name": "Alejandro de Miquel", "author_url": "", "post_date": "2020-05-29T02:57:48.937000", "content": "Hi icecuber, thanks a lot for sharing these details about your solution, and congratulations for winning the competition. It feels that there are many common things with our solution (very briefly described here), but you get a much better result. We also do the initial recoloring, for example, and then try a lot of different functions to see what works best.\nOf course, there are some differences. I didn't think of adding the diagonally flipped images; maybe that would have given us a better score. Maybe parallelizing and restricting ourselves to only unary functions would have helped too. Many of our functions started having many paramenters quickly, in order to make everything as general as possible; but, as you say, the execution time grows very fast.\nAlso, we allow certain operations to be executed before we impose a final output size. We found that it can be useful for operatios like switchColors, cropAllBackground or minimize (which deletes adjacent rows/columns that are identical), which are quite simple but give a different view of the task. Another difference is that we do some kind of BFS, but storing only 3 candidates at the same time at maximum. Therefore, we didn't run in to memory issues. However, I don't know if we missed any solutions for tasks because of that.
I believe if you keep on working this way, you finally maybe can crack ARC without AGI.
", "votes": 4, "replies": [] }, { "id": 865954, "author_name": "hamlet", "author_url": "", "post_date": "2020-05-29T02:12:31.427000", "content": "You are a genius! in all the published solutions up to now, i think this is the only solution without ensemble, but reached the incredible best score. Thanks for sharing your great solution.
", "votes": 4, "replies": [] }, { "id": 869800, "author_name": "Animesh Nareda", "author_url": "", "post_date": "2020-06-01T09:43:24.330000", "content": "Thanks a lot for sharing your solution and Congrats on winning
", "votes": 2, "replies": [] }, { "id": 866979, "author_name": "Kepler456b", "author_url": "", "post_date": "2020-05-29T21:02:42.533000", "content": "@icecuber, congratulations! One more question: if you were not limited by time and memory, do you see solving thr entire set with your approach?
", "votes": 2, "replies": [] }, { "id": 866013, "author_name": "Froggy McFrogson", "author_url": "", "post_date": "2020-05-29T03:27:33.993000", "content": "Hello, a well deserved win! You derserve it for all the hard work. (By the way how many hours do you think it took you?)
\n\nThe only one I got from my algorithm is when I added diagonal flipped transformations too!
\n\nOne question I have, is you talked a lot about the transformations, but not so much about the selection. e.g. for a problem like \"colour all the shapes with the most holes, green\". Or \"colour all the pixels nearest a line the same color as the line\". Or \"crop round the symmetrical shapes\". Or \"select a multicoloured shape with the most blue pixels\". Did you have some list of important properties for shapes/blobs/etc. And did you have a good method of filtering out the correct rules, or did you go for a more brute force approach and try as many rules as possible?
\n\nAlso, why not write up your result and submit it to an Artificial Intelligence Journal? Just an idea.
\n\nMy view is that because a human can look at the images and quickly deduce the solution, that there should be a way to deduce the answer in the same way without too much processing.
", "votes": 2, "replies": [] }, { "id": 868330, "author_name": "kenluck2001", "author_url": "", "post_date": "2020-05-31T06:37:00.993000", "content": "If you can re-organized your writing to meet a machine learning journal format, then you could have a scientific paper from this competition. You can also get another research paper from your last competition (https://www.kaggle.com/c/trackml-particle-identification/discussion/63249)
", "votes": 1, "replies": [] }, { "id": 2752165, "author_name": "Mahmoud Abdulhamid", "author_url": "", "post_date": "2024-04-14T18:57:36.807000", "content": "how can i run and save all 400 task
", "votes": 0, "replies": [] }, { "id": 2723503, "author_name": "Ahmad Jajan", "author_url": "", "post_date": "2024-03-30T09:26:32.510000", "content": "You are a genius!
", "votes": 0, "replies": [] }, { "id": 1027713, "author_name": "Naim Mhedhbi", "author_url": "", "post_date": "2020-09-26T09:57:28.507000", "content": "Bravo ! GOOD job
", "votes": 0, "replies": [] }, { "id": 871926, "author_name": "Somesh Sharma", "author_url": "", "post_date": "2020-06-02T18:21:07.463000", "content": "Congratulations for 🥇 and thanks for sharing this information.
", "votes": 0, "replies": [] }, { "id": 869272, "author_name": "James Gadoury", "author_url": "", "post_date": "2020-05-31T22:08:32.100000", "content": "Thank you so much for taking the time to write out and detail your solution. It was very thorough and pleasant to read.
", "votes": 0, "replies": [] }, { "id": 867506, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-05-30T11:26:20.963000", "content": "Thank you very much for elobarating on your strategy for solving this task and your willingness to share the code @icecuber .\nSome ideas like the diagonal flipping are genious
", "votes": 0, "replies": [] }, { "id": 867380, "author_name": "liuze", "author_url": "", "post_date": "2020-05-30T08:24:30.890000", "content": "impressive!
", "votes": 0, "replies": [] }, { "id": 867167, "author_name": "adityaork", "author_url": "", "post_date": "2020-05-30T04:17:49.487000", "content": "@icecuber Even with the described picking strategy for candidates, it will be very hard to pick top 3 candidates given the size of candidates, can you elaborate more on how you picked top 3 ?
", "votes": 0, "replies": [] }, { "id": 867124, "author_name": "Giang", "author_url": "", "post_date": "2020-05-30T02:49:16.700000", "content": "oh nice
", "votes": 0, "replies": [] }, { "id": 866382, "author_name": "Stefan Mandl", "author_url": "", "post_date": "2020-05-29T10:55:34.373000", "content": "Hey @icecuber congratulations ... well deserved!
\n\nReading you description, I'm not sure I get how you actually \"parse\" the input. You talk about images
\n\n\n\n\nAn image has a position, a size, and a 2d array of colors (0-9).
\n
Are those parts of the input or the whole input grid?
\n\nIf they are parts, could you elaborate on how you choose the relevant ones for a given input?
", "votes": 0, "replies": [] }, { "id": 866342, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-05-29T10:00:56.330000", "content": "Awesome work. I have few questions:
\n\nMany thanks in advance!
", "votes": 0, "replies": [ { "id": 866495, "author_name": "Froggy McFrogson", "author_url": "", "post_date": "2020-05-29T13:01:45.610000", "content": "Hello, you may be interested in my notebook which tells you how to call C++ functions from Python so you get the best of both worlds. You want to use C++ as it's 30x faster than Python in some cases.
", "votes": 1, "replies": [] }, { "id": 866550, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-05-29T13:37:14.543000", "content": "Thanks for the reference. Indeed, this is one way to do it but I was thinking more about doing it in a Python fashion (using CPython or Numba if necessary for example).
", "votes": 0, "replies": [] } ] }, { "id": 866280, "author_name": "ahmed gabr", "author_url": "", "post_date": "2020-05-29T08:59:10.350000", "content": "Congratulations icecuber for the brilliant solution ! Will the code be open sourced github ?
", "votes": 0, "replies": [] }, { "id": 866062, "author_name": "Mertony Sarceda", "author_url": "", "post_date": "2020-05-29T04:45:32.350000", "content": "Kudos to you @icecuber for such brilliant moves in solving this challenging competition. Thank you for the inspiration
", "votes": 0, "replies": [] }, { "id": 866483, "author_name": "", "author_url": "", "post_date": "2020-05-29T12:51:32.627000", "content": "", "votes": 0, "replies": [] }, { "id": 872094, "author_name": "corochann", "author_url": "", "post_date": "2020-06-02T23:01:49.873000", "content": "Congratulations, thanks for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 24, "messages": [ { "id": 865917, "postDate": "2020-05-29T01:28:29.223Z", "content": "Thanks to @fchollet for making this dataset, and organizing the competition. I think understanding and making intelligence is extremely interesting, and working on this problem and reading the dataset's paper significantly changed my view on AGI. The problem of squeezing prior knowledge and generalization into your model really becomes clear when you need to do it yourself. I had actually read the paper before the contest was announced, but working on the contest gave me a much more detailed view of the practical side of things.
\n\nUnfortunately, I don't feel like my solution itself brings us closer to AGI. The main component is a DSL which applies up to 4 of 142 unary transformations (42 different functions, some have different variants). This DSL is solved by enumeration (exploiting duplicates) + a greedy stacking combiner. Everything is implemented efficiently in C++ (with no dependencies) and running in parallel.
\n\nMy code is currently > 10k lines long, but (I think) most of it is not used. I will publish it after removing the unused / deprecated / failed code.
\n\nEdit: Code and official documentation now available at\nhttps://github.com/top-quarks/ARC-solution and \nhttps://github.com/top-quarks/ARC-solution/blob/master/ARC-solution_documentation.pdf\n
\n\nEdit 2: Public notebook scoring 0.794\nhttps://www.kaggle.com/icecuber/arc-1st-place-solution\n
\n\nIn practice I started out with implementing the first 100 training tasks by hand, extracting and generalizing useful functions from that. I made a brute force combiner of these transformations, choosing to only support transformations from images to a single image. Originally I wanted more types such as positions and colors, but it turned out that it was easier to represent these as images (such as a dot at the position, or a dot of a given color). This decision made it much easier to make an DSL, and got 10 tasks on the LB.
\n\nI then noticed that distribution of training, evaluation and LB were quite different, so I decided the evaluation dataset was better used as training data. I hand-coded 100 evaluation tasks, which I used to add more transformations, and improve my DSL. I noticed that functions taking more than 1 argument were growing my search space super-exponentially, and that they usually just took either the input or output image size as second argument. This led me to only keep unary functions. I also added lists of images as a type, to solve tasks that required looping. I also started representing my DSL in a DAG to exploit duplicates (multiple transformations giving same output). This required me to rewrite most of my code, but gave me 14 tasks on LB.
\n\nOptimizations, multithreading, reducing memory usage, etc. gave me 17 on LB. And augmenting my samples with diagonally flipped tasks, boosted me up to 21.
\n\nNote that the above (and below) is simplified, and there are many, many details I'm forgetting / omitting as I write this.
\n\nI remap colors in each sample input to be consistent across samples. A typical use of this is when each sample has a single color, but it is different for each sample. The recoloring would make all of them equal. I do this through iteratively, greedily recoloring the most extreme color across all images using features such as size, number of objects or position. So if in each sample input there is a single-colored object considerably larger than all others, I make all those objects the same color. This works surprisingly well, even for cases I did not think I accounted for. It even straight up solves some easy tasks from the training set by making all outputs equal.
\n\nI double the sample inputs by adding the same tasks flipped along a diagonal. This makes for 3 full runs (depth 3 for performance reasons): one normal, and one for each diagonal. This simple addition moved me from 17 to 21 tasks solved on the LB (and hence helped more than all the optimizations needed for doing depth 4 search instead of depth 3).
\n\nI first generate pieces as explained in \"Overview of DSL\". I do combinations and transformations of piece sizes to fit the training samples, and pick the one consistent with the most training samples. I break ties by approximately ordering the combinations by complexity (so transforming a piece with size WxH to get (2W)x(H+1) is more likely than (W+2)x(3H). The output size is correct for about 97% of the evaluation tasks.
\n\nDetails in \"Overview of DSL\". Output size is used to give the second argument to functions like embed, which crops / pads images to a given size.
\n\nSee \"Overview of DSL\".
\n\nI order candidates by the following criteria in decreasing order of importance:\n - Fits most training samples\n - Uses fewest transformations (max depth across pieces)\n - Uses fewest pieces in the stack
\n\nAll the nodes in my DSL are either images, or lists of images. An image has a position, a size, and a 2d array of colors (0-9). Black (0) is treated as transparent in most of the functions.
\n\nFirst, sequentially apply up to 3 or 4 unary transformations (see \"List of transformations\") to the input image. Then optionally apply a subset of the following (in order):\n - Stack lists of images\n - Move image to origin\n - Color all non-black pixels some color (for each color in every training output)\n - Resize image (crop / pad) to fit output size
\n\nI call result of the above transformations \"pieces\". Finally, stack some pieces to produce a final output.
\n\nMy solver mainly enumerates all the pieces, and applies a greedy solver to stack them. Since there are 142 image transformations at each depth, plus transformations at the end, we can't naively generate all the pieces. Fortunately, many different paths of transformations lead to the same image. I therefore make a DAG (Directed Acyclic Graph) for each training input, where each node is a image, and edges are transformations. This typically leaves on the order of 10^7 nodes in the DAGs and a few million unique pieces (where pieces are different if they differ for any sample input image or the test image).
\n\nI run multiple tasks in parallel (4 at depth 3, 2 at depth 4) using a script which tries to schedule all the tasks to fit within memory and 9 hour time limit, this runs maybe 50% of the images at depth 4 and the rest at depth 3 (I didn't really benchmark it, except 67% were run at depth 4 in an old version). The memory limit was typically more restrictive than the time limit, so I reduced memory usage through several tricks, such as huffman coding the images in the DAGs and using a custom memory-efficient hashmap.
\n\nMy greedy stacker produces each candidate output as follows:\n - Split pieces into black parts and non-black parts (parts not picked are treated as transparent)\n - Pick a guiding sample\n - Pick some subset of samples to \"care about\"\n - Up to 10 times:\n - Among all the pieces which do not contradict any of the images we care about\n - Add the piece which fits the most new pixels in the guiding image\n - If the test output image contains no holes, add this as a candidate
\n\nThe stacker is optimized using bitsets to be fast and memory efficient.
\n\nGreedily stacking outputs worked surprisingly well, even solving many tasks I didn't think it could solve. It consistently beat all my attempts at alternatives and improvements, so it just stayed mostly the same from the beginning.
\n\nI tried several ideas which didn't make it. One of these was adding general functions of more than one variable. However, none of my many attempts seemed to be able to give a significant improvement.
\n\nAnother idea was a system of deducing transformations from the training outputs, which I really wanted to work. If I could make this in a general way, I could maybe apply meet-in-the-middle to search double the depth! However, after many attempts I found that deductions are way harder to stack and generalize than transformations. This made it much harder to generalize deductions, and ultimately only a single deduction was kept (deducing \"kronecker products\" as last step).
\n\nYet another approach was generating a large number of sample tasks to train ML algorithms and fit hyperparameters (f.ex. different costs for each transformation). However, I concluded that generating new tasks was harder than solving them.
\n\nI always felt ranking candidates was a much easier problem than generating candidates (f.ex. if all inputs are symmetrical, then output should probably be symmetrical). Ideally this could be used to fill in parts of the image, or generate a huge number of candidates and leave most of the work to the ranking algorithm. However, I never found a way to exploit powerful ranking to my advantage. My simple ranking heuristics worked well enough for its purpose.
\n\nDepth 3, without adding flipped images. Faster, but probably worse than a full run.
\n\nTotal: 419 (Some of the 400 tasks have multiple tests)\nSize : 409 (97.6% deduced the correct output size)\nCands: 176 (42% had the correct answer among the generated candidates)\nCorrect: 164 (39% gave the correct answer among top 3 answers)
\n\nMany functions have many variations, but are only listed once (f.ex. there are 14 variations of pickMax for different criteria).\nFill, Move, border, broadcast, center, colShape, composeGrowing, compress, compress2, compress3, connect, count, cut, embed, eraseCol, filterCol, getPos, getSize, getSize0, gravity, half, hull, hull0, insideMarked, interior, interior2, majCol, makeBorder, makeBorder2, mirror, myStack, pickMax, pickMaxes, pickNotMaxes, pickUnique, repeat, rigid, smear, splitAll, splitCols, splitColumns, splitRows, spreadCols, stackLine, toOrigin, wrap.
\n\nEach transformation takes either an image or a list of images, and produces either an image or a list of images. Image -> image transformations are also broadcast across lists of images to produce list -> list transformations. Transformations which take multiple inputs are given the original sample input image, or a black image with the dimensions of the output image, to make them unary.
\n\nTransformations are implemented to be O(#input pixels) to ensure no transformation takes significantly more time than any other, although some exceptions were made when they take negligible time for other reasons.
", "rawMarkdown": "Thanks to @fchollet for making this dataset, and organizing the competition. I think understanding and making intelligence is extremely interesting, and working on this problem and reading the dataset's paper significantly changed my view on AGI. The problem of squeezing prior knowledge and generalization into your model really becomes clear when you need to do it yourself. I had actually read the paper before the contest was announced, but working on the contest gave me a much more detailed view of the practical side of things.\n\nUnfortunately, I don't feel like my solution itself brings us closer to AGI. The main component is a DSL which applies up to 4 of 142 unary transformations (42 different functions, some have different variants). This DSL is solved by enumeration (exploiting duplicates) + a greedy stacking combiner. Everything is implemented efficiently in C++ (with no dependencies) and running in parallel.\n\nMy code is currently > 10k lines long, but (I think) most of it is not used. I will publish it after removing the unused / deprecated / failed code.\n\nEdit: Code and official documentation now available at\n[https://github.com/top-quarks/ARC-solution](https://github.com/top-quarks/ARC-solution) and \n[https://github.com/top-quarks/ARC-solution/blob/master/ARC-solution_documentation.pdf\n](https://github.com/top-quarks/ARC-solution/blob/master/ARC-solution_documentation.pdf)\n\nEdit 2: Public notebook scoring 0.794\n[https://www.kaggle.com/icecuber/arc-1st-place-solution\n](https://www.kaggle.com/icecuber/arc-1st-place-solution)\n\n###My path\n\n\nIn practice I started out with implementing the first 100 training tasks by hand, extracting and generalizing useful functions from that. I made a brute force combiner of these transformations, choosing to only support transformations from images to a single image. Originally I wanted more types such as positions and colors, but it turned out that it was easier to represent these as images (such as a dot at the position, or a dot of a given color). This decision made it much easier to make an DSL, and got 10 tasks on the LB.\n\nI then noticed that distribution of training, evaluation and LB were quite different, so I decided the evaluation dataset was better used as training data. I hand-coded 100 evaluation tasks, which I used to add more transformations, and improve my DSL. I noticed that functions taking more than 1 argument were growing my search space super-exponentially, and that they usually just took either the input or output image size as second argument. This led me to only keep unary functions. I also added lists of images as a type, to solve tasks that required looping. I also started representing my DSL in a DAG to exploit duplicates (multiple transformations giving same output). This required me to rewrite most of my code, but gave me 14 tasks on LB.\n\nOptimizations, multithreading, reducing memory usage, etc. gave me 17 on LB. And augmenting my samples with diagonally flipped tasks, boosted me up to 21.\n\nNote that the above (and below) is simplified, and there are many, many details I'm forgetting / omitting as I write this.\n\n\n## Main solver steps\n### 1. Recolor inputs\nI remap colors in each sample input to be consistent across samples. A typical use of this is when each sample has a single color, but it is different for each sample. The recoloring would make all of them equal. I do this through iteratively, greedily recoloring the most extreme color across all images using features such as size, number of objects or position. So if in each sample input there is a single-colored object considerably larger than all others, I make all those objects the same color. This works surprisingly well, even for cases I did not think I accounted for. It even straight up solves some easy tasks from the training set by making all outputs equal.\n\n### 2. Maybe add diagonally flipped images\nI double the sample inputs by adding the same tasks flipped along a diagonal. This makes for 3 full runs (depth 3 for performance reasons): one normal, and one for each diagonal. This simple addition moved me from 17 to 21 tasks solved on the LB (and hence helped more than all the optimizations needed for doing depth 4 search instead of depth 3).\n\n### 3. Solve for output size\nI first generate pieces as explained in \"Overview of DSL\". I do combinations and transformations of piece sizes to fit the training samples, and pick the one consistent with the most training samples. I break ties by approximately ordering the combinations by complexity (so transforming a piece with size WxH to get (2W)x(H+1) is more likely than (W+2)x(3H). The output size is correct for about 97% of the evaluation tasks.\n\n### 4. Generate transformed pieces using output size\nDetails in \"Overview of DSL\". Output size is used to give the second argument to functions like embed, which crops / pads images to a given size.\n\n### 5. Greedily stack pieces to fit a training sample\nSee \"Overview of DSL\".\n\n### 6. Pick the top 3 candidates\nI order candidates by the following criteria in decreasing order of importance:\n - Fits most training samples\n - Uses fewest transformations (max depth across pieces)\n - Uses fewest pieces in the stack\n\n\n\n## Overview of DSL\nAll the nodes in my DSL are either images, or lists of images. An image has a position, a size, and a 2d array of colors (0-9). Black (0) is treated as transparent in most of the functions.\n\nFirst, sequentially apply up to 3 or 4 unary transformations (see \"List of transformations\") to the input image. Then optionally apply a subset of the following (in order):\n - Stack lists of images\n - Move image to origin\n - Color all non-black pixels some color (for each color in every training output)\n - Resize image (crop / pad) to fit output size\n\nI call result of the above transformations \"pieces\". Finally, stack some pieces to produce a final output.\n\n\nMy solver mainly enumerates all the pieces, and applies a greedy solver to stack them. Since there are 142 image transformations at each depth, plus transformations at the end, we can't naively generate all the pieces. Fortunately, many different paths of transformations lead to the same image. I therefore make a DAG (Directed Acyclic Graph) for each training input, where each node is a image, and edges are transformations. This typically leaves on the order of 10^7 nodes in the DAGs and a few million unique pieces (where pieces are different if they differ for any sample input image or the test image).\n\nI run multiple tasks in parallel (4 at depth 3, 2 at depth 4) using a script which tries to schedule all the tasks to fit within memory and 9 hour time limit, this runs maybe 50% of the images at depth 4 and the rest at depth 3 (I didn't really benchmark it, except 67% were run at depth 4 in an old version). The memory limit was typically more restrictive than the time limit, so I reduced memory usage through several tricks, such as huffman coding the images in the DAGs and using a custom memory-efficient hashmap.\n\nMy greedy stacker produces each candidate output as follows:\n - Split pieces into black parts and non-black parts (parts not picked are treated as transparent)\n - Pick a guiding sample\n - Pick some subset of samples to \"care about\"\n - Up to 10 times:\n - Among all the pieces which do not contradict any of the images we care about\n - Add the piece which fits the most new pixels in the guiding image\n - If the test output image contains no holes, add this as a candidate\n\nThe stacker is optimized using bitsets to be fast and memory efficient.\n\n\n\n##Some notes about choices and failed ideas:\n\nGreedily stacking outputs worked surprisingly well, even solving many tasks I didn't think it could solve. It consistently beat all my attempts at alternatives and improvements, so it just stayed mostly the same from the beginning.\n\nI tried several ideas which didn't make it. One of these was adding general functions of more than one variable. However, none of my many attempts seemed to be able to give a significant improvement.\n\nAnother idea was a system of deducing transformations from the training outputs, which I really wanted to work. If I could make this in a general way, I could maybe apply meet-in-the-middle to search double the depth! However, after many attempts I found that deductions are way harder to stack and generalize than transformations. This made it much harder to generalize deductions, and ultimately only a single deduction was kept (deducing \"kronecker products\" as last step).\n\nYet another approach was generating a large number of sample tasks to train ML algorithms and fit hyperparameters (f.ex. different costs for each transformation). However, I concluded that generating new tasks was harder than solving them.\n\nI always felt ranking candidates was a much easier problem than generating candidates (f.ex. if all inputs are symmetrical, then output should probably be symmetrical). Ideally this could be used to fill in parts of the image, or generate a huge number of candidates and leave most of the work to the ranking algorithm. However, I never found a way to exploit powerful ranking to my advantage. My simple ranking heuristics worked well enough for its purpose.\n\n\n##Other\n\n###Statistics for the evaluation dataset\nDepth 3, without adding flipped images. Faster, but probably worse than a full run.\n\nTotal: 419 (Some of the 400 tasks have multiple tests)\nSize : 409 (97.6% deduced the correct output size)\nCands: 176 (42% had the correct answer among the generated candidates)\nCorrect: 164 (39% gave the correct answer among top 3 answers)\n\n\n\n###List of transformations\nMany functions have many variations, but are only listed once (f.ex. there are 14 variations of pickMax for different criteria).\nFill, Move, border, broadcast, center, colShape, composeGrowing, compress, compress2, compress3, connect, count, cut, embed, eraseCol, filterCol, getPos, getSize, getSize0, gravity, half, hull, hull0, insideMarked, interior, interior2, majCol, makeBorder, makeBorder2, mirror, myStack, pickMax, pickMaxes, pickNotMaxes, pickUnique, repeat, rigid, smear, splitAll, splitCols, splitColumns, splitRows, spreadCols, stackLine, toOrigin, wrap.\n\nEach transformation takes either an image or a list of images, and produces either an image or a list of images. Image -> image transformations are also broadcast across lists of images to produce list -> list transformations. Transformations which take multiple inputs are given the original sample input image, or a black image with the dimensions of the output image, to make them unary.\n\nTransformations are implemented to be O(#input pixels) to ensure no transformation takes significantly more time than any other, although some exceptions were made when they take negligible time for other reasons.\n", "votes": 264 }, { "id": 865995, "postDate": "2020-05-29T02:57:48.937Z", "content": "Hi icecuber, thanks a lot for sharing these details about your solution, and congratulations for winning the competition. It feels that there are many common things with our solution (very briefly described here), but you get a much better result. We also do the initial recoloring, for example, and then try a lot of different functions to see what works best.\nOf course, there are some differences. I didn't think of adding the diagonally flipped images; maybe that would have given us a better score. Maybe parallelizing and restricting ourselves to only unary functions would have helped too. Many of our functions started having many paramenters quickly, in order to make everything as general as possible; but, as you say, the execution time grows very fast.\nAlso, we allow certain operations to be executed before we impose a final output size. We found that it can be useful for operatios like switchColors, cropAllBackground or minimize (which deletes adjacent rows/columns that are identical), which are quite simple but give a different view of the task. Another difference is that we do some kind of BFS, but storing only 3 candidates at the same time at maximum. Therefore, we didn't run in to memory issues. However, I don't know if we missed any solutions for tasks because of that.
I believe if you keep on working this way, you finally maybe can crack ARC without AGI.
", "rawMarkdown": "I believe if you keep on working this way, you finally maybe can crack ARC without AGI. ", "votes": 4 }, { "id": 865954, "postDate": "2020-05-29T02:12:31.427Z", "content": "You are a genius! in all the published solutions up to now, i think this is the only solution without ensemble, but reached the incredible best score. Thanks for sharing your great solution.
", "rawMarkdown": "You are a genius! in all the published solutions up to now, i think this is the only solution without ensemble, but reached the incredible best score. Thanks for sharing your great solution.", "votes": 4 }, { "id": 869800, "postDate": "2020-06-01T09:43:24.330Z", "content": "Thanks a lot for sharing your solution and Congrats on winning
", "rawMarkdown": "Thanks a lot for sharing your solution and Congrats on winning", "votes": 2 }, { "id": 866979, "postDate": "2020-05-29T21:02:42.533Z", "content": "@icecuber, congratulations! One more question: if you were not limited by time and memory, do you see solving thr entire set with your approach?
", "rawMarkdown": "@icecuber, congratulations! One more question: if you were not limited by time and memory, do you see solving thr entire set with your approach?", "votes": 2 }, { "id": 866013, "postDate": "2020-05-29T03:27:33.993Z", "content": "Hello, a well deserved win! You derserve it for all the hard work. (By the way how many hours do you think it took you?)
\n\nThe only one I got from my algorithm is when I added diagonal flipped transformations too!
\n\nOne question I have, is you talked a lot about the transformations, but not so much about the selection. e.g. for a problem like \"colour all the shapes with the most holes, green\". Or \"colour all the pixels nearest a line the same color as the line\". Or \"crop round the symmetrical shapes\". Or \"select a multicoloured shape with the most blue pixels\". Did you have some list of important properties for shapes/blobs/etc. And did you have a good method of filtering out the correct rules, or did you go for a more brute force approach and try as many rules as possible?
\n\nAlso, why not write up your result and submit it to an Artificial Intelligence Journal? Just an idea.
\n\nMy view is that because a human can look at the images and quickly deduce the solution, that there should be a way to deduce the answer in the same way without too much processing.
", "rawMarkdown": "Hello, a well deserved win! You derserve it for all the hard work. (By the way how many hours do you think it took you?)\n\nThe only one I got from my algorithm is when I added diagonal flipped transformations too!\n\nOne question I have, is you talked a lot about the transformations, but not so much about the selection. e.g. for a problem like \"colour all the shapes with the most holes, green\". Or \"colour all the pixels nearest a line the same color as the line\". Or \"crop round the symmetrical shapes\". Or \"select a multicoloured shape with the most blue pixels\". Did you have some list of important properties for shapes/blobs/etc. And did you have a good method of filtering out the correct rules, or did you go for a more brute force approach and try as many rules as possible?\n\nAlso, why not write up your result and submit it to an Artificial Intelligence Journal? Just an idea.\n\nMy view is that because a human can look at the images and quickly deduce the solution, that there should be a way to deduce the answer in the same way without too much processing. ", "votes": 2 }, { "id": 868330, "postDate": "2020-05-31T06:37:00.993Z", "content": "If you can re-organized your writing to meet a machine learning journal format, then you could have a scientific paper from this competition. You can also get another research paper from your last competition (https://www.kaggle.com/c/trackml-particle-identification/discussion/63249)
", "rawMarkdown": "If you can re-organized your writing to meet a machine learning journal format, then you could have a scientific paper from this competition. You can also get another research paper from your last competition (https://www.kaggle.com/c/trackml-particle-identification/discussion/63249)", "votes": 1 }, { "id": 2752165, "postDate": "2024-04-14T18:57:36.807Z", "content": "how can i run and save all 400 task
", "rawMarkdown": "how can i run and save all 400 task" }, { "id": 2723503, "postDate": "2024-03-30T09:26:32.510Z", "content": "You are a genius!
", "rawMarkdown": "You are a genius! " }, { "id": 1027713, "postDate": "2020-09-26T09:57:28.507Z", "content": "Bravo ! GOOD job
", "rawMarkdown": "Bravo ! GOOD job " }, { "id": 871926, "postDate": "2020-06-02T18:21:07.463Z", "content": "Congratulations for 🥇 and thanks for sharing this information.
", "rawMarkdown": "Congratulations for 🥇 and thanks for sharing this information." }, { "id": 869272, "postDate": "2020-05-31T22:08:32.100Z", "content": "Thank you so much for taking the time to write out and detail your solution. It was very thorough and pleasant to read.
", "rawMarkdown": "Thank you so much for taking the time to write out and detail your solution. It was very thorough and pleasant to read." }, { "id": 867506, "postDate": "2020-05-30T11:26:20.963Z", "content": "Thank you very much for elobarating on your strategy for solving this task and your willingness to share the code @icecuber .\nSome ideas like the diagonal flipping are genious
", "rawMarkdown": "Thank you very much for elobarating on your strategy for solving this task and your willingness to share the code @icecuber .\nSome ideas like the diagonal flipping are genious " }, { "id": 867380, "postDate": "2020-05-30T08:24:30.890Z", "content": "impressive!
", "rawMarkdown": "impressive!" }, { "id": 867167, "postDate": "2020-05-30T04:17:49.487Z", "content": "@icecuber Even with the described picking strategy for candidates, it will be very hard to pick top 3 candidates given the size of candidates, can you elaborate more on how you picked top 3 ?
", "rawMarkdown": "@icecuber Even with the described picking strategy for candidates, it will be very hard to pick top 3 candidates given the size of candidates, can you elaborate more on how you picked top 3 ?" }, { "id": 867124, "postDate": "2020-05-30T02:49:16.700Z", "content": "oh nice
", "rawMarkdown": "oh nice\n" }, { "id": 866382, "postDate": "2020-05-29T10:55:34.373Z", "content": "Hey @icecuber congratulations ... well deserved!
\n\nReading you description, I'm not sure I get how you actually \"parse\" the input. You talk about images
\n\n\n\n\nAn image has a position, a size, and a 2d array of colors (0-9).
\n
Are those parts of the input or the whole input grid?
\n\nIf they are parts, could you elaborate on how you choose the relevant ones for a given input?
", "rawMarkdown": "Hey @icecuber congratulations ... well deserved!\n\nReading you description, I'm not sure I get how you actually \"parse\" the input. You talk about images\n\n> An image has a position, a size, and a 2d array of colors (0-9).\n\nAre those parts of the input or the whole input grid?\n\nIf they are parts, could you elaborate on how you choose the relevant ones for a given input?" }, { "id": 866342, "postDate": "2020-05-29T10:00:56.330Z", "content": "Awesome work. I have few questions:
\n\nMany thanks in advance!
", "rawMarkdown": "Awesome work. I have few questions: \n\n- Is it possible to port the work to Python for example (with JIT + CPython for performance if necessary) or is the performance penalty too much?\n- Do you have any books or papers that you recommend for doing similar work?\n- Have you tried combining your work with deep learning models or other ML models?\n\nMany thanks in advance!", "replies": [ { "id": 866495, "postDate": "2020-05-29T13:01:45.610Z", "content": "Hello, you may be interested in my notebook which tells you how to call C++ functions from Python so you get the best of both worlds. You want to use C++ as it's 30x faster than Python in some cases.
", "rawMarkdown": "Hello, you may be interested in my [notebook ](https://www.kaggle.com/aidetective/calling-c-functions)which tells you how to call C++ functions from Python so you get the best of both worlds. You want to use C++ as it's 30x faster than Python in some cases.", "votes": 1 }, { "id": 866550, "postDate": "2020-05-29T13:37:14.543Z", "content": "Thanks for the reference. Indeed, this is one way to do it but I was thinking more about doing it in a Python fashion (using CPython or Numba if necessary for example).
", "rawMarkdown": "Thanks for the reference. Indeed, this is one way to do it but I was thinking more about doing it in a Python fashion (using CPython or Numba if necessary for example)." } ] }, { "id": 866280, "postDate": "2020-05-29T08:59:10.350Z", "content": "Congratulations icecuber for the brilliant solution ! Will the code be open sourced github ?
", "rawMarkdown": "Congratulations icecuber for the brilliant solution ! Will the code be open sourced github ?" }, { "id": 866062, "postDate": "2020-05-29T04:45:32.350Z", "content": "Kudos to you @icecuber for such brilliant moves in solving this challenging competition. Thank you for the inspiration
", "rawMarkdown": "Kudos to you @icecuber for such brilliant moves in solving this challenging competition. Thank you for the inspiration" }, { "id": 866483, "postDate": "2020-05-29T12:51:32.627Z", "rawMarkdown": "", "isDeleted": true }, { "id": 872094, "postDate": "2020-06-02T23:01:49.873Z", "content": "Congratulations, thanks for sharing.
", "rawMarkdown": "Congratulations, thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:06:57Z", "competition_id": 18329, "competition_slug": "abstraction-and-reasoning-challenge", "team_id": 4393753, "solution_url": "/competitions/abstraction-and-reasoning-challenge/writeups/icecuber-1st-place-solution-code-and-official-docu" }