In many ways, I feel I had only just gotten started with this competition by the time the submission deadline hit. This is a very interesting problem that has let me start understanding the machine learning toolbox in a completely new way. I am curious now about what this dataset can show us about the limits of each of the other tools in the machine learning toolbox.
\n\nMy first attempt was written inside a Kaggle notebook, attempting to solve geometry and tessellation problems by hand visualizing the results as I went along. I had reached the level \n- https://www.kaggle.com/jamesmcguigan/arc-geometry-solvers/
\n\nThis got me sufficiently noticed to be invited to join my first team: \"Mathematicians + Experts\" who gave me new perspectives on how to approach Kaggle Competitions as well as introducing me to new ideas. Thank you.
\n\nI realized at this point that I had reached the complexity limit for code written in a jupyter notebook, so ported my code to a multi-file IDE github repo and used my previously written kaggle_compile.py script to create a single submission.py which could easily be copy/pasted into Kaggle. \n- https://www.kaggle.com/jamesmcguigan/kaggle-compile-py-python-ide-to-kaggle-compiler/
My next approach was to write a typed Object Oriented Class Model around the dataset, to allow for static typing (yay for linters!) and ease of navigaton between related data-structures. I also converted all the grids into numpy.array(dtype=np.int8).
Attempting to solve the general problem of f(g(h(x))), I wrote proof of concept code using inspect.signiture() figure out all possible permutations of f(g(h(x))) and implement an IoC dependency injection solver. Then the deadline started getting close.
I then went back to my original notebook code and properly ported it into the IDE. This resulted in Solver and ProblemSetSorter and ProblemEncoder base classes containing all the common case logic for looping over the dataset and applying the .detect(task), .fit(task), .predict(task), .test(task), .solve_grid(np.ndarray, *args) lifecycle. This significantly reduces the effort required to write new solver algorithms, when all that is needed is to implement a subset of the lifecycle methods. The rest of the autowiring is handled by the base class.
Combined with the OO DataModel, and a modified plot_task() function (that also shows the numbers and algorithm solutions), this provides a very clean interface for showcasing new Solvers within a Jupiter Notebook.
Looking through the teams collective codebase, I encounter XGBoost for the first time. I ported the existing code into my Solver framework with significant refactoring and then started playing around with greatly increasing the featuremap and hyperparaeter tuning. I was able to greatly improve the success rate of the algorithm
\n```
training : {'correct': 18, 'guesses': 49, 'total': 416, 'error': 0.9567, 'name': 'training'}\nevaluation : {'correct': 3, 'guesses': 19, 'total': 419, 'error': 0.9928, 'name': 'evaluation'}\ntest : {'correct': 0, 'guesses': 8, 'total': 104, 'error': 1.0, 'name': 'test'}\ntime : 00:00:30
\n\ntraining : {'correct': 43, 'guesses': 266, 'total': 416, 'error': 0.8966, 'name': 'training'}\nevaluation : {'correct': 14, 'guesses': 266, 'total': 419, 'error': 0.9666, 'name': 'evaluation'}\ntest : {'correct': 3.4, 'guesses': 65, 'total': 104, 'error': 1.0, 'name': 'test'}\ntime : 00:14:53\n```
\n\nWhat surprised me was how many seemingly unrelated classes of problem XGBoost could autosolve (in the training and evaluation dataset) when given a sufficiently large multi-dimensional (flattened) feature map, including each pixel's relative \"view\" of neighbouring pixels. However, this XGBoost approach also led to a large amount of random guessing, as it could quite often find rules that matched all the train examples, but incorrectly matched the test. This approach was able to score 0.990 == 1/104 on the public leaderboard.
I did a complete rewrite of the XGBoostSolver, turning it into a fully subclassable ProblemSetSolver and Encoder base classes, that would allow configuration based definition of typed univariate functions that could be used to auto-generate a feature map. However, it turns out that XGBoost had signifcant trouble when faced with the seemingly simple Single Color Problem.
\n\nI also showed how to implement Hyperopt Bayesian Hyperparameter Optimization was also performed on XGBoost.
\n\nI also wrote a few silly Solvers, as the DoNothingSolver(), and the GlobSolver() which did a hashmap lookup on the training and evaluation datasets incase any of the hidden submission dataset. Alas, these did not find anything.
The full list of solved training and evaluation tasks is visualized in the notebook, along with full code and an extended writeup.\n- https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers
", "comments": [], "comment_count": 0, "messages": [ { "id": 866443, "postDate": "2020-05-29T12:14:41.307Z", "content": "In many ways, I feel I had only just gotten started with this competition by the time the submission deadline hit. This is a very interesting problem that has let me start understanding the machine learning toolbox in a completely new way. I am curious now about what this dataset can show us about the limits of each of the other tools in the machine learning toolbox.
\n\nMy first attempt was written inside a Kaggle notebook, attempting to solve geometry and tessellation problems by hand visualizing the results as I went along. I had reached the level \n- https://www.kaggle.com/jamesmcguigan/arc-geometry-solvers/
\n\nThis got me sufficiently noticed to be invited to join my first team: \"Mathematicians + Experts\" who gave me new perspectives on how to approach Kaggle Competitions as well as introducing me to new ideas. Thank you.
\n\nI realized at this point that I had reached the complexity limit for code written in a jupyter notebook, so ported my code to a multi-file IDE github repo and used my previously written kaggle_compile.py script to create a single submission.py which could easily be copy/pasted into Kaggle. \n- https://www.kaggle.com/jamesmcguigan/kaggle-compile-py-python-ide-to-kaggle-compiler/
My next approach was to write a typed Object Oriented Class Model around the dataset, to allow for static typing (yay for linters!) and ease of navigaton between related data-structures. I also converted all the grids into numpy.array(dtype=np.int8).
Attempting to solve the general problem of f(g(h(x))), I wrote proof of concept code using inspect.signiture() figure out all possible permutations of f(g(h(x))) and implement an IoC dependency injection solver. Then the deadline started getting close.
I then went back to my original notebook code and properly ported it into the IDE. This resulted in Solver and ProblemSetSorter and ProblemEncoder base classes containing all the common case logic for looping over the dataset and applying the .detect(task), .fit(task), .predict(task), .test(task), .solve_grid(np.ndarray, *args) lifecycle. This significantly reduces the effort required to write new solver algorithms, when all that is needed is to implement a subset of the lifecycle methods. The rest of the autowiring is handled by the base class.
Combined with the OO DataModel, and a modified plot_task() function (that also shows the numbers and algorithm solutions), this provides a very clean interface for showcasing new Solvers within a Jupiter Notebook.
Looking through the teams collective codebase, I encounter XGBoost for the first time. I ported the existing code into my Solver framework with significant refactoring and then started playing around with greatly increasing the featuremap and hyperparaeter tuning. I was able to greatly improve the success rate of the algorithm
\n```
training : {'correct': 18, 'guesses': 49, 'total': 416, 'error': 0.9567, 'name': 'training'}\nevaluation : {'correct': 3, 'guesses': 19, 'total': 419, 'error': 0.9928, 'name': 'evaluation'}\ntest : {'correct': 0, 'guesses': 8, 'total': 104, 'error': 1.0, 'name': 'test'}\ntime : 00:00:30
\n\ntraining : {'correct': 43, 'guesses': 266, 'total': 416, 'error': 0.8966, 'name': 'training'}\nevaluation : {'correct': 14, 'guesses': 266, 'total': 419, 'error': 0.9666, 'name': 'evaluation'}\ntest : {'correct': 3.4, 'guesses': 65, 'total': 104, 'error': 1.0, 'name': 'test'}\ntime : 00:14:53\n```
\n\nWhat surprised me was how many seemingly unrelated classes of problem XGBoost could autosolve (in the training and evaluation dataset) when given a sufficiently large multi-dimensional (flattened) feature map, including each pixel's relative \"view\" of neighbouring pixels. However, this XGBoost approach also led to a large amount of random guessing, as it could quite often find rules that matched all the train examples, but incorrectly matched the test. This approach was able to score 0.990 == 1/104 on the public leaderboard.
I did a complete rewrite of the XGBoostSolver, turning it into a fully subclassable ProblemSetSolver and Encoder base classes, that would allow configuration based definition of typed univariate functions that could be used to auto-generate a feature map. However, it turns out that XGBoost had signifcant trouble when faced with the seemingly simple Single Color Problem.
\n\nI also showed how to implement Hyperopt Bayesian Hyperparameter Optimization was also performed on XGBoost.
\n\nI also wrote a few silly Solvers, as the DoNothingSolver(), and the GlobSolver() which did a hashmap lookup on the training and evaluation datasets incase any of the hidden submission dataset. Alas, these did not find anything.
The full list of solved training and evaluation tasks is visualized in the notebook, along with full code and an extended writeup.\n- https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers
", "rawMarkdown": "# Visualized Notebook Kernel\n- https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers\n\n# Introduction\n\nIn many ways, I feel I had only just gotten started with this competition by the time the submission deadline hit. This is a very interesting problem that has let me start understanding the machine learning toolbox in a completely new way. I am curious now about what this dataset can show us about the limits of each of the other tools in the machine learning toolbox.\n\nMy first attempt was written inside a Kaggle notebook, attempting to solve geometry and tessellation problems by hand visualizing the results as I went along. I had reached the level \n- https://www.kaggle.com/jamesmcguigan/arc-geometry-solvers/\n\nThis got me sufficiently noticed to be invited to join my first team: \"Mathematicians + Experts\" who gave me new perspectives on how to approach Kaggle Competitions as well as introducing me to new ideas. Thank you.\n\nI realized at this point that I had reached the complexity limit for code written in a jupyter notebook, so ported my code to a multi-file IDE github repo and used my previously written `kaggle_compile.py` script to create a single `submission.py` which could easily be copy/pasted into Kaggle. \n- https://www.kaggle.com/jamesmcguigan/kaggle-compile-py-python-ide-to-kaggle-compiler/\n\nMy next approach was to write a typed Object Oriented Class Model around the dataset, to allow for static typing (yay for linters!) and ease of navigaton between related data-structures. I also converted all the grids into `numpy.array(dtype=np.int8)`.\n\nAttempting to solve the general problem of `f(g(h(x)))`, I wrote proof of concept code using `inspect.signiture()` figure out all possible permutations of `f(g(h(x)))` and implement an IoC dependency injection solver. Then the deadline started getting close.\n\nI then went back to my original notebook code and properly ported it into the IDE. This resulted in `Solver` and `ProblemSetSorter` and `ProblemEncoder` base classes containing all the common case logic for looping over the dataset and applying the `.detect(task)`, `.fit(task)`, `.predict(task)`, `.test(task)`, `.solve_grid(np.ndarray, *args)` lifecycle. This significantly reduces the effort required to write new solver algorithms, when all that is needed is to implement a subset of the lifecycle methods. The rest of the autowiring is handled by the base class. \n\nCombined with the OO DataModel, and a modified `plot_task()` function (that also shows the numbers and algorithm solutions), this provides a very clean interface for showcasing new Solvers within a Jupiter Notebook.\n\nLooking through the teams collective codebase, I encounter XGBoost for the first time. I ported the existing code into my Solver framework with significant refactoring and then started playing around with greatly increasing the featuremap and hyperparaeter tuning. I was able to greatly improve the success rate of the algorithm \n```\n### Original\ntraining : {'correct': 18, 'guesses': 49, 'total': 416, 'error': 0.9567, 'name': 'training'}\nevaluation : {'correct': 3, 'guesses': 19, 'total': 419, 'error': 0.9928, 'name': 'evaluation'}\ntest : {'correct': 0, 'guesses': 8, 'total': 104, 'error': 1.0, 'name': 'test'}\ntime : 00:00:30\n\n\n### XGBGridSolver(n_estimators=32, max_depth=10)\ntraining : {'correct': 43, 'guesses': 266, 'total': 416, 'error': 0.8966, 'name': 'training'}\nevaluation : {'correct': 14, 'guesses': 266, 'total': 419, 'error': 0.9666, 'name': 'evaluation'}\ntest : {'correct': 3.4, 'guesses': 65, 'total': 104, 'error': 1.0, 'name': 'test'}\ntime : 00:14:53\n```\n\nWhat surprised me was how many seemingly unrelated classes of problem XGBoost could autosolve (in the training and evaluation dataset) when given a sufficiently large multi-dimensional (flattened) feature map, including each pixel's relative \"view\" of neighbouring pixels. However, this XGBoost approach also led to a large amount of random guessing, as it could quite often find rules that matched all the `train` examples, but incorrectly matched the `test`. This approach was able to score `0.990` == `1/104` on the public leaderboard.\n\nI did a complete rewrite of the XGBoostSolver, turning it into a fully subclassable ProblemSetSolver and Encoder base classes, that would allow configuration based definition of typed univariate functions that could be used to auto-generate a feature map. However, it turns out that XGBoost had signifcant trouble when faced with the seemingly simple Single Color Problem. \n\nI also showed how to implement Hyperopt Bayesian Hyperparameter Optimization was also performed on XGBoost.\n\nI also wrote a few silly Solvers, as the `DoNothingSolver()`, and the `GlobSolver()` which did a hashmap lookup on the `training` and `evaluation` datasets incase any of the hidden submission dataset. Alas, these did not find anything.\n\n\n# Onwards\n\nThe full list of solved training and evaluation tasks is visualized in the notebook, along with full code and an extended writeup.\n- https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers\n", "votes": 4 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:07:10Z", "competition_id": 18329, "competition_slug": "abstraction-and-reasoning-challenge", "team_id": 4392504, "solution_url": "/competitions/abstraction-and-reasoning-challenge/writeups/mathematicians-experts-arc-oo-framework-xgboost-mu" }