# ARC - OO Framework + XGBoost Multimodel Solvers

Visualized Notebook Kernel
[https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers](https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers)
Introduction
In many ways, I feel I had only just gotten started with this competition by the time the submission deadline hit. This is a very interesting problem that has let me start understanding the machine learning toolbox in a completely new way. I am curious now about what this dataset can show us about the limits of each of the other tools in the machine learning toolbox.
My first attempt was written inside a Kaggle notebook, attempting to solve geometry and tessellation problems by hand visualizing the results as I went along. I had reached the level 
-
[https://www.kaggle.com/jamesmcguigan/arc-geometry-solvers/](https://www.kaggle.com/jamesmcguigan/arc-geometry-solvers/)
This got me sufficiently noticed to be invited to join my first team: "Mathematicians + Experts" who gave me new perspectives on how to approach Kaggle Competitions as well as introducing me to new ideas. Thank you.
I realized at this point that I had reached the complexity limit for code written in a jupyter notebook,  so ported my code to a multi-file IDE github repo and used my previously written
kaggle_compile.py
script to create a single
submission.py
which could easily be copy/pasted into Kaggle. 
-
[https://www.kaggle.com/jamesmcguigan/kaggle-compile-py-python-ide-to-kaggle-compiler/](https://www.kaggle.com/jamesmcguigan/kaggle-compile-py-python-ide-to-kaggle-compiler/)
My next approach was to write a typed Object Oriented Class Model around the dataset, to allow for static typing (yay for linters!) and ease of navigaton  between related data-structures. I also converted all the grids into
numpy.array(dtype=np.int8)
.
Attempting to solve the general problem of
f(g(h(x)))
, I wrote proof of concept code using
inspect.signiture()
figure out all possible permutations of
f(g(h(x)))
and implement an IoC dependency injection solver. Then the deadline started getting close.
I then went back to my original notebook code and properly ported it into the IDE. This resulted in
Solver
and
ProblemSetSorter
and
ProblemEncoder
base classes containing all the common case logic for looping over the dataset and applying the
.detect(task)
,
.fit(task)
,
.predict(task)
,
.test(task)
,
.solve_grid(np.ndarray, *args)
lifecycle. This significantly reduces the effort required to write new solver algorithms, when all that is needed is to implement a subset of the lifecycle methods. The rest of the autowiring is handled by the base class.
Combined with the OO DataModel, and a modified
plot_task()
function (that also shows the numbers and algorithm solutions), this provides a very clean interface for showcasing new Solvers within a Jupiter Notebook.
Looking through the teams collective codebase, I encounter XGBoost for the first time. I ported the existing code into my Solver framework with significant refactoring and then started playing around with greatly increasing the featuremap and hyperparaeter  tuning. I was able to greatly improve the success rate of the algorithm
```
Original
training   : {'correct': 18, 'guesses': 49, 'total': 416, 'error': 0.9567, 'name': 'training'}
evaluation : {'correct': 3, 'guesses': 19, 'total': 419, 'error': 0.9928, 'name': 'evaluation'}
test       : {'correct': 0, 'guesses': 8, 'total': 104, 'error': 1.0, 'name': 'test'}
time       : 00:00:30
XGBGridSolver(n_estimators=32, max_depth=10)
training   : {'correct': 43, 'guesses': 266, 'total': 416, 'error': 0.8966, 'name': 'training'}
evaluation : {'correct': 14, 'guesses': 266, 'total': 419, 'error': 0.9666, 'name': 'evaluation'}
test       : {'correct': 3.4, 'guesses': 65, 'total': 104, 'error': 1.0, 'name': 'test'}
time       : 00:14:53
```
What surprised me was how many seemingly unrelated classes of problem XGBoost could autosolve (in the training and evaluation dataset) when given a sufficiently large multi-dimensional (flattened) feature map, including each pixel's relative "view" of neighbouring pixels. However, this XGBoost approach also led to a large amount of random guessing, as it could quite often find rules that matched all the
train
examples, but incorrectly matched the
test
. This approach was able to score
0.990
==
1/104
on the public leaderboard.
I did a complete rewrite of the XGBoostSolver, turning it into a fully subclassable ProblemSetSolver and Encoder base classes, that would allow configuration based definition of typed univariate functions that could be used to auto-generate a feature map. However,  it turns out that XGBoost had signifcant trouble when faced with the seemingly simple Single Color Problem.
I also showed how to implement Hyperopt Bayesian Hyperparameter Optimization was also performed on XGBoost.
I also wrote a few silly Solvers, as the
DoNothingSolver()
, and the
GlobSolver()
which did a hashmap lookup on the
training
and
evaluation
datasets incase any of the hidden submission dataset. Alas,  these did not find anything.
Onwards
The full list of solved training and evaluation tasks is visualized in the notebook, along with full code and an extended writeup.
-
[https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers](https://www.kaggle.com/jamesmcguigan/arc-oo-framework-xgboost-multimodel-solvers)