The competition uses top-3 error rate for the evaluation metric. For each task in the test set, you can predict up to 3 outputs for each test input grid. Each task output has one ground truth. For a given task output, if the ground truth is contained in any of the 3 predicted outputs, then the error for that task is 0, otherwise it is 1. The final score is the error averaged across all tasks.

Mathematically, for each task \\(i\\), your algorithm can make up to 3 predictions \\( o\_{ij} \\), where \\(1 \le j \le 3\\). The error for task \\(i\\) with ground truth \\( g\_i\\) is:

$$
e\_i = \min\_j d\(o\_{ij},  g\_i)
$$

where \\( d\(x,  y) \text{ is } 0 \text{ if } x=y, \text{ otherwise } 1 \\). The overall error score is the average over all \\(N\\) task outputs:

$$
\text{score } = \frac{1}{N} \sum\_i e\_i 
$$

## Format of output predictions
The training, evaluation, and as well as test _input_ data all in the same JSON format. The inputs and outputs are stored in 2d python lists. Your _output predictions_, though, must be flattened into strings, with list rows delimited with an `|`.

For example, the output `[[1,2], [3,4]]` should be reformatted to `|12|34|` as a prediction. You can have up to 3 output predictions per task, and they should be space delimited. See the `sample_solution.csv` file as an example.

The following python code converts a 2d list `pred` into the correct format:
```
def flattener(pred):
    str_pred = str([row for row in pred])
    str_pred = str_pred.replace(', ', '')
    str_pred = str_pred.replace('[[', '|')
    str_pred = str_pred.replace('][', '|')
    str_pred = str_pred.replace(']]', '|')
    return str_pred
```

## Submission File
For each task output `output_id` in the test set, you can make up to 3 predictions. The `output_id` is the `id` of the task, with an indicator of which output you are predicting for that task. Most tasks only have a single output (i.e., `0`), although some tasks have two outputs that must be predicted (i.e., `0, 1`). The file should contain a header and have the following format:

    output_id,output
    00576224_0,|32|78| |32|78| |00|00|
    ...
    12997ef3_0,|00000000000|01100000000|11000000000|...
    12997ef3_1,|00000000000|01100000000|11000000000|...
    etc.