The objective of the competition is to predict the data series represented by four kinds of scientific figures (or charts): **bar graphs**, **dot plots**, **line graphs**, and **scatter plots**.

Please see this page for details on what kind of predictions to make for each chart type: [**Graph Conventions**](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/overview/graph-conventions).

You may find a Python implementation of the metric in this notebook: [Benetech Competition Metric](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match).

## The Evaluation Metric ##

The data series for a single figure comprises two instances for evaluation: the series of values along the x-axis and the corresponding series of values along the y-axis. Each data series will be either of **numerical** type or of **categorical** type as determined by the type of chart. (See the [Graph Conventions](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/overview/graph-conventions).)

Predicted data series are evaluated by a combination of two metrics, **Levenshtein distance** for categorical (that is, string) data types and **RMSE** for numerical data types, with an initial **exact-match** criterion for the chart type and the number of values in the series. Each of these distances is rescaled and mapped to a common similarity scale by a sigmoid-type transform with an optimum value of `1`:
$$
\sigma(x) = 2 - \frac{2}{1 + e^{-x}}
$$

Evaluation of a single instance proceeds as follows:

1. We first check the **number of values and type of chart** predicted for a series. If either of these is different than that in the ground-truth, the score for that series is 0. Otherwise, we evaluate the predicted series by the data type of that series as given in the field `axes/{x|y}-axis/values-type`.

2. a) For series of numerical type, we evaluate predictions by a normalized RMSE:
$$
\textrm{NRMSE} = \sigma\left(\frac{\textrm{RMSE}(y, \widehat{y})}{\textrm{RMSE}(y, \bar{y})}\right)
$$
Note that the argument to the sigmoid transform is equal to the square root of one minus the R2 score.
    b) For series of categorical type, we evaluate predictions by a normalized Levenshtein distance:
$$
\textrm{NLev} = \sigma\left(\frac{\sum_i \textrm{Lev}(y_i, \widehat{y}_i)}{\sum_i \textrm{length}(y_i)}\right)
$$
In other words, NLev is the sum of the Levenshtein distances for each category string divided by the total length of all strings for that instance.

The **overall score** is the mean of the similarity scores over all instances.

## Submission File ##

Each row in the submission file should contain predicted series for one axis of a figure in the test set, where, for instance, `abc123_x` would give predictions for axis `x` in figure `abc123`. The values of the series should be within a single string and delimited by `;`. You must also give the appropriate type for the chart the axis belongs to.

The file should contain a header and have the following format:

```
id,data_series,chart_type
abc123_x,2;3;4;5,horizontal_bar
abc123_y,a;b;c;d,horizontal_bar
```