Evaluation
Evaluation metric
The evaluation metric for this competition is top-30  error rate.
Each observation \( i \) is associated with a single ground-truth label \( y_i \) corresponding to the observed species.
For each observation, the submissions will provide 30 candidate labels \( \hat{y}_{i,1}, \hat{y}_{i,2}, \dots, {\hat{y}}_{i,30} \).
The top-30 error rate is then computed using
\[
  \text{Top-30 error rate} = \frac{1}{N} \sum_{i=1}^N e_i
  \quad \text{where} \quad
  e_i = \begin{cases}
  1 & \text{if } \, \forall k \in \{1,\dots,30\}, \, \hat{y}_{i,k} \neq y_i \\
  0 & \text{otherwise}
  \end{cases}
  \]
Train/test split
![Illustration of train/test split](https://raw.githubusercontent.com/maximiliense/GLC/GLC21/images/train_test_split.png)
In order to limit the spatial bias during evaluation, the observations were split into training and test sets using a spatial block holdout procedure.
This procedure is illustrated in the previous figure: the test data - in red - is drawn from different spatial blocks than the training data - in blue.
In more details, all the observations are assigned into a grid of 5km×5km quadrats.
2.5% of these quadrats are randomly sampled for the test set while the remaining ones are used for the training set.
A validation set built using the same splitting procedure is provided (column
subset
of
observations_*_train.csv
).
The code for this splitting procedure is also available on the
[GitHub](https://github.com/maximiliense/GLC)
.
Leaderboard baselines
We provide 3 baselines in the leaderboard based on last year's results:
top-30 most present species: baseline always predicting the most present species;
random forest on environmental vectors: classical method used in ecology using only the environmental vectors;
CNN on patches: CNN trained on the patches but not the environmental vectors.
These methods are detailed in
[this paper](https://hal.inria.fr/hal-02989084/file/paper_192.pdf)
.
We also provide a
[notebook](https://www.kaggle.com/tlorieul/geolifeclef2021-baselines-and-submission)
explaining how to compute some simpler baselines.
Submission Format
The submission format is a CSV file containing two columns for each observation:
Id
column containing integers corresponding to the observations ids
Predicted
column containing space-delimited lists of the 30 predicted labels made for each observation
NOTE: due to a misalignment of some test observation ids, please use the mapping provided in
test_observation_ids_mapping.csv
before submitting.
The file should contain a header and have the following format:
Id, Predicted
1, 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
2, 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
...
See also the
[following notebook](https://www.kaggle.com/tlorieul/geolifeclef2021-baselines-and-submission)
showing how submission files can be generated.