Evaluation Metric
The evaluation metric for this competition is the top-30 error rate.
Each observation \( i \) is associated with a single ground-truth label \( y_i \) corresponding to the observed species.
For each observation, the submissions will provide 30 candidate labels \( \hat{y}_{i,1}, \hat{y}_{i,2}, \dots, {\hat{y}}_{i,30} \).
The top-30 error rate is then computed using
\[
  \text{Top-30 error rate} =
 \frac{1}{N} \sum_{i=1}^N e_i
  \quad \text{where} \quad
  e_i = 
\begin{cases}
  1 & \text{if } \, \forall k \in \{1,\dots,30\}, \, \hat{y}_{i,k} \neq y_i \\
  0 & \text{otherwise}
  \end{cases}
  \]
Train/test Split
![Illustration of train/test split](https://raw.githubusercontent.com/maximiliense/GLC/master/images/train_test_split.png)
In order to limit the spatial bias during evaluation, the observations were split into training and test sets using a spatial block holdout procedure.
This procedure is illustrated in the previous figure: the test data - in red - is drawn from different spatial blocks than the training data - in blue.
In more detail, all the observations are assigned into a grid of 5km×5km quadrats.
2.5% of these quadrats are randomly sampled for the test set while the remaining ones are used for the training set.
A validation set built using the same splitting procedure is provided (column
subset
of
observations_*_train.csv
).
The code for this splitting procedure is also available on the
[GitHub repository](https://github.com/maximiliense/GLC)
.
Leaderboard Baselines
We provide 4 baselines in the leaderboard:
top-30 most present species: baseline always predicting the most present species;
random forest on environmental vectors: a classical method used in ecology using only the environmental vectors,
scikit-learn random forest of 100 trees with max depth 16 and minimum samples split of 30;
CNN on RGB patches: CNN trained on the RGB patches without any environmental vectors,
standard ResNet-50 pretrained on ImageNet and finetuned using a learning rate of 0.01, a batch size of 32 and an early stopping strategy on the validation top-30 error rate;
CNN on Red, Green and Near-IR patches: same as above but the blue channel was replaced by near-infrared highlighting the importance of this wavelength for the task.
We also provide a
[notebook](https://www.kaggle.com/tlorieul/geolifeclef2022-baselines-and-submission)
explaining how to compute some simpler baselines.
Submission Format
The submission format is a CSV file containing two columns for each observation:
Id
column containing integers corresponding to the observations ids
Predicted
column containing space-delimited lists of the 30 predicted labels made for each observation
The file should contain a header and have the following format:
Id, Predicted
1, 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
2, 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
...
See also the
[following notebook](https://www.kaggle.com/tlorieul/geolifeclef2022-baselines-and-submission)
showing how submission files can be generated.