Evaluation Metric
The evaluation metric for this competition is the micro \(F_1\)-score computed on the test set made of species presence-absence (PA) samples. In terms of machine learning, it is a multi-label classification task. The  \(F_1\)-score is an average measure of overlap between the predicted and actual set of species present at a given location and time.
Each test PA sample \( i \) is associated with a set of ground-truth labels \( Y_i \), namely the set of plant species (=speciesId) associated with a given combination of the columns
patchID
and
dayOfYear
(see the Data tab for details on the species observation data structure).
For each sample, the submission will provide a list of labels, i.e. the set of species predicted present \( \hat{Y}_{i,1}, \hat{Y}_{i,2}, \dots, {\hat{Y}}_{i,R_i} \).
The micro \(F_1\)-score is then computed using
\[
  F_1 = \frac{1}{N} \sum_{i=1}^N  \frac{\text{TP}_i}{\text{TP}_i+(\text{FP}_i+\text{FN}_i)/2} \\ \quad \text{Where} 
\begin{cases}
   \text{TP}_i =\text{ Number of predicted species truly present, i.e. }|\hat{Y}_i \cap Y_i |\\
   \text{FP}_i =\text{ Number of species predicted but absent, i.e. } |\hat{Y}_i \setminus Y_i | \\
   \text{FN}_i =\text{ Number of species not predicted but present, i.e. } | Y_i \setminus \hat{Y}_i |\\
  \end{cases}
  \]
Validation/Test Split of PA data and train set of PO data
![Illustration of validation/test split](https://filedn.com/ltHjVlFrSfSJL1S4hXqqTzy/split_points.png)
In order to limit the spatial bias during evaluation, the presence-absence data (PA) were split into validation and test sets using a spatial block holdout procedure, and the presence-only data (PO) were filtered to remove PO near test samples.
This procedure is illustrated in the previous figure: The test samples - in blue - are located in randomly drawn blocks of a large spatial grid while the other blocks constitute the validation PA samples - in red.
The train set is fully made of PO samples, i.e. each is the record of one species at a certain location and date, while other species might have been present. Nevertheless, PO samples falling at the exact location of test PA sample would inform on its composition. Hence, we have filtered all PO samples near the test samples, inside a radius of a few hundred meters.
Leaderboard Baselines
We provide XX baselines in the leaderboard:
Constant composition: A constant set of species is predicted, i.e. the K most common species across samples of the validation dataset so that K maximises the \(F_1\)-score on this same dataset. K turns out to be XX;
...
Submission Format
The submission format is a CSV file containing two columns for each sample (row):
Id
column containing integers corresponding to the test sample ids, corresponding to unique combinations of patchID and dayOfYear column values.
Predicted
column containing space-delimited lists of the predicted species identifiers (column spId in training/validation datasets)
The blinded test CSV, associating the "Id" column with the various input data, will be provided later in the competition.
The file should contain a header and have the following format:
Id, Predicted
1,1 52 10231
2,78 201 1243 1333 2310 4841
...
For each sample (row), the predicted species identifiers must be ordered by increasing value from left to right. No test sample is empty and the test set only contain species that are present in the train or validation set.