The objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the publications that appear to note a dataset.

Submissions are evaluated on a [Jaccard-based](https://en.wikipedia.org/wiki/Jaccard_index) [FBeta](https://en.wikipedia.org/wiki/F-score) score between predicted texts and ground truth texts, with `Beta = 0.5` (a `micro F0.5` score). Multiple predictions are delineated with a pipe (`|`) character in the submission file.

The following is Python code for calculating the Jaccard score for a single prediction string against a single ground truth string. Note that the overall score for a sample uses Jaccard to compare multiple ground truth and prediction strings that are pipe-delimited - this code does not handle that process or the final `micro F-beta` calculation.
    
    def jaccard(str1, str2): 
        a = set(str1.lower().split()) 
        b = set(str2.lower().split())
        c = a.intersection(b)
        return float(len(c)) / (len(a) + len(b) - len(c))

Note that ALL ground truth texts have been cleaned for matching purposes using the following code:
    
    def clean_text(txt):
        return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower())

For each publication's set of predictions, a token-based Jaccard score is calculated for each potential prediction / ground truth pair. The prediction with the highest score for a given ground truth is **matched** with that ground truth.
- Predicted strings for each publication are **sorted alphabetically** and processed in that order. Any scoring ties are resolved on the basis of that sort.
- Any matched predictions where the Jaccard score meets or exceeds the threshold of `0.5` are counted as true positives (`TP`), the remainder as false positives (`FP`).
- Any unmatched predictions are counted as false positives (`FP`).
- Any ground truths with no nearest predictions are counted as false negatives (`FN`).

All `TP`, `FP` and `FN` across all samples are used to calculate a final `micro F0.5` score. (Note that a `micro` F score does precisely this, creating one pool of `TP`, `FP` and `FN` that is used to calculate a score for the entire set of predictions.) 

## Submission File
For each publication Id in the test set, you must predict excerpts (multiple excerpts divided by a pipe character) for `PredictionString` variable. The file should contain a header and have the following format:

    Id,PredictionString
    000e04d6-d6ef-442f-b070-4309493221ba,space objects dataset|small objects data
    0176e38e-2286-4ea2-914f-0583808a98aa,small objects dataset
    01860fa5-2c39-4ea2-9124-74458ae4a4b4,large objects
    01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset
    01fea149-a6b8-4b01-8af9-51e02f46f03f,a dataset of large objects
    etc.