Evaluation Metrics
The classifiers will be evaluated by metrics that reflect the two objectives stated above:
Maximize the amount of relevant information presented to the analyst
. This means presenting the highest possible fraction of tweets that were correctly classified as relevant, which corresponds to maximizing the True Positive Rate (TPR) or sensitivity;
Minimize the amount of irrelevant information presented to the analyst
. This means presenting the smallest possible fraction of tweets that were wrongly classified as relevant, which corresponds to maximizing the True Negative Rate (TNR) or specificity.
Denoting the number of relevant tweets correctly classified as relevant (True Positives) by **TP**, the number of relevant tweets incorrectly classified as irrelevant (False Negatives) by **FN**, the number of irrelevant tweets correctly classified as irrelevant (True Negatives) by **TN**, and the number of irrelevant tweets incorrectly classified as relevant (False Positives) by **FP**, the metrics are given by:

* **_TPR = TP/(TP+FN)_**
* **_TNR = TN/(TN+FP)_**
Ranking the results
By using the evaluation data sets, participants will be ranked in each case study according to the Euclidean distance of their **_(TPR, TNR)_** classifier pair to the ideal performance pair, **_(1.0, 1.0)_**, from smallest to highest value. Then a number of points from 1 to the number of participants, (_n_), is awarded according to the ranking. Finally the participant receiving the smallest total number of points considering all the case studies, wins the competition.
Competition Rules
The participants must comply with the following rules:
Participants will use only the data sets provided (DSA, DSB and DSC) to train and design their classifier(s). Two options are available:
train a single classifier for the three case studies, therefore using the data sets altogether;
train one classifier per case study, therefore using the data sets individually. In this case evaluation metrics will be computed considering the aggregated results.
The integer number that relates uniquely to the Twitter account that posted a tweet, cannot be used at the classifier input.
Participants must use only freely available tools/frameworks to train and design their classification models.
All the results have to be reproducible by using solely code provided by the participants and the data sets provided by the competition.
Once the results are published in tis web page, the participants will receive the evaluation data sets ESA, ESB and ESC, allowing them to verify the results published. At the same time the competition organizers will also verify that the participants submissions comply with the competition rules. Only after these verifications the results will be considered final.
Submissions
The following guidelines should be followed when submitting the results:
As soon as participants decide to enter the competition, we kindly ask that an email be sent to the organizers, stating the willingness to participate;
Before the participation archive submission deadline participants will deliver an archive (e.g., a ZIP file) with the following elements:
A short report naming the team and the authors, a contact person and corresponding email address, a description of the software tools/platforms employed to train and design the classifier(s), a description of the methodologies employed, and instructions on how to reproduce the design of the classifier model(s) submitted;
A computer program in an interpreted language (e.g., Java, Python, R, ...) for each classifier, that is able to take an input text file with one tweet per line and produce an output text file with the corresponding classification of each tweet. Classifier parameters may be stored in accompanying files to the programs;
A source code package that is able to execute by taking the design data sets as input and reproduce the design and training of the classifier(s) submitted. Instructions on how to execute the code will be given in the report mentioned above. Off course, authorship of the code, procedures and methodology remains with the authors participating.