The labeled training dataset contains 3000 instances, annotated using the ACT platform.\nThe dataset is provided in csv format and contains the following fields:\n\n
\nwhere citation context represents the sentence containing the citations. Each ciation context in the dataset contains the label, \"#AUTHOR_TAG\", which represents the citation that is being considered. The citation class labels correspond to values 0 and 1, where
\n
\n
The test dataset, in csv format, contains 1000 instances, with all the above mentioned fields, except labels.
\n\nBesides the training and the test sets, we also provide the participants with the full text files corresponding to the coreid's in our dataset. This can be accessed from here. The name of the text files in the full text dump represents the COREID. The participants are encouraged to use this additional information for enhancing the performance of the classifiers. The fulltext for each paper in the dataset is derived from the CORE API.
\n\nAny use of the above mentioned full text dataset or additional resources, besides the ACT dataset for this competition should be clearly documented in the short description paper. Also, please make sure, the short paper includes the appropriate references mentioned in this page.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Description", "content": "Welcome to the second edition of the 3C Citation Context Classification shared task based on Influence. This year, we organise this competition as part of the Second Workshop on Scholarly Document Processing (SDP) @ NAACL 2021.
\nThe subtask B of the 3C Shared task aims at classifying citations based on their influence. This binary classification problem has the following classes:\n
More details about the shared task can be viewed here.
\n\nTask participants are required to:\n
The paper and code should be submitted via Softconf. More details regarding the submission will be made available on the SDP website. Please make sure, your team is registered here (in case you have not done this already).
\n\nThe overview paper describing the first 3C Shared task and the submitted systems can be accessed from \nhere. To view the last year's competition leaderboard, please click this \nlink.
\nTo participate in the subtask A of this shared task, please click here.
\n\nKaggle Submission deadline: 30 April, 2021\nPaper and Code Submission deadline: 17 May, 2021 (Extended)\n\nThe submitted systems will be evaluation using the withheld test data, containing 1000 instances. The evaluation metric used will be the F1-macro score.
\n$$F-macro = \\frac{1}{n}\\sum_{i=1}^{n}\\frac{2\\times P _{i} \\times R _{i}}{P _{i} + R _{i}}$$\nWe use F1-macro score in order to give more weight to minority classes and to encourage the task participants to focus on the detection of these minority classes, which are particularly important for advancing the field of research metrics beyond just counting citations. However, we recognize that micro-F might be a better metric if our objective was to build a classifier making the least number of mistakes in production.
\nWe will use 50% of the test set for evaluation initially and this will appear on the public leaderboard as the competition progresses. The rest of the dataset will be used for the final scoring. This result will be displayed on the private leaderboard, based on which the winners will be decided.
\nThe submission files need to be in CSV format. Each row in the submission file should contain two columns, with values corresponding to the fields, unique_id, and citation_influence_label. The unique_id values are textual numbers with a preceding \"CCT\", followed by a number in the range 1 and 1000. The predicted labels should be a single value, either 0 or 1, representing the different classes.\n\nThe file should contain a header and have the following format:
\n\nunique_id,citation_influence_label\nCCT1,0\nCCT2,0\nCCT3,1\nCCT4,0\nCCT5,0\nCCT6,0\nCCT7,0\nCCT8,1\nCCT9,0\nCCT10,0\n......\nCCT997,1\nCCT998,0\nCCT999,0\nCCT1000,0\nPlease make sure that the values in the unique_id and the citation_class_labels field lie in the range mentioned above. Wrong values in the submission file could result in a failed submission. Also, verify the number of instances in the submissions is equal to 1000 before making submissions.
\nSubmit your CSV file using the 'Submit Predictions' section.\nYou can check your score from the 'My Submissions' section and choose the one to make it final score by checking the 'Use for Final Score' checkbox.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "All teams are required to submit their code and a short paper describing the methodology used. Promising solutions will be invited to give a presentation at SDP 2021.The final interpretation/decision right belongs to the organizing committee of the 3C Shared task.
\n\nThe limit on the number of submissions per day is set to 5. All teams are allowed to submit a maximum of 5 runs to the competition for the final evaluation.\n\n
\n\nTeam submissions should be made through one Kaggle account.
\n\nTeam mergers are not allowed in this competition.
\n\nParticipants are encouraged to use external datasets (ex: ACL-ARC), which is compatible with the ACT data, for the purpose of training. Use of external datasets should be mentioned in the submitted paper.
\n\nStart Date: 26/02/2021
\nEntry Deadline: None
\nEnd Date: 30/04/2021 11:59 PM UTC
\n\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }