{ "fetched_at": "2026-10-24T12:23:11Z", "source_url": "https://www.kaggle.com/competitions/competition-on-open-source-intelligence-discovery/data", "pages": [ { "name": "rules", "content": "\r\n\t\r\n

One account per participant

\r\n

You cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.

\r\n

No private sharing outside teams

\r\n

\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n

\r\n

Team Mergers

\r\n

Team mergers are not allowed in this competition.

\r\n\r\n

Team Limits

\r\n

There is no maximum team size.

\r\n

Submission Limits

\r\n

You may submit a maximum of 2 entries per day.

\r\n

You may select up to 2 final submissions for judging.

\r\n

Competition Timeline

\r\n

Start Date: TBA

\r\n

Merger Deadline: None

\r\n

Entry Deadline: None

\r\n

End Date: 2/1/2014 12:00 AM UTC

\r\n\t\r\n

The rules page is a bit special: it will auto-update the text based on the parameters of the competition. The rules you need to add here are any additional rules relating to dataset usage. You may also put grading information here.

", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "data-description", "content": "The data sets are structured in line-based text files with lines containing an hyperlink, an integer number that uniquely corresponds to one tweeter, and one label. The hyperlink references the tweet, the integer number relates to the account that posted the tweet, and the label provides its class. A simple program written in Python is provided together with the datasets, that receives these files and outputs files with the actual tweets text, the corresponding labels and the integer referencing the tweeter account. Please read carefully the bundled README.txt file.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Description", "content": "

WCCI 2018 COMPETITION ON OPEN SOURCE INTELLIGENCE DISCOVERY FOR CYBERSECURITY THREAT AWARENESS

\n\n[![wcci](http://www.ecomp.poli.br/~wcci2018/wp-content/uploads/2017/06/logo-wcci-2018.png)](http://www.ecomp.poli.br/~wcci2018/)\n\n

Introduction

\nThis competition is organized within the scope of the European Union H2020 project DiSIEM - Diversity Enhancements for Security Information and Event Management. **_Note that participation in the competition does not imply participation in the WCCI 2018 conference._**\n

Sponsorship

\nThis competition is kindly sponsored by AMADEUS, who sponsors the following prizes in Amazon gift cards:\n\n[![amadeus](https://mms.businesswire.com/media/20171005005541/en/616752/5/origAmadeus_Logo.jpg)](http://www.amadeus.com)\n\n

About the DiSIEM project

\n

The DiSIEM project objective is to improve Security Information and Event Management (SIEM) systems' capabilities using diversity-related technology. The project's expected contributions are to:

\n\n1. Integrate diverse Open Source INTelligence (OSINT) to identify relationships, trends and anomalies, and hence help reacting to new vulnerabilities affecting an IT infrastructure or even predict possible emerging threats against the infrastructure monitored by the SIEM, hence improving organizations cybersecurity threat awareness capabilities.\n2. Develop novel probabilistic security models and risk-based metrics to help security analysts decide which infrastructure configurations offer better security guarantees and increase the capacity of Security Operation Centers (SOCs) to communicate the status of the organization to C-level managers.\n3. Design novel visualization methods to present the diverse live and archival data sets, to better support the decision-making process by enabling the extraction of high-level security insight from the data which will be used by the security analysts working with SOCs that operate the SIEM.\n4. Integrate diverse, redundant and enhanced monitoring capabilities to the SIEM ecosystem.\n5. Add support for long term archival of events in public cloud storage services.\n\n

Cybersecurity Threat Awareness

\nThe main goal of cybersecurity threat awareness tools is to provide security analysts timely information about security threats to the IT infrastructures under their responsibility. This translates into two important objectives:\n\n\nFor this purpose OSINT information is collected about the security of a specific IT infrastructure, then each piece of information is classified as relevant or not to the security of that infrastructure.\n\nAs an example, in DiSIEM we are collecting tweets concerning the security of various case-study IT infrastructures. These tweets have to be classified as relevant or not to fulfil the two objectives above. A tweet is relevant if it mentions a threat to an element of an IT infrastructure (e.g., a vulnerability or an exploit) or a security measure to protect that element (e.g., an update or a software patch).\n

Why Twitter?

\nAlthough there are many sources of OSINT, including security-related ones, Twitter was used for two main reasons. First, Twitter is well-recognized as an important information hub for short notices (almost in real-time) about cutting edge information on events regarding many subjects. These include cybersecurity-related events as demonstrated by the highly-active accounts of most security feeds and researchers, where they tweet security-related news. Second, since a tweet is limited to 280 characters (mostly 40–60 words), these messages are simple to process automatically.\n\nMore importantly, our work also demonstrates that it is possible to obtain valuable security-related information from Twitter before it becomes available on established databases as confirmed threats (e.g., [National Vulnerability Database](https://nvd.nist.gov/), [ExploitDB](https://www.exploit-db.com/). As an example consider some results related to the various [NSA](https://www.nsa.gov/) tools [leaked in August 2016](https://blog.comae.io/shadow-brokers-nsa-exploits-of-the-week-3f7e17bdc216). From those, two threats called EGREGIOUSBLUNDER and ESCALATEPLOWMAN were tweeted 9 ([see publication](https://twitter.com/clucianomartins/status/765288624044802048)) and 6 ([see publication](https://twitter.com/evanderburg/status/766309829476429824)) days ahead of official confirmation on NVD.\n

Problem Statement

\nA key part in any software component designed to provide end-to-end OSINT-based cybersecurity threat awareness is a binary classifier that takes as input one piece of information, e.g., a tweet, and assigns it to one of the classes, relevant (1) or irrelevant (-1).\n\nThis competition consists in using previously labelled tweet data sets concerning three case-studies, in order to design binary classification models for the case studies. Therefore participants will develop models that take tweets as input and produce the corresponding classification for each tweet: -1 (not relevant) or 1 (relevant).\n

Datasets

\nBoth for the classifiers design stage (before the deadline for submission of results) and the classifiers evaluation stage (after the deadline for submission of results) we provide three data sets, each corresponding to a different case study (case studies A, B, and C) and corresponding IT infrastructure. The sets consist of tweets that have been manually labelled as relevant or not to the security of the IT infrastructures represented in the case studies.\n\nFor the design stage we call these sets Design Set A, B and C (DSA, DSB and DSC). The tweets in DSA, DSB and DSC have been collected from a set of Twitter accounts designated Account Set 1 (AS1).\n\nIn the evaluation stage the classifiers proposed by the competition participants will be tested using Evaluation Set A, B and C (ESA, ESB and ESC). The tweets in ESA, ESB and ESC have been collected from a set of Twitter accounts designated Account Set 2 (AS2).\n\nESA, ESB and ESC are such that all the tweets were posted in Twitter after the tweets in DSA, DSB and DSC. AS2 includes all accounts from AS1, but extends it with an additional set of accounts, having in total 223 tweeters. Hence, the evaluation procedure tests the classifiers generalization performance not only in future unseen data, but also considering data from additional Twitter accounts.\n\nThe data sets are structured in line-based text files with lines containing an hyperlink, an integer number that uniquely corresponds to one tweeter, and one label. The hyperlink references the tweet, the integer number relates to the account that posted the tweet, and the label provides its class. A simple program written in Python is provided together with the datasets, that receives these files and outputs files with the actual tweets text, the corresponding labels and the integer referencing the tweeter account. **Please read carefully the bundled README.txt**.\n\n**PLEASE USE ONLY THE DATASETS AVAILABLE BELOW OR ON THE WCCI PAGE**\n\nZip file containing the datasets.\n\nWe kindly ask all participants to send an email to [Fernando Alves and Pedro M. Ferreira] simply mentioning your participation, for management reasons.\n\n

Important dates

\nThe date for the submission of the participation archive is June 24, 2018.\n\nThe preliminary results will be published until July 1, 2018.\n\nThe validations of the results and of the submissions have to be concluded before July 5, 2018.\n

Organizers

\nPedro M. Ferreira, Alysson Bessani, Fernando Alves\nLASIGE, Faculdade de Ciências, Universidade de Lisboa, Portugal\n\nEmail addresses:\n\nPMF: pmf (at) ciencias (dot) ulisboa (dot) pt\n\nAB: anbessani (at) ciencias (dot) ulisboa (dot) pt\n\nFA: falves (at) lasige (dot) di (dot) fc (dot) ul (dot) pt\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "

Evaluation Metrics

\nThe classifiers will be evaluated by metrics that reflect the two objectives stated above:\n\nDenoting the number of relevant tweets correctly classified as relevant (True Positives) by **TP**, the number of relevant tweets incorrectly classified as irrelevant (False Negatives) by **FN**, the number of irrelevant tweets correctly classified as irrelevant (True Negatives) by **TN**, and the number of irrelevant tweets incorrectly classified as relevant (False Positives) by **FP**, the metrics are given by:\n\n* **_TPR = TP/(TP+FN)_**\n* **_TNR = TN/(TN+FP)_**\n\n

Ranking the results

\nBy using the evaluation data sets, participants will be ranked in each case study according to the Euclidean distance of their **_(TPR, TNR)_** classifier pair to the ideal performance pair, **_(1.0, 1.0)_**, from smallest to highest value. Then a number of points from 1 to the number of participants, (_n_), is awarded according to the ranking. Finally the participant receiving the smallest total number of points considering all the case studies, wins the competition.\n\n

Competition Rules

\nThe participants must comply with the following rules:\n
    \n \t
  1. Participants will use only the data sets provided (DSA, DSB and DSC) to train and design their classifier(s). Two options are available:\n
      \n \t
    • train a single classifier for the three case studies, therefore using the data sets altogether;
      • \n \t
    • train one classifier per case study, therefore using the data sets individually. In this case evaluation metrics will be computed considering the aggregated results.
      • \n
    \n
    2. \n \t
  2. The integer number that relates uniquely to the Twitter account that posted a tweet, cannot be used at the classifier input.
    3. \n \t
  3. Participants must use only freely available tools/frameworks to train and design their classification models.
    4. \n \t
  4. All the results have to be reproducible by using solely code provided by the participants and the data sets provided by the competition.
    5. \n
\nOnce the results are published in tis web page, the participants will receive the evaluation data sets ESA, ESB and ESC, allowing them to verify the results published. At the same time the competition organizers will also verify that the participants submissions comply with the competition rules. Only after these verifications the results will be considered final.\n

Submissions

\nThe following guidelines should be followed when submitting the results:\n
    \n \t
  1. As soon as participants decide to enter the competition, we kindly ask that an email be sent to the organizers, stating the willingness to participate;
    2. \n \t
  2. Before the participation archive submission deadline participants will deliver an archive (e.g., a ZIP file) with the following elements:\n
      \n \t
    • A short report naming the team and the authors, a contact person and corresponding email address, a description of the software tools/platforms employed to train and design the classifier(s), a description of the methodologies employed, and instructions on how to reproduce the design of the classifier model(s) submitted;
      • \n \t
    • A computer program in an interpreted language (e.g., Java, Python, R, ...) for each classifier, that is able to take an input text file with one tweet per line and produce an output text file with the corresponding classification of each tweet. Classifier parameters may be stored in accompanying files to the programs;
      • \n \t
    • A source code package that is able to execute by taking the design data sets as input and reproduce the design and training of the classifier(s) submitted. Instructions on how to execute the code will be given in the report mentioned above. Off course, authorship of the code, procedures and methodology remains with the authors participating.
      • \n
    \n
    3. \n
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }