A significant proportion of web usage relates to discussions, research, and purchase of consumer products. Currently, hundreds of thousands of blogs, forums, product review sites, and e-commerce merchants currently exist, in part, to service consumer's need\r\n to access product related information and demand to share experiences with products.
\r\nThe goal of this competition is to determine the state-of-the-art methods to automatically recognize product mentions in such textual content and to also disambiguate which product(s) in product catalogs are being referenced. Specifically, the task is to\r\n automatically identify all mentions of consumer products in a largely user generated collection of web-content, and to correctly identify the product(s) that each product mention refers to from a large catalog of products. The datasets provided includes hundreds\r\n of thousands of text items, a product catalog with over fifteen million products, and hundreds of manually annotated product mentions to support data-driven approaches.
\r\nThe prize pool for the contest is $10,000 and is divided as follows: $6,000 for first, $3,000 for second and $1,000 for third place submissions.
\r\nNote that the contest is colocated with the ICDM-2012 conference. There will be a workshop on the contest results on December 10th.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\nKaggle and the competition host have the right to publicly disseminate any entries or models.
\r\n\r\n Winning solutions need to be made available under\r\n a popular OSI-approved license\r\n in order to be eligible for recognition and prize money.\r\n
\r\n\r\n Prizes will be awarded after the winners have posted their solutions to the competition forum.\r\n Winners must post or link to their solutions within seven (7) days of being notified of their winning status.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
There is no maximum team size.
\r\nYou may submit a maximum of 2 entries per day.
\r\nYou may select up to 10 final submissions for judging.
\r\n\r\n Because the data can be manually decoded and thus an entry can be created by hand, scoring data will be withheld until you have locked down a model.\r\n
\r\n\r\n You must upload all kernels and model files alongside documentation in a single zip file\r\n in order for your entry to be eligible. You can do this at the time of submission, or append\r\n it to your submission at a later time (before the competition end).\r\n
\r\nStart Date: 7/2/2012 10:38 PM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 9/24/2012 11:59 PM UTC
\r\n\t\r\nBeyond the Kaggle terms and conditions you must abide by the following rules to participate in the CPROD-1 contest:
\r\nEach contestant submission of disambiguated product mentions will be scored based on the following correctness metric. In summary, the score is the F1 value\r\n of product predictions for the union of predicted and true disambiguated product mentions. The metric ranges from 0 to 1, with higher numbers representing better performance.
\r\nThe table below illustrates the performance calculation for a single contestant who scored 0.414 on a simplified test set. The table covers all possible cases of predictions versus true outcomes. In this scenario the contestant submitted six disambiguated\r\n product mentions (pm1 … pm6) while the truth set contained six manually annotated product mentions that were hidden from the contestant (tm1 … tm6). Notice that one of the predicted mentions, pm6, is not\r\n in the truth set (their start and end tokens do not align), and that one of the mentions in the truth set, tm3, is not in the predicted set. Both of these outcomes are assigned an F1 score of 0. The remaining five predictions can be scored based\r\n on the F1 calculation based on the predicted products. (See \r\nwikipedia or our wiki page on the F1 score).
\r\n| \r\n Predicted Mention \r\n | \r\n\r\n True Mention \r\n | \r\n\r\n Predicted Product \r\n | \r\n\r\n True Product \r\n | \r\n\r\n Correctness | \r\n\r\n Precision \r\n | \r\n\r\n Recall \r\n | \r\n\r\n F1 \r\n | \r\n
| \r\n pm1 \r\n | \r\n\r\n tm1 \r\n | \r\n\r\n #484946 \r\n | \r\n\r\n #484946 \r\n | \r\n\r\n TP \r\n | \r\n\r\n 100% \r\n | \r\n\r\n 100% \r\n | \r\n\r\n 100% \r\n | \r\n
| \r\n pm2 \r\n | \r\n\r\n tm2 \r\n | \r\n\r\n 0 \r\n | \r\n\r\n 0 \r\n | \r\n\r\n TP \r\n | \r\n\r\n 100% \r\n | \r\n\r\n 100% \r\n | \r\n\r\n 100% \r\n | \r\n
| \r\n Not | \r\n\r\n tm3 \r\n | \r\n\r\n Not | \r\n\r\n #103492 \r\n | \r\n\r\n FP \r\n | \r\n\r\n 0% \r\n | \r\n\r\n 0% \r\n | \r\n\r\n 0% \r\n | \r\n
| \r\n pm3 \r\n | \r\n\r\n tm4 \r\n | \r\n\r\n Not | \r\n\r\n 0 \r\n | \r\n\r\n FN \r\n | \r\n\r\n 0% \r\n | \r\n\r\n 0% \r\n | \r\n\r\n 0% \r\n | \r\n
| \r\n #223801 \r\n | \r\n\r\n Not in | \r\n\r\n FP \r\n | \r\n|||||
| \r\n pm4 \r\n | \r\n\r\n tm5 \r\n | \r\n\r\n #167712 \r\n | \r\n\r\n #167712 \r\n | \r\n\r\n TP \r\n | \r\n\r\n 50% \r\n | \r\n\r\n 50% \r\n | \r\n\r\n 50% \r\n | \r\n
| \r\n Not | \r\n\r\n #385994 \r\n | \r\n\r\n FN \r\n | \r\n|||||
| \r\n 194730 \r\n | \r\n\r\n Not in | \r\n\r\n FP \r\n | \r\n|||||
| \r\n pm5 \r\n | \r\n\r\n tm6 \r\n | \r\n\r\n #250747 \r\n | \r\n\r\n #250747 \r\n | \r\n\r\n TP \r\n | \r\n\r\n 50% \r\n | \r\n\r\n 33% \r\n | \r\n\r\n 40% \r\n | \r\n
| \r\n Not | \r\n\r\n #237004 \r\n | \r\n\r\n FN \r\n | \r\n|||||
| \r\n Not | \r\n\r\n #482721 \r\n | \r\n\r\n FN \r\n | \r\n|||||
| \r\n #722416 \r\n | \r\n\r\n Not in | \r\n\r\n FP \r\n | \r\n|||||
| \r\n pm6 \r\n | \r\n\r\n Not | \r\n\r\n #416094 \r\n | \r\n\r\n Not in | \r\n\r\n FP \r\n | \r\n\r\n 0% \r\n | \r\n\r\n 0% \r\n | \r\n\r\n 0% \r\n | \r\n
| \r\n \r\n | \r\n\r\n \r\n | \r\n\r\n \r\n | \r\n\r\n \r\n | \r\n\r\n \r\n | \r\n\r\n \r\n | \r\n\r\n avg(F1)= \r\n | \r\n\r\n 41.4% \r\n | \r\n
Solution submissions must be in the same file format as the disambiguated training data.
\r\nSubmission Reproducibility Requirements
\r\nSolution submissions and model code will be accepted throughout the competition, but will be required by the release of the model evaluation set (see Timeline). The model code submission must contain all code, data, and parameter settings necessary to evaluate\r\n your models on validation set items, and include a README file with step-by-step instructions on how to do so. The purpose of this reproducibility requirement is to ensure that solutions do not use any manual annotation of product mentions and product candidates. The\r\n model submission is required to be eligible to win prize money (see Rules).
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Prizes", "content": "1st – A cash prize of $6,000
\r\n2nd – A cash prize of $3,000
\r\n3rd – A cash prize of $1,000
Prizes will only be awarded after the winners have posted their code on the Competition Forum and agree to make the code available under the\r\nGPLv3 license.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Timeline", "content": "Monday, July 2 to Sunday, September 16: Model training period against a public leaderboard. Code submission available throughout.
\r\nSunday, September 16: Deadline to submit final models. Final code must be submitted by this time (midnight PST)
\r\nMonday, September 17: Release of the Model Evaluation Set for winner selection (10am PST)
\r\nMonday, September 24: Deadline to submit Model Evaluation Set solutions. (midnight PST)
\r\nMonday, October 1: Winners are announced
\r\nSaturday, October 13: Deadline for submission of 2-3 page report for consideration. This option is only available for contestants who submitted a model and solutions by their deadline. The paper will be referreed for admitance into the contest\r\n workshop publication. The paper must be in \r\nICDM format.
\r\nMonday, December 10: Conference workshop paper presentation and official awards ceremony.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "data-description", "content": "The CPROD1 competition involves the release of six data files to contestants. Five of the files are provided immediately, while the model evalulation set of text items will be released later to determine the contest winners. Files are in two formats: JSON\r\n format and .CSV format The six files are as follows:
\r\n| leaderboard-text.json | \r\nThis .json file contains the text items that contestants must disambiguate to determine their leaderboard score. | \r\n
| products.json | \r\nThis .json file contains a product catalog. The items in this catalog were referenced by annotator to manually create the training-disambiguated-product-mentions.csv file. It must also be referenced by contestants to create their submission file. | \r\n
| training-annotated-text.json | \r\nThis .json file contains the text items that were manually reviewed for product mentions. This file, along with the training-disambiguated-product-mentions.csv file, could be used to train a supervised model. | \r\n
| training-disambiguated-product-mentions.csv | \r\nThis .csv file contains disambiguated product mentions. This file, along with the training-annotated-text.json file could be used to train a supervised model. This file is in the same format as the solutions that contestants must submit to determine their\r\n average F1 performance.. | \r\n
| training-non-annotated-text.json | \r\nThis .json file contains supplementary text item data drawn from the same domain as the other text items in the contest. They are provided for contestants who may opt to produce semi-supervised models. | \r\n
| evaluation-text.json | \r\nThis .json file (to be provided near the end of the contest) will contain the text item to be annotated for the final submission. | \r\n
Sample Perl Code is provided in the file CPROD1_baseline.120707.pl, which also produces a simple benchmark.
\r\nThe files to be used for model training have been bundled together in TrainingSet.zip/.7z: products.json, training-annotated-text.json, training-disambiguated-text.json, training-non-annotated-text.json. The file PublicLeaderboardSet.zip/.7z contains leaderboard-text.json\r\n for determining the leaderboard rankings.
\r\nBelow we describe the key data entities involved (text items, products, and disambiguated product mentions), along with the process used to generate the data.
\r\nFor the contest, a “text item” stands for a tokenized representation of a portion or entirety of a web page or a web-forum postings page. We processed each web item to create text items as follows:
\r\n. This step is known to produce imperfect results.
\r\nHere is an example of a text_item: "TextItem": {"0c1edc5b2ed5abb25e25b966ccdb01d2": ["Here","'s","an","example","of","a","(","pre-tokenized",")","text”, “item",".","", "
", "Check","out","the","new","iPhone","4s","!"]}
\r\nNotice: 1) how the word "Here's" has been divided into two tokens: "Here" and "’s"; 2) how the end-of-sentence punctuation have been placed into their own tokens; and 3) how sentences have been seperated by both a "” token representing an end-of-sentence\r\n and a "
" token representing an end of paragraph.
\r\nFor the contest a “product item” is a semi-structured record that represents some purchasable consumer product from either the consumer electronics (CE) or automotive (AU) verticals. Each record has: 1) a unique string-based identifier,\r\n and 2) an array composed of a string-based “name”, a two character-based product category, and a two digit-based price. A sample of some of the products records is presented below (in tabular format):
\r\n{"1QlV3Pe6T0W":["iphone 3","CE",399.95],"op6rsUEYjWc":["ifone case","CE",21.25],"P7Ntsvaer1Y":["hawk break pads for SUVs","AU",110.00]}
\r\n\r\n
For the contest a disambiguated product mention is a structured record composed of two fields: a product mention identifier, and a space-separated set of product item identifiers. The identiffier represents some specific product mention\r\n within some specific text item, for example: 0c1edc5b2ed5abb25e25b966ccdb01d2:0-2 represents the product mention that begins on the first token and ends on the third token in textitem 0c1edc5b2ed5abb25e25b966ccdb01d2. (product mentions are always complete\r\n substrings of one or more tokens). Finally, the set of space-separated product identifiers represents the products within the product catalog that have been deemed to refer to the same product as the mention.
\r\n\r\n
We used the following process to annotate text items. The annotation task involved two phases: 1) the identification of product mention
\r\nwithin text items, and 2) the labeling of product records for each annotated product mention with True match (or False match).
During the first phase a set of text items were randomly selected. Each text item was reviewed by at least two different annotators. In cases where there was disagreement about mentions a third annotator broke ties.
\r\nDuring the second phase the human annotators were asked to classify which products were legitimate references for each of the product mentions. This phase was significantly more time consuming so only a small portion of product candidates were reviewed by\r\n two or more annotators.
\r\n\r\n
We randomly separated the annotated text items into training set, leaderboard set and model evaluation set using 50%, 25% and 25% proportions.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Winners", "content": "