{ "id": 2152, "title": "General contest questions - summary", "url": "/competitions/cprod1/discussion/2152", "author_name": "", "post_date": "2012-07-09T16:44:33.807Z", "votes": null, "comment_count": 9, "views": 4904, "content": "

Hello,

\r\n

After reading through the description + data tabs several times, I am still a bit fuzzy on the details, and I'd really appreciate feedback (from fellow contenders / contest admins):

\r\n
    \r\n
  1. are all products from training-annotated-text.json available (= present as entries) in the products.json\r\ndictionary?
  2. the format of the annotated and non-annotated files seems identical = where is the human input\r\n
\r\n
rg,
\r\n
K
", "messages": [ { "id": "12232", "postDate": "07/09/2012 16:44:33", "content": "

Hello,

\r\n

After reading through the description + data tabs several times, I am still a bit fuzzy on the details, and I'd really appreciate feedback (from fellow contenders / contest admins):

\r\n
    \r\n
  1. are all products from training-annotated-text.json available (= present as entries) in the products.json\r\ndictionary?
  2. the format of the annotated and non-annotated files seems identical = where is the human input\r\n
\r\n
rg,
\r\n
K
", "rawMarkdown": "", "votes": null }, { "id": "12244", "postDate": "07/09/2012 22:30:48", "content": "

The training-annotated-text.json file contains the tokens of the text items that were manually annotated.

\r\n

The actual results of the manual annotation are in training-disambiguated-product-mentions.csv 

\r\n

This separation mimicks what needs to be done for the leaderboard phase.

\r\n

The leaderboard-text.json file contains the tokens of the text items that you must automatically annotate.

\r\n

You must produce a .CSV file that contains the product mentions in these text items and the products for each mention.

\r\n

Note: the products in training-disambiguated-product-mentions.csv are all in the product catalog.

\r\n

Cheers!

", "rawMarkdown": "", "votes": null }, { "id": "12263", "postDate": "07/10/2012 13:13:26", "content": "

Ok, thank you very much for the clarification. Two more follow-up question: in the file produced by baseline solution script there are lines like

\r\n
8c8e3756f1c2e33e2bdf17cff0c41344:494-498,0
8c8e3756f1c2e33e2bdf17cff0c41344:529-530,RgKfuLlgHtQ
\r\n

\r\n

1. what do the lines ending with zero mean: there was a product mention (some product) in this item, but it was not possible to match it to anything in the product file? if so, do such lines\r\n need to be included in the submission as well (I don't really see how removing the zeros would impact the score)?

\r\n

2. there are ~200 tokens (before processing) in item 8c8e3756f1c2e33e2bdf17cff0c41344 - how can there be a mention indexed 529 - 530? Does the baseline solution increase the number of tokens\r\n so much?

\r\n

regards,

\r\n
Konrad \r\n
", "rawMarkdown": "", "votes": null }, { "id": "12292", "postDate": "07/11/2012 02:05:56", "content": "

Here is an answer to your first question. Think of product 0 as a catch-all ID for the product not being in the catalog. Product mentions that do not have reference product in the catalog must be included. Knowing about product mentions that reference no\r\n products can help users of your system to identify gaps in the product catalog.

", "rawMarkdown": "", "votes": null }, { "id": "12295", "postDate": "07/11/2012 02:26:04", "content": "

Here is a reply to your question about product mention 8c8e3756f1c2e33e2bdf17cff0c41344:529-530

\r\n

First, the text_item's tokens are in training-annotated-text.json file beginning at line number 182392
\r\n$ grep -n "8c8e3756f1c2e33e2bdf17cff0c41344" training-annotated-text.json
\r\n182392: "8c8e3756f1c2e33e2bdf17cff0c41344": [

\r\n

A visual inspection of the files shows that the next text_item in the file is ee82f330ed4231e00398de2fba54ba94 which is at line number 183159
\r\n$ grep -n "ee82f330ed4231e00398de2fba54ba94" training-annotated-text.json
\r\n183159: "ee82f330ed4231e00398de2fba54ba94": [

\r\n

The difference in line numbers indicates the text_item has 767 tokens
\r\n$ echo 183159-182392 | bc
\r\n767

\r\n

Note that the mention's term (529-530) is "Lowepro CompuDaypack"
\r\n$ echo 182392+1+529+1 | bc
\r\n182923
\r\n$ head -182923 training-annotated-text.json | tail -2
\r\n"Lowepro",
\r\n"CompuDaypack",

\r\n

Hope this helps.

", "rawMarkdown": "", "votes": null }, { "id": "12306", "postDate": "07/11/2012 12:59:13", "content": "

How about, from disambiguated csv:

\r\n

520a21c6ab559e7cd4da063b13b52406:171-172,0

\r\n

Where 171-172 is "Denon AVR-3311", which seems to match the product

\r\n

"pWkVcNvHk1Y": [
\r\n"AVR-3311CI Receiver (7.2 Channels, 125 W/Channel, 0.08% THD, Supports 10 Devices)"

\r\n

John

", "rawMarkdown": "", "votes": null }, { "id": "12309", "postDate": "07/11/2012 13:20:29", "content": "

Unfortunately, given the large number of annotations the data is imperfect. You appear to have found an example were the annotators failed to locate the matching product. In such cases it may be impossible to get the highest scoring answer. Note that because\r\n of random assignment of text items to dataset, any errors will be consistent among the training, leaderboard and validation files..

", "rawMarkdown": "", "votes": null }, { "id": "12358", "postDate": "07/13/2012 12:55:54", "content": "

Are there any guidelines as to what constitutes a "product"?  Were the human annotators given any instructions on what qualifies as a product?

\r\n

For example: why is "LG TVs LG Firmware" considered a product and "Sony 3D glasses" is not?

", "rawMarkdown": "", "votes": null }, { "id": "12359", "postDate": "07/13/2012 13:10:42", "content": "

The general guideline that we gave annotators was to select term X if they might go up to a sales clerk and state "I want to buy an X" and that X must be specific enough that they would just have to decide among some features (such as color and those with\r\n bundled extras).

\r\n

That said, given the subjectivity of interpretation solutions may want to simply apply data-driven approaches that can reverse engineer what the annotators opted to do using data-driven approaches.

\r\n

The example that you give looks like noisy labeling (that at least two annotators agreed to).

", "rawMarkdown": "", "votes": null }, { "id": "12589", "postDate": "07/22/2012 05:43:23", "content": "

May be this question is too silly to be asked here, but whats the use of R language in relation to this contest as i am pretty new to it.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 12244, "author_name": "cprod1", "author_url": "", "post_date": "07/09/2012 22:30:48", "content": "

The training-annotated-text.json file contains the tokens of the text items that were manually annotated.

\r\n

The actual results of the manual annotation are in training-disambiguated-product-mentions.csv 

\r\n

This separation mimicks what needs to be done for the leaderboard phase.

\r\n

The leaderboard-text.json file contains the tokens of the text items that you must automatically annotate.

\r\n

You must produce a .CSV file that contains the product mentions in these text items and the products for each mention.

\r\n

Note: the products in training-disambiguated-product-mentions.csv are all in the product catalog.

\r\n

Cheers!

", "votes": null, "replies": [] }, { "id": 12263, "author_name": "konradb", "author_url": "", "post_date": "07/10/2012 13:13:26", "content": "

Ok, thank you very much for the clarification. Two more follow-up question: in the file produced by baseline solution script there are lines like

\r\n
8c8e3756f1c2e33e2bdf17cff0c41344:494-498,0
8c8e3756f1c2e33e2bdf17cff0c41344:529-530,RgKfuLlgHtQ
\r\n

\r\n

1. what do the lines ending with zero mean: there was a product mention (some product) in this item, but it was not possible to match it to anything in the product file? if so, do such lines\r\n need to be included in the submission as well (I don't really see how removing the zeros would impact the score)?

\r\n

2. there are ~200 tokens (before processing) in item 8c8e3756f1c2e33e2bdf17cff0c41344 - how can there be a mention indexed 529 - 530? Does the baseline solution increase the number of tokens\r\n so much?

\r\n

regards,

\r\n
Konrad \r\n
", "votes": null, "replies": [] }, { "id": 12292, "author_name": "cprod1", "author_url": "", "post_date": "07/11/2012 02:05:56", "content": "

Here is an answer to your first question. Think of product 0 as a catch-all ID for the product not being in the catalog. Product mentions that do not have reference product in the catalog must be included. Knowing about product mentions that reference no\r\n products can help users of your system to identify gaps in the product catalog.

", "votes": null, "replies": [] }, { "id": 12295, "author_name": "cprod1", "author_url": "", "post_date": "07/11/2012 02:26:04", "content": "

Here is a reply to your question about product mention 8c8e3756f1c2e33e2bdf17cff0c41344:529-530

\r\n

First, the text_item's tokens are in training-annotated-text.json file beginning at line number 182392
\r\n$ grep -n "8c8e3756f1c2e33e2bdf17cff0c41344" training-annotated-text.json
\r\n182392: "8c8e3756f1c2e33e2bdf17cff0c41344": [

\r\n

A visual inspection of the files shows that the next text_item in the file is ee82f330ed4231e00398de2fba54ba94 which is at line number 183159
\r\n$ grep -n "ee82f330ed4231e00398de2fba54ba94" training-annotated-text.json
\r\n183159: "ee82f330ed4231e00398de2fba54ba94": [

\r\n

The difference in line numbers indicates the text_item has 767 tokens
\r\n$ echo 183159-182392 | bc
\r\n767

\r\n

Note that the mention's term (529-530) is "Lowepro CompuDaypack"
\r\n$ echo 182392+1+529+1 | bc
\r\n182923
\r\n$ head -182923 training-annotated-text.json | tail -2
\r\n"Lowepro",
\r\n"CompuDaypack",

\r\n

Hope this helps.

", "votes": null, "replies": [] }, { "id": 12306, "author_name": "j1o1h1n", "author_url": "", "post_date": "07/11/2012 12:59:13", "content": "

How about, from disambiguated csv:

\r\n

520a21c6ab559e7cd4da063b13b52406:171-172,0

\r\n

Where 171-172 is "Denon AVR-3311", which seems to match the product

\r\n

"pWkVcNvHk1Y": [
\r\n"AVR-3311CI Receiver (7.2 Channels, 125 W/Channel, 0.08% THD, Supports 10 Devices)"

\r\n

John

", "votes": null, "replies": [] }, { "id": 12309, "author_name": "cprod1", "author_url": "", "post_date": "07/11/2012 13:20:29", "content": "

Unfortunately, given the large number of annotations the data is imperfect. You appear to have found an example were the annotators failed to locate the matching product. In such cases it may be impossible to get the highest scoring answer. Note that because\r\n of random assignment of text items to dataset, any errors will be consistent among the training, leaderboard and validation files..

", "votes": null, "replies": [] }, { "id": 12358, "author_name": "gkoundry", "author_url": "", "post_date": "07/13/2012 12:55:54", "content": "

Are there any guidelines as to what constitutes a "product"?  Were the human annotators given any instructions on what qualifies as a product?

\r\n

For example: why is "LG TVs LG Firmware" considered a product and "Sony 3D glasses" is not?

", "votes": null, "replies": [] }, { "id": 12359, "author_name": "cprod1", "author_url": "", "post_date": "07/13/2012 13:10:42", "content": "

The general guideline that we gave annotators was to select term X if they might go up to a sales clerk and state "I want to buy an X" and that X must be specific enough that they would just have to decide among some features (such as color and those with\r\n bundled extras).

\r\n

That said, given the subjectivity of interpretation solutions may want to simply apply data-driven approaches that can reverse engineer what the annotators opted to do using data-driven approaches.

\r\n

The example that you give looks like noisy labeling (that at least two annotators agreed to).

", "votes": null, "replies": [] }, { "id": 12589, "author_name": "shekharsaha1", "author_url": "", "post_date": "07/22/2012 05:43:23", "content": "

May be this question is too silly to be asked here, but whats the use of R language in relation to this contest as i am pretty new to it.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "12232": "", "12244": "", "12263": "", "12292": "", "12295": "", "12306": "", "12309": "", "12358": "", "12359": "", "12589": "" }, "source": "meta" }