{
  "id": 10043,
  "title": "Data Formats",
  "url": "/competitions/billion-word-imputation/discussion/10043",
  "author_name": "",
  "post_date": "2014-08-14T21:01:49.137Z",
  "votes": null,
  "comment_count": 3,
  "views": 1559,
  "content": "<p>Hello, I have a few questions about the data formatting:</p>\n<p>i. Are there headers and columns in the training and test sets?</p>\n<p>ii. For the training set we're expected to remove one word (randomly) from each sentence, correct?</p>\n<p>iii. How should the submission file be formatted? Can you provide an example?</p>\n\n<p>Thank you,</p>\n<p>Alex</p>",
  "messages": [
    {
      "id": "52026",
      "postDate": "08/14/2014 21:01:49",
      "content": "<p>Hello, I have a few questions about the data formatting:</p>\n<p>i. Are there headers and columns in the training and test sets?</p>\n<p>ii. For the training set we're expected to remove one word (randomly) from each sentence, correct?</p>\n<p>iii. How should the submission file be formatted? Can you provide an example?</p>\n\n<p>Thank you,</p>\n<p>Alex</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52027",
      "postDate": "08/14/2014 21:04:05",
      "content": "<p>Question (iii) is answered here:&nbsp;http://www.kaggle.com/c/billion-word-imputation/details/evaluation</p>\n<p>My apologies for not searching more before posting.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52030",
      "postDate": "08/14/2014 22:07:32",
      "content": "<p>(i) you can easily download the data to see how it is formatted. &nbsp; &nbsp;It's in CSV format, so pretty easy to parse but if you are not using a library for this then watch out for &quot;&quot;&quot; and the like - that is either read up on the matter or use a library.</p>\n<p>(ii) you've got the task completely wrong. &nbsp; You are not expected to remove a word from the training set but to add one to the test set so as to restore the original sentence.</p>\n<p>There are subtleties to (ii) in the case of short sentences which you can read about in the posts here, but first you've got to understand the basic task.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52118",
      "postDate": "08/17/2014 19:34:39",
      "content": "<p>The escaped double quotes (&quot;&quot;) are worth highlighting because they only appear in the test data. &nbsp;Double quotes in the training data are not escaped. &nbsp;This caused me some trouble until I figured out what was going on.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 52027,
      "author_name": "boltzmannbrain",
      "author_url": "",
      "post_date": "08/14/2014 21:04:05",
      "content": "<p>Question (iii) is answered here:&nbsp;http://www.kaggle.com/c/billion-word-imputation/details/evaluation</p>\n<p>My apologies for not searching more before posting.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52030,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "08/14/2014 22:07:32",
      "content": "<p>(i) you can easily download the data to see how it is formatted. &nbsp; &nbsp;It's in CSV format, so pretty easy to parse but if you are not using a library for this then watch out for &quot;&quot;&quot; and the like - that is either read up on the matter or use a library.</p>\n<p>(ii) you've got the task completely wrong. &nbsp; You are not expected to remove a word from the training set but to add one to the test set so as to restore the original sentence.</p>\n<p>There are subtleties to (ii) in the case of short sentences which you can read about in the posts here, but first you've got to understand the basic task.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52118,
      "author_name": "onemillionmonkeys",
      "author_url": "",
      "post_date": "08/17/2014 19:34:39",
      "content": "<p>The escaped double quotes (&quot;&quot;) are worth highlighting because they only appear in the test data. &nbsp;Double quotes in the training data are not escaped. &nbsp;This caused me some trouble until I figured out what was going on.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "52026": "",
    "52027": "",
    "52030": "",
    "52118": ""
  },
  "source": "meta"
}