{
  "id": 13289,
  "title": "What qualifies as a word that can removed from the test file?",
  "url": "/competitions/billion-word-imputation/discussion/13289",
  "author_name": "",
  "post_date": "2015-04-09T02:45:54.573Z",
  "votes": 1,
  "comment_count": 2,
  "views": 1099,
  "content": "<p>When splitting up information in either text files into words, I notice that my code, which is using python's str().split() function for each line, classifies punctuation marks, specifically commas, semicolons, and quotation marks, as individual words. When test_v2.txt was generated, were punctuation marks like this considered viable words to remove?</p>",
  "messages": [
    {
      "id": "70073",
      "postDate": "04/09/2015 02:45:54",
      "content": "<p>When splitting up information in either text files into words, I notice that my code, which is using python's str().split() function for each line, classifies punctuation marks, specifically commas, semicolons, and quotation marks, as individual words. When test_v2.txt was generated, were punctuation marks like this considered viable words to remove?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "70074",
      "postDate": "04/09/2015 03:08:48",
      "content": "<p>Also, an elaboration on the above:</p>\n<p>The training set also appears to contain some sets of words separated by hyphens, and in some cases some large blocks of pure punctuation without any actual letters. How are these treated? Does a word qualify as something with whitespace on either side, or is a word considered differently?</p>\n\n<p>Examples of the first issue (these all may or may not qualify as one &quot;word&quot;):</p>\n<p>middle-of-nowhere-for-no-discernible-bleeding-reason-whatever</p>\n<p>jump-up-and-jam-the-aisle-as-soon-as-the-plane-touches-down</p>\n<p>daffodil-followed-by-weeks-and-weeks-of-scruffy-grass</p>\n\n<p>Examples of the second issue:</p>\n<p>-------------------------------------------------------------------------</p>\n<p>.................................................................................................</p>\n<p>China.---------------------------------------------------------------------------------Where</p>\n<p>-------------------------DecemberDecemberDecemberDecember31</p>\n\n<p>Also, another unusual &quot;word&quot; that I didn't mention above but I think is also worth noting:</p>\n<p>yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "70075",
      "postDate": "04/09/2015 03:28:31",
      "content": "<p>A word means a sequence of non-whitespace characters.</p>\n<p>One subtle twist to this is that the specs say that neither the first word nor the last word can be removed.&nbsp;&nbsp; However, this sometimes happens as there are one word left so these specs have been violated.&nbsp;&nbsp; If anyone has reverse engineered the real algorithm used please share it!</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 70074,
      "author_name": "trevorkirkby",
      "author_url": "",
      "post_date": "04/09/2015 03:08:48",
      "content": "<p>Also, an elaboration on the above:</p>\n<p>The training set also appears to contain some sets of words separated by hyphens, and in some cases some large blocks of pure punctuation without any actual letters. How are these treated? Does a word qualify as something with whitespace on either side, or is a word considered differently?</p>\n\n<p>Examples of the first issue (these all may or may not qualify as one &quot;word&quot;):</p>\n<p>middle-of-nowhere-for-no-discernible-bleeding-reason-whatever</p>\n<p>jump-up-and-jam-the-aisle-as-soon-as-the-plane-touches-down</p>\n<p>daffodil-followed-by-weeks-and-weeks-of-scruffy-grass</p>\n\n<p>Examples of the second issue:</p>\n<p>-------------------------------------------------------------------------</p>\n<p>.................................................................................................</p>\n<p>China.---------------------------------------------------------------------------------Where</p>\n<p>-------------------------DecemberDecemberDecemberDecember31</p>\n\n<p>Also, another unusual &quot;word&quot; that I didn't mention above but I think is also worth noting:</p>\n<p>yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 70075,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "04/09/2015 03:28:31",
      "content": "<p>A word means a sequence of non-whitespace characters.</p>\n<p>One subtle twist to this is that the specs say that neither the first word nor the last word can be removed.&nbsp;&nbsp; However, this sometimes happens as there are one word left so these specs have been violated.&nbsp;&nbsp; If anyone has reverse engineered the real algorithm used please share it!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "70073": "",
    "70074": "",
    "70075": ""
  },
  "source": "meta"
}