{
  "id": 10862,
  "title": "lowercase?",
  "url": "/competitions/billion-word-imputation/discussion/10862",
  "author_name": "",
  "post_date": "2014-11-07T17:54:57.650Z",
  "votes": null,
  "comment_count": 2,
  "views": 1573,
  "content": "<p>New to this forum. I browsed the data quickly in order to get some sense of the competition. I noticed that both training and test texts are not lower cased. Will lowercase help the model training and prediction?</p>",
  "messages": [
    {
      "id": "57477",
      "postDate": "11/07/2014 17:54:57",
      "content": "<p>New to this forum. I browsed the data quickly in order to get some sense of the competition. I noticed that both training and test texts are not lower cased. Will lowercase help the model training and prediction?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57537",
      "postDate": "11/09/2014 00:23:13",
      "content": "<p>I lower cased everything before building a model. &nbsp;I suspect that's best although I didn't try not lower casing.</p>\n<p>Of course, when you actually output an imputed word, you're going to want to guess the right case, and not just always output the lower case version.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57542",
      "postDate": "11/09/2014 05:56:39",
      "content": "<p>Ah, now that's interesting as I think we've both built very similar systems judging by the fact we had almost exactly the same score until recently.</p>\n<p>Our system to date is very very simple. &nbsp; We've built a cased language model from the training text then we look at the likelihood of every word fitting every available slot. &nbsp; If there is one completion that's a lot better than the rest we output it, else we leave it alone (actually we also delete some whitespace that hurts us - we'll fix this one day).</p>\n<p>I assume that's what you are doing and so there isn't much difference between using a cased language model and a lowercased one. &nbsp; Given the time to work on this we'd probably use lowercased language models and then case afterwards as this is the flow we use in our speech recognition systems.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 57537,
      "author_name": "onemillionmonkeys",
      "author_url": "",
      "post_date": "11/09/2014 00:23:13",
      "content": "<p>I lower cased everything before building a model. &nbsp;I suspect that's best although I didn't try not lower casing.</p>\n<p>Of course, when you actually output an imputed word, you're going to want to guess the right case, and not just always output the lower case version.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57542,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "11/09/2014 05:56:39",
      "content": "<p>Ah, now that's interesting as I think we've both built very similar systems judging by the fact we had almost exactly the same score until recently.</p>\n<p>Our system to date is very very simple. &nbsp; We've built a cased language model from the training text then we look at the likelihood of every word fitting every available slot. &nbsp; If there is one completion that's a lot better than the rest we output it, else we leave it alone (actually we also delete some whitespace that hurts us - we'll fix this one day).</p>\n<p>I assume that's what you are doing and so there isn't much difference between using a cased language model and a lowercased one. &nbsp; Given the time to work on this we'd probably use lowercased language models and then case afterwards as this is the flow we use in our speech recognition systems.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "57477": "",
    "57537": "",
    "57542": ""
  },
  "source": "meta"
}