{
  "id": 12977,
  "title": "Cantab's approach",
  "url": "/competitions/billion-word-imputation/discussion/12977",
  "author_name": "",
  "post_date": "2015-03-22T14:39:49.233Z",
  "votes": 1,
  "comment_count": 9,
  "views": 2774,
  "content": "<p>We really haven't done any work on this task since last summer when we slipped from the top of the leaderboard.</p>\n\n<p>As there is only 40 days to go and we almost certainly have the best language model anyone has created on this task, a combination of a well trained huge recurrent neural network and a KN 5-gram (see http://code.google.com/p/1-billion-word-language-modeling-benchmark/ and our ICASSP paper athttp://arxiv.org/abs/1502.00512 &nbsp;for details) I thought I'd throw out what we've done and what we haven't done.</p>\n\n<p>What we've done is the (to me) obvious approach. &nbsp; We have the best language model, so we built a network that contained all words in all possible locations and then asked the language model to return us the 100 most likely sentences. &nbsp; Each of these has a corresponding likelihood so we divide by the sum to get an approximate posterior of each. &nbsp; If the posterior for the top one is greater than 0.5 we use it, else we do nothing. &nbsp; That's it - one big fancy language model, one search and one comparison - there was almost no Kaggle specific work involved as we had all the tools and language models already.</p>\n\n<p>Now we planned all sorts of work, but then my company took off big time and it doesn't look like it's going to slow down any time soon. &nbsp; Some easy things to do given what we have are:</p>\n<p>a) maintain any extraneous white space in the input, we assumed that things were clean when they were not.&nbsp;</p>\n<p>b) if the posterior for a slot is greater than 0.5 then put in two spaces for this slot.</p>\n<p>c) extend (b) to say for the most likely slot, put in any character that has a posterior greater than 0.5, whether the resulting word makes sense or not.</p>\n\n<p>So with very little work we got 1.1 better than the benchmark, whether these simple extensions would progress us far I don't know (we need another 0.4 to get to the top again). &nbsp; But I would like to find out in the next 40 days, so if anyone (a) understands everything I've said without needing tuition, (b) has a reasonably specced Linux machine with good connectivity and (c) has the time right now then do get in touch.</p>\n\n<p>Tony</p>\n<p>www.speechmatics.com</p>",
  "messages": [
    {
      "id": "67612",
      "postDate": "03/22/2015 14:39:49",
      "content": "<p>We really haven't done any work on this task since last summer when we slipped from the top of the leaderboard.</p>\n\n<p>As there is only 40 days to go and we almost certainly have the best language model anyone has created on this task, a combination of a well trained huge recurrent neural network and a KN 5-gram (see http://code.google.com/p/1-billion-word-language-modeling-benchmark/ and our ICASSP paper athttp://arxiv.org/abs/1502.00512 &nbsp;for details) I thought I'd throw out what we've done and what we haven't done.</p>\n\n<p>What we've done is the (to me) obvious approach. &nbsp; We have the best language model, so we built a network that contained all words in all possible locations and then asked the language model to return us the 100 most likely sentences. &nbsp; Each of these has a corresponding likelihood so we divide by the sum to get an approximate posterior of each. &nbsp; If the posterior for the top one is greater than 0.5 we use it, else we do nothing. &nbsp; That's it - one big fancy language model, one search and one comparison - there was almost no Kaggle specific work involved as we had all the tools and language models already.</p>\n\n<p>Now we planned all sorts of work, but then my company took off big time and it doesn't look like it's going to slow down any time soon. &nbsp; Some easy things to do given what we have are:</p>\n<p>a) maintain any extraneous white space in the input, we assumed that things were clean when they were not.&nbsp;</p>\n<p>b) if the posterior for a slot is greater than 0.5 then put in two spaces for this slot.</p>\n<p>c) extend (b) to say for the most likely slot, put in any character that has a posterior greater than 0.5, whether the resulting word makes sense or not.</p>\n\n<p>So with very little work we got 1.1 better than the benchmark, whether these simple extensions would progress us far I don't know (we need another 0.4 to get to the top again). &nbsp; But I would like to find out in the next 40 days, so if anyone (a) understands everything I've said without needing tuition, (b) has a reasonably specced Linux machine with good connectivity and (c) has the time right now then do get in touch.</p>\n\n<p>Tony</p>\n<p>www.speechmatics.com</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68072",
      "postDate": "03/24/2015 20:12:15",
      "content": "<p>First, bravo, it really is an impressive language model to generate candidate sentences like that and get such a good leaderboard score&nbsp;so quickly.</p>\n<p>I had a question, though, more from&nbsp;the Kaggle perspective (perhaps an Admin?)... the rules state &quot;You should use only the sentences in the training set to build you model&quot;, which I've taken to mean that I can't train on a different massive corpus, but if I read your explanation properly, you trained on the &quot;Monolingual language model training data&quot; -- an 11GB (compressed?!) dataset.</p>\n<p>Is that, then, allowed?... Is this a valid approach? &nbsp;My reading of the rules&nbsp;months ago was that this disallowed&nbsp;language models designed from other training sets,&nbsp;which would include this approach, but I'd be interested to hear that I was incorrect.</p>\n<p>And I really don't mean to take away from an excellent solution; I just understood the rules differently and was looking for clarification.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68085",
      "postDate": "03/24/2015 21:31:26",
      "content": "<p>I built the corpus used in this evaluation (and debugged the Kaggle version when they included the test set in with the training set) so I've no doubt that we've done things properly.</p>\n\n<p>Tony</p>\n<p>www.speechmatics.com</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68089",
      "postDate": "03/24/2015 21:47:22",
      "content": "<p>Ah, got it. &nbsp;Fantastic! &nbsp;Thanks for clarifying!</p>\n<p>All the more impressive then! &nbsp;Kudos!</p>\n<p>---Chip</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68838",
      "postDate": "03/29/2015 11:02:46",
      "content": "<p>&quot;a) maintain any extraneous white space in the input, we assumed that things were clean when they were not&quot;</p>\n<p>Can you explain what does this mean?</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68845",
      "postDate": "03/29/2015 14:16:30",
      "content": "<p>Yes. sure. &nbsp; From looking at the data it would seem that there is one space between words, with no extra space at the end. &nbsp; So we wrote code that parses everything and outputs in this assumed format. &nbsp;It turns out things aren't as clean as we assumed (as documented here in another thread) and maintaining this extraneous white space gives some gains.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68854",
      "postDate": "03/29/2015 16:16:06",
      "content": "<p>Thanks!</p>\n<p>mmm -&nbsp; I don't see any extra spaces in the data (which makes me worry I'm doing something terribly wrong somewhere...) or maybe this was fixed already in the v2 version of the data?</p>\n<p>Cheers</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68870",
      "postDate": "03/29/2015 19:32:01",
      "content": "<p>Sorry, it's been a long time since we looked at this data. &nbsp; The list was just a list of things that I thought we should look at. &nbsp; IIRC the spaces issue was mentioned in one of the threads here.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68872",
      "postDate": "03/29/2015 19:35:52",
      "content": "<p>If we take &quot;space&quot; to mean the ascii value 32, then I think there are no problems in the data. &nbsp;For example, you will never see more than two spaces (defined this way) between words.</p>\n<p>However, there is some non-ascii text in the data, encoded in UTF-8. &nbsp;And there are a few cases of a special Unicode space character that is represented in UTF-8 with some sort of two or three byte sequence.</p>\n<p>I wouldn't worry about it. &nbsp;There are very few of these cases.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68889",
      "postDate": "03/29/2015 20:40:06",
      "content": "<p>I should have said &quot;you will never see more than *one* space (defined this way) between words&quot;.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 68072,
      "author_name": "chipmonkey",
      "author_url": "",
      "post_date": "03/24/2015 20:12:15",
      "content": "<p>First, bravo, it really is an impressive language model to generate candidate sentences like that and get such a good leaderboard score&nbsp;so quickly.</p>\n<p>I had a question, though, more from&nbsp;the Kaggle perspective (perhaps an Admin?)... the rules state &quot;You should use only the sentences in the training set to build you model&quot;, which I've taken to mean that I can't train on a different massive corpus, but if I read your explanation properly, you trained on the &quot;Monolingual language model training data&quot; -- an 11GB (compressed?!) dataset.</p>\n<p>Is that, then, allowed?... Is this a valid approach? &nbsp;My reading of the rules&nbsp;months ago was that this disallowed&nbsp;language models designed from other training sets,&nbsp;which would include this approach, but I'd be interested to hear that I was incorrect.</p>\n<p>And I really don't mean to take away from an excellent solution; I just understood the rules differently and was looking for clarification.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68085,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "03/24/2015 21:31:26",
      "content": "<p>I built the corpus used in this evaluation (and debugged the Kaggle version when they included the test set in with the training set) so I've no doubt that we've done things properly.</p>\n\n<p>Tony</p>\n<p>www.speechmatics.com</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68089,
      "author_name": "chipmonkey",
      "author_url": "",
      "post_date": "03/24/2015 21:47:22",
      "content": "<p>Ah, got it. &nbsp;Fantastic! &nbsp;Thanks for clarifying!</p>\n<p>All the more impressive then! &nbsp;Kudos!</p>\n<p>---Chip</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68838,
      "author_name": "meron1",
      "author_url": "",
      "post_date": "03/29/2015 11:02:46",
      "content": "<p>&quot;a) maintain any extraneous white space in the input, we assumed that things were clean when they were not&quot;</p>\n<p>Can you explain what does this mean?</p>\n\n<p>Thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68845,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "03/29/2015 14:16:30",
      "content": "<p>Yes. sure. &nbsp; From looking at the data it would seem that there is one space between words, with no extra space at the end. &nbsp; So we wrote code that parses everything and outputs in this assumed format. &nbsp;It turns out things aren't as clean as we assumed (as documented here in another thread) and maintaining this extraneous white space gives some gains.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68854,
      "author_name": "meron1",
      "author_url": "",
      "post_date": "03/29/2015 16:16:06",
      "content": "<p>Thanks!</p>\n<p>mmm -&nbsp; I don't see any extra spaces in the data (which makes me worry I'm doing something terribly wrong somewhere...) or maybe this was fixed already in the v2 version of the data?</p>\n<p>Cheers</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68870,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "03/29/2015 19:32:01",
      "content": "<p>Sorry, it's been a long time since we looked at this data. &nbsp; The list was just a list of things that I thought we should look at. &nbsp; IIRC the spaces issue was mentioned in one of the threads here.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68872,
      "author_name": "onemillionmonkeys",
      "author_url": "",
      "post_date": "03/29/2015 19:35:52",
      "content": "<p>If we take &quot;space&quot; to mean the ascii value 32, then I think there are no problems in the data. &nbsp;For example, you will never see more than two spaces (defined this way) between words.</p>\n<p>However, there is some non-ascii text in the data, encoded in UTF-8. &nbsp;And there are a few cases of a special Unicode space character that is represented in UTF-8 with some sort of two or three byte sequence.</p>\n<p>I wouldn't worry about it. &nbsp;There are very few of these cases.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68889,
      "author_name": "onemillionmonkeys",
      "author_url": "",
      "post_date": "03/29/2015 20:40:06",
      "content": "<p>I should have said &quot;you will never see more than *one* space (defined this way) between words&quot;.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "67612": "",
    "68072": "",
    "68085": "",
    "68089": "",
    "68838": "",
    "68845": "",
    "68854": "",
    "68870": "",
    "68872": "",
    "68889": ""
  },
  "source": "meta"
}