{
  "id": 9478,
  "title": "More problems: 797,\"!\"",
  "url": "/competitions/billion-word-imputation/discussion/9478",
  "author_name": "",
  "post_date": "2014-06-16T10:39:07.283Z",
  "votes": null,
  "comment_count": 6,
  "views": 1530,
  "content": "<p>Test line 797 is:&nbsp; 797,&quot;!&quot;</p>\n\n<p>From https://www.kaggle.com/c/billion-word-imputation/data :</p>\n\n<p><em><strong>We have removed one word from each sentence in the test set. The location of the removed word was chosen uniformly randomly and is never the first or last word of the sentence (in this dataset, the last word is always a period). You must attempt to submit the sentences in the test set with the correct missing word located in the correct location. </strong></em></p>\n\n<p>What are we expected to do here?&nbsp;&nbsp; The above text was written assuming the last word was a period and that there were always more than two words in a sentence and both assumptions are demonstrably false.&nbsp;&nbsp; Perhaps it's time to convert the bug into a feature and properly document the code that was used to generate the test set.</p>\n\n<p>Tony</p>",
  "messages": [
    {
      "id": "49189",
      "postDate": "06/16/2014 10:39:07",
      "content": "<p>Test line 797 is:&nbsp; 797,&quot;!&quot;</p>\n\n<p>From https://www.kaggle.com/c/billion-word-imputation/data :</p>\n\n<p><em><strong>We have removed one word from each sentence in the test set. The location of the removed word was chosen uniformly randomly and is never the first or last word of the sentence (in this dataset, the last word is always a period). You must attempt to submit the sentences in the test set with the correct missing word located in the correct location. </strong></em></p>\n\n<p>What are we expected to do here?&nbsp;&nbsp; The above text was written assuming the last word was a period and that there were always more than two words in a sentence and both assumptions are demonstrably false.&nbsp;&nbsp; Perhaps it's time to convert the bug into a feature and properly document the code that was used to generate the test set.</p>\n\n<p>Tony</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49224",
      "postDate": "06/17/2014 00:45:30",
      "content": "<p>There are too few of these singletons to make a significant difference. I also think there could be two words (not more than two words) and have the text hold: if not the first or last token, then you obviously impute in between (it's how I beat the &quot;benchmark&quot;).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49235",
      "postDate": "06/17/2014 06:08:29",
      "content": "<p>[quote=Triskelion;49224]</p>\n<p>There are too few of these singletons to make a significant difference.</p>\n<p>[/quote]</p>\n\n<p>Agreed - that's not the problem. &nbsp;We read the description literally and so our code couldn't deal with this case which is why I posted here.</p>\n\n<p>[quote=Triskelion;49224]</p>\n<p>I also think there could be two words (not more than two words) and have the text hold: if not the first or last token, then you obviously impute in between (it's how I beat the &quot;benchmark&quot;).</p>\n<p>[/quote]</p>\n\n<p>I disagree. &nbsp; The sentence &quot;<strong><em>We have removed one word from each sentence in the test set&quot;</em></strong></p>\n<p>clearly defines a sentence as an original complete sentence. &nbsp; If you never remove the first or last word (where word is a token as made clear elsewhere) and you always remove a word then a sentence must have at least a first word, a word removed and a last word and so must be at least three words long.</p>\n\n<p>Anyway, how do you impute in between one word? &nbsp; (the subject line is a line from the test data showing that there is only one word)</p>\n\n<p>It's just a minor bug which is best fixed by documenting it, although if I were running the test I'd have taken out all one word and two word sentences from the test data (there is only one single word test sentence and only 192 two word test sentences in the original data so removing them would have been easy), and that would be the cleaner way of doing things.</p>\n\n\n<p>Tony</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "155301",
      "postDate": "01/10/2017 14:13:13",
      "content": "<p>Tony,</p>\n\n<p>For the !B Word paper, who carried out the normalization of SBO scores?</p>",
      "rawMarkdown": "Tony,\r\n\r\nFor the !B Word paper, who carried out the normalization of SBO scores?",
      "votes": null
    },
    {
      "id": "155321",
      "postDate": "01/10/2017 16:22:20",
      "content": "<p>Sorry - It's been a while - what does SBO mean?    Probabilities are only probabilities if they sum to one - so in some neural net architectures you've got to compute everything and make it sum to one. </p>",
      "rawMarkdown": "Sorry - It's been a while - what does SBO mean?    Probabilities are only probabilities if they sum to one - so in some neural net architectures you've got to compute everything and make it sum to one.",
      "votes": null
    },
    {
      "id": "155335",
      "postDate": "01/10/2017 17:52:47",
      "content": "<p>Thanks for the reply. I think the normalization was done as follows. (1) Score each n-gram in the Test sewt using Stupid Backoff. yielding Sigma. (2) Sum the scores calculated in (1). (3) Divide each n-gram score by Sigma. No need to respond unless I'm in error. This seems obvious to me now, but it didn't before, so I'm asking for a sanity check if needed.</p>",
      "rawMarkdown": "Thanks for the reply. I think the normalization was done as follows. (1) Score each n-gram in the Test sewt using Stupid Backoff. yielding Sigma. (2) Sum the scores calculated in (1). (3) Divide each n-gram score by Sigma. No need to respond unless I'm in error. This seems obvious to me now, but it didn't before, so I'm asking for a sanity check if needed.",
      "votes": null
    },
    {
      "id": "155342",
      "postDate": "01/10/2017 18:31:41",
      "content": "<p>Ah, sorry, I didn't do that bit.   A quick google says that Stupid BackOff was introduced in <a href=\"http://www.aclweb.org/anthology/D07-1090.pdf\">http://www.aclweb.org/anthology/D07-1090.pdf</a> and a few people have asked about the implementation already, e.g.  <a href=\"http://stackoverflow.com/questions/16383194/stupid-backoff-implementation-clarification\">http://stackoverflow.com/questions/16383194/stupid-backoff-implementation-clarification</a></p>",
      "rawMarkdown": "Ah, sorry, I didn't do that bit.   A quick google says that Stupid BackOff was introduced in http://www.aclweb.org/anthology/D07-1090.pdf and a few people have asked about the implementation already, e.g.  http://stackoverflow.com/questions/16383194/stupid-backoff-implementation-clarification",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 49224,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "06/17/2014 00:45:30",
      "content": "<p>There are too few of these singletons to make a significant difference. I also think there could be two words (not more than two words) and have the text hold: if not the first or last token, then you obviously impute in between (it's how I beat the &quot;benchmark&quot;).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49235,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "06/17/2014 06:08:29",
      "content": "<p>[quote=Triskelion;49224]</p>\n<p>There are too few of these singletons to make a significant difference.</p>\n<p>[/quote]</p>\n\n<p>Agreed - that's not the problem. &nbsp;We read the description literally and so our code couldn't deal with this case which is why I posted here.</p>\n\n<p>[quote=Triskelion;49224]</p>\n<p>I also think there could be two words (not more than two words) and have the text hold: if not the first or last token, then you obviously impute in between (it's how I beat the &quot;benchmark&quot;).</p>\n<p>[/quote]</p>\n\n<p>I disagree. &nbsp; The sentence &quot;<strong><em>We have removed one word from each sentence in the test set&quot;</em></strong></p>\n<p>clearly defines a sentence as an original complete sentence. &nbsp; If you never remove the first or last word (where word is a token as made clear elsewhere) and you always remove a word then a sentence must have at least a first word, a word removed and a last word and so must be at least three words long.</p>\n\n<p>Anyway, how do you impute in between one word? &nbsp; (the subject line is a line from the test data showing that there is only one word)</p>\n\n<p>It's just a minor bug which is best fixed by documenting it, although if I were running the test I'd have taken out all one word and two word sentences from the test data (there is only one single word test sentence and only 192 two word test sentences in the original data so removing them would have been easy), and that would be the cleaner way of doing things.</p>\n\n\n<p>Tony</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155301,
      "author_name": "smedleykagnovitch",
      "author_url": "",
      "post_date": "01/10/2017 14:13:13",
      "content": "<p>Tony,</p>\n\n<p>For the !B Word paper, who carried out the normalization of SBO scores?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155321,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "01/10/2017 16:22:20",
      "content": "<p>Sorry - It's been a while - what does SBO mean?    Probabilities are only probabilities if they sum to one - so in some neural net architectures you've got to compute everything and make it sum to one. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155335,
      "author_name": "smedleykagnovitch",
      "author_url": "",
      "post_date": "01/10/2017 17:52:47",
      "content": "<p>Thanks for the reply. I think the normalization was done as follows. (1) Score each n-gram in the Test sewt using Stupid Backoff. yielding Sigma. (2) Sum the scores calculated in (1). (3) Divide each n-gram score by Sigma. No need to respond unless I'm in error. This seems obvious to me now, but it didn't before, so I'm asking for a sanity check if needed.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155342,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "01/10/2017 18:31:41",
      "content": "<p>Ah, sorry, I didn't do that bit.   A quick google says that Stupid BackOff was introduced in <a href=\"http://www.aclweb.org/anthology/D07-1090.pdf\">http://www.aclweb.org/anthology/D07-1090.pdf</a> and a few people have asked about the implementation already, e.g.  <a href=\"http://stackoverflow.com/questions/16383194/stupid-backoff-implementation-clarification\">http://stackoverflow.com/questions/16383194/stupid-backoff-implementation-clarification</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "49189": "",
    "49224": "",
    "49235": "",
    "155301": "Tony,\r\n\r\nFor the !B Word paper, who carried out the normalization of SBO scores?",
    "155321": "Sorry - It's been a while - what does SBO mean?    Probabilities are only probabilities if they sum to one - so in some neural net architectures you've got to compute everything and make it sum to one.",
    "155335": "Thanks for the reply. I think the normalization was done as follows. (1) Score each n-gram in the Test sewt using Stupid Backoff. yielding Sigma. (2) Sum the scores calculated in (1). (3) Divide each n-gram score by Sigma. No need to respond unless I'm in error. This seems obvious to me now, but it didn't before, so I'm asking for a sanity check if needed.",
    "155342": "Ah, sorry, I didn't do that bit.   A quick google says that Stupid BackOff was introduced in http://www.aclweb.org/anthology/D07-1090.pdf and a few people have asked about the implementation already, e.g.  http://stackoverflow.com/questions/16383194/stupid-backoff-implementation-clarification"
  },
  "source": "meta"
}