{
  "id": 10574,
  "title": "Reproducing the Evaluation Criterion",
  "url": "/competitions/billion-word-imputation/discussion/10574",
  "author_name": "",
  "post_date": "2014-10-11T02:29:57.097Z",
  "votes": null,
  "comment_count": 2,
  "views": 1757,
  "content": "<p>Hello All,</p>\n<p>I am trying to reproduce the evaluation criterion.</p>\n<p>I am using the following code to compute the levenshtein&nbsp;distance between two sentences:&nbsp;http://hetland.org/coding/python/levenshtein.py</p>\n<p>On the leaderboard, submitting the test data set (without any modifications), yields a score of 5.55.</p>\n<p>So my idea was to take the training data set and:</p>\n<ul>\n<li>For n sentences, randomly delete a word other than the 1st or last word.</li>\n<li>Then compute the levenshtein distance between the original sentence, and the deleted version of the sentence.</li>\n<li>Then get the average levenshtein for the n sentences and their counterparts containing a deletion.</li>\n</ul>\n<p>If n is high enough, the average&nbsp;levenshtein distance should be similar to what is on the leaderboard when submitting the unaltered test data set: 5.55. However, for n=100000, I get an average&nbsp;levenshtein distance of 6.57 on the training data set. So it seems I am off by one.</p>\n<p>I am wondering how exactly the levenshtein distance is computed for evaluating submissions. Are the&nbsp;weights for&nbsp;insertions deletions and substitutions all 1, or is a different weighting scheme involved?&nbsp;</p>",
  "messages": [
    {
      "id": "55909",
      "postDate": "10/11/2014 02:29:57",
      "content": "<p>Hello All,</p>\n<p>I am trying to reproduce the evaluation criterion.</p>\n<p>I am using the following code to compute the levenshtein&nbsp;distance between two sentences:&nbsp;http://hetland.org/coding/python/levenshtein.py</p>\n<p>On the leaderboard, submitting the test data set (without any modifications), yields a score of 5.55.</p>\n<p>So my idea was to take the training data set and:</p>\n<ul>\n<li>For n sentences, randomly delete a word other than the 1st or last word.</li>\n<li>Then compute the levenshtein distance between the original sentence, and the deleted version of the sentence.</li>\n<li>Then get the average levenshtein for the n sentences and their counterparts containing a deletion.</li>\n</ul>\n<p>If n is high enough, the average&nbsp;levenshtein distance should be similar to what is on the leaderboard when submitting the unaltered test data set: 5.55. However, for n=100000, I get an average&nbsp;levenshtein distance of 6.57 on the training data set. So it seems I am off by one.</p>\n<p>I am wondering how exactly the levenshtein distance is computed for evaluating submissions. Are the&nbsp;weights for&nbsp;insertions deletions and substitutions all 1, or is a different weighting scheme involved?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "55910",
      "postDate": "10/11/2014 03:01:53",
      "content": "<p>It indeed sounds like you are &quot;off by one&quot;. &nbsp;I wonder if you mistakenly deleted both the space before the word and the space after the word. &nbsp;You should only delete one space. &nbsp;If the original sentence was:</p>\n<p>&nbsp; &quot;man bites dog&quot;</p>\n<p>and you delete the word &quot;bites&quot;, you should be left with:</p>\n<p>&nbsp; &quot;man dog&quot;</p>\n<p>(To answer your question, insertions, deletions and substitutions all &quot;count as 1&quot;. &nbsp;I doubt your problem is in the Python code you linked to which looks OK at first glance.)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "55911",
      "postDate": "10/11/2014 03:21:45",
      "content": "<p>Hey Eric,</p>\n<p>Thanks for your comment. You are right, the code I linked to is okay. I found the source of the error:&nbsp;some of the training examples have whitespaces after the end of the sentence, which were removed in my process of deleting a word. I accounted for this, and the results look all right now, I'm getting around 5.56 on the training examples.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 55910,
      "author_name": "onemillionmonkeys",
      "author_url": "",
      "post_date": "10/11/2014 03:01:53",
      "content": "<p>It indeed sounds like you are &quot;off by one&quot;. &nbsp;I wonder if you mistakenly deleted both the space before the word and the space after the word. &nbsp;You should only delete one space. &nbsp;If the original sentence was:</p>\n<p>&nbsp; &quot;man bites dog&quot;</p>\n<p>and you delete the word &quot;bites&quot;, you should be left with:</p>\n<p>&nbsp; &quot;man dog&quot;</p>\n<p>(To answer your question, insertions, deletions and substitutions all &quot;count as 1&quot;. &nbsp;I doubt your problem is in the Python code you linked to which looks OK at first glance.)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 55911,
      "author_name": "shuvrobiswas",
      "author_url": "",
      "post_date": "10/11/2014 03:21:45",
      "content": "<p>Hey Eric,</p>\n<p>Thanks for your comment. You are right, the code I linked to is okay. I found the source of the error:&nbsp;some of the training examples have whitespaces after the end of the sentence, which were removed in my process of deleting a word. I accounted for this, and the results look all right now, I'm getting around 5.56 on the training examples.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "55909": "",
    "55910": "",
    "55911": ""
  },
  "source": "meta"
}