{
  "id": 9500,
  "title": "Evaluation criterion ?",
  "url": "/competitions/billion-word-imputation/discussion/9500",
  "author_name": "",
  "post_date": "2014-06-19T15:19:32.953Z",
  "votes": null,
  "comment_count": 5,
  "views": 2115,
  "content": "<p>The evaluation criterion says&nbsp;&quot;average levenshteins distance&quot; between sentences. I'm wondering if it is at the word level or at the character level.&nbsp;</p>",
  "messages": [
    {
      "id": "49306",
      "postDate": "06/19/2014 15:19:32",
      "content": "<p>The evaluation criterion says&nbsp;&quot;average levenshteins distance&quot; between sentences. I'm wondering if it is at the word level or at the character level.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49307",
      "postDate": "06/19/2014 15:22:55",
      "content": "<p>It's got to be at the character level else the baseline would have a score of 1.0.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49308",
      "postDate": "06/19/2014 15:37:35",
      "content": "<p>Correct, it is character level.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49309",
      "postDate": "06/19/2014 15:37:56",
      "content": "<p>Thanks for clarifying that.</p>\n\n<p>Well, to me the evaluation criterion seems quite wrong. Take the following sentences as an example:</p>\n<p>Original sentence: &quot;cat sat on the mat&quot;</p>\n<p>Imputed sentence: &quot;sat on the mat&quot; (edit distance: 4)</p>\n<p>Predicted sentence: &quot;baby sat on the mat&quot; ( edit distance: 5)</p>\n<p>A perfectly valid sentence has higher score than just submitting the imputed sentence. This is possible not just in this contrived example, it's also possible in case of common nouns, proper nouns, adjectives, adverbs, pronouns imputed. So basically it's testing for conjunctions, prepositions and may be verbs.&nbsp;</p>\n<p>Shouldn't evaluation criteria contain some semantic aspect ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49310",
      "postDate": "06/19/2014 15:59:58",
      "content": "<p>Why this is an issue? Part of the challenge&nbsp;is using sentence&nbsp;context to assign probabilities to the possible words. While &quot;cat sat on the mat&quot; is just as likely as &quot;baby sat on the mat&quot;, you'll agree that &quot;Obama is the US president&quot; is more probable than&nbsp;&quot;cat is the US president&quot;. If there are more sentence like the former, scores will be lower. If there are more like the latter, scores will be better. It's still a level playing field.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49311",
      "postDate": "06/19/2014 16:02:11",
      "content": "<p>Yes, you are right. agree.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 49307,
      "author_name": "tonyrobinson",
      "author_url": "",
      "post_date": "06/19/2014 15:22:55",
      "content": "<p>It's got to be at the character level else the baseline would have a score of 1.0.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49308,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "06/19/2014 15:37:35",
      "content": "<p>Correct, it is character level.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49309,
      "author_name": "darshanhegde",
      "author_url": "",
      "post_date": "06/19/2014 15:37:56",
      "content": "<p>Thanks for clarifying that.</p>\n\n<p>Well, to me the evaluation criterion seems quite wrong. Take the following sentences as an example:</p>\n<p>Original sentence: &quot;cat sat on the mat&quot;</p>\n<p>Imputed sentence: &quot;sat on the mat&quot; (edit distance: 4)</p>\n<p>Predicted sentence: &quot;baby sat on the mat&quot; ( edit distance: 5)</p>\n<p>A perfectly valid sentence has higher score than just submitting the imputed sentence. This is possible not just in this contrived example, it's also possible in case of common nouns, proper nouns, adjectives, adverbs, pronouns imputed. So basically it's testing for conjunctions, prepositions and may be verbs.&nbsp;</p>\n<p>Shouldn't evaluation criteria contain some semantic aspect ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49310,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "06/19/2014 15:59:58",
      "content": "<p>Why this is an issue? Part of the challenge&nbsp;is using sentence&nbsp;context to assign probabilities to the possible words. While &quot;cat sat on the mat&quot; is just as likely as &quot;baby sat on the mat&quot;, you'll agree that &quot;Obama is the US president&quot; is more probable than&nbsp;&quot;cat is the US president&quot;. If there are more sentence like the former, scores will be lower. If there are more like the latter, scores will be better. It's still a level playing field.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49311,
      "author_name": "darshanhegde",
      "author_url": "",
      "post_date": "06/19/2014 16:02:11",
      "content": "<p>Yes, you are right. agree.&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "49306": "",
    "49307": "",
    "49308": "",
    "49309": "",
    "49310": "",
    "49311": ""
  },
  "source": "meta"
}