{
  "id": 229734,
  "title": "About evaluation.",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/229734",
  "author_name": "Alexey Savin",
  "post_date": "2021-03-31T12:52:40.306000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Someone could explain to me how is evaluation happening? Please.<br>\nI red some topics here and decided to make a little experiment.<br>\nI have assumed that found more datasets labels than was mentioned in train data labels.(score by them 0.701)</p>\n<p>So, at first step I have added to 2100032a-7c33-4bff-97ef-690822c43466 PredictionString two more dataset labels(became 4 prediction for this at the moment) and score does not changed.<br>\nAnd second step was to adding 2 more labels(became 6) and still no changes in score.<br>\nThat’s what I don't understand.</p>\n<p>I Assume:<br>\n1 - If this two new labels true - score must increase.<br>\n2 - If this two new labels false - score must decrease.</p>\n<p>for example:<br>\nLet's imagine that we have 3 dataset labels in the text with total 9 words. <br>\n'alzheimer s disease neuroimaging initiative adni|adni' give to as 7 words, let's imagine again this correct words. our score 7/(9+7-7) = 0.7777 etc.<br>\n'alzheimer s disease neuroimaging initiative adni|adni|new label' 9 words, 7 correct ,2 not<br>\nscore 7/(9+9-7)=0.6363<br>\nIf all words correct score should be 1.0.</p>\n<p>I'm understand that's F0.5 Score not the same, but it should change too anyway.</p>\n<p>Back to real.<br>\nI have added 4 more labels with 10+ words in total and Score does not changes.<br>\nWhat should it mean?</p>\n<ol>\n<li>I am totally don't understand how it is calculating.<br>\nThen someone explain me please it and how it is happening when multiple labels separated by '|' calculating?<br>\nCalculating for each labels by posting multiple labels for one text happen independently to each other, further calculating mean or?</li>\n<li>Here extremely huge amount of dataset labels in test text, that's 10 word can't change evaluation by 0.001.</li>\n<li>If last version shows lower score than older version -&gt; public score displaying your top score?</li>\n</ol>",
  "messages": [
    {
      "id": 1258208,
      "postDate": "2021-03-31T12:52:40.307Z",
      "content": "<p>Someone could explain to me how is evaluation happening? Please.<br>\nI red some topics here and decided to make a little experiment.<br>\nI have assumed that found more datasets labels than was mentioned in train data labels.(score by them 0.701)</p>\n<p>So, at first step I have added to 2100032a-7c33-4bff-97ef-690822c43466 PredictionString two more dataset labels(became 4 prediction for this at the moment) and score does not changed.<br>\nAnd second step was to adding 2 more labels(became 6) and still no changes in score.<br>\nThat’s what I don't understand.</p>\n<p>I Assume:<br>\n1 - If this two new labels true - score must increase.<br>\n2 - If this two new labels false - score must decrease.</p>\n<p>for example:<br>\nLet's imagine that we have 3 dataset labels in the text with total 9 words. <br>\n'alzheimer s disease neuroimaging initiative adni|adni' give to as 7 words, let's imagine again this correct words. our score 7/(9+7-7) = 0.7777 etc.<br>\n'alzheimer s disease neuroimaging initiative adni|adni|new label' 9 words, 7 correct ,2 not<br>\nscore 7/(9+9-7)=0.6363<br>\nIf all words correct score should be 1.0.</p>\n<p>I'm understand that's F0.5 Score not the same, but it should change too anyway.</p>\n<p>Back to real.<br>\nI have added 4 more labels with 10+ words in total and Score does not changes.<br>\nWhat should it mean?</p>\n<ol>\n<li>I am totally don't understand how it is calculating.<br>\nThen someone explain me please it and how it is happening when multiple labels separated by '|' calculating?<br>\nCalculating for each labels by posting multiple labels for one text happen independently to each other, further calculating mean or?</li>\n<li>Here extremely huge amount of dataset labels in test text, that's 10 word can't change evaluation by 0.001.</li>\n<li>If last version shows lower score than older version -&gt; public score displaying your top score?</li>\n</ol>",
      "rawMarkdown": "Someone could explain to me how is evaluation happening? Please.\nI red some topics here and decided to make a little experiment.\nI have assumed that found more datasets labels than was mentioned in train data labels.(score by them 0.701)\n\nSo, at first step I have added to 2100032a-7c33-4bff-97ef-690822c43466 PredictionString two more dataset labels(became 4 prediction for this at the moment) and score does not changed.\nAnd second step was to adding 2 more labels(became 6) and still no changes in score.\nThat’s what I don't understand.\n\nI Assume:\n1 - If this two new labels true - score must increase.\n2 - If this two new labels false - score must decrease.\n\nfor example:\nLet's imagine that we have 3 dataset labels in the text with total 9 words. \n'alzheimer s disease neuroimaging initiative adni|adni' give to as 7 words, let's imagine again this correct words. our score 7/(9+7-7) = 0.7777 etc.\n'alzheimer s disease neuroimaging initiative adni|adni|new label' 9 words, 7 correct ,2 not\nscore 7/(9+9-7)=0.6363\nIf all words correct score should be 1.0.\n\nI'm understand that's F0.5 Score not the same, but it should change too anyway.\n\nBack to real.\nI have added 4 more labels with 10+ words in total and Score does not changes.\nWhat should it mean?\n1. I am totally don't understand how it is calculating.\nThen someone explain me please it and how it is happening when multiple labels separated by '|' calculating?\nCalculating for each labels by posting multiple labels for one text happen independently to each other, further calculating mean or?\n2. Here extremely huge amount of dataset labels in test text, that's 10 word can't change evaluation by 0.001.\n3. If last version shows lower score than older version -> public score displaying your top score?\n\n"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1258208": "Someone could explain to me how is evaluation happening? Please.\nI red some topics here and decided to make a little experiment.\nI have assumed that found more datasets labels than was mentioned in train data labels.(score by them 0.701)\n\nSo, at first step I have added to 2100032a-7c33-4bff-97ef-690822c43466 PredictionString two more dataset labels(became 4 prediction for this at the moment) and score does not changed.\nAnd second step was to adding 2 more labels(became 6) and still no changes in score.\nThat’s what I don't understand.\n\nI Assume:\n1 - If this two new labels true - score must increase.\n2 - If this two new labels false - score must decrease.\n\nfor example:\nLet's imagine that we have 3 dataset labels in the text with total 9 words. \n'alzheimer s disease neuroimaging initiative adni|adni' give to as 7 words, let's imagine again this correct words. our score 7/(9+7-7) = 0.7777 etc.\n'alzheimer s disease neuroimaging initiative adni|adni|new label' 9 words, 7 correct ,2 not\nscore 7/(9+9-7)=0.6363\nIf all words correct score should be 1.0.\n\nI'm understand that's F0.5 Score not the same, but it should change too anyway.\n\nBack to real.\nI have added 4 more labels with 10+ words in total and Score does not changes.\nWhat should it mean?\n1. I am totally don't understand how it is calculating.\nThen someone explain me please it and how it is happening when multiple labels separated by '|' calculating?\nCalculating for each labels by posting multiple labels for one text happen independently to each other, further calculating mean or?\n2. Here extremely huge amount of dataset labels in test text, that's 10 word can't change evaluation by 0.001.\n3. If last version shows lower score than older version -> public score displaying your top score?\n\n"
  }
}