{
  "id": 228501,
  "title": "Metric discussion",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/228501",
  "author_name": "",
  "post_date": "2021-03-24T23:51:58.182290400Z",
  "votes": 4,
  "comment_count": 5,
  "views": 0,
  "content": "<p><code>jaccard_similarity</code>, detailed in <a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\" target=\"_blank\">evaluation</a>, clearly compute the score based on words and not in labels.</p>\n<p>If my algorithm predict <code>\"national education longitudinal study | education longitudinal study\"</code>, (<em>i.e.</em>, two classes), and the metric computes only words, is it the same of predict only one class <code>\"national education longitudinal study\"</code>?</p>\n<p>How is it really works? 🤔</p>",
  "messages": [
    {
      "id": "1251598",
      "postDate": "03/24/2021 23:51:58",
      "content": "<p><code>jaccard_similarity</code>, detailed in <a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\" target=\"_blank\">evaluation</a>, clearly compute the score based on words and not in labels.</p>\n<p>If my algorithm predict <code>\"national education longitudinal study | education longitudinal study\"</code>, (<em>i.e.</em>, two classes), and the metric computes only words, is it the same of predict only one class <code>\"national education longitudinal study\"</code>?</p>\n<p>How is it really works? 🤔</p>",
      "rawMarkdown": "`jaccard_similarity`, detailed in [evaluation](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation), clearly compute the score based on words and not in labels.\n\nIf my algorithm predict `\"national education longitudinal study | education longitudinal study\"`, (_i.e._, two classes), and the metric computes only words, is it the same of predict only one class `\"national education longitudinal study\"`?\n\nHow is it really works? 🤔",
      "votes": null
    },
    {
      "id": "1251758",
      "postDate": "03/25/2021 05:29:29",
      "content": "<p>Classification metrics may work. However, I think of BLEU score because the problem statement says \"extract the dataset\". I may scope this challenge as NLG. Correct if I am wrong somewhere.</p>",
      "rawMarkdown": "Classification metrics may work. However, I think of BLEU score because the problem statement says \"extract the dataset\". I may scope this challenge as NLG. Correct if I am wrong somewhere.",
      "votes": null
    },
    {
      "id": "1252138",
      "postDate": "03/25/2021 12:58:46",
      "content": "<p>I'm a little bit confused by the metric, as well the test.. heheh 😬</p>",
      "rawMarkdown": "I'm a little bit confused by the metric, as well the test.. heheh 😬",
      "votes": null
    },
    {
      "id": "1252313",
      "postDate": "03/25/2021 15:21:14",
      "content": "<p>Hi! Good question. If your two predictions are both separate mentions of that dataset in the document in question, then you should make both predictions. The ground truth includes multiple different mentions of the same dataset in a document. As mentioned in the Evaluation page, multiple predictions are checked against each of the multiple ground truths, with the highest Jaccard score determining a \"match\", and a threshold determining whether that match is a true or false positive.</p>\n<p>So basically - those two mentions are potentially separate classes that just happen to point to the same dataset. If you think they both exist in the document, make that prediction.</p>",
      "rawMarkdown": "Hi! Good question. If your two predictions are both separate mentions of that dataset in the document in question, then you should make both predictions. The ground truth includes multiple different mentions of the same dataset in a document. As mentioned in the Evaluation page, multiple predictions are checked against each of the multiple ground truths, with the highest Jaccard score determining a \"match\", and a threshold determining whether that match is a true or false positive.\n\nSo basically - those two mentions are potentially separate classes that just happen to point to the same dataset. If you think they both exist in the document, make that prediction.",
      "votes": null
    },
    {
      "id": "1252323",
      "postDate": "03/25/2021 15:32:04",
      "content": "<p>Am I right to assume that a document may mention a dataset multiple times? And the ground truth reflect exactly the number of times that the dataset is mentioned in that dataset?</p>",
      "rawMarkdown": "Am I right to assume that a document may mention a dataset multiple times? And the ground truth reflect exactly the number of times that the dataset is mentioned in that dataset?",
      "votes": null
    },
    {
      "id": "1283127",
      "postDate": "04/24/2021 15:49:50",
      "content": "<p><a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a></p>\n<p>So if we have the following 3 references in a paper:</p>\n<p>noaa storm surge inundation<br>\nnational weather service nws storm surge risk<br>\nsea lake and overland surges from hurricanes</p>\n<p>we should predict the name of the dataset \"sea lake and overland surges from hurricanes\" 3 times?</p>\n<p>or should we include the cleaned label as the prediction when we find a reference like this?</p>\n<p>noaa storm surge inundation<br>\nnational weather service nws storm surge risk<br>\nsea lake and overland surges from hurricanes</p>\n<p>I think this may be what you were referring to when you said \"class\" below?</p>",
      "rawMarkdown": "philculliton\n\nSo if we have the following 3 references in a paper:\n\nnoaa storm surge inundation\nnational weather service nws storm surge risk\nsea lake and overland surges from hurricanes\n\nwe should predict the name of the dataset \"sea lake and overland surges from hurricanes\" 3 times?\n\nor should we include the cleaned label as the prediction when we find a reference like this?\n\nnoaa storm surge inundation\nnational weather service nws storm surge risk\nsea lake and overland surges from hurricanes\n\nI think this may be what you were referring to when you said \"class\" below?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1251758,
      "author_name": "datqngo",
      "author_url": "",
      "post_date": "03/25/2021 05:29:29",
      "content": "<p>Classification metrics may work. However, I think of BLEU score because the problem statement says \"extract the dataset\". I may scope this challenge as NLG. Correct if I am wrong somewhere.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1252138,
      "author_name": "leomauro",
      "author_url": "",
      "post_date": "03/25/2021 12:58:46",
      "content": "<p>I'm a little bit confused by the metric, as well the test.. heheh 😬</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1252313,
      "author_name": "philculliton",
      "author_url": "",
      "post_date": "03/25/2021 15:21:14",
      "content": "<p>Hi! Good question. If your two predictions are both separate mentions of that dataset in the document in question, then you should make both predictions. The ground truth includes multiple different mentions of the same dataset in a document. As mentioned in the Evaluation page, multiple predictions are checked against each of the multiple ground truths, with the highest Jaccard score determining a \"match\", and a threshold determining whether that match is a true or false positive.</p>\n<p>So basically - those two mentions are potentially separate classes that just happen to point to the same dataset. If you think they both exist in the document, make that prediction.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1252323,
          "author_name": "datqngo",
          "author_url": "",
          "post_date": "03/25/2021 15:32:04",
          "content": "<p>Am I right to assume that a document may mention a dataset multiple times? And the ground truth reflect exactly the number of times that the dataset is mentioned in that dataset?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1283127,
      "author_name": "mlconsult",
      "author_url": "",
      "post_date": "04/24/2021 15:49:50",
      "content": "<p><a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a></p>\n<p>So if we have the following 3 references in a paper:</p>\n<p>noaa storm surge inundation<br>\nnational weather service nws storm surge risk<br>\nsea lake and overland surges from hurricanes</p>\n<p>we should predict the name of the dataset \"sea lake and overland surges from hurricanes\" 3 times?</p>\n<p>or should we include the cleaned label as the prediction when we find a reference like this?</p>\n<p>noaa storm surge inundation<br>\nnational weather service nws storm surge risk<br>\nsea lake and overland surges from hurricanes</p>\n<p>I think this may be what you were referring to when you said \"class\" below?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1251598": "`jaccard_similarity`, detailed in [evaluation](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation), clearly compute the score based on words and not in labels.\n\nIf my algorithm predict `\"national education longitudinal study | education longitudinal study\"`, (_i.e._, two classes), and the metric computes only words, is it the same of predict only one class `\"national education longitudinal study\"`?\n\nHow is it really works? 🤔",
    "1251758": "Classification metrics may work. However, I think of BLEU score because the problem statement says \"extract the dataset\". I may scope this challenge as NLG. Correct if I am wrong somewhere.",
    "1252138": "I'm a little bit confused by the metric, as well the test.. heheh 😬",
    "1252313": "Hi! Good question. If your two predictions are both separate mentions of that dataset in the document in question, then you should make both predictions. The ground truth includes multiple different mentions of the same dataset in a document. As mentioned in the Evaluation page, multiple predictions are checked against each of the multiple ground truths, with the highest Jaccard score determining a \"match\", and a threshold determining whether that match is a true or false positive.\n\nSo basically - those two mentions are potentially separate classes that just happen to point to the same dataset. If you think they both exist in the document, make that prediction.",
    "1252323": "Am I right to assume that a document may mention a dataset multiple times? And the ground truth reflect exactly the number of times that the dataset is mentioned in that dataset?",
    "1283127": "philculliton\n\nSo if we have the following 3 references in a paper:\n\nnoaa storm surge inundation\nnational weather service nws storm surge risk\nsea lake and overland surges from hurricanes\n\nwe should predict the name of the dataset \"sea lake and overland surges from hurricanes\" 3 times?\n\nor should we include the cleaned label as the prediction when we find a reference like this?\n\nnoaa storm surge inundation\nnational weather service nws storm surge risk\nsea lake and overland surges from hurricanes\n\nI think this may be what you were referring to when you said \"class\" below?"
  },
  "source": "meta"
}