{
  "id": 243905,
  "title": "About False Positives",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/243905",
  "author_name": "",
  "post_date": "2021-06-04T12:58:50.229029300Z",
  "votes": 7,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I made a plot using various TP and FP counts ,which shows that making few FPs will increase your private LB score by a large margin. In short, you need to aim for a high precision, which is expected.</p>\n<p>1- How much false positives do string search methods make?<br>\nIf they are fairly precise, wouldn't this make a string search method win this competition?</p>\n<p>Let's say they don't make any FPs, then finding 50% of the positive labels will give you a 0.83 score.<br>\nIn my plot(100 labels), a submission made by a model can beat this score by predicting 75 TPs and less than 12-13 FPs. (Precision would be 0.85)<br>\nWouldn't this make web scraping and data cleaning hard to beat for this competition?<br>\nOr am I missing something?</p>\n<p>2- Are all datasets labeled in public LB? If this is not the case, submissions that use only string search are getting artificially low scores in public LB, because their correct predictions are counted as FPs. (Same goes for other models, of course. But they are probably predicting more FPs in correctly labeled parts.)</p>\n<p>What are your thoughts about this?</p>\n<p><img src=\"https://i.ibb.co/34PqGxP/fp-plot.png\" alt=\"\"></p>",
  "messages": [
    {
      "id": "1335802",
      "postDate": "06/04/2021 12:58:50",
      "content": "<p>I made a plot using various TP and FP counts ,which shows that making few FPs will increase your private LB score by a large margin. In short, you need to aim for a high precision, which is expected.</p>\n<p>1- How much false positives do string search methods make?<br>\nIf they are fairly precise, wouldn't this make a string search method win this competition?</p>\n<p>Let's say they don't make any FPs, then finding 50% of the positive labels will give you a 0.83 score.<br>\nIn my plot(100 labels), a submission made by a model can beat this score by predicting 75 TPs and less than 12-13 FPs. (Precision would be 0.85)<br>\nWouldn't this make web scraping and data cleaning hard to beat for this competition?<br>\nOr am I missing something?</p>\n<p>2- Are all datasets labeled in public LB? If this is not the case, submissions that use only string search are getting artificially low scores in public LB, because their correct predictions are counted as FPs. (Same goes for other models, of course. But they are probably predicting more FPs in correctly labeled parts.)</p>\n<p>What are your thoughts about this?</p>\n<p><img src=\"https://i.ibb.co/34PqGxP/fp-plot.png\" alt=\"\"></p>",
      "rawMarkdown": "I made a plot using various TP and FP counts ,which shows that making few FPs will increase your private LB score by a large margin. In short, you need to aim for a high precision, which is expected.\n\n1- How much false positives do string search methods make?\nIf they are fairly precise, wouldn't this make a string search method win this competition?\n\nLet's say they don't make any FPs, then finding 50% of the positive labels will give you a 0.83 score.\nIn my plot(100 labels), a submission made by a model can beat this score by predicting 75 TPs and less than 12-13 FPs. (Precision would be 0.85)\nWouldn't this make web scraping and data cleaning hard to beat for this competition?\nOr am I missing something?\n\n2- Are all datasets labeled in public LB? If this is not the case, submissions that use only string search are getting artificially low scores in public LB, because their correct predictions are counted as FPs. (Same goes for other models, of course. But they are probably predicting more FPs in correctly labeled parts.)\n\nWhat are your thoughts about this?\n\n![](https://i.ibb.co/34PqGxP/fp-plot.png)",
      "votes": null
    },
    {
      "id": "1336131",
      "postDate": "06/04/2021 16:38:36",
      "content": "<p>I think you need to have true positive, true negative, false positive, and false negative add up to your total.  It looks like you keep track of TP and FP, but you also need to factor in FN</p>",
      "rawMarkdown": "I think you need to have true positive, true negative, false positive, and false negative add up to your total.  It looks like you keep track of TP and FP, but you also need to factor in FN",
      "votes": null
    },
    {
      "id": "1336238",
      "postDate": "06/04/2021 18:25:19",
      "content": "<p>We are definitely not shown all the labels in the public dataset.  There are many datasets unlabelled. This doesn't mean that they are not in the ground truths for the hidden public LB test. I have experimented with matching and added datsets that are not labelled and my score went up.</p>",
      "rawMarkdown": "We are definitely not shown all the labels in the public dataset.  There are many datasets unlabelled. This doesn't mean that they are not in the ground truths for the hidden public LB test. I have experimented with matching and added datsets that are not labelled and my score went up.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1336131,
      "author_name": "nbroad",
      "author_url": "",
      "post_date": "06/04/2021 16:38:36",
      "content": "<p>I think you need to have true positive, true negative, false positive, and false negative add up to your total.  It looks like you keep track of TP and FP, but you also need to factor in FN</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1336238,
      "author_name": "mlconsult",
      "author_url": "",
      "post_date": "06/04/2021 18:25:19",
      "content": "<p>We are definitely not shown all the labels in the public dataset.  There are many datasets unlabelled. This doesn't mean that they are not in the ground truths for the hidden public LB test. I have experimented with matching and added datsets that are not labelled and my score went up.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1335802": "I made a plot using various TP and FP counts ,which shows that making few FPs will increase your private LB score by a large margin. In short, you need to aim for a high precision, which is expected.\n\n1- How much false positives do string search methods make?\nIf they are fairly precise, wouldn't this make a string search method win this competition?\n\nLet's say they don't make any FPs, then finding 50% of the positive labels will give you a 0.83 score.\nIn my plot(100 labels), a submission made by a model can beat this score by predicting 75 TPs and less than 12-13 FPs. (Precision would be 0.85)\nWouldn't this make web scraping and data cleaning hard to beat for this competition?\nOr am I missing something?\n\n2- Are all datasets labeled in public LB? If this is not the case, submissions that use only string search are getting artificially low scores in public LB, because their correct predictions are counted as FPs. (Same goes for other models, of course. But they are probably predicting more FPs in correctly labeled parts.)\n\nWhat are your thoughts about this?\n\n![](https://i.ibb.co/34PqGxP/fp-plot.png)",
    "1336131": "I think you need to have true positive, true negative, false positive, and false negative add up to your total.  It looks like you keep track of TP and FP, but you also need to factor in FN",
    "1336238": "We are definitely not shown all the labels in the public dataset.  There are many datasets unlabelled. This doesn't mean that they are not in the ground truths for the hidden public LB test. I have experimented with matching and added datsets that are not labelled and my score went up."
  },
  "source": "meta"
}