{
  "id": 236562,
  "title": "Downward pressure on the LB after re-annotation",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/236562",
  "author_name": "",
  "post_date": "2021-05-05T00:19:43.111680600Z",
  "votes": 16,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Don't be surprised if there's a general drop on the public LB - most teams are likely to get hit with an increased penalty for False Negatives assuming there are more labels now. </p>\n<p>Edited :- The baseline string matching approach has dropped from ~0.7 to ~0.533 </p>",
  "messages": [
    {
      "id": "1293501",
      "postDate": "05/05/2021 00:19:43",
      "content": "<p>Don't be surprised if there's a general drop on the public LB - most teams are likely to get hit with an increased penalty for False Negatives assuming there are more labels now. </p>\n<p>Edited :- The baseline string matching approach has dropped from ~0.7 to ~0.533 </p>",
      "rawMarkdown": "Don't be surprised if there's a general drop on the public LB - most teams are likely to get hit with an increased penalty for False Negatives assuming there are more labels now. \n\nEdited :- The baseline string matching approach has dropped from ~0.7 to ~0.533",
      "votes": null
    },
    {
      "id": "1293548",
      "postDate": "05/05/2021 02:20:37",
      "content": "<p>I'm watching the LB change. I should go to bed, but I can't look away from the train wreck. I'll be plummeting soon.</p>",
      "rawMarkdown": "I'm watching the LB change. I should go to bed, but I can't look away from the train wreck. I'll be plummeting soon.",
      "votes": null
    },
    {
      "id": "1293554",
      "postDate": "05/05/2021 02:33:49",
      "content": "<p>Same, I'm taking a pretty brutal hit. </p>\n<p>It's a bit funny how the re-scoring algo doesn't clear out a team's submissions so you have \"loose ends\" hanging waiting to be rescored.</p>",
      "rawMarkdown": "Same, I'm taking a pretty brutal hit. \n\nIt's a bit funny how the re-scoring algo doesn't clear out a team's submissions so you have \"loose ends\" hanging waiting to be rescored.",
      "votes": null
    },
    {
      "id": "1293559",
      "postDate": "05/05/2021 02:41:30",
      "content": "<p>Looking at my submissions that I can tell have been rescored, non-string matching is doing better than it was before, but string matching is still the best. (This may not be true for the private LB of course.)</p>",
      "rawMarkdown": "Looking at my submissions that I can tell have been rescored, non-string matching is doing better than it was before, but string matching is still the best. (This may not be true for the private LB of course.)",
      "votes": null
    },
    {
      "id": "1293564",
      "postDate": "05/05/2021 02:47:00",
      "content": "<p>I'm even more skeptical of the LB (public and private) if anything. <br>\nModifying the annotations mid-way thru a competition doesn't inspire much confidence in the thoroughness/consistency of it. </p>",
      "rawMarkdown": "I'm even more skeptical of the LB (public and private) if anything. \nModifying the annotations mid-way thru a competition doesn't inspire much confidence in the thoroughness/consistency of it.",
      "votes": null
    },
    {
      "id": "1293931",
      "postDate": "05/05/2021 09:24:16",
      "content": "<p>Yes, you basically have to re-assess all your submissions (unless you stop using the LB as a signal of course).</p>",
      "rawMarkdown": "Yes, you basically have to re-assess all your submissions (unless you stop using the LB as a signal of course).",
      "votes": null
    },
    {
      "id": "1294187",
      "postDate": "05/05/2021 13:27:21",
      "content": "<p>Is the test evaluation completed for all participants. The First 7 participants Score is unaffected. Quite interesting?</p>",
      "rawMarkdown": "Is the test evaluation completed for all participants. The First 7 participants Score is unaffected. Quite interesting?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1293548,
      "author_name": "trentb",
      "author_url": "",
      "post_date": "05/05/2021 02:20:37",
      "content": "<p>I'm watching the LB change. I should go to bed, but I can't look away from the train wreck. I'll be plummeting soon.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1293554,
          "author_name": "leecming",
          "author_url": "",
          "post_date": "05/05/2021 02:33:49",
          "content": "<p>Same, I'm taking a pretty brutal hit. </p>\n<p>It's a bit funny how the re-scoring algo doesn't clear out a team's submissions so you have \"loose ends\" hanging waiting to be rescored.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1293559,
          "author_name": "trentb",
          "author_url": "",
          "post_date": "05/05/2021 02:41:30",
          "content": "<p>Looking at my submissions that I can tell have been rescored, non-string matching is doing better than it was before, but string matching is still the best. (This may not be true for the private LB of course.)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1293564,
          "author_name": "leecming",
          "author_url": "",
          "post_date": "05/05/2021 02:47:00",
          "content": "<p>I'm even more skeptical of the LB (public and private) if anything. <br>\nModifying the annotations mid-way thru a competition doesn't inspire much confidence in the thoroughness/consistency of it. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1293931,
          "author_name": "hannes82",
          "author_url": "",
          "post_date": "05/05/2021 09:24:16",
          "content": "<p>Yes, you basically have to re-assess all your submissions (unless you stop using the LB as a signal of course).</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1294187,
      "author_name": "pranayteaches",
      "author_url": "",
      "post_date": "05/05/2021 13:27:21",
      "content": "<p>Is the test evaluation completed for all participants. The First 7 participants Score is unaffected. Quite interesting?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1293501": "Don't be surprised if there's a general drop on the public LB - most teams are likely to get hit with an increased penalty for False Negatives assuming there are more labels now. \n\nEdited :- The baseline string matching approach has dropped from ~0.7 to ~0.533",
    "1293548": "I'm watching the LB change. I should go to bed, but I can't look away from the train wreck. I'll be plummeting soon.",
    "1293554": "Same, I'm taking a pretty brutal hit. \n\nIt's a bit funny how the re-scoring algo doesn't clear out a team's submissions so you have \"loose ends\" hanging waiting to be rescored.",
    "1293559": "Looking at my submissions that I can tell have been rescored, non-string matching is doing better than it was before, but string matching is still the best. (This may not be true for the private LB of course.)",
    "1293564": "I'm even more skeptical of the LB (public and private) if anything. \nModifying the annotations mid-way thru a competition doesn't inspire much confidence in the thoroughness/consistency of it.",
    "1293931": "Yes, you basically have to re-assess all your submissions (unless you stop using the LB as a signal of course).",
    "1294187": "Is the test evaluation completed for all participants. The First 7 participants Score is unaffected. Quite interesting?"
  },
  "source": "meta"
}