{
  "id": 236508,
  "title": "Revision to annotations",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/236508",
  "author_name": "",
  "post_date": "2021-05-04T17:30:23.746200900Z",
  "votes": 37,
  "comment_count": 28,
  "views": 0,
  "content": "<p>Dear colleagues<br>\nThanks for all your hard work on the competition – we are thrilled by the engagement and interest.  We thought our annotation efforts were sufficient but we realized that we needed to capture more. Kaggle has received the updated public test set and your submissions will be re-run against this new set. Re-running all of your submissions may take several days.  The private test set will also be updated.   Thanks again for your engagement in this important effort!</p>",
  "messages": [
    {
      "id": "1293250",
      "postDate": "05/04/2021 17:30:23",
      "content": "<p>Dear colleagues<br>\nThanks for all your hard work on the competition – we are thrilled by the engagement and interest.  We thought our annotation efforts were sufficient but we realized that we needed to capture more. Kaggle has received the updated public test set and your submissions will be re-run against this new set. Re-running all of your submissions may take several days.  The private test set will also be updated.   Thanks again for your engagement in this important effort!</p>",
      "rawMarkdown": "Dear colleagues\nThanks for all your hard work on the competition – we are thrilled by the engagement and interest.  We thought our annotation efforts were sufficient but we realized that we needed to capture more. Kaggle has received the updated public test set and your submissions will be re-run against this new set. Re-running all of your submissions may take several days.  The private test set will also be updated.   Thanks again for your engagement in this important effort!",
      "votes": null
    },
    {
      "id": "1293263",
      "postDate": "05/04/2021 17:43:19",
      "content": "<p>Thanks very much Julia! You and your team's hard work is much appreciated.</p>\n<p>Note that only scoring test labels have been updated on our back end - the competition's publications and other data remain the same. As Julia mentioned, I'll be rerunning all existing submissions against the updated test labels - any new submissions starting now will be using the update.</p>",
      "rawMarkdown": "Thanks very much Julia! You and your team's hard work is much appreciated.\n\nNote that only scoring test labels have been updated on our back end - the competition's publications and other data remain the same. As Julia mentioned, I'll be rerunning all existing submissions against the updated test labels - any new submissions starting now will be using the update.",
      "votes": null
    },
    {
      "id": "1293387",
      "postDate": "05/04/2021 19:43:19",
      "content": "<p>Thanks for update. Would it be possible to describe more in detail the kind of revision that has been carried out? </p>",
      "rawMarkdown": "Thanks for update. Would it be possible to describe more in detail the kind of revision that has been carried out?",
      "votes": null
    },
    {
      "id": "1293542",
      "postDate": "05/05/2021 02:09:19",
      "content": "<p><a href=\"https://www.kaggle.com/julialane\" target=\"_blank\">@julialane</a> is an updated public testset completely disjoint with training labels (including missing training label) ? </p>",
      "rawMarkdown": "julialane is an updated public testset completely disjoint with training labels (including missing training label) ?",
      "votes": null
    },
    {
      "id": "1293785",
      "postDate": "05/05/2021 06:57:48",
      "content": "<p>Thank You for this effort and update.</p>\n<p>Are there any changes to the various discussions / responses regarding the overlap / non-overlap of publications and labels?</p>\n<p>Is it safe to assume all of that still remains the same and the only update being done is including more labels to the public and private test sets?</p>",
      "rawMarkdown": "Thank You for this effort and update.\n\nAre there any changes to the various discussions / responses regarding the overlap / non-overlap of publications and labels?\n\nIs it safe to assume all of that still remains the same and the only update being done is including more labels to the public and private test sets?",
      "votes": null
    },
    {
      "id": "1293891",
      "postDate": "05/05/2021 08:35:12",
      "content": "<p>Looking forward to seeing the details of this change！👀</p>",
      "rawMarkdown": "Looking forward to seeing the details of this change！👀",
      "votes": null
    },
    {
      "id": "1293897",
      "postDate": "05/05/2021 08:41:44",
      "content": "<p>Will you let us know when you have rerun all the submissions please? I'm trying not to get unduly excited by my jump of 500 positions up the public LB… 😄</p>",
      "rawMarkdown": "Will you let us know when you have rerun all the submissions please? I'm trying not to get unduly excited by my jump of 500 positions up the public LB... 😄",
      "votes": null
    },
    {
      "id": "1293906",
      "postDate": "05/05/2021 08:51:43",
      "content": "<p>Same here 😊</p>\n<p>Btw, never seen two deleted users on top of the LB before..</p>",
      "rawMarkdown": "Same here 😊\n\nBtw, never seen two deleted users on top of the LB before..",
      "votes": null
    },
    {
      "id": "1293943",
      "postDate": "05/05/2021 09:40:06",
      "content": "<p>I haven't spent much time on Kaggle recently - what's up with people deleting their accounts so early in a competition.</p>\n<p>I know that cheaters tend to create dummy accounts but that seemed to be a thing at the tail-end, not at the start of a competition.</p>\n<p>Odd…</p>",
      "rawMarkdown": "I haven't spent much time on Kaggle recently - what's up with people deleting their accounts so early in a competition.\n\nI know that cheaters tend to create dummy accounts but that seemed to be a thing at the tail-end, not at the start of a competition.\n\nOdd...",
      "votes": null
    },
    {
      "id": "1293952",
      "postDate": "05/05/2021 09:50:15",
      "content": "<p><a href=\"https://www.kaggle.com/julialane\" target=\"_blank\">@julialane</a>  can you clarify please how annotations are done when the paper refers to a dataset incorrectly? For example if the paper misspells Alzheimers as Alzhiemers will it be annotated as the correct spelling?</p>\n<p>I have found many mispellings/misreferences/permutations in the training data papers - below a small subset - will these all be labelled as :  \"Alzheimer's Disease NeuroImaging Initiative ADNI\"?</p>\n<p>Alzheimer s Disease Neuro Initiative ADNI<br>\nAlzheimer s Disease NeuroImaging Initiative ADNI<br>\nAlzheimer s Disease Neuroimage Initiative ADNI<br>\nAlzheimer s Disease Neuroimaging Initative ADNI<br>\nAlzheimer s Disease Neuroimaging Initiative ANDI<br>\nAlzheimer s Disease Neuroimaging Initiative s ADNI<br>\nAlzheimer s Disease Neuroimaging Initiatives ADNI<br>\nAlzheimer s Disease Neuroimaging Initive ADNI<br>\nAlzheimer s Disease Neuroimaging Study ADNI<br>\nAlzheimer s Disease Neuroimaing Initiative ADNI<br>\nAlzheimer s Diseases Neuroimaging Initiative ADNI<br>\nAlzheimer s Neuroimaging Initiative ADNI<br>\nAlzheimer3s Disease Neuroimaging Initiative ADNI<br>\nAlzheimer3s Disease Neuromaging Initiative ADNI<br>\nAlzheimers Disease Neuroimaging Initiative ADNI<br>\nAlzheimers Disease NeuroimagingInitiative ADNI<br>\nAlzhimer s Disease Neuroimaging Initiative ADNI</p>",
      "rawMarkdown": "julialane  can you clarify please how annotations are done when the paper refers to a dataset incorrectly? For example if the paper misspells Alzheimers as Alzhiemers will it be annotated as the correct spelling?\n\nI have found many mispellings/misreferences/permutations in the training data papers - below a small subset - will these all be labelled as :  \"Alzheimer's Disease NeuroImaging Initiative ADNI\"?\n\nAlzheimer s Disease Neuro Initiative ADNI\nAlzheimer s Disease NeuroImaging Initiative ADNI\nAlzheimer s Disease Neuroimage Initiative ADNI\nAlzheimer s Disease Neuroimaging Initative ADNI\nAlzheimer s Disease Neuroimaging Initiative ANDI\nAlzheimer s Disease Neuroimaging Initiative s ADNI\nAlzheimer s Disease Neuroimaging Initiatives ADNI\nAlzheimer s Disease Neuroimaging Initive ADNI\nAlzheimer s Disease Neuroimaging Study ADNI\nAlzheimer s Disease Neuroimaing Initiative ADNI\nAlzheimer s Diseases Neuroimaging Initiative ADNI\nAlzheimer s Neuroimaging Initiative ADNI\nAlzheimer3s Disease Neuroimaging Initiative ADNI\nAlzheimer3s Disease Neuromaging Initiative ADNI\nAlzheimers Disease Neuroimaging Initiative ADNI\nAlzheimers Disease NeuroimagingInitiative ADNI\nAlzhimer s Disease Neuroimaging Initiative ADNI",
      "votes": null
    },
    {
      "id": "1294098",
      "postDate": "05/05/2021 12:15:27",
      "content": "<p>The problem of only updating the testing dataset but leaving the training dataset untouched increases the difficulty of understanding which is treated as corrects and which is not. I had a quick look and have the feeling that after the update, the usefulness of the training data becomes less…</p>",
      "rawMarkdown": "The problem of only updating the testing dataset but leaving the training dataset untouched increases the difficulty of understanding which is treated as corrects and which is not. I had a quick look and have the feeling that after the update, the usefulness of the training data becomes less...",
      "votes": null
    },
    {
      "id": "1294609",
      "postDate": "05/05/2021 19:17:29",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/cascadenite\" target=\"_blank\">@cascadenite</a>. Great question. The correct labels for a given publication are drawn directly from that publication's text - misspellings, different wordings, and all. So if your model sees something that looks like a dataset mention in a publication, no matter how it's spelled, you should submit the predicted mention verbatim.</p>",
      "rawMarkdown": "Hi @cascadenite. Great question. The correct labels for a given publication are drawn directly from that publication's text - misspellings, different wordings, and all. So if your model sees something that looks like a dataset mention in a publication, no matter how it's spelled, you should submit the predicted mention verbatim.",
      "votes": null
    },
    {
      "id": "1294636",
      "postDate": "05/05/2021 19:37:48",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a> -  does the same thinking apply to the length of the mention?  What I mean is:</p>\n<ol>\n<li>if the following are both mentioned (at different parts of the text): <br>\naging integrated database<br>\naging integrated database agid </li>\n</ol>\n<p>Should I have both in my predictions or just the longest one?</p>\n<ol>\n<li>If there is only one mention of the agid dataset and it is written like this in the paper:<br>\naging integrated database agid </li>\n</ol>\n<p>Should I have the full string in my prediction?</p>",
      "rawMarkdown": "Thanks @philculliton -  does the same thinking apply to the length of the mention?  What I mean is:\n\n1.  if the following are both mentioned (at different parts of the text): \naging integrated database\naging integrated database agid \n\nShould I have both in my predictions or just the longest one?\n\n\n2. If there is only one mention of the agid dataset and it is written like this in the paper:\naging integrated database agid \n\nShould I have the full string in my prediction?",
      "votes": null
    },
    {
      "id": "1294650",
      "postDate": "05/05/2021 19:47:38",
      "content": "<p>There hasn't been much movement on the LB today as far as I can tell, but I'm thinking the rescoring must still be ongoing because there is a group of participants with 0.701 scores and generally few submissions near the top that are unlikely to have been rescored.</p>",
      "rawMarkdown": "There hasn't been much movement on the LB today as far as I can tell, but I'm thinking the rescoring must still be ongoing because there is a group of participants with 0.701 scores and generally few submissions near the top that are unlikely to have been rescored.",
      "votes": null
    },
    {
      "id": "1294796",
      "postDate": "05/05/2021 23:46:46",
      "content": "<p><a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a> </p>\n<p>Is there a write-up of the Kaggle backend for stuff like managing and scoring submissions anywhere (e.g., mechanics, tech stack, resource considerations etc.)? Or would you be able to describe it in brief? </p>\n<p>Separate from this specific competition, I think many of us would be interested in understanding what goes on behind the curtain. </p>",
      "rawMarkdown": "philculliton \n\nIs there a write-up of the Kaggle backend for stuff like managing and scoring submissions anywhere (e.g., mechanics, tech stack, resource considerations etc.)? Or would you be able to describe it in brief? \n\nSeparate from this specific competition, I think many of us would be interested in understanding what goes on behind the curtain.",
      "votes": null
    },
    {
      "id": "1295925",
      "postDate": "05/06/2021 20:06:35",
      "content": "<p>I confirm this. I'm one happy owner of a 0.701 submission. As it turns out this is a notebook that I deleted and therefore can't be re-run. See you at the bottom in a few days when they catch the glitch!</p>\n<p>Is your current submission also from a deleted notebook too?, because it is we'll meet at the 600th. position soon.</p>",
      "rawMarkdown": "I confirm this. I'm one happy owner of a 0.701 submission. As it turns out this is a notebook that I deleted and therefore can't be re-run. See you at the bottom in a few days when they catch the glitch!\n\nIs your current submission also from a deleted notebook too?, because it is we'll meet at the 600th. position soon.",
      "votes": null
    },
    {
      "id": "1295960",
      "postDate": "05/06/2021 20:39:51",
      "content": "<p>Hahaha! So that's what's going on with the 0.701 submissions near the top. I should delete all of my notebooks now in case there is another rescore. 🙂</p>\n<p>My current score is from a non-deleted notebook, which is why I'm keeping an eye on those submissions above me.</p>",
      "rawMarkdown": "Hahaha! So that's what's going on with the 0.701 submissions near the top. I should delete all of my notebooks now in case there is another rescore. 🙂\n\nMy current score is from a non-deleted notebook, which is why I'm keeping an eye on those submissions above me.",
      "votes": null
    },
    {
      "id": "1295980",
      "postDate": "05/06/2021 21:08:09",
      "content": "<p><a href=\"https://www.kaggle.com/cascadenite\" target=\"_blank\">@cascadenite</a> Unfortunately I guess you won't receive a reply to your questions because it seems there is very little consistency in the way the data are labelled (see <a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236972\" target=\"_blank\">here</a>). So the answer would be \"nobody knows\" (<a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236753#1294510\" target=\"_blank\">cit</a>)</p>",
      "rawMarkdown": "cascadenite Unfortunately I guess you won't receive a reply to your questions because it seems there is very little consistency in the way the data are labelled (see [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236972)). So the answer would be \"nobody knows\" ([cit](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236753#1294510))",
      "votes": null
    },
    {
      "id": "1296022",
      "postDate": "05/06/2021 22:12:00",
      "content": "<p>It must be odd for a newcomer to the competition to see the current top two…</p>",
      "rawMarkdown": "It must be odd for a newcomer to the competition to see the current top two...",
      "votes": null
    },
    {
      "id": "1296119",
      "postDate": "05/07/2021 02:13:59",
      "content": "<p><a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a> It looks like deleted submissions could not be updated thus the leaderboard is frozen at the moment, are we going to get an update soon?</p>",
      "rawMarkdown": "philculliton It looks like deleted submissions could not be updated thus the leaderboard is frozen at the moment, are we going to get an update soon?",
      "votes": null
    },
    {
      "id": "1296191",
      "postDate": "05/07/2021 04:10:22",
      "content": "<p>The more I train my models the worse score I get.</p>\n<p>I can visually see it getting better and better and yet, I score lower and lower.</p>",
      "rawMarkdown": "The more I train my models the worse score I get.\n\nI can visually see it getting better and better and yet, I score lower and lower.",
      "votes": null
    },
    {
      "id": "1296820",
      "postDate": "05/07/2021 14:16:05",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/suicaokhoailang\" target=\"_blank\">@suicaokhoailang</a> - the removal of submissions that couldn't be re-run is the final part of the process, and the re-run has taken (at this point) several days. Should be completed today.</p>",
      "rawMarkdown": "Hi @suicaokhoailang - the removal of submissions that couldn't be re-run is the final part of the process, and the re-run has taken (at this point) several days. Should be completed today.",
      "votes": null
    },
    {
      "id": "1296849",
      "postDate": "05/07/2021 14:29:01",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/leecming\" target=\"_blank\">@leecming</a> - that's a great question. As you can imagine, it's extensive and depends on what type of competition is being run. I'll ask around and see if we have a public write-up of it anywhere.</p>",
      "rawMarkdown": "Hi @leecming - that's a great question. As you can imagine, it's extensive and depends on what type of competition is being run. I'll ask around and see if we have a public write-up of it anywhere.",
      "votes": null
    },
    {
      "id": "1297441",
      "postDate": "05/08/2021 03:39:00",
      "content": "<p>This may be too late. My guess is that the winner at this point will be a simple string matching strategy, I think those of us trying models will not have time to come up with anything good. I have been so confused with everything I have tried over the past two months…</p>",
      "rawMarkdown": "This may be too late. My guess is that the winner at this point will be a simple string matching strategy, I think those of us trying models will not have time to come up with anything good. I have been so confused with everything I have tried over the past two months...",
      "votes": null
    },
    {
      "id": "1297647",
      "postDate": "05/08/2021 07:50:25",
      "content": "<p>What makes you think everyone else is doing string matching?</p>",
      "rawMarkdown": "What makes you think everyone else is doing string matching?",
      "votes": null
    },
    {
      "id": "1297932",
      "postDate": "05/08/2021 12:37:16",
      "content": "<p>I have to admit that the winning solution being some kind of string matching lookup table would be pretty funny :)</p>",
      "rawMarkdown": "I have to admit that the winning solution being some kind of string matching lookup table would be pretty funny :)",
      "votes": null
    },
    {
      "id": "1297986",
      "postDate": "05/08/2021 13:28:30",
      "content": "<p>That's a possibility, doesn't mean that ML approach wouldn't work here though, because it does. At least I can have a model that can apply to non-crawlable problems in the future.</p>",
      "rawMarkdown": "That's a possibility, doesn't mean that ML approach wouldn't work here though, because it does. At least I can have a model that can apply to non-crawlable problems in the future.",
      "votes": null
    },
    {
      "id": "1298257",
      "postDate": "05/08/2021 17:19:02",
      "content": "<p>One Question: is right now the new public test dataset containing that 4 leakage data?</p>",
      "rawMarkdown": "One Question: is right now the new public test dataset containing that 4 leakage data?",
      "votes": null
    },
    {
      "id": "1321268",
      "postDate": "05/24/2021 15:21:01",
      "content": "<p>Your Welcome Sir.!</p>",
      "rawMarkdown": "Your Welcome Sir.!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1293263,
      "author_name": "philculliton",
      "author_url": "",
      "post_date": "05/04/2021 17:43:19",
      "content": "<p>Thanks very much Julia! You and your team's hard work is much appreciated.</p>\n<p>Note that only scoring test labels have been updated on our back end - the competition's publications and other data remain the same. As Julia mentioned, I'll be rerunning all existing submissions against the updated test labels - any new submissions starting now will be using the update.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1294796,
          "author_name": "leecming",
          "author_url": "",
          "post_date": "05/05/2021 23:46:46",
          "content": "<p><a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a> </p>\n<p>Is there a write-up of the Kaggle backend for stuff like managing and scoring submissions anywhere (e.g., mechanics, tech stack, resource considerations etc.)? Or would you be able to describe it in brief? </p>\n<p>Separate from this specific competition, I think many of us would be interested in understanding what goes on behind the curtain. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1296119,
          "author_name": "suicaokhoailang",
          "author_url": "",
          "post_date": "05/07/2021 02:13:59",
          "content": "<p><a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a> It looks like deleted submissions could not be updated thus the leaderboard is frozen at the moment, are we going to get an update soon?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1296820,
          "author_name": "philculliton",
          "author_url": "",
          "post_date": "05/07/2021 14:16:05",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/suicaokhoailang\" target=\"_blank\">@suicaokhoailang</a> - the removal of submissions that couldn't be re-run is the final part of the process, and the re-run has taken (at this point) several days. Should be completed today.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1296849,
          "author_name": "philculliton",
          "author_url": "",
          "post_date": "05/07/2021 14:29:01",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/leecming\" target=\"_blank\">@leecming</a> - that's a great question. As you can imagine, it's extensive and depends on what type of competition is being run. I'll ask around and see if we have a public write-up of it anywhere.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1293387,
      "author_name": "eguidotti",
      "author_url": "",
      "post_date": "05/04/2021 19:43:19",
      "content": "<p>Thanks for update. Would it be possible to describe more in detail the kind of revision that has been carried out? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1293542,
      "author_name": "dathudeptrai",
      "author_url": "",
      "post_date": "05/05/2021 02:09:19",
      "content": "<p><a href=\"https://www.kaggle.com/julialane\" target=\"_blank\">@julialane</a> is an updated public testset completely disjoint with training labels (including missing training label) ? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1293785,
      "author_name": "rohanrao",
      "author_url": "",
      "post_date": "05/05/2021 06:57:48",
      "content": "<p>Thank You for this effort and update.</p>\n<p>Are there any changes to the various discussions / responses regarding the overlap / non-overlap of publications and labels?</p>\n<p>Is it safe to assume all of that still remains the same and the only update being done is including more labels to the public and private test sets?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1293891,
      "author_name": "vanle73",
      "author_url": "",
      "post_date": "05/05/2021 08:35:12",
      "content": "<p>Looking forward to seeing the details of this change！👀</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1293897,
      "author_name": "cascadenite",
      "author_url": "",
      "post_date": "05/05/2021 08:41:44",
      "content": "<p>Will you let us know when you have rerun all the submissions please? I'm trying not to get unduly excited by my jump of 500 positions up the public LB… 😄</p>",
      "votes": null,
      "replies": [
        {
          "id": 1293906,
          "author_name": "hannes82",
          "author_url": "",
          "post_date": "05/05/2021 08:51:43",
          "content": "<p>Same here 😊</p>\n<p>Btw, never seen two deleted users on top of the LB before..</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1293943,
          "author_name": "leecming",
          "author_url": "",
          "post_date": "05/05/2021 09:40:06",
          "content": "<p>I haven't spent much time on Kaggle recently - what's up with people deleting their accounts so early in a competition.</p>\n<p>I know that cheaters tend to create dummy accounts but that seemed to be a thing at the tail-end, not at the start of a competition.</p>\n<p>Odd…</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1294650,
          "author_name": "trentb",
          "author_url": "",
          "post_date": "05/05/2021 19:47:38",
          "content": "<p>There hasn't been much movement on the LB today as far as I can tell, but I'm thinking the rescoring must still be ongoing because there is a group of participants with 0.701 scores and generally few submissions near the top that are unlikely to have been rescored.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1295925,
          "author_name": "verracodeguacas",
          "author_url": "",
          "post_date": "05/06/2021 20:06:35",
          "content": "<p>I confirm this. I'm one happy owner of a 0.701 submission. As it turns out this is a notebook that I deleted and therefore can't be re-run. See you at the bottom in a few days when they catch the glitch!</p>\n<p>Is your current submission also from a deleted notebook too?, because it is we'll meet at the 600th. position soon.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1295960,
          "author_name": "trentb",
          "author_url": "",
          "post_date": "05/06/2021 20:39:51",
          "content": "<p>Hahaha! So that's what's going on with the 0.701 submissions near the top. I should delete all of my notebooks now in case there is another rescore. 🙂</p>\n<p>My current score is from a non-deleted notebook, which is why I'm keeping an eye on those submissions above me.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1296022,
          "author_name": "leecming",
          "author_url": "",
          "post_date": "05/06/2021 22:12:00",
          "content": "<p>It must be odd for a newcomer to the competition to see the current top two…</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1293952,
      "author_name": "cascadenite",
      "author_url": "",
      "post_date": "05/05/2021 09:50:15",
      "content": "<p><a href=\"https://www.kaggle.com/julialane\" target=\"_blank\">@julialane</a>  can you clarify please how annotations are done when the paper refers to a dataset incorrectly? For example if the paper misspells Alzheimers as Alzhiemers will it be annotated as the correct spelling?</p>\n<p>I have found many mispellings/misreferences/permutations in the training data papers - below a small subset - will these all be labelled as :  \"Alzheimer's Disease NeuroImaging Initiative ADNI\"?</p>\n<p>Alzheimer s Disease Neuro Initiative ADNI<br>\nAlzheimer s Disease NeuroImaging Initiative ADNI<br>\nAlzheimer s Disease Neuroimage Initiative ADNI<br>\nAlzheimer s Disease Neuroimaging Initative ADNI<br>\nAlzheimer s Disease Neuroimaging Initiative ANDI<br>\nAlzheimer s Disease Neuroimaging Initiative s ADNI<br>\nAlzheimer s Disease Neuroimaging Initiatives ADNI<br>\nAlzheimer s Disease Neuroimaging Initive ADNI<br>\nAlzheimer s Disease Neuroimaging Study ADNI<br>\nAlzheimer s Disease Neuroimaing Initiative ADNI<br>\nAlzheimer s Diseases Neuroimaging Initiative ADNI<br>\nAlzheimer s Neuroimaging Initiative ADNI<br>\nAlzheimer3s Disease Neuroimaging Initiative ADNI<br>\nAlzheimer3s Disease Neuromaging Initiative ADNI<br>\nAlzheimers Disease Neuroimaging Initiative ADNI<br>\nAlzheimers Disease NeuroimagingInitiative ADNI<br>\nAlzhimer s Disease Neuroimaging Initiative ADNI</p>",
      "votes": null,
      "replies": [
        {
          "id": 1294609,
          "author_name": "philculliton",
          "author_url": "",
          "post_date": "05/05/2021 19:17:29",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/cascadenite\" target=\"_blank\">@cascadenite</a>. Great question. The correct labels for a given publication are drawn directly from that publication's text - misspellings, different wordings, and all. So if your model sees something that looks like a dataset mention in a publication, no matter how it's spelled, you should submit the predicted mention verbatim.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1294636,
          "author_name": "cascadenite",
          "author_url": "",
          "post_date": "05/05/2021 19:37:48",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/philculliton\" target=\"_blank\">@philculliton</a> -  does the same thinking apply to the length of the mention?  What I mean is:</p>\n<ol>\n<li>if the following are both mentioned (at different parts of the text): <br>\naging integrated database<br>\naging integrated database agid </li>\n</ol>\n<p>Should I have both in my predictions or just the longest one?</p>\n<ol>\n<li>If there is only one mention of the agid dataset and it is written like this in the paper:<br>\naging integrated database agid </li>\n</ol>\n<p>Should I have the full string in my prediction?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1295980,
          "author_name": "eguidotti",
          "author_url": "",
          "post_date": "05/06/2021 21:08:09",
          "content": "<p><a href=\"https://www.kaggle.com/cascadenite\" target=\"_blank\">@cascadenite</a> Unfortunately I guess you won't receive a reply to your questions because it seems there is very little consistency in the way the data are labelled (see <a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236972\" target=\"_blank\">here</a>). So the answer would be \"nobody knows\" (<a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236753#1294510\" target=\"_blank\">cit</a>)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1296191,
          "author_name": "verracodeguacas",
          "author_url": "",
          "post_date": "05/07/2021 04:10:22",
          "content": "<p>The more I train my models the worse score I get.</p>\n<p>I can visually see it getting better and better and yet, I score lower and lower.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1294098,
      "author_name": "xiaonanji",
      "author_url": "",
      "post_date": "05/05/2021 12:15:27",
      "content": "<p>The problem of only updating the testing dataset but leaving the training dataset untouched increases the difficulty of understanding which is treated as corrects and which is not. I had a quick look and have the feeling that after the update, the usefulness of the training data becomes less…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1297441,
      "author_name": "verracodeguacas",
      "author_url": "",
      "post_date": "05/08/2021 03:39:00",
      "content": "<p>This may be too late. My guess is that the winner at this point will be a simple string matching strategy, I think those of us trying models will not have time to come up with anything good. I have been so confused with everything I have tried over the past two months…</p>",
      "votes": null,
      "replies": [
        {
          "id": 1297647,
          "author_name": "suicaokhoailang",
          "author_url": "",
          "post_date": "05/08/2021 07:50:25",
          "content": "<p>What makes you think everyone else is doing string matching?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1297932,
          "author_name": "leecming",
          "author_url": "",
          "post_date": "05/08/2021 12:37:16",
          "content": "<p>I have to admit that the winning solution being some kind of string matching lookup table would be pretty funny :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1297986,
          "author_name": "suicaokhoailang",
          "author_url": "",
          "post_date": "05/08/2021 13:28:30",
          "content": "<p>That's a possibility, doesn't mean that ML approach wouldn't work here though, because it does. At least I can have a model that can apply to non-crawlable problems in the future.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1298257,
      "author_name": "httpwwwfszyc",
      "author_url": "",
      "post_date": "05/08/2021 17:19:02",
      "content": "<p>One Question: is right now the new public test dataset containing that 4 leakage data?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1321268,
      "author_name": "dilawarali316",
      "author_url": "",
      "post_date": "05/24/2021 15:21:01",
      "content": "<p>Your Welcome Sir.!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1293250": "Dear colleagues\nThanks for all your hard work on the competition – we are thrilled by the engagement and interest.  We thought our annotation efforts were sufficient but we realized that we needed to capture more. Kaggle has received the updated public test set and your submissions will be re-run against this new set. Re-running all of your submissions may take several days.  The private test set will also be updated.   Thanks again for your engagement in this important effort!",
    "1293263": "Thanks very much Julia! You and your team's hard work is much appreciated.\n\nNote that only scoring test labels have been updated on our back end - the competition's publications and other data remain the same. As Julia mentioned, I'll be rerunning all existing submissions against the updated test labels - any new submissions starting now will be using the update.",
    "1293387": "Thanks for update. Would it be possible to describe more in detail the kind of revision that has been carried out?",
    "1293542": "julialane is an updated public testset completely disjoint with training labels (including missing training label) ?",
    "1293785": "Thank You for this effort and update.\n\nAre there any changes to the various discussions / responses regarding the overlap / non-overlap of publications and labels?\n\nIs it safe to assume all of that still remains the same and the only update being done is including more labels to the public and private test sets?",
    "1293891": "Looking forward to seeing the details of this change！👀",
    "1293897": "Will you let us know when you have rerun all the submissions please? I'm trying not to get unduly excited by my jump of 500 positions up the public LB... 😄",
    "1293906": "Same here 😊\n\nBtw, never seen two deleted users on top of the LB before..",
    "1293943": "I haven't spent much time on Kaggle recently - what's up with people deleting their accounts so early in a competition.\n\nI know that cheaters tend to create dummy accounts but that seemed to be a thing at the tail-end, not at the start of a competition.\n\nOdd...",
    "1293952": "julialane  can you clarify please how annotations are done when the paper refers to a dataset incorrectly? For example if the paper misspells Alzheimers as Alzhiemers will it be annotated as the correct spelling?\n\nI have found many mispellings/misreferences/permutations in the training data papers - below a small subset - will these all be labelled as :  \"Alzheimer's Disease NeuroImaging Initiative ADNI\"?\n\nAlzheimer s Disease Neuro Initiative ADNI\nAlzheimer s Disease NeuroImaging Initiative ADNI\nAlzheimer s Disease Neuroimage Initiative ADNI\nAlzheimer s Disease Neuroimaging Initative ADNI\nAlzheimer s Disease Neuroimaging Initiative ANDI\nAlzheimer s Disease Neuroimaging Initiative s ADNI\nAlzheimer s Disease Neuroimaging Initiatives ADNI\nAlzheimer s Disease Neuroimaging Initive ADNI\nAlzheimer s Disease Neuroimaging Study ADNI\nAlzheimer s Disease Neuroimaing Initiative ADNI\nAlzheimer s Diseases Neuroimaging Initiative ADNI\nAlzheimer s Neuroimaging Initiative ADNI\nAlzheimer3s Disease Neuroimaging Initiative ADNI\nAlzheimer3s Disease Neuromaging Initiative ADNI\nAlzheimers Disease Neuroimaging Initiative ADNI\nAlzheimers Disease NeuroimagingInitiative ADNI\nAlzhimer s Disease Neuroimaging Initiative ADNI",
    "1294098": "The problem of only updating the testing dataset but leaving the training dataset untouched increases the difficulty of understanding which is treated as corrects and which is not. I had a quick look and have the feeling that after the update, the usefulness of the training data becomes less...",
    "1294609": "Hi @cascadenite. Great question. The correct labels for a given publication are drawn directly from that publication's text - misspellings, different wordings, and all. So if your model sees something that looks like a dataset mention in a publication, no matter how it's spelled, you should submit the predicted mention verbatim.",
    "1294636": "Thanks @philculliton -  does the same thinking apply to the length of the mention?  What I mean is:\n\n1.  if the following are both mentioned (at different parts of the text): \naging integrated database\naging integrated database agid \n\nShould I have both in my predictions or just the longest one?\n\n\n2. If there is only one mention of the agid dataset and it is written like this in the paper:\naging integrated database agid \n\nShould I have the full string in my prediction?",
    "1294650": "There hasn't been much movement on the LB today as far as I can tell, but I'm thinking the rescoring must still be ongoing because there is a group of participants with 0.701 scores and generally few submissions near the top that are unlikely to have been rescored.",
    "1294796": "philculliton \n\nIs there a write-up of the Kaggle backend for stuff like managing and scoring submissions anywhere (e.g., mechanics, tech stack, resource considerations etc.)? Or would you be able to describe it in brief? \n\nSeparate from this specific competition, I think many of us would be interested in understanding what goes on behind the curtain.",
    "1295925": "I confirm this. I'm one happy owner of a 0.701 submission. As it turns out this is a notebook that I deleted and therefore can't be re-run. See you at the bottom in a few days when they catch the glitch!\n\nIs your current submission also from a deleted notebook too?, because it is we'll meet at the 600th. position soon.",
    "1295960": "Hahaha! So that's what's going on with the 0.701 submissions near the top. I should delete all of my notebooks now in case there is another rescore. 🙂\n\nMy current score is from a non-deleted notebook, which is why I'm keeping an eye on those submissions above me.",
    "1295980": "cascadenite Unfortunately I guess you won't receive a reply to your questions because it seems there is very little consistency in the way the data are labelled (see [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236972)). So the answer would be \"nobody knows\" ([cit](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236753#1294510))",
    "1296022": "It must be odd for a newcomer to the competition to see the current top two...",
    "1296119": "philculliton It looks like deleted submissions could not be updated thus the leaderboard is frozen at the moment, are we going to get an update soon?",
    "1296191": "The more I train my models the worse score I get.\n\nI can visually see it getting better and better and yet, I score lower and lower.",
    "1296820": "Hi @suicaokhoailang - the removal of submissions that couldn't be re-run is the final part of the process, and the re-run has taken (at this point) several days. Should be completed today.",
    "1296849": "Hi @leecming - that's a great question. As you can imagine, it's extensive and depends on what type of competition is being run. I'll ask around and see if we have a public write-up of it anywhere.",
    "1297441": "This may be too late. My guess is that the winner at this point will be a simple string matching strategy, I think those of us trying models will not have time to come up with anything good. I have been so confused with everything I have tried over the past two months...",
    "1297647": "What makes you think everyone else is doing string matching?",
    "1297932": "I have to admit that the winning solution being some kind of string matching lookup table would be pretty funny :)",
    "1297986": "That's a possibility, doesn't mean that ML approach wouldn't work here though, because it does. At least I can have a model that can apply to non-crawlable problems in the future.",
    "1298257": "One Question: is right now the new public test dataset containing that 4 leakage data?",
    "1321268": "Your Welcome Sir.!"
  },
  "source": "meta"
}