{
  "id": 246649,
  "title": "Terror of private LB",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/246649",
  "author_name": "",
  "post_date": "2021-06-16T10:15:27.465012500Z",
  "votes": 4,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Simply, if the ratios of existing labels are same within train:public LB:private LB, number of cleaned_label are estimated to be about 19661 : 1318 : 9668.<br>\nand about numbers of unique label, are estimated to be<br>\nmore (cuz public govt notebook shows higher score than normal string match, it means there are lots of unknown labels there),<br>\nand I'm still thinking about how many unknown labels are in private test set… it will determine our last result,</p>",
  "messages": [
    {
      "id": "1351444",
      "postDate": "06/16/2021 10:15:27",
      "content": "<p>Simply, if the ratios of existing labels are same within train:public LB:private LB, number of cleaned_label are estimated to be about 19661 : 1318 : 9668.<br>\nand about numbers of unique label, are estimated to be<br>\nmore (cuz public govt notebook shows higher score than normal string match, it means there are lots of unknown labels there),<br>\nand I'm still thinking about how many unknown labels are in private test set… it will determine our last result,</p>",
      "rawMarkdown": "Simply, if the ratios of existing labels are same within train:public LB:private LB, number of cleaned_label are estimated to be about 19661 : 1318 : 9668.\nand about numbers of unique label, are estimated to be\nmore (cuz public govt notebook shows higher score than normal string match, it means there are lots of unknown labels there),\nand I'm still thinking about how many unknown labels are in private test set... it will determine our last result,",
      "votes": null
    },
    {
      "id": "1351479",
      "postDate": "06/16/2021 10:51:26",
      "content": "<p>I am curious about the ratios :19661 : 1318 : 9668, if OK, how do you get it? </p>\n<p>Otherwise, if we pushed the processing of the competition with string matching, it is nosense for improvement of understanding to data science and  just a medal we will take…    It tired to do this including me.</p>",
      "rawMarkdown": "I am curious about the ratios :19661 : 1318 : 9668, if OK, how do you get it? \n\nOtherwise, if we pushed the processing of the competition with string matching, it is nosense for improvement of understanding to data science and  just a medal we will take...    It tired to do this including me.",
      "votes": null
    },
    {
      "id": "1351495",
      "postDate": "06/16/2021 11:02:22",
      "content": "<p>Ok, I'll answer.<br>\nI heard that total Ids of test are about 8000,<br>\nand train.Ids are 14316, len(train)==19661, and LB says public : private == 12% : 88%,<br>\nthen just simply<br>\npublic : 1318 == int(19661x(8000/14316)x0.12)<br>\nprivate : 9668 == int(19661x(8000/14316)x0.88)<br>\nyeah I think only string matching approach with public probing is not interesting, but I think some teams will use nlp-based model so I expect them,</p>",
      "rawMarkdown": "Ok, I'll answer.\nI heard that total Ids of test are about 8000,\nand train.Ids are 14316, len(train)==19661, and LB says public : private == 12% : 88%,\nthen just simply\npublic : 1318 == int(19661x(8000/14316)x0.12)\nprivate : 9668 == int(19661x(8000/14316)x0.88)\nyeah I think only string matching approach with public probing is not interesting, but I think some teams will use nlp-based model so I expect them,",
      "votes": null
    },
    {
      "id": "1351513",
      "postDate": "06/16/2021 11:13:48",
      "content": "<p>thx, but there are many unlabeled mentions in train data which proved to be right. <br>\nThere is still a lot of uncertainty in second stage of com. :(</p>",
      "rawMarkdown": "thx, but there are many unlabeled mentions in train data which proved to be right. \nThere is still a lot of uncertainty in second stage of com. :(",
      "votes": null
    },
    {
      "id": "1351518",
      "postDate": "06/16/2021 11:17:02",
      "content": "<p>Yeah, the last labels will be more,<br>\nthe more I improve public LB, the less my confidence becomes…<br>\n(although I'm not doing one-by-one public probing) orz</p>",
      "rawMarkdown": "Yeah, the last labels will be more,\nthe more I improve public LB, the less my confidence becomes...\n(although I'm not doing one-by-one public probing) orz",
      "votes": null
    },
    {
      "id": "1357635",
      "postDate": "06/19/2021 20:48:44",
      "content": "<p>I see. Just try not to overfit in the public lb by purely string matching because those matched dataset names won't be in the private data set. You are actually wasting running time and resources. I think the key of the competition is still about building models. I joined late, so I am not going to train any model right now.</p>",
      "rawMarkdown": "I see. Just try not to overfit in the public lb by purely string matching because those matched dataset names won't be in the private data set. You are actually wasting running time and resources. I think the key of the competition is still about building models. I joined late, so I am not going to train any model right now.",
      "votes": null
    },
    {
      "id": "1357680",
      "postDate": "06/19/2021 22:01:49",
      "content": "<p>You have made a good point here.</p>",
      "rawMarkdown": "You have made a good point here.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1351479,
      "author_name": "vanle73",
      "author_url": "",
      "post_date": "06/16/2021 10:51:26",
      "content": "<p>I am curious about the ratios :19661 : 1318 : 9668, if OK, how do you get it? </p>\n<p>Otherwise, if we pushed the processing of the competition with string matching, it is nosense for improvement of understanding to data science and  just a medal we will take…    It tired to do this including me.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1351495,
          "author_name": "ryunosukeishizaki",
          "author_url": "",
          "post_date": "06/16/2021 11:02:22",
          "content": "<p>Ok, I'll answer.<br>\nI heard that total Ids of test are about 8000,<br>\nand train.Ids are 14316, len(train)==19661, and LB says public : private == 12% : 88%,<br>\nthen just simply<br>\npublic : 1318 == int(19661x(8000/14316)x0.12)<br>\nprivate : 9668 == int(19661x(8000/14316)x0.88)<br>\nyeah I think only string matching approach with public probing is not interesting, but I think some teams will use nlp-based model so I expect them,</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1351513,
          "author_name": "vanle73",
          "author_url": "",
          "post_date": "06/16/2021 11:13:48",
          "content": "<p>thx, but there are many unlabeled mentions in train data which proved to be right. <br>\nThere is still a lot of uncertainty in second stage of com. :(</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1351518,
          "author_name": "ryunosukeishizaki",
          "author_url": "",
          "post_date": "06/16/2021 11:17:02",
          "content": "<p>Yeah, the last labels will be more,<br>\nthe more I improve public LB, the less my confidence becomes…<br>\n(although I'm not doing one-by-one public probing) orz</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1357635,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "06/19/2021 20:48:44",
      "content": "<p>I see. Just try not to overfit in the public lb by purely string matching because those matched dataset names won't be in the private data set. You are actually wasting running time and resources. I think the key of the competition is still about building models. I joined late, so I am not going to train any model right now.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1357680,
      "author_name": "koearn",
      "author_url": "",
      "post_date": "06/19/2021 22:01:49",
      "content": "<p>You have made a good point here.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1351444": "Simply, if the ratios of existing labels are same within train:public LB:private LB, number of cleaned_label are estimated to be about 19661 : 1318 : 9668.\nand about numbers of unique label, are estimated to be\nmore (cuz public govt notebook shows higher score than normal string match, it means there are lots of unknown labels there),\nand I'm still thinking about how many unknown labels are in private test set... it will determine our last result,",
    "1351479": "I am curious about the ratios :19661 : 1318 : 9668, if OK, how do you get it? \n\nOtherwise, if we pushed the processing of the competition with string matching, it is nosense for improvement of understanding to data science and  just a medal we will take...    It tired to do this including me.",
    "1351495": "Ok, I'll answer.\nI heard that total Ids of test are about 8000,\nand train.Ids are 14316, len(train)==19661, and LB says public : private == 12% : 88%,\nthen just simply\npublic : 1318 == int(19661x(8000/14316)x0.12)\nprivate : 9668 == int(19661x(8000/14316)x0.88)\nyeah I think only string matching approach with public probing is not interesting, but I think some teams will use nlp-based model so I expect them,",
    "1351513": "thx, but there are many unlabeled mentions in train data which proved to be right. \nThere is still a lot of uncertainty in second stage of com. :(",
    "1351518": "Yeah, the last labels will be more,\nthe more I improve public LB, the less my confidence becomes...\n(although I'm not doing one-by-one public probing) orz",
    "1357635": "I see. Just try not to overfit in the public lb by purely string matching because those matched dataset names won't be in the private data set. You are actually wasting running time and resources. I think the key of the competition is still about building models. I joined late, so I am not going to train any model right now.",
    "1357680": "You have made a good point here."
  },
  "source": "meta"
}