{
  "id": 163645,
  "title": "Test Set Same Distribution?",
  "url": "/competitions/global-wheat-detection/discussion/163645",
  "author_name": "",
  "post_date": "2020-07-02T21:20:55.069126500Z",
  "votes": 1,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Do you guys think the test, final leaderboard, set comes from the same distribution as the training set, or public leaderboard or different? </p>\n\n<p>Meaning do you think they got all the pictures, mixed it up, and then put it in train/public LB/private LB, or you do think the private leaderboard images come from a separate file that wasn't mixed in with the training and public LB data?</p>\n\n<p>Thanks. </p>",
  "messages": [
    {
      "id": "912969",
      "postDate": "07/02/2020 21:20:55",
      "content": "<p>Do you guys think the test, final leaderboard, set comes from the same distribution as the training set, or public leaderboard or different? </p>\n\n<p>Meaning do you think they got all the pictures, mixed it up, and then put it in train/public LB/private LB, or you do think the private leaderboard images come from a separate file that wasn't mixed in with the training and public LB data?</p>\n\n<p>Thanks. </p>",
      "rawMarkdown": "Do you guys think the test, final leaderboard, set comes from the same distribution as the training set, or public leaderboard or different? \n\nMeaning do you think they got all the pictures, mixed it up, and then put it in train/public LB/private LB, or you do think the private leaderboard images come from a separate file that wasn't mixed in with the training and public LB data?\n\nThanks.",
      "votes": null
    },
    {
      "id": "912992",
      "postDate": "07/02/2020 21:45:36",
      "content": "<p>Training and test data is certainly distinct. From the competition description:</p>\n\n<blockquote>\n  <p>You will use more than 3,000 images from Europe (France, UK, Switzerland) and North America (Canada). The test data includes about 1,000 images from Australia, Japan, and China.</p>\n</blockquote>\n\n<p>However, the split between public LB and private LB is less certain. I'm inclined to believe it's a random split of the test data though.</p>",
      "rawMarkdown": "Training and test data is certainly distinct. From the competition description:\n&gt; You will use more than 3,000 images from Europe (France, UK, Switzerland) and North America (Canada). The test data includes about 1,000 images from Australia, Japan, and China.\n\nHowever, the split between public LB and private LB is less certain. I'm inclined to believe it's a random split of the test data though.",
      "votes": null
    },
    {
      "id": "912993",
      "postDate": "07/02/2020 21:46:44",
      "content": "<p>To be fair, I seem to have pretty transferable performance between the public train dataset and the public test set, so I would argue that they are pretty similar.</p>\n\n<p>However, they recently announced big news about the private set regarding its distribution. The train set is infamous for having mistakes in its bounding boxes, multiple threads have been talking about it (e.g. <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/159578\">this post</a>).</p>\n\n<p>While this may have had an impact on training (some people that had put aside images with wrongly created BBoxes experienced lower lb score), the organizer decided (in <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/163291\">this post</a>) to correct the bounding boxes from the private test and from the private test set only ! This means that they may, by the same occasion, create distribution variance between the training set, the public test set and the private test set. </p>\n\n<p>This is in no means a critic of their decision as I consider that they chose the best thing to do. However, you'd need to take this into account.</p>",
      "rawMarkdown": "To be fair, I seem to have pretty transferable performance between the public train dataset and the public test set, so I would argue that they are pretty similar.\n\nHowever, they recently announced big news about the private set regarding its distribution. The train set is infamous for having mistakes in its bounding boxes, multiple threads have been talking about it (e.g. [this post](https://www.kaggle.com/c/global-wheat-detection/discussion/159578)).\n\nWhile this may have had an impact on training (some people that had put aside images with wrongly created BBoxes experienced lower lb score), the organizer decided (in [this post](https://www.kaggle.com/c/global-wheat-detection/discussion/163291)) to correct the bounding boxes from the private test and from the private test set only ! This means that they may, by the same occasion, create distribution variance between the training set, the public test set and the private test set. \n\nThis is in no means a critic of their decision as I consider that they chose the best thing to do. However, you'd need to take this into account.",
      "votes": null
    },
    {
      "id": "913003",
      "postDate": "07/02/2020 22:00:32",
      "content": "<p>Great job Tomas, Matthew. Yea it would be interesting to see if the public/private leaderboard comes from the same distribution or different. I.E. take pictures, mix it up and divide into public/private LB, or have 2 separate folders with different type of images or whatever. </p>",
      "rawMarkdown": "Great job Tomas, Matthew. Yea it would be interesting to see if the public/private leaderboard comes from the same distribution or different. I.E. take pictures, mix it up and divide into public/private LB, or have 2 separate folders with different type of images or whatever.",
      "votes": null
    },
    {
      "id": "913011",
      "postDate": "07/02/2020 22:08:53",
      "content": "<p>As <a href=\"/matthewmasters\">@matthewmasters</a> mentioned the sources are distinct (confirmed by the host). I also think that the public LB and the private LB have (almost) different sources as well. (IMO, it is not a random split)</p>\n\n<p><a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/155933#872939\">Read my comment here</a></p>",
      "rawMarkdown": "As @matthewmasters mentioned the sources are distinct (confirmed by the host). I also think that the public LB and the private LB have (almost) different sources as well. (IMO, it is not a random split)\n\n[Read my comment here](https://www.kaggle.com/c/global-wheat-detection/discussion/155933#872939)",
      "votes": null
    },
    {
      "id": "913260",
      "postDate": "07/03/2020 04:59:24",
      "content": "<p><a href=\"/pestipeti\">@pestipeti</a> I agree with your opinion that test is almost divided by source. Here's my guess.</p>\n\n<p>The competition host said \"&lt;20 images in test that are not the same size as the others\" in <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/163291\">here</a></p>\n\n<p>If test data is randomly divided, it's natural to think both public and private include some part of the different-size images. So we should adjust predicted bbox coordinates for these images, or you'll get lower LB score. However, my LB score unchanged whether adjusting it or not.</p>\n\n<p>In addition, I guess all the different-size images are generated from the same source.</p>\n\n<p>Based on the above, I believe there is a higher chance that test is divied by source.</p>",
      "rawMarkdown": "pestipeti I agree with your opinion that test is almost divided by source. Here's my guess.\n\nThe competition host said \"&lt;20 images in test that are not the same size as the others\" in [here](https://www.kaggle.com/c/global-wheat-detection/discussion/163291)\n\nIf test data is randomly divided, it's natural to think both public and private include some part of the different-size images. So we should adjust predicted bbox coordinates for these images, or you'll get lower LB score. However, my LB score unchanged whether adjusting it or not.\n\nIn addition, I guess all the different-size images are generated from the same source.\n\nBased on the above, I believe there is a higher chance that test is divied by source.",
      "votes": null
    },
    {
      "id": "913282",
      "postDate": "07/03/2020 05:31:37",
      "content": "<p>Another thing with kaggle is that the leaderboard tends to be pretty saturated, especially when lots of people are based off a similar baseline. So even if the test set have a similar distribution as the training set (and your local results align with PB), considerable shakeups can still take place.</p>",
      "rawMarkdown": "Another thing with kaggle is that the leaderboard tends to be pretty saturated, especially when lots of people are based off a similar baseline. So even if the test set have a similar distribution as the training set (and your local results align with PB), considerable shakeups can still take place.",
      "votes": null
    },
    {
      "id": "913482",
      "postDate": "07/03/2020 08:34:43",
      "content": "<p>Why is Public test data(62%) much bigger than private test data(38%)?</p>",
      "rawMarkdown": "Why is Public test data(62%) much bigger than private test data(38%)?",
      "votes": null
    },
    {
      "id": "913519",
      "postDate": "07/03/2020 08:57:33",
      "content": "<p>The distribution between the public and private test set is also slightly skewed but comes from the same data sources :) </p>",
      "rawMarkdown": "The distribution between the public and private test set is also slightly skewed but comes from the same data sources :)",
      "votes": null
    },
    {
      "id": "914325",
      "postDate": "07/03/2020 18:58:52",
      "content": "<p>Take a look at the <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/163645#913519\">comment below</a>. I think this indicates all sources are present in both public and private test data</p>",
      "rawMarkdown": "Take a look at the [comment below](https://www.kaggle.com/c/global-wheat-detection/discussion/163645#913519). I think this indicates all sources are present in both public and private test data",
      "votes": null
    },
    {
      "id": "919742",
      "postDate": "07/08/2020 04:53:19",
      "content": "<p>Ok David, so you mean that private test and public test is both from Australia, Japan, and China, but the proportion of Australia, Japan, and China is different from public and private test although both public and private test has images from all 3 regions?</p>\n\n<p>thnaks</p>",
      "rawMarkdown": "Ok David, so you mean that private test and public test is both from Australia, Japan, and China, but the proportion of Australia, Japan, and China is different from public and private test although both public and private test has images from all 3 regions?\n\nthnaks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 912992,
      "author_name": "matthewmasters",
      "author_url": "",
      "post_date": "07/02/2020 21:45:36",
      "content": "<p>Training and test data is certainly distinct. From the competition description:</p>\n\n<blockquote>\n  <p>You will use more than 3,000 images from Europe (France, UK, Switzerland) and North America (Canada). The test data includes about 1,000 images from Australia, Japan, and China.</p>\n</blockquote>\n\n<p>However, the split between public LB and private LB is less certain. I'm inclined to believe it's a random split of the test data though.</p>",
      "votes": null,
      "replies": [
        {
          "id": 913519,
          "author_name": "bendvd",
          "author_url": "",
          "post_date": "07/03/2020 08:57:33",
          "content": "<p>The distribution between the public and private test set is also slightly skewed but comes from the same data sources :) </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 919742,
          "author_name": "bopengiowa",
          "author_url": "",
          "post_date": "07/08/2020 04:53:19",
          "content": "<p>Ok David, so you mean that private test and public test is both from Australia, Japan, and China, but the proportion of Australia, Japan, and China is different from public and private test although both public and private test has images from all 3 regions?</p>\n\n<p>thnaks</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 912993,
      "author_name": "dimartinot",
      "author_url": "",
      "post_date": "07/02/2020 21:46:44",
      "content": "<p>To be fair, I seem to have pretty transferable performance between the public train dataset and the public test set, so I would argue that they are pretty similar.</p>\n\n<p>However, they recently announced big news about the private set regarding its distribution. The train set is infamous for having mistakes in its bounding boxes, multiple threads have been talking about it (e.g. <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/159578\">this post</a>).</p>\n\n<p>While this may have had an impact on training (some people that had put aside images with wrongly created BBoxes experienced lower lb score), the organizer decided (in <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/163291\">this post</a>) to correct the bounding boxes from the private test and from the private test set only ! This means that they may, by the same occasion, create distribution variance between the training set, the public test set and the private test set. </p>\n\n<p>This is in no means a critic of their decision as I consider that they chose the best thing to do. However, you'd need to take this into account.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 913003,
      "author_name": "bopengiowa",
      "author_url": "",
      "post_date": "07/02/2020 22:00:32",
      "content": "<p>Great job Tomas, Matthew. Yea it would be interesting to see if the public/private leaderboard comes from the same distribution or different. I.E. take pictures, mix it up and divide into public/private LB, or have 2 separate folders with different type of images or whatever. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 913011,
      "author_name": "pestipeti",
      "author_url": "",
      "post_date": "07/02/2020 22:08:53",
      "content": "<p>As <a href=\"/matthewmasters\">@matthewmasters</a> mentioned the sources are distinct (confirmed by the host). I also think that the public LB and the private LB have (almost) different sources as well. (IMO, it is not a random split)</p>\n\n<p><a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/155933#872939\">Read my comment here</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 913260,
          "author_name": "kmizunoster",
          "author_url": "",
          "post_date": "07/03/2020 04:59:24",
          "content": "<p><a href=\"/pestipeti\">@pestipeti</a> I agree with your opinion that test is almost divided by source. Here's my guess.</p>\n\n<p>The competition host said \"&lt;20 images in test that are not the same size as the others\" in <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/163291\">here</a></p>\n\n<p>If test data is randomly divided, it's natural to think both public and private include some part of the different-size images. So we should adjust predicted bbox coordinates for these images, or you'll get lower LB score. However, my LB score unchanged whether adjusting it or not.</p>\n\n<p>In addition, I guess all the different-size images are generated from the same source.</p>\n\n<p>Based on the above, I believe there is a higher chance that test is divied by source.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 914325,
          "author_name": "matthewmasters",
          "author_url": "",
          "post_date": "07/03/2020 18:58:52",
          "content": "<p>Take a look at the <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/163645#913519\">comment below</a>. I think this indicates all sources are present in both public and private test data</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 913282,
      "author_name": "alexanderliao",
      "author_url": "",
      "post_date": "07/03/2020 05:31:37",
      "content": "<p>Another thing with kaggle is that the leaderboard tends to be pretty saturated, especially when lots of people are based off a similar baseline. So even if the test set have a similar distribution as the training set (and your local results align with PB), considerable shakeups can still take place.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 913482,
      "author_name": "mcggood",
      "author_url": "",
      "post_date": "07/03/2020 08:34:43",
      "content": "<p>Why is Public test data(62%) much bigger than private test data(38%)?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "912969": "Do you guys think the test, final leaderboard, set comes from the same distribution as the training set, or public leaderboard or different? \n\nMeaning do you think they got all the pictures, mixed it up, and then put it in train/public LB/private LB, or you do think the private leaderboard images come from a separate file that wasn't mixed in with the training and public LB data?\n\nThanks.",
    "912992": "Training and test data is certainly distinct. From the competition description:\n&gt; You will use more than 3,000 images from Europe (France, UK, Switzerland) and North America (Canada). The test data includes about 1,000 images from Australia, Japan, and China.\n\nHowever, the split between public LB and private LB is less certain. I'm inclined to believe it's a random split of the test data though.",
    "912993": "To be fair, I seem to have pretty transferable performance between the public train dataset and the public test set, so I would argue that they are pretty similar.\n\nHowever, they recently announced big news about the private set regarding its distribution. The train set is infamous for having mistakes in its bounding boxes, multiple threads have been talking about it (e.g. [this post](https://www.kaggle.com/c/global-wheat-detection/discussion/159578)).\n\nWhile this may have had an impact on training (some people that had put aside images with wrongly created BBoxes experienced lower lb score), the organizer decided (in [this post](https://www.kaggle.com/c/global-wheat-detection/discussion/163291)) to correct the bounding boxes from the private test and from the private test set only ! This means that they may, by the same occasion, create distribution variance between the training set, the public test set and the private test set. \n\nThis is in no means a critic of their decision as I consider that they chose the best thing to do. However, you'd need to take this into account.",
    "913003": "Great job Tomas, Matthew. Yea it would be interesting to see if the public/private leaderboard comes from the same distribution or different. I.E. take pictures, mix it up and divide into public/private LB, or have 2 separate folders with different type of images or whatever.",
    "913011": "As @matthewmasters mentioned the sources are distinct (confirmed by the host). I also think that the public LB and the private LB have (almost) different sources as well. (IMO, it is not a random split)\n\n[Read my comment here](https://www.kaggle.com/c/global-wheat-detection/discussion/155933#872939)",
    "913260": "pestipeti I agree with your opinion that test is almost divided by source. Here's my guess.\n\nThe competition host said \"&lt;20 images in test that are not the same size as the others\" in [here](https://www.kaggle.com/c/global-wheat-detection/discussion/163291)\n\nIf test data is randomly divided, it's natural to think both public and private include some part of the different-size images. So we should adjust predicted bbox coordinates for these images, or you'll get lower LB score. However, my LB score unchanged whether adjusting it or not.\n\nIn addition, I guess all the different-size images are generated from the same source.\n\nBased on the above, I believe there is a higher chance that test is divied by source.",
    "913282": "Another thing with kaggle is that the leaderboard tends to be pretty saturated, especially when lots of people are based off a similar baseline. So even if the test set have a similar distribution as the training set (and your local results align with PB), considerable shakeups can still take place.",
    "913482": "Why is Public test data(62%) much bigger than private test data(38%)?",
    "913519": "The distribution between the public and private test set is also slightly skewed but comes from the same data sources :)",
    "914325": "Take a look at the [comment below](https://www.kaggle.com/c/global-wheat-detection/discussion/163645#913519). I think this indicates all sources are present in both public and private test data",
    "919742": "Ok David, so you mean that private test and public test is both from Australia, Japan, and China, but the proportion of Australia, Japan, and China is different from public and private test although both public and private test has images from all 3 regions?\n\nthnaks"
  },
  "source": "meta"
}