{ "id": 149770, "title": "Number of samples in test (by source)", "url": "/competitions/global-wheat-detection/discussion/149770", "author_name": "", "post_date": "2020-05-09T20:10:46.731527Z", "votes": 31, "comment_count": 7, "views": 0, "content": "

| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 379 | 22.20% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 49 | 2.87% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 3 | 0.18% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 58.23% |\n|UTokyo_2 | 0 | 0% | 120 | 7.03% |\n|NAU_1 | 0 | 0% | 20 | 1.17% |\n|UQ_1 | 0 | 0% | 142 | 8.32% |

\n\n
\n\n
\n

The GWHD dataset represents 4,698 squared patches extracted from the 2219 original high-resolution RGB images acquired across the 11 sub-datasets.

\n
\n\n\n\n

My assumptions\n- The 49 test samples are actually the 49 in the train folder without bb.

\n\n
\n

About 100 images contain no heads ...

\n
\n\n\n\n

And here is a quote from the Welcome post:

\n\n
\n

Winners will be selected based on their performance on a blind test set of images from different world regions than in the training set — so a true generalization challenge!

\n
\n\n

In this case the test distribution is a bit different:

\n\n

| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 0 | 0% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 0 | 0% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 0 | 0% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 77.90% |\n|UTokyo_2 | 0 | 0% | 120 | 9.04% |\n|NAU_1 | 0 | 0% | 20 | 1.57% |\n|UQ_1 | 0 | 0% | 142 | 11.13% |

\n\n
\n\n

Source: GWHD Dataset publication

", "messages": [ { "id": "840176", "postDate": "05/09/2020 20:10:46", "content": "

| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 379 | 22.20% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 49 | 2.87% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 3 | 0.18% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 58.23% |\n|UTokyo_2 | 0 | 0% | 120 | 7.03% |\n|NAU_1 | 0 | 0% | 20 | 1.17% |\n|UQ_1 | 0 | 0% | 142 | 8.32% |

\n\n
\n\n
\n

The GWHD dataset represents 4,698 squared patches extracted from the 2219 original high-resolution RGB images acquired across the 11 sub-datasets.

\n
\n\n\n\n

My assumptions\n- The 49 test samples are actually the 49 in the train folder without bb.

\n\n
\n

About 100 images contain no heads ...

\n
\n\n\n\n

And here is a quote from the Welcome post:

\n\n
\n

Winners will be selected based on their performance on a blind test set of images from different world regions than in the training set — so a true generalization challenge!

\n
\n\n

In this case the test distribution is a bit different:

\n\n

| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 0 | 0% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 0 | 0% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 0 | 0% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 77.90% |\n|UTokyo_2 | 0 | 0% | 120 | 9.04% |\n|NAU_1 | 0 | 0% | 20 | 1.57% |\n|UQ_1 | 0 | 0% | 142 | 11.13% |

\n\n
\n\n

Source: GWHD Dataset publication

", "rawMarkdown": "| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 379 | 22.20% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 49 | 2.87% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 3 | 0.18% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 58.23% |\n|UTokyo_2 | 0 | 0% | 120 | 7.03% |\n|NAU_1 | 0 | 0% | 20 | 1.17% |\n|UQ_1 | 0 | 0% | 142 | 8.32% | \n\n------\n\n> The GWHD dataset represents 4,698 squared patches extracted from the 2219 original high-resolution RGB images acquired across the 11 sub-datasets.\n\n- In the table above the total number of patches are 5080.\n- We have 49 extra images in the train folder (without bounding boxes)\n- We have samples in both set from only the `arvalis_1`, `arvalis_3` and `ethz_1` sources. \n\n**My assumptions**\n- The 49 test samples are actually the 49 in the train folder without bb. \n> About 100 images contain no heads ...\n\n- If we assume (1) then we only have `arvalis_1 (379)` + `ethz_1 (3)` in both set. The difference between my calculations and the number in the publication is 381.\n- Probably the number in the publication is correct (4698) and there is no overlap (by source) between the train and test set.\n\nAnd here is a quote from the [Welcome](https://www.kaggle.com/c/global-wheat-detection/discussion/148592) post:\n\n> Winners will be selected based on their performance on a blind test set of images from different world regions than in the training set — so a true generalization challenge!\n\nIn this case the test distribution is a bit different:\n\n| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 0 | 0% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 0 | 0% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 0 | 0% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 77.90% |\n|UTokyo_2 | 0 | 0% | 120 | 9.04% |\n|NAU_1 | 0 | 0% | 20 | 1.57% |\n|UQ_1 | 0 | 0% | 142 | 11.13% | \n\n-----\n\n\nSource: [GWHD Dataset publication](https://arxiv.org/pdf/2005.02162.pdf)", "votes": null }, { "id": "840219", "postDate": "05/09/2020 20:52:03", "content": "

I'm not sure that you are correct that Arvalis and ETHZ are in the test set at all. I'm assuming you arrived at these numbers by looking at the paper's totals in Table 3 and then subtracting what is present in the training set, correct? They noted in the paper that some patches from those two sites were removed during the labeling process, which could account for the discrepancy. Perhaps @bendvd or @stavness could answer this better

", "rawMarkdown": "I'm not sure that you are correct that Arvalis and ETHZ are in the test set at all. I'm assuming you arrived at these numbers by looking at the paper's totals in Table 3 and then subtracting what is present in the training set, correct? They noted in the paper that some patches from those two sites were removed during the labeling process, which could account for the discrepancy. Perhaps @bendvd or @stavness could answer this better", "votes": null }, { "id": "840228", "postDate": "05/09/2020 21:01:23", "content": "

Yes you are correct, I simple subtracting the number (without reading the full paper)\nI think the 2nd table is correct. I just finished the paper, here is a quote from 4.1

\n\n
\n

To evaluate model performance, including robustness against unseen images, the test data set includes all the images from Australia, Japan, and China, representing 1276 images.

\n
", "rawMarkdown": "Yes you are correct, I simple subtracting the number (without reading the full paper)\nI think the 2nd table is correct. I just finished the paper, here is a quote from 4.1\n\n> To evaluate model performance, including robustness against unseen images, the test data set includes all the images from Australia, Japan, and China, representing 1276 images.", "votes": null }, { "id": "840231", "postDate": "05/09/2020 21:06:10", "content": "

Oh I'm sorry, I should have looked at your post more carefully, I see that you corrected for this in your second table now.

\n\n

Good analysis of the data, you are a strong competitor in the challenge!

", "rawMarkdown": "Oh I'm sorry, I should have looked at your post more carefully, I see that you corrected for this in your second table now.\n\nGood analysis of the data, you are a strong competitor in the challenge!", "votes": null }, { "id": "916369", "postDate": "07/05/2020 15:29:09", "content": "

Hey! Great work, upvoted all your notebooks

", "rawMarkdown": "Hey! Great work, upvoted all your notebooks", "votes": null }, { "id": "938043", "postDate": "07/21/2020 10:05:01", "content": "

Hi guys. I still don't understand the amount of files in the test set. I can only find 10. When I use my model and predict teh test folder it only generates 10 rows corresponding to the 10 images. This is giving me a submission error. Peter, do you mean that there are total 49 images in the test folder?

", "rawMarkdown": "Hi guys. I still don't understand the amount of files in the test set. I can only find 10. When I use my model and predict teh test folder it only generates 10 rows corresponding to the 10 images. This is giving me a submission error. Peter, do you mean that there are total 49 images in the test folder?", "votes": null }, { "id": "938061", "postDate": "07/21/2020 10:15:33", "content": "

@luisvivas we have 10 publicly available images in the test set, but when you submit your submission.csv (with the 10 prediction row), Kaggle will re-run your kernel on a private test set. ~1200 images. And they calculate the public/private (62%-38%) LB based on the private samples

", "rawMarkdown": "luisvivas we have 10 publicly available images in the test set, but when you submit your `submission.csv` (with the 10 prediction row), Kaggle will re-run your kernel on a private test set. ~1200 images. And they calculate the public/private (62%-38%) LB based on the private samples", "votes": null }, { "id": "938074", "postDate": "07/21/2020 10:32:04", "content": "

Thanks @pestipeti now I think understand what my error could be. I created a separate kernel which is just sumbmitting the file and it does not have the model or the code to predict. I wil use the same kernel that predicts to send submit the file.

", "rawMarkdown": "Thanks @pestipeti now I think understand what my error could be. I created a separate kernel which is just sumbmitting the file and it does not have the model or the code to predict. I wil use the same kernel that predicts to send submit the file.", "votes": null } ], "comments": [ { "id": 840219, "author_name": "matthewmasters", "author_url": "", "post_date": "05/09/2020 20:52:03", "content": "

I'm not sure that you are correct that Arvalis and ETHZ are in the test set at all. I'm assuming you arrived at these numbers by looking at the paper's totals in Table 3 and then subtracting what is present in the training set, correct? They noted in the paper that some patches from those two sites were removed during the labeling process, which could account for the discrepancy. Perhaps @bendvd or @stavness could answer this better

", "votes": null, "replies": [ { "id": 840228, "author_name": "pestipeti", "author_url": "", "post_date": "05/09/2020 21:01:23", "content": "

Yes you are correct, I simple subtracting the number (without reading the full paper)\nI think the 2nd table is correct. I just finished the paper, here is a quote from 4.1

\n\n
\n

To evaluate model performance, including robustness against unseen images, the test data set includes all the images from Australia, Japan, and China, representing 1276 images.

\n
", "votes": null, "replies": [] }, { "id": 840231, "author_name": "matthewmasters", "author_url": "", "post_date": "05/09/2020 21:06:10", "content": "

Oh I'm sorry, I should have looked at your post more carefully, I see that you corrected for this in your second table now.

\n\n

Good analysis of the data, you are a strong competitor in the challenge!

", "votes": null, "replies": [] }, { "id": 938043, "author_name": "luisvivas", "author_url": "", "post_date": "07/21/2020 10:05:01", "content": "

Hi guys. I still don't understand the amount of files in the test set. I can only find 10. When I use my model and predict teh test folder it only generates 10 rows corresponding to the 10 images. This is giving me a submission error. Peter, do you mean that there are total 49 images in the test folder?

", "votes": null, "replies": [] }, { "id": 938061, "author_name": "pestipeti", "author_url": "", "post_date": "07/21/2020 10:15:33", "content": "

@luisvivas we have 10 publicly available images in the test set, but when you submit your submission.csv (with the 10 prediction row), Kaggle will re-run your kernel on a private test set. ~1200 images. And they calculate the public/private (62%-38%) LB based on the private samples

", "votes": null, "replies": [] }, { "id": 938074, "author_name": "luisvivas", "author_url": "", "post_date": "07/21/2020 10:32:04", "content": "

Thanks @pestipeti now I think understand what my error could be. I created a separate kernel which is just sumbmitting the file and it does not have the model or the code to predict. I wil use the same kernel that predicts to send submit the file.

", "votes": null, "replies": [] } ] }, { "id": 916369, "author_name": "salmaneunus", "author_url": "", "post_date": "07/05/2020 15:29:09", "content": "

Hey! Great work, upvoted all your notebooks

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "840176": "| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 379 | 22.20% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 49 | 2.87% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 3 | 0.18% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 58.23% |\n|UTokyo_2 | 0 | 0% | 120 | 7.03% |\n|NAU_1 | 0 | 0% | 20 | 1.17% |\n|UQ_1 | 0 | 0% | 142 | 8.32% | \n\n------\n\n> The GWHD dataset represents 4,698 squared patches extracted from the 2219 original high-resolution RGB images acquired across the 11 sub-datasets.\n\n- In the table above the total number of patches are 5080.\n- We have 49 extra images in the train folder (without bounding boxes)\n- We have samples in both set from only the `arvalis_1`, `arvalis_3` and `ethz_1` sources. \n\n**My assumptions**\n- The 49 test samples are actually the 49 in the train folder without bb. \n> About 100 images contain no heads ...\n\n- If we assume (1) then we only have `arvalis_1 (379)` + `ethz_1 (3)` in both set. The difference between my calculations and the number in the publication is 381.\n- Probably the number in the publication is correct (4698) and there is no overlap (by source) between the train and test set.\n\nAnd here is a quote from the [Welcome](https://www.kaggle.com/c/global-wheat-detection/discussion/148592) post:\n\n> Winners will be selected based on their performance on a blind test set of images from different world regions than in the training set — so a true generalization challenge!\n\nIn this case the test distribution is a bit different:\n\n| Source | Train | % | Test | % |\n|:---------:|:-----:|:---:|------|---|\n|Arvalis_1 | 1055 | 31.28% | 0 | 0% |\n|Arvalis_2 | 204 | 6.05% | 0 | 0% |\n|Arvalis_3 | 559 | 16.57% | 0 | 0% |\n|INRAE_1 | 176 | 5.22% | 0 | 0% |\n|Rres_1 | 432 | 12.81% | 0 | 0% |\n|ETHZ_1 | 747 | 22.15% | 0 | 0% |\n|USask_1 | 200 | 5.93% | 0 | 0% |\n|UTokyo_1 | 0 | 0% | 994 | 77.90% |\n|UTokyo_2 | 0 | 0% | 120 | 9.04% |\n|NAU_1 | 0 | 0% | 20 | 1.57% |\n|UQ_1 | 0 | 0% | 142 | 11.13% | \n\n-----\n\n\nSource: [GWHD Dataset publication](https://arxiv.org/pdf/2005.02162.pdf)", "840219": "I'm not sure that you are correct that Arvalis and ETHZ are in the test set at all. I'm assuming you arrived at these numbers by looking at the paper's totals in Table 3 and then subtracting what is present in the training set, correct? They noted in the paper that some patches from those two sites were removed during the labeling process, which could account for the discrepancy. Perhaps @bendvd or @stavness could answer this better", "840228": "Yes you are correct, I simple subtracting the number (without reading the full paper)\nI think the 2nd table is correct. I just finished the paper, here is a quote from 4.1\n\n> To evaluate model performance, including robustness against unseen images, the test data set includes all the images from Australia, Japan, and China, representing 1276 images.", "840231": "Oh I'm sorry, I should have looked at your post more carefully, I see that you corrected for this in your second table now.\n\nGood analysis of the data, you are a strong competitor in the challenge!", "916369": "Hey! Great work, upvoted all your notebooks", "938043": "Hi guys. I still don't understand the amount of files in the test set. I can only find 10. When I use my model and predict teh test folder it only generates 10 rows corresponding to the 10 images. This is giving me a submission error. Peter, do you mean that there are total 49 images in the test folder?", "938061": "luisvivas we have 10 publicly available images in the test set, but when you submit your `submission.csv` (with the 10 prediction row), Kaggle will re-run your kernel on a private test set. ~1200 images. And they calculate the public/private (62%-38%) LB based on the private samples", "938074": "Thanks @pestipeti now I think understand what my error could be. I created a separate kernel which is just sumbmitting the file and it does not have the model or the code to predict. I wil use the same kernel that predicts to send submit the file." }, "source": "meta" }