{ "id": 12326, "title": "Duplicate images", "url": "/competitions/datasciencebowl/discussion/12326", "author_name": "", "post_date": "2015-01-25T05:12:28.780Z", "votes": 9, "comment_count": 5, "views": 1912, "content": "

If this were a "normal" data analysis project (and not a competition), I would remove a number of the images as duplicates.

\n

I found duplicates by comparing md5sums of the image files.

\n

Train set.  Image 14864.jpg in chaetognath_non_sagittal and Image 271.jpg in chaetognath_other have the same md5sum, the same number of bytes, and visually look identical.

\n

Test set.  In the 93,789 test images I found only 72,397 unique md5sums. 

\n

A count of the number of duplicates shows this:

\n

counts
    1     2     3     4     5     6     7     8
57513  9931  3695  1004   218    30     5     1     

\n

Only 57,513 images had no duplicates.  There were 9931 cases where two images shared the same md5sum. ... 5 cases of 7 images matching, and 1 case of 8 images matching.

\n

Here are the image numbers of the 8 identical images in the test set: 

\n

10084
117336
120878
137310
159251
3105
39527
56867

\n

Here are the image numbers of one of the 5 cases of 7 duplicates:

\n

11574
12828
19976
36521
4847
49518
59124

\n

Prediction is likely not that CPU intensive once a model is built, so the 21,391 duplicate test images that could be eliminated are perhaps not that big of a deal, but do introduce a likely minor overrepresentation bias.

\n

Duplicates between Train and Test Sets?  Luckily, there was only one:  159128.jpg from appendicularian_straight matches test set image 108193.jpg.

\n

For additional details on the duplicates and how they were found, see https://github.com/EarlGlynn/kaggle-plankton

\n\n

  

", "messages": [ { "id": "63126", "postDate": "01/25/2015 05:12:28", "content": "

If this were a "normal" data analysis project (and not a competition), I would remove a number of the images as duplicates.

\n

I found duplicates by comparing md5sums of the image files.

\n

Train set.  Image 14864.jpg in chaetognath_non_sagittal and Image 271.jpg in chaetognath_other have the same md5sum, the same number of bytes, and visually look identical.

\n

Test set.  In the 93,789 test images I found only 72,397 unique md5sums. 

\n

A count of the number of duplicates shows this:

\n

counts
    1     2     3     4     5     6     7     8
57513  9931  3695  1004   218    30     5     1     

\n

Only 57,513 images had no duplicates.  There were 9931 cases where two images shared the same md5sum. ... 5 cases of 7 images matching, and 1 case of 8 images matching.

\n

Here are the image numbers of the 8 identical images in the test set: 

\n

10084
117336
120878
137310
159251
3105
39527
56867

\n

Here are the image numbers of one of the 5 cases of 7 duplicates:

\n

11574
12828
19976
36521
4847
49518
59124

\n

Prediction is likely not that CPU intensive once a model is built, so the 21,391 duplicate test images that could be eliminated are perhaps not that big of a deal, but do introduce a likely minor overrepresentation bias.

\n

Duplicates between Train and Test Sets?  Luckily, there was only one:  159128.jpg from appendicularian_straight matches test set image 108193.jpg.

\n

For additional details on the duplicates and how they were found, see https://github.com/EarlGlynn/kaggle-plankton

\n\n

  

", "rawMarkdown": "", "votes": null }, { "id": "63129", "postDate": "01/25/2015 06:48:57", "content": "

good catch :)  thanks for sharing.

", "rawMarkdown": "", "votes": null }, { "id": "63171", "postDate": "01/26/2015 03:12:00", "content": "

Any explainations of these duplicates? And the duplicates in Test Dataset belong to the same category? @Admin

", "rawMarkdown": "", "votes": null }, { "id": "63191", "postDate": "01/26/2015 15:23:49", "content": "

[quote=efg2;63126]

\n

the 21,391 duplicate test images that could be eliminated are perhaps not that big of a deal, but do introduce a likely minor overrepresentation bias.

\n

[/quote]

Rest assured, the duplicates are not an accident and do not introduce bias. On the data page we cautioned that, "Kaggle has supplemented the test set with additional "ignored" images. These are not counted in the scoring." This is what you've found here. Specifically, the exact duplicates are either intentional or images which were, by chance, transformed back to their original state (e.g. flipped twice). These ignored images are there to discourage hand labeling. Anyone who tries to hand label has both more images and less certainty about which are actually part of the scoring.

", "rawMarkdown": "", "votes": null }, { "id": "63284", "postDate": "01/28/2015 19:20:15", "content": "

I discovered the stats I originally reported were incomplete.  I discovered my test.zip was corrupt. 

\n

After getting the complete set of 130,400 test images from a new copy of test.zip, the duplicate count in the test set is now 36,898 -- there are only 93,502 unique test set images.

\n

The counts of the number of kinds of duplicates is now:

\n

[1] 93502
counts
    1     2     3     4     5     6     7     8     9
71244 12346  6393  2597   684   195    38     4     1   

\n

So, 71,244 test images are completely unique.  At the other extreme, there is one case of 9 images being identically the same.  There are four cases of 8 images being identically the same.

\n

I would have thought minor variations in the test set (e.g, a variety of rotations of a particular image) would be a better test strategy than just repeating the same images so many times. 

\n

A second test image was found to match a train image.

\n

I updated the scripts and stats in the GitHub repository mentioned above.

", "rawMarkdown": "", "votes": null }, { "id": "63285", "postDate": "01/28/2015 19:48:19", "content": "

[quote=efg2;63284]

\n

I would have thought minor variations in the test set (e.g, a variety of rotations of a particular image) would be a better test strategy than just repeating the same images so many times. 

\n

[/quote]

\n

And you would be correct :) There are many other ignored images with many types of linear and nonlinear transformations.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 63129, "author_name": "drpatrickchan", "author_url": "", "post_date": "01/25/2015 06:48:57", "content": "

good catch :)  thanks for sharing.

", "votes": null, "replies": [] }, { "id": 63171, "author_name": "liangpang", "author_url": "", "post_date": "01/26/2015 03:12:00", "content": "

Any explainations of these duplicates? And the duplicates in Test Dataset belong to the same category? @Admin

", "votes": null, "replies": [] }, { "id": 63191, "author_name": "wcukierski", "author_url": "", "post_date": "01/26/2015 15:23:49", "content": "

[quote=efg2;63126]

\n

the 21,391 duplicate test images that could be eliminated are perhaps not that big of a deal, but do introduce a likely minor overrepresentation bias.

\n

[/quote]

Rest assured, the duplicates are not an accident and do not introduce bias. On the data page we cautioned that, "Kaggle has supplemented the test set with additional "ignored" images. These are not counted in the scoring." This is what you've found here. Specifically, the exact duplicates are either intentional or images which were, by chance, transformed back to their original state (e.g. flipped twice). These ignored images are there to discourage hand labeling. Anyone who tries to hand label has both more images and less certainty about which are actually part of the scoring.

", "votes": null, "replies": [] }, { "id": 63284, "author_name": "efglynn", "author_url": "", "post_date": "01/28/2015 19:20:15", "content": "

I discovered the stats I originally reported were incomplete.  I discovered my test.zip was corrupt. 

\n

After getting the complete set of 130,400 test images from a new copy of test.zip, the duplicate count in the test set is now 36,898 -- there are only 93,502 unique test set images.

\n

The counts of the number of kinds of duplicates is now:

\n

[1] 93502
counts
    1     2     3     4     5     6     7     8     9
71244 12346  6393  2597   684   195    38     4     1   

\n

So, 71,244 test images are completely unique.  At the other extreme, there is one case of 9 images being identically the same.  There are four cases of 8 images being identically the same.

\n

I would have thought minor variations in the test set (e.g, a variety of rotations of a particular image) would be a better test strategy than just repeating the same images so many times. 

\n

A second test image was found to match a train image.

\n

I updated the scripts and stats in the GitHub repository mentioned above.

", "votes": null, "replies": [] }, { "id": 63285, "author_name": "wcukierski", "author_url": "", "post_date": "01/28/2015 19:48:19", "content": "

[quote=efg2;63284]

\n

I would have thought minor variations in the test set (e.g, a variety of rotations of a particular image) would be a better test strategy than just repeating the same images so many times. 

\n

[/quote]

\n

And you would be correct :) There are many other ignored images with many types of linear and nonlinear transformations.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "63126": "", "63129": "", "63171": "", "63191": "", "63284": "", "63285": "" }, "source": "meta" }