Hi all,
\nAs promised, here are the overall classification accuracies for the top 10 teams in the competition. I used all of the test data for this - Public and Private.
\n1. Deep Sea 81.52%
\n2. Happy Lantern Festival 81.01%
\n3. Poisson Process 80.84%
\n4. Junonia 80.49%
\n5. Deepsea Challenger 80.17%
\n6. AuroraXie 80.32%
\n7. Maxim Milakov 80.13%
\n8. Ilya Kostrikov 80.50%
\n9. oldufo 80.27%
\n10. nagadomi 80.09%
\nBut perhaps more interesting to you is the following figure (attached). I calculated the average precision per class in the top 10 teams and plotted each team's per-class precision relative to the mean. The scaling is per row, and the green values are high precision, and red values are low precision. White values are NAs - meaning that the team did not successfully predict any of the images in that class. (This was fairly rare and only occurred for hydromedusae_other, fish_larvae_deep_body, and a couple more scattered classes.) I thought this heatmap was a more interesting way of looking at how the top 10 teams performed rather than a big table of numbers.
\nAll of us in the Plankton team (Robert Cowen's research group at Hatfield Marine Science Center, OSU) were incredibly impressed by the performance of all the participants (not just the top 10!) in this difficult classification challenge. We look forward to seeing the winners' code and documentation - as well as anyone else who wants to share their code on github. We are also encouraging teams to publish their approaches in scientific journals and conferences. Feel free to contact me (jessica.luo [at] rsmas.miami.edu) or Kelly Robinson (kelly.robinson [at] oregonstate.edu) if you need any details on the images (collection, processing, segmentation, etc). We are currently planning a publication on our initial processing scheme for submission in June, so if you need any information prior to that, just let us know.
\nThanks and congratulations,
\nJessica
", "messages": [ { "id": "67763", "postDate": "03/23/2015 19:42:45", "content": "Hi all,
\nAs promised, here are the overall classification accuracies for the top 10 teams in the competition. I used all of the test data for this - Public and Private.
\n1. Deep Sea 81.52%
\n2. Happy Lantern Festival 81.01%
\n3. Poisson Process 80.84%
\n4. Junonia 80.49%
\n5. Deepsea Challenger 80.17%
\n6. AuroraXie 80.32%
\n7. Maxim Milakov 80.13%
\n8. Ilya Kostrikov 80.50%
\n9. oldufo 80.27%
\n10. nagadomi 80.09%
\nBut perhaps more interesting to you is the following figure (attached). I calculated the average precision per class in the top 10 teams and plotted each team's per-class precision relative to the mean. The scaling is per row, and the green values are high precision, and red values are low precision. White values are NAs - meaning that the team did not successfully predict any of the images in that class. (This was fairly rare and only occurred for hydromedusae_other, fish_larvae_deep_body, and a couple more scattered classes.) I thought this heatmap was a more interesting way of looking at how the top 10 teams performed rather than a big table of numbers.
\nAll of us in the Plankton team (Robert Cowen's research group at Hatfield Marine Science Center, OSU) were incredibly impressed by the performance of all the participants (not just the top 10!) in this difficult classification challenge. We look forward to seeing the winners' code and documentation - as well as anyone else who wants to share their code on github. We are also encouraging teams to publish their approaches in scientific journals and conferences. Feel free to contact me (jessica.luo [at] rsmas.miami.edu) or Kelly Robinson (kelly.robinson [at] oregonstate.edu) if you need any details on the images (collection, processing, segmentation, etc). We are currently planning a publication on our initial processing scheme for submission in June, so if you need any information prior to that, just let us know.
\nThanks and congratulations,
\nJessica
", "rawMarkdown": "", "votes": null }, { "id": "67773", "postDate": "03/23/2015 20:10:35", "content": "Addendum:
\nFor those of you who might want to compare your precision scores to the average of the top 10, attached is the raw file.
\nNB: I used all the test data (public & private), and derived the precision value (= PPV) from a confusion matrix constructed using the caret package in R (for formulas see pg 24).
\n\n- Jessica
", "rawMarkdown": "", "votes": null }, { "id": "67909", "postDate": "03/24/2015 07:13:02", "content": "Wa! Thank you Jessica for sharing this. The top-10 team beat the 80% classification accuracy. My best model only gets 72% for cross validation accuracy. A huge gap and motivates me to improve and learn more!
", "rawMarkdown": "", "votes": null }, { "id": "67956", "postDate": "03/24/2015 12:54:03", "content": "Hi Jessica,
\nI find the heat map very interesting indeed. This is how I understand the computation:
\nBut there are still things I don't understand:
\nThat said, assuming that the general patterns are solid, this reveals interesting differences between competitors. I would be curious to know if those differences are voluntary, strategic, or just the by product of different techniques and "blind" (i.e. not directed) optimisation.
\nFor exemple, DeepSea, the best solution, is not that far from DeepseaChallenger (good name by the way!) in the final ranking, both in terms of accuracy and log-loss. However, DeepSea is good almost everywhere but quite bad for decapods, a probably quite rare group. On the other hand, DeepseaChallenger is mostly average but very very good on copepods, a very common group. Two very different ways of getting to comparable overall results!
", "rawMarkdown": "", "votes": null }, { "id": "67991", "postDate": "03/24/2015 15:50:08", "content": "Hi Jean-Olivier,
\n[quote=Jean-Olivier Irisson;67956]
\nBut there are still things I don't understand:
\n[/quote]
\nYes - I wanted to look at how the top 10 teams compared against each other, and decided to just use precision, which is just one metric (though fairly commonly used). Since a big table of numbers can be boring/ difficult to interpret, I decided to use a graphical approach and plot anomalies relative to the mean. I computed a single value for precision per class for each team, then averaged amongst the teams, and then calculated each team's difference relative to the average.
\nScaling is by row. No scaling results in a graph where you can only see one or two high values and the rest looks like a (relative) sea of homogeny. I used the heatmap() command in the base R package, and from the source code it looks like it centers the values to the mean and then scales it by a calculated value of standard deviation.
\nI'm not sure if this was exactly your point -- but yes - the extra step I did in explicitly calculating the deviances relative to a mean precision per class, given the scaling in the plotting package, is redundant. I wasn't aware of the scaling options when I started and just went with what seemed most logical to me at the time! :)
\n[quote]
\nThat said, assuming that the general patterns are solid, this reveals interesting differences between competitors. I would be curious to know if those differences are voluntary, strategic, or just the by product of different techniques and "blind" (i.e. not directed) optimisation.
\nFor exemple, DeepSea, the best solution, is not that far from DeepseaChallenger (good name by the way!) in the final ranking, both in terms of accuracy and log-loss. However, DeepSea is good almost everywhere but quite bad for decapods, a probably quite rare group. On the other hand, DeepseaChallenger is mostly average but very very good on copepods, a very common group. Two very different ways of getting to comparable overall results!
\n[/quote]
\nYup - I read about the different techniques that some of the top 10 teams used, and was interested in seeing how they compared against each other. My initial approach overly highlights the differences between the teams, which is actually quite small -- which you can see in the attached figure. These are the raw precision values, unscaled.
\nAs you can see, echinoderms as a group were predicted fairly well, as well as salps and doliolids. Oithona copepods were pretty easy as well. Some of the hydromedusae were easier to predict than others, and the fish were all over the map. This makes sense to me -- the groups that were easier to predict were also easier for us to label. The ones that we struggled again and again to differentiate (and in some cases, just threw up our hands i.e. "unknowns") -- performed poorly. Except the detritus -- that surprised me.
\nThis goes back to what Sander and SR were saying in the other thread -- more training images are better. We now have a set of 60K training images through this dataset, but if we can get another 30K more, or 60K more, the models should be running a lot more smoothly with the real data.
\ncheers
\nJessica
", "rawMarkdown": "", "votes": null }, { "id": "67998", "postDate": "03/24/2015 16:14:35", "content": "[quote=Jessica Luo;67991]
\nI used the heatmap() command in the base R package, and from the source code it looks like it centers the values to the mean and then scales it by a calculated value of standard deviation.
\n[/quote]
\n\n?heatmap mentions dendrograms and plenty of other complex stuff. Computing and scaling the data manually is easy enough and would allow you to control what is displayed. If everything is scaled by the variance also, that makes it difficult to compare between classes (rows) actually. Would you mind sharing the data (precision per team per class)?
\n[quote]
\nI'm not sure if this was exactly your point -- but yes - the extra step I did in explicitly calculating the deviances relative to a mean precision per class, given the scaling in the plotting package, is redundant.
\n[/quote]
\nIt was not my point but indeed, it seems redundant now ;)
\n\nAll your other points indeed show that we need better and larger training sets. Now those weekly emails from prospective interns will come in handy!
", "rawMarkdown": "", "votes": null }, { "id": "68024", "postDate": "03/24/2015 17:13:18", "content": "[quote=Jean-Olivier Irisson;67998]
\nWould you mind sharing the data (precision per team per class)?
\n[/quote]
\n\nYup! Here you go - attached. Now... back to my dissertation :)
", "rawMarkdown": "", "votes": null }, { "id": "68078", "postDate": "03/24/2015 21:09:34", "content": "@Jessica Luo,
\nFrom heatmap it looks like if someone averages predictions of top-3..5 teams, 82% threshold will be done. Just curious :)
", "rawMarkdown": "", "votes": null }, { "id": "68309", "postDate": "03/25/2015 21:17:12", "content": "Thanks Jessica,
On the subject of classes & heatmaps …
As part of DeepSea Challenger team (#5), we wanted to better understand the individual class behavior as shown in the attachments.
The plot with all the various shades of blue is a visual representation of a confusion matrix summarizing the real class id data (the rows) vs the predicted class id data (the columns). For example, a perfect match would be for a dark blue diagonal (100% correlation between real and predicted class values) and blank squares (zero correlation) everywhere else. Our models had problems distinguishing between things like:
55,fish_larvae_very_thin_body (average real class id)
11,chaetognath_other (average predicted id)
If you take a look at these images in the train sets you’ll understand why.
Interestingly, not all the classes that we confused behaved symmetrically e.g. average real id = 11 vs average predicted id = 55 … doesn’t show up as dark blue (although this is partly due to the averaging technique we used).
The big purple squares are used to emphasize a very basic class hierarchy - this hierarchy is simply based on shared class prefixes e.g. the three last class ids for the classes that begin with "unknown" (118-120).
A while ago I also posted a plot for all the miss stats and it contains 3 charts for: (i) total counts (ii) miss counts (iii) miss percentages – all as a function of class id (121 plankton classes, alphabetically sorted). For each image, a “hit” is attributed to a class only if the highest probability class prediction matches the actual class in the training set (last layer is CNN softmax).
The charts were created using 80% of the train data on an older iteration of our models but towards the end we still had most problems with “other” and “unknown” classes.
Thanks for hosting this very interesting competition and in particular to Sander and his team for sharing their solution methods – well done!
[quote=Gal A.;68309]
\nOur models had problems distinguishing between things like:
55,fish_larvae_very_thin_body (average real class id)
11,chaetognath_other (average predicted id)
If you take a look at these images in the train sets you’ll understand why.
[/quote]
\n\nWhat if we told you that the fish larvae vs. chaetognath distinction is one of the most important distinctions that we need to make for our research? Chaetognaths are really abundant plankton predators, but completely unimportant socioeconomically, and fish larvae are, well, baby fish that can be hugely important to society. Chaetognaths and fish larvae look totally different in a regular plankton sample, but in our images, they look surprisingly similar -- particularly these two groups: the poorly resolved "other" chaetognaths, and the very thin bodied fish larvae. But, there are at least 1-2 orders of magnitude more chaetognaths than fish larvae in the ocean, and we need a way to detect the fish larvae without having chaetognaths pollute our fish groups. And similarly, we need to not be sorting through millions and millions of chaetognath images to find just a few fish larvae.
\n\nOne example is from a SVM model that we developed for a cruise a few years back. About 500K "fish" were detected out of about 160M images, but only about 25K were actually fish. In contrast, there were 7M chaetognaths detected, and if even 1% of those images were fish, that would mean 70K additional fish, or twice as much of the true positives in the fish categories. Some poor schmuck (... grad student...) would have to sort through all the low probability chaetognaths to find those fish, because well, the fish were the focal species of the whole study. That grad student might still be bitter at the task and ... then just found a kaggle competition to run. :)
", "rawMarkdown": "", "votes": null }, { "id": "68387", "postDate": "03/26/2015 08:05:31", "content": "Hi Jessica,
As a plankton-ignoramus, I like to let the data science tools uncover these connections :)
A couple of ways to get around the sampling problems that you mention, let’s say you have just two classes with 9000 type A and 1000 type B training images:
1. If you were to under-sample the majority class (type A) and provide competitors with 1000 type A and 1000 type B images, and at the same time declare that the ratio in the population is 9:1 (i.e. provide all the class priors to the best of your knowledge) then that could have saved you some valuable resources in the collection process. This assumes that the 8000 type B images being discarded are essentially "more of the same" information (although we don't know the real impact of the discarded information without experimenting). Maybe you can devise a special capture procedure at locations where the rarer fish concentrations are known to exist and aim for disproportionate class samples to begin with (over-sample the rare classes) - as long as you also provide the "real" class priors then that should be fine.
2. For the competition, I understand that you sampled the population uniformly (but did you consider stratified sampling in order to get consistently sized samples?). In any case, the researcher can then decide to either over-sample the minority classes (like type B) or under-sample the majority classes (like type A) in order to rebalance the training sets.
Alexandre Dalyac and his colleagues give a good account of their experiments using CNNs (Chapter 6 on Tackling Class Imbalance):
\nhttp://www.academia.edu/8472416/Tackling_Class_Imbalance_with_Deep_Convolutional_Neural_Networks
\nThey say they had much more success with under-sampling (p44):
\n“The results show that under-sampling is very effective: the model does not get stuck in any seeming critical point, obtains an additional 15 percentage points in classification accuracy, to obtain 80.8% validation accuracy. This is substantial and shows how strikingly detrimental class imbalance is: we have eliminated 58.7% of the data to get rid of it, and yet this sacrifice is highly worthwhile. This sets a high benchmark for other techniques for tackling class imbalance. Since we are throwing data, it also implies that if there exists a technique for solving class imbalance without losing data, performance could be even higher.”
", "rawMarkdown": "", "votes": null }, { "id": "69228", "postDate": "03/31/2015 22:57:51", "content": "Hi Gal,
\n\nThank you for this suggestion on how to deal with class imbalance. We actually did use a mixed approach in selecting the images for this competition - a completely uniform / even sampling of the classes would have resulted in 4-5 order of magnitude differences between the most abundant group and the least abundant group. So we strove for a balance between non-selective and highly selective sampling.
\n\nIt's great to see this approach for providing a "prior" for the class distributions. I would actually be interested in seeing how this performs under more 'extreme' imbalances (so - instead of 9:1, how about 9999:1?) -- these cases are actually fairly common in the natural world.
\n\ncheers,
\n\nJessica
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 67773, "author_name": "jessicaluo", "author_url": "", "post_date": "03/23/2015 20:10:35", "content": "Addendum:
\nFor those of you who might want to compare your precision scores to the average of the top 10, attached is the raw file.
\nNB: I used all the test data (public & private), and derived the precision value (= PPV) from a confusion matrix constructed using the caret package in R (for formulas see pg 24).
\n\n- Jessica
", "votes": null, "replies": [] }, { "id": 67909, "author_name": "pengpai", "author_url": "", "post_date": "03/24/2015 07:13:02", "content": "Wa! Thank you Jessica for sharing this. The top-10 team beat the 80% classification accuracy. My best model only gets 72% for cross validation accuracy. A huge gap and motivates me to improve and learn more!
", "votes": null, "replies": [] }, { "id": 67956, "author_name": "jeanolivieririsson", "author_url": "", "post_date": "03/24/2015 12:54:03", "content": "Hi Jessica,
\nI find the heat map very interesting indeed. This is how I understand the computation:
\nBut there are still things I don't understand:
\nThat said, assuming that the general patterns are solid, this reveals interesting differences between competitors. I would be curious to know if those differences are voluntary, strategic, or just the by product of different techniques and "blind" (i.e. not directed) optimisation.
\nFor exemple, DeepSea, the best solution, is not that far from DeepseaChallenger (good name by the way!) in the final ranking, both in terms of accuracy and log-loss. However, DeepSea is good almost everywhere but quite bad for decapods, a probably quite rare group. On the other hand, DeepseaChallenger is mostly average but very very good on copepods, a very common group. Two very different ways of getting to comparable overall results!
", "votes": null, "replies": [] }, { "id": 67991, "author_name": "jessicaluo", "author_url": "", "post_date": "03/24/2015 15:50:08", "content": "Hi Jean-Olivier,
\n[quote=Jean-Olivier Irisson;67956]
\nBut there are still things I don't understand:
\n[/quote]
\nYes - I wanted to look at how the top 10 teams compared against each other, and decided to just use precision, which is just one metric (though fairly commonly used). Since a big table of numbers can be boring/ difficult to interpret, I decided to use a graphical approach and plot anomalies relative to the mean. I computed a single value for precision per class for each team, then averaged amongst the teams, and then calculated each team's difference relative to the average.
\nScaling is by row. No scaling results in a graph where you can only see one or two high values and the rest looks like a (relative) sea of homogeny. I used the heatmap() command in the base R package, and from the source code it looks like it centers the values to the mean and then scales it by a calculated value of standard deviation.
\nI'm not sure if this was exactly your point -- but yes - the extra step I did in explicitly calculating the deviances relative to a mean precision per class, given the scaling in the plotting package, is redundant. I wasn't aware of the scaling options when I started and just went with what seemed most logical to me at the time! :)
\n[quote]
\nThat said, assuming that the general patterns are solid, this reveals interesting differences between competitors. I would be curious to know if those differences are voluntary, strategic, or just the by product of different techniques and "blind" (i.e. not directed) optimisation.
\nFor exemple, DeepSea, the best solution, is not that far from DeepseaChallenger (good name by the way!) in the final ranking, both in terms of accuracy and log-loss. However, DeepSea is good almost everywhere but quite bad for decapods, a probably quite rare group. On the other hand, DeepseaChallenger is mostly average but very very good on copepods, a very common group. Two very different ways of getting to comparable overall results!
\n[/quote]
\nYup - I read about the different techniques that some of the top 10 teams used, and was interested in seeing how they compared against each other. My initial approach overly highlights the differences between the teams, which is actually quite small -- which you can see in the attached figure. These are the raw precision values, unscaled.
\nAs you can see, echinoderms as a group were predicted fairly well, as well as salps and doliolids. Oithona copepods were pretty easy as well. Some of the hydromedusae were easier to predict than others, and the fish were all over the map. This makes sense to me -- the groups that were easier to predict were also easier for us to label. The ones that we struggled again and again to differentiate (and in some cases, just threw up our hands i.e. "unknowns") -- performed poorly. Except the detritus -- that surprised me.
\nThis goes back to what Sander and SR were saying in the other thread -- more training images are better. We now have a set of 60K training images through this dataset, but if we can get another 30K more, or 60K more, the models should be running a lot more smoothly with the real data.
\ncheers
\nJessica
", "votes": null, "replies": [] }, { "id": 67998, "author_name": "jeanolivieririsson", "author_url": "", "post_date": "03/24/2015 16:14:35", "content": "[quote=Jessica Luo;67991]
\nI used the heatmap() command in the base R package, and from the source code it looks like it centers the values to the mean and then scales it by a calculated value of standard deviation.
\n[/quote]
\n\n?heatmap mentions dendrograms and plenty of other complex stuff. Computing and scaling the data manually is easy enough and would allow you to control what is displayed. If everything is scaled by the variance also, that makes it difficult to compare between classes (rows) actually. Would you mind sharing the data (precision per team per class)?
\n[quote]
\nI'm not sure if this was exactly your point -- but yes - the extra step I did in explicitly calculating the deviances relative to a mean precision per class, given the scaling in the plotting package, is redundant.
\n[/quote]
\nIt was not my point but indeed, it seems redundant now ;)
\n\nAll your other points indeed show that we need better and larger training sets. Now those weekly emails from prospective interns will come in handy!
", "votes": null, "replies": [] }, { "id": 68024, "author_name": "jessicaluo", "author_url": "", "post_date": "03/24/2015 17:13:18", "content": "[quote=Jean-Olivier Irisson;67998]
\nWould you mind sharing the data (precision per team per class)?
\n[/quote]
\n\nYup! Here you go - attached. Now... back to my dissertation :)
", "votes": null, "replies": [] }, { "id": 68078, "author_name": "oldufo", "author_url": "", "post_date": "03/24/2015 21:09:34", "content": "@Jessica Luo,
\nFrom heatmap it looks like if someone averages predictions of top-3..5 teams, 82% threshold will be done. Just curious :)
", "votes": null, "replies": [] }, { "id": 68309, "author_name": "gala1276881", "author_url": "", "post_date": "03/25/2015 21:17:12", "content": "Thanks Jessica,
On the subject of classes & heatmaps …
As part of DeepSea Challenger team (#5), we wanted to better understand the individual class behavior as shown in the attachments.
The plot with all the various shades of blue is a visual representation of a confusion matrix summarizing the real class id data (the rows) vs the predicted class id data (the columns). For example, a perfect match would be for a dark blue diagonal (100% correlation between real and predicted class values) and blank squares (zero correlation) everywhere else. Our models had problems distinguishing between things like:
55,fish_larvae_very_thin_body (average real class id)
11,chaetognath_other (average predicted id)
If you take a look at these images in the train sets you’ll understand why.
Interestingly, not all the classes that we confused behaved symmetrically e.g. average real id = 11 vs average predicted id = 55 … doesn’t show up as dark blue (although this is partly due to the averaging technique we used).
The big purple squares are used to emphasize a very basic class hierarchy - this hierarchy is simply based on shared class prefixes e.g. the three last class ids for the classes that begin with "unknown" (118-120).
A while ago I also posted a plot for all the miss stats and it contains 3 charts for: (i) total counts (ii) miss counts (iii) miss percentages – all as a function of class id (121 plankton classes, alphabetically sorted). For each image, a “hit” is attributed to a class only if the highest probability class prediction matches the actual class in the training set (last layer is CNN softmax).
The charts were created using 80% of the train data on an older iteration of our models but towards the end we still had most problems with “other” and “unknown” classes.
Thanks for hosting this very interesting competition and in particular to Sander and his team for sharing their solution methods – well done!
[quote=Gal A.;68309]
\nOur models had problems distinguishing between things like:
55,fish_larvae_very_thin_body (average real class id)
11,chaetognath_other (average predicted id)
If you take a look at these images in the train sets you’ll understand why.
[/quote]
\n\nWhat if we told you that the fish larvae vs. chaetognath distinction is one of the most important distinctions that we need to make for our research? Chaetognaths are really abundant plankton predators, but completely unimportant socioeconomically, and fish larvae are, well, baby fish that can be hugely important to society. Chaetognaths and fish larvae look totally different in a regular plankton sample, but in our images, they look surprisingly similar -- particularly these two groups: the poorly resolved "other" chaetognaths, and the very thin bodied fish larvae. But, there are at least 1-2 orders of magnitude more chaetognaths than fish larvae in the ocean, and we need a way to detect the fish larvae without having chaetognaths pollute our fish groups. And similarly, we need to not be sorting through millions and millions of chaetognath images to find just a few fish larvae.
\n\nOne example is from a SVM model that we developed for a cruise a few years back. About 500K "fish" were detected out of about 160M images, but only about 25K were actually fish. In contrast, there were 7M chaetognaths detected, and if even 1% of those images were fish, that would mean 70K additional fish, or twice as much of the true positives in the fish categories. Some poor schmuck (... grad student...) would have to sort through all the low probability chaetognaths to find those fish, because well, the fish were the focal species of the whole study. That grad student might still be bitter at the task and ... then just found a kaggle competition to run. :)
", "votes": null, "replies": [] }, { "id": 68387, "author_name": "gala1276881", "author_url": "", "post_date": "03/26/2015 08:05:31", "content": "Hi Jessica,
As a plankton-ignoramus, I like to let the data science tools uncover these connections :)
A couple of ways to get around the sampling problems that you mention, let’s say you have just two classes with 9000 type A and 1000 type B training images:
1. If you were to under-sample the majority class (type A) and provide competitors with 1000 type A and 1000 type B images, and at the same time declare that the ratio in the population is 9:1 (i.e. provide all the class priors to the best of your knowledge) then that could have saved you some valuable resources in the collection process. This assumes that the 8000 type B images being discarded are essentially "more of the same" information (although we don't know the real impact of the discarded information without experimenting). Maybe you can devise a special capture procedure at locations where the rarer fish concentrations are known to exist and aim for disproportionate class samples to begin with (over-sample the rare classes) - as long as you also provide the "real" class priors then that should be fine.
2. For the competition, I understand that you sampled the population uniformly (but did you consider stratified sampling in order to get consistently sized samples?). In any case, the researcher can then decide to either over-sample the minority classes (like type B) or under-sample the majority classes (like type A) in order to rebalance the training sets.
Alexandre Dalyac and his colleagues give a good account of their experiments using CNNs (Chapter 6 on Tackling Class Imbalance):
\nhttp://www.academia.edu/8472416/Tackling_Class_Imbalance_with_Deep_Convolutional_Neural_Networks
\nThey say they had much more success with under-sampling (p44):
\n“The results show that under-sampling is very effective: the model does not get stuck in any seeming critical point, obtains an additional 15 percentage points in classification accuracy, to obtain 80.8% validation accuracy. This is substantial and shows how strikingly detrimental class imbalance is: we have eliminated 58.7% of the data to get rid of it, and yet this sacrifice is highly worthwhile. This sets a high benchmark for other techniques for tackling class imbalance. Since we are throwing data, it also implies that if there exists a technique for solving class imbalance without losing data, performance could be even higher.”
", "votes": null, "replies": [] }, { "id": 69228, "author_name": "jessicaluo", "author_url": "", "post_date": "03/31/2015 22:57:51", "content": "Hi Gal,
\n\nThank you for this suggestion on how to deal with class imbalance. We actually did use a mixed approach in selecting the images for this competition - a completely uniform / even sampling of the classes would have resulted in 4-5 order of magnitude differences between the most abundant group and the least abundant group. So we strove for a balance between non-selective and highly selective sampling.
\n\nIt's great to see this approach for providing a "prior" for the class distributions. I would actually be interested in seeing how this performs under more 'extreme' imbalances (so - instead of 9:1, how about 9999:1?) -- these cases are actually fairly common in the natural world.
\n\ncheers,
\n\nJessica
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "67763": "", "67773": "", "67909": "", "67956": "", "67991": "", "67998": "", "68024": "", "68078": "", "68309": "", "68342": "", "68387": "", "69228": "" }, "source": "meta" }