Hi All,
\nFirst, everyone here in the Cowen-Sponaugle Research Group at Hatfield Marine Science Center would like to thank all of the participants of the National Data Science Bowl. Amazing to see what the data science community is capable of.
\nA question for our top three teams: "Deep Sea", "HappyLanternFestival" and "Possion Process" as well as the top US university team.
\nWhat is the average run-time for your top submissions?
\nBecause one of our goals is to acquire and process the plankton imagery data in near real-time, we are curious to know how close we are to reaching that goal.
\nAll the best,
\nKelly Robinson, OSU Hatfield
", "messages": [ { "id": "67806", "postDate": "03/23/2015 22:26:50", "content": "Hi All,
\nFirst, everyone here in the Cowen-Sponaugle Research Group at Hatfield Marine Science Center would like to thank all of the participants of the National Data Science Bowl. Amazing to see what the data science community is capable of.
\nA question for our top three teams: "Deep Sea", "HappyLanternFestival" and "Possion Process" as well as the top US university team.
\nWhat is the average run-time for your top submissions?
\nBecause one of our goals is to acquire and process the plankton imagery data in near real-time, we are curious to know how close we are to reaching that goal.
\nAll the best,
\nKelly Robinson, OSU Hatfield
", "rawMarkdown": "", "votes": null }, { "id": "67808", "postDate": "03/23/2015 22:45:14", "content": "It depends on what level of accuracy you want and what kind of hardware can be used. As with many things it's a story of diminishing returns: the last few percent of accuracy cost exponentially more than the first 75-80%.
\nWe used extensive model averaging to achieve our result. This means that each prediction is actually an average of many predictions from different models (or different transformations of the input). I don't have the exact numbers on hand, but a rough calculation shows that each prediction in our final submission was a weighted average of between 2000 and 3000 individual predictions per image.
\nIn addition, we used GPUs to train the models and generate the predictions. Doing this on a CPU, while possible, would take orders of magnitude longer. So using this kind of model for real-time predictions probably won't be feasible for another 10 years or so :)
\nLuckily, this complexity was only necessary to get us the Kaggle win - it's not actually necessary to achieve a good (i.e. usable) prediction accuracy. You could generate predictions using only our single best model and probably get 80% classification accuracy. In fact, I'd wager you could take any of the models in the top 10 and get good enough accuracy from them for all practical purposes. Those last few percent aren't going to make a difference in practice, but they make all the difference in the world on Kaggle! (On a somewhat related note, it might actually pay off to re-train the model to optimize classification accuracy instead of the log-loss, if that is the actual goal of the system.)
\nOn a GPU, generating a single prediction will take on the order of milliseconds - on the condition that you batch several images together and then generate predictions for them simultaneously. This is because the GPU gets its speed advantage through massive parallelism. This parallelism can only be exploited if the GPU is allowed to do the same computation on many different inputs in one go.
\nOn a CPU, generating a single prediction will probably take on the order of 100s of milliseconds - maybe seconds in the worst case. This is just a wild guess based on prior experience, I didn't actually do the calculations.
\nDepending on how fast the imagery is acquired (and needs to be processed), the single best model approach may already run fast enough on commodity hardware (i.e. CPUs). If not, good GPUs are pretty cheap nowadays, so that shouldn't be too big a hurdle.
", "rawMarkdown": "", "votes": null }, { "id": "67897", "postDate": "03/24/2015 05:42:52", "content": "Sander - Thanks a lot for your explanation! It makes sense that the actual implementation of the winning solutions will look quite a bit different. We probably won't be doing a ton of ensembling, and might just stick with a single best model and then try to parallelize it so we can run it on a cluster. A lot more work to be done, for sure.
\nA side question - in your team's solution (and also quite a few other solutions) you guys mentioned data augmentation. (But also that CNNs did not appear to need the quantity of data that you guys originally expected - interesting.) We actually struggled quite a bit with the issue of the dataset size for this competition. Early on, we were told that we only needed 100 images per class to train a classifier, but I was skeptical of that number -- given the diversity of the orientations and morphologies of some of these groups. So we ended up upping that target number significantly to ~100 for the rarer classes and 500-1000 for the abundant classes (though in biological systems, species abundance curves tend to fall along a negative binomial distribution - so some groups just had 20-30 total images, much to your collective chagrin, I'm sure).
\nSo I'm wondering -- would anyone care to share their impressions (or data) on what size of a training set is necessary for their models to run? Obviously you can't know how your models would perform if our training dataset were larger, but roughly -- what size would be ideal for these CNNs to perform best? Alternatively, what happened when you trained on less than the full training set (25% / 50% / 75%)? Was there a threshold beyond which the models could not converge, or the accuracy/logloss score dropped significantly?
\nAll this information is extremely valuable for us, as we probably have ~150 - 200 million images in our current dataset to process, and will go collect another 200 million more in a few more months. Labeling images for a training set is extremely time-consuming, so is it worth our effort and manpower to try to increase the size of the training set for the next run? Or is data augmentation going to be roughly ok? What about in the future -- would a smaller training set also work?
\nThanks in advance for your comments and insights!
\nJessica
", "rawMarkdown": "", "votes": null }, { "id": "67899", "postDate": "03/24/2015 06:35:50", "content": "Data augmentation was instrumental to train the kind of models we trained on this dataset, because the dataset of 30k examples is relatively small by modern standards. CNNs are known to scale extremely well with dataset size, so I would guess that labeling additional data will probably pay off considerably (especially for the smaller classes).
\nCleaning up the labels may also help: we noticed some examples in the training set were mislabeled, as well as a lot of confusion with the 'unknown' and 'other' classes - it seems that the classes overlap quite a bit. The log loss metric is pretty sensitive to these things.
\nWe never tried training models with a smaller subset of the training set (we always used 90% for training and 10% for evaluation), but I would guess that we'd see much more overfitting and would need to increase regularization further (i.e. more data augmentation, smaller models, more weight decay, more dropout, ...).
\nI appreciate that labeling images is very tedious and time-consuming, but I think labeling even just 30k more will already result in a considerable accuracy improvement.
\nAs we mentioned in our blog post, we found that our current model is able to achieve ~98% accuracy if it is allowed 5 guesses - in other words, the correct class is amongst the top 5 most probable predictions 98% of the time. This means you could use the model predictions to build a tool for speeding up the labeling process.
\nI don't know what the current workflow looks like, but I imagine experts just sit and stare at images a lot, and then look at the taxonomy tree to determine which of 121 classes to pick. Using our model, you could initially present the labeler with 5 options and an "other..." button, and 98% of the time they will just have to pick which of those 5 options is the most correct one. This could speed up the manual labeling process considerably.
\nIt could turn into some kind of feedback loop: you could label 30k more images using this process, and then maybe the next model you train with this bigger dataset is able to achieve 99% accuracy, allowing you to label the next 30k even faster, ... and so on :)
\nAnother nice property of models that are trained to optimize the log loss, is that the entropy of the class distributions they predict can be used as a measure of confidence (alternatively you could just use the probability assigned to the most likely class as a measure of confidence as well).
\nConfident predictions are more likely to be correct. This means that you could set some kind of confidence threshold and treat all labels the model is confident about as truth. It will probably still make some mistakes, but if the threshold is high enough, it might be on par with a human labeler. This could considerably reduce the size of the set that requires manual attention.
", "rawMarkdown": "", "votes": null }, { "id": "67922", "postDate": "03/24/2015 08:49:23", "content": "Hi everyone,
\nI'm Jean-Olivier Irisson, from the Ocean Observatory of Villefranche-sur-Mer. We used ISIIS in the past with Bob, Jessica, et al. (more at http://puf.rsmas.obs-vlfr.fr/tags/visufront/). We have ~25Tb of images waiting for your solutions so thanks a lot for the effort!
\nTo give you an idea, ISIIS generated an average of about 150 particle images per second in our quite poor Mediterranean waters. Most of them are detritus or physical noise (but still need to be classified!). I don't know if you have seen full ISIIS frames so let me link to examples of a typical frame (segmented particles highlighted in red), a noisy one (with some kind pre-filter to not segment most of the noise), and a frame very rich in organisms. Each of those correspond to ~0.7s of ISIIS imaging. We have days worth of this!
\nProcessing and segmenting the images is done with various solutions at present, none of them being real time (mine is about 10x real time, but it is beginner-level python code: https://github.com/jiho/apeep). Yet, this is typically done once. Training and prediction will be done multiple times, and probably iteratively as sedielem suggests above (start from a small training set, predict, manually validate the result of the prediction, use that as a new training set, etc.).
\nWe also thought about the use of the probability in the chosen class as an indicator of the confidence in prediction. Actually, we have used this with predictions from a Random Forest model based on ~40 image features and we get very good results without throwing too much of the data away, so I suspect this strategy will be even better with your solutions.
\nIn Villefranche, we have ~1.5 million particles which class has been checked by a human (in ~40, general classes). I suspect Jessica et al have about the same at OSU. So there is room for much larger training sets without additional effort (which may even alleviate the need for data augmentation).
\nAs a side note, we are building a machine here which will be used for the processing and classification of ISIIS images. From the various answers and my experience, I figure that I/O and GPU performance are the two main points to consider. Other aspects?
\nDo you have recommandations in terms of I/O? What do you use for real-life, large datasets? (the thing is that SSDs are kind of out of the question for ~25Tb worth of images...).
\nIn terms of GPU, the choices (and prices) we have for a Dell server are:
\nDo you have a (reasonably priced) recommendation?
\nThanks again for all the work. Now I need to learn about CNNs ;)
", "rawMarkdown": "", "votes": null }, { "id": "67946", "postDate": "03/24/2015 11:44:20", "content": "As a reference point I think AlexNet is a good standard. Per hardware being used, there was a lot of optimization work done.
\n\nI got AUC of 0.662760 (public LB) on a single slightly modified AlexNet (I added additional image features almost identical to sedielem.) I used 8 times TTA so you have to multiple by 8 the time it takes to make a single standard AlexNet prediction.
\n\nTo train this model I used pseudo-labeling again almost identical to sedielem, so re-training over the entire data set (or even more data) will give an even better network for the same prediction run-time.
", "rawMarkdown": "", "votes": null }, { "id": "67957", "postDate": "03/24/2015 12:59:20", "content": "In our case, model validation results when training on 80% of labels and testing on rest 20% were about 0.035-0.04 worse than LB scores of models trained on 100% labels, which may give some idea of the potential improvement for increased training set.
", "rawMarkdown": "", "votes": null }, { "id": "67980", "postDate": "03/24/2015 15:04:22", "content": "[quote=Jean-Olivier Irisson;67922]
\nIn Villefranche, we have ~1.5 million particles which class has been checked by a human (in ~40, general classes). I suspect Jessica et al have about the same at OSU. So there is room for much larger training sets without additional effort (which may even alleviate the need for data augmentation).
\n[/quote]That is great news. I expect that there is still a ton of room for improvement simply by scaling up the dataset.
\n[quote]As a side note, we are building a machine here which will be used for the processing and classification of ISIIS images. From the various answers and my experience, I figure that I/O and GPU performance are the two main points to consider. Other aspects?[/quote]
\nI think those are the main two things. Tim Dettmers has a very nice "hardware guide" for deep learning: https://timdettmers.wordpress.com/2015/03/09/deep-learning-hardware-guide/
\nWorth a read! The only thing I disagree with is his recommendation to get a second-hand GTX 580 if you're short on cash. While these cards are fast and cheap, they do not support the cuDNN library because the architecture is too old. As more and more deep learning libraries are starting to rely on cuDNN, I think it's a bad idea get a card that isn't able to run it. A 680, while not much faster, will at least run cuDNN so that's a better bet in that market segment :)
\n[quote]Do you have recommandations in terms of I/O? What do you use for real-life, large datasets? (the thing is that SSDs are kind of out of the question for ~25Tb worth of images...).[/quote]You only need to be able to load the data as fast as the model can process it, so I guess there is some leeway there, at least for training the models. For prediction generation, I guess I/O will end up being the bottleneck.
\nYou could also set up some kind of "reservoir sampling" approach, where a subset of the data is copied to an SSD for fast training, and this subset is periodically refreshed.
\n[quote]In terms of GPU, the choices (and prices) we have for a Dell server are:
\nDo you have a (reasonably priced) recommendation?[/quote]From these options, the K10, K20M, K20X and K40M are going to be the best match for deep learning. AMD or Xeon Phi-based solutions aren't a good idea at this point because NVIDIA has pretty much cornered the market. Maybe that will change at some point, but I think right now everyone is heavily invested in NVIDIA hardware and cuDNN.
\nThe most cost-effective option would actually be to go out and buy a gamer GPU (or multiple) and insert that into the server instead. A GeForce 780Ti will give you better performance than a Tesla K40M (it has less video RAM, but that shouldn't be a huge issue) at a fraction of the price. A GeForce 980 is currently the best option price-performance wise and will set you back about EUR 550 a piece.
\nOr if you want the fastest of the fastest, the just-released GeForce Titan X is what you want, I believe that one is in the EUR 1100-1300 range.
\nEDIT:
\n[quote=zzspar;67946]
\nTo train this model I used pseudo-labeling again almost identical to sedielem, so re-training over the entire data set (or even more data) will give an even better network for the same prediction run-time.[/quote]This is an excellent point that I neglected to mention previously. We found that our models trained with pseudo-labeling get ~80% of the performance of the ensemble using only the test-time augmentation (TTA) averaging. This agrees with the results in Hinton et al.'s "knowledge distillation" paper. So it would allow for a very fast prediction model which is only slightly less accurate than the full ensemble we submitted to Kaggle.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 67808, "author_name": "sedielem", "author_url": "", "post_date": "03/23/2015 22:45:14", "content": "It depends on what level of accuracy you want and what kind of hardware can be used. As with many things it's a story of diminishing returns: the last few percent of accuracy cost exponentially more than the first 75-80%.
\nWe used extensive model averaging to achieve our result. This means that each prediction is actually an average of many predictions from different models (or different transformations of the input). I don't have the exact numbers on hand, but a rough calculation shows that each prediction in our final submission was a weighted average of between 2000 and 3000 individual predictions per image.
\nIn addition, we used GPUs to train the models and generate the predictions. Doing this on a CPU, while possible, would take orders of magnitude longer. So using this kind of model for real-time predictions probably won't be feasible for another 10 years or so :)
\nLuckily, this complexity was only necessary to get us the Kaggle win - it's not actually necessary to achieve a good (i.e. usable) prediction accuracy. You could generate predictions using only our single best model and probably get 80% classification accuracy. In fact, I'd wager you could take any of the models in the top 10 and get good enough accuracy from them for all practical purposes. Those last few percent aren't going to make a difference in practice, but they make all the difference in the world on Kaggle! (On a somewhat related note, it might actually pay off to re-train the model to optimize classification accuracy instead of the log-loss, if that is the actual goal of the system.)
\nOn a GPU, generating a single prediction will take on the order of milliseconds - on the condition that you batch several images together and then generate predictions for them simultaneously. This is because the GPU gets its speed advantage through massive parallelism. This parallelism can only be exploited if the GPU is allowed to do the same computation on many different inputs in one go.
\nOn a CPU, generating a single prediction will probably take on the order of 100s of milliseconds - maybe seconds in the worst case. This is just a wild guess based on prior experience, I didn't actually do the calculations.
\nDepending on how fast the imagery is acquired (and needs to be processed), the single best model approach may already run fast enough on commodity hardware (i.e. CPUs). If not, good GPUs are pretty cheap nowadays, so that shouldn't be too big a hurdle.
", "votes": null, "replies": [] }, { "id": 67897, "author_name": "jessicaluo", "author_url": "", "post_date": "03/24/2015 05:42:52", "content": "Sander - Thanks a lot for your explanation! It makes sense that the actual implementation of the winning solutions will look quite a bit different. We probably won't be doing a ton of ensembling, and might just stick with a single best model and then try to parallelize it so we can run it on a cluster. A lot more work to be done, for sure.
\nA side question - in your team's solution (and also quite a few other solutions) you guys mentioned data augmentation. (But also that CNNs did not appear to need the quantity of data that you guys originally expected - interesting.) We actually struggled quite a bit with the issue of the dataset size for this competition. Early on, we were told that we only needed 100 images per class to train a classifier, but I was skeptical of that number -- given the diversity of the orientations and morphologies of some of these groups. So we ended up upping that target number significantly to ~100 for the rarer classes and 500-1000 for the abundant classes (though in biological systems, species abundance curves tend to fall along a negative binomial distribution - so some groups just had 20-30 total images, much to your collective chagrin, I'm sure).
\nSo I'm wondering -- would anyone care to share their impressions (or data) on what size of a training set is necessary for their models to run? Obviously you can't know how your models would perform if our training dataset were larger, but roughly -- what size would be ideal for these CNNs to perform best? Alternatively, what happened when you trained on less than the full training set (25% / 50% / 75%)? Was there a threshold beyond which the models could not converge, or the accuracy/logloss score dropped significantly?
\nAll this information is extremely valuable for us, as we probably have ~150 - 200 million images in our current dataset to process, and will go collect another 200 million more in a few more months. Labeling images for a training set is extremely time-consuming, so is it worth our effort and manpower to try to increase the size of the training set for the next run? Or is data augmentation going to be roughly ok? What about in the future -- would a smaller training set also work?
\nThanks in advance for your comments and insights!
\nJessica
", "votes": null, "replies": [] }, { "id": 67899, "author_name": "sedielem", "author_url": "", "post_date": "03/24/2015 06:35:50", "content": "Data augmentation was instrumental to train the kind of models we trained on this dataset, because the dataset of 30k examples is relatively small by modern standards. CNNs are known to scale extremely well with dataset size, so I would guess that labeling additional data will probably pay off considerably (especially for the smaller classes).
\nCleaning up the labels may also help: we noticed some examples in the training set were mislabeled, as well as a lot of confusion with the 'unknown' and 'other' classes - it seems that the classes overlap quite a bit. The log loss metric is pretty sensitive to these things.
\nWe never tried training models with a smaller subset of the training set (we always used 90% for training and 10% for evaluation), but I would guess that we'd see much more overfitting and would need to increase regularization further (i.e. more data augmentation, smaller models, more weight decay, more dropout, ...).
\nI appreciate that labeling images is very tedious and time-consuming, but I think labeling even just 30k more will already result in a considerable accuracy improvement.
\nAs we mentioned in our blog post, we found that our current model is able to achieve ~98% accuracy if it is allowed 5 guesses - in other words, the correct class is amongst the top 5 most probable predictions 98% of the time. This means you could use the model predictions to build a tool for speeding up the labeling process.
\nI don't know what the current workflow looks like, but I imagine experts just sit and stare at images a lot, and then look at the taxonomy tree to determine which of 121 classes to pick. Using our model, you could initially present the labeler with 5 options and an "other..." button, and 98% of the time they will just have to pick which of those 5 options is the most correct one. This could speed up the manual labeling process considerably.
\nIt could turn into some kind of feedback loop: you could label 30k more images using this process, and then maybe the next model you train with this bigger dataset is able to achieve 99% accuracy, allowing you to label the next 30k even faster, ... and so on :)
\nAnother nice property of models that are trained to optimize the log loss, is that the entropy of the class distributions they predict can be used as a measure of confidence (alternatively you could just use the probability assigned to the most likely class as a measure of confidence as well).
\nConfident predictions are more likely to be correct. This means that you could set some kind of confidence threshold and treat all labels the model is confident about as truth. It will probably still make some mistakes, but if the threshold is high enough, it might be on par with a human labeler. This could considerably reduce the size of the set that requires manual attention.
", "votes": null, "replies": [] }, { "id": 67922, "author_name": "jeanolivieririsson", "author_url": "", "post_date": "03/24/2015 08:49:23", "content": "Hi everyone,
\nI'm Jean-Olivier Irisson, from the Ocean Observatory of Villefranche-sur-Mer. We used ISIIS in the past with Bob, Jessica, et al. (more at http://puf.rsmas.obs-vlfr.fr/tags/visufront/). We have ~25Tb of images waiting for your solutions so thanks a lot for the effort!
\nTo give you an idea, ISIIS generated an average of about 150 particle images per second in our quite poor Mediterranean waters. Most of them are detritus or physical noise (but still need to be classified!). I don't know if you have seen full ISIIS frames so let me link to examples of a typical frame (segmented particles highlighted in red), a noisy one (with some kind pre-filter to not segment most of the noise), and a frame very rich in organisms. Each of those correspond to ~0.7s of ISIIS imaging. We have days worth of this!
\nProcessing and segmenting the images is done with various solutions at present, none of them being real time (mine is about 10x real time, but it is beginner-level python code: https://github.com/jiho/apeep). Yet, this is typically done once. Training and prediction will be done multiple times, and probably iteratively as sedielem suggests above (start from a small training set, predict, manually validate the result of the prediction, use that as a new training set, etc.).
\nWe also thought about the use of the probability in the chosen class as an indicator of the confidence in prediction. Actually, we have used this with predictions from a Random Forest model based on ~40 image features and we get very good results without throwing too much of the data away, so I suspect this strategy will be even better with your solutions.
\nIn Villefranche, we have ~1.5 million particles which class has been checked by a human (in ~40, general classes). I suspect Jessica et al have about the same at OSU. So there is room for much larger training sets without additional effort (which may even alleviate the need for data augmentation).
\nAs a side note, we are building a machine here which will be used for the processing and classification of ISIIS images. From the various answers and my experience, I figure that I/O and GPU performance are the two main points to consider. Other aspects?
\nDo you have recommandations in terms of I/O? What do you use for real-life, large datasets? (the thing is that SSDs are kind of out of the question for ~25Tb worth of images...).
\nIn terms of GPU, the choices (and prices) we have for a Dell server are:
\nDo you have a (reasonably priced) recommendation?
\nThanks again for all the work. Now I need to learn about CNNs ;)
", "votes": null, "replies": [] }, { "id": 67946, "author_name": "udibr1", "author_url": "", "post_date": "03/24/2015 11:44:20", "content": "As a reference point I think AlexNet is a good standard. Per hardware being used, there was a lot of optimization work done.
\n\nI got AUC of 0.662760 (public LB) on a single slightly modified AlexNet (I added additional image features almost identical to sedielem.) I used 8 times TTA so you have to multiple by 8 the time it takes to make a single standard AlexNet prediction.
\n\nTo train this model I used pseudo-labeling again almost identical to sedielem, so re-training over the entire data set (or even more data) will give an even better network for the same prediction run-time.
", "votes": null, "replies": [] }, { "id": 67957, "author_name": "shaironen", "author_url": "", "post_date": "03/24/2015 12:59:20", "content": "In our case, model validation results when training on 80% of labels and testing on rest 20% were about 0.035-0.04 worse than LB scores of models trained on 100% labels, which may give some idea of the potential improvement for increased training set.
", "votes": null, "replies": [] }, { "id": 67980, "author_name": "sedielem", "author_url": "", "post_date": "03/24/2015 15:04:22", "content": "[quote=Jean-Olivier Irisson;67922]
\nIn Villefranche, we have ~1.5 million particles which class has been checked by a human (in ~40, general classes). I suspect Jessica et al have about the same at OSU. So there is room for much larger training sets without additional effort (which may even alleviate the need for data augmentation).
\n[/quote]That is great news. I expect that there is still a ton of room for improvement simply by scaling up the dataset.
\n[quote]As a side note, we are building a machine here which will be used for the processing and classification of ISIIS images. From the various answers and my experience, I figure that I/O and GPU performance are the two main points to consider. Other aspects?[/quote]
\nI think those are the main two things. Tim Dettmers has a very nice "hardware guide" for deep learning: https://timdettmers.wordpress.com/2015/03/09/deep-learning-hardware-guide/
\nWorth a read! The only thing I disagree with is his recommendation to get a second-hand GTX 580 if you're short on cash. While these cards are fast and cheap, they do not support the cuDNN library because the architecture is too old. As more and more deep learning libraries are starting to rely on cuDNN, I think it's a bad idea get a card that isn't able to run it. A 680, while not much faster, will at least run cuDNN so that's a better bet in that market segment :)
\n[quote]Do you have recommandations in terms of I/O? What do you use for real-life, large datasets? (the thing is that SSDs are kind of out of the question for ~25Tb worth of images...).[/quote]You only need to be able to load the data as fast as the model can process it, so I guess there is some leeway there, at least for training the models. For prediction generation, I guess I/O will end up being the bottleneck.
\nYou could also set up some kind of "reservoir sampling" approach, where a subset of the data is copied to an SSD for fast training, and this subset is periodically refreshed.
\n[quote]In terms of GPU, the choices (and prices) we have for a Dell server are:
\nDo you have a (reasonably priced) recommendation?[/quote]From these options, the K10, K20M, K20X and K40M are going to be the best match for deep learning. AMD or Xeon Phi-based solutions aren't a good idea at this point because NVIDIA has pretty much cornered the market. Maybe that will change at some point, but I think right now everyone is heavily invested in NVIDIA hardware and cuDNN.
\nThe most cost-effective option would actually be to go out and buy a gamer GPU (or multiple) and insert that into the server instead. A GeForce 780Ti will give you better performance than a Tesla K40M (it has less video RAM, but that shouldn't be a huge issue) at a fraction of the price. A GeForce 980 is currently the best option price-performance wise and will set you back about EUR 550 a piece.
\nOr if you want the fastest of the fastest, the just-released GeForce Titan X is what you want, I believe that one is in the EUR 1100-1300 range.
\nEDIT:
\n[quote=zzspar;67946]
\nTo train this model I used pseudo-labeling again almost identical to sedielem, so re-training over the entire data set (or even more data) will give an even better network for the same prediction run-time.[/quote]This is an excellent point that I neglected to mention previously. We found that our models trained with pseudo-labeling get ~80% of the performance of the ensemble using only the test-time augmentation (TTA) averaging. This agrees with the results in Hinton et al.'s "knowledge distillation" paper. So it would allow for a very fast prediction model which is only slightly less accurate than the full ensemble we submitted to Kaggle.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "67806": "", "67808": "", "67897": "", "67899": "", "67922": "", "67946": "", "67957": "", "67980": "" }, "source": "meta" }![typical frame](\"https://www.dropbox.com/s/ru6r9xcc131ftqr/20130726235329_709996_mask.png?dl=0\"){kind=link}
![noisy one](\"https://www.dropbox.com/s/d1dc0qulj8smcyv/20130726233552_560756_mask.png?dl=0\"){kind=link}
![frame very rich in organisms](\"https://www.dropbox.com/s/3hdj1g7ji8l5da0/20130726232035_235964_mask.png?dl=0\"){kind=link}