Kaggle thinks there is something fishy going on if your leaderboard score exceeds 0.99514 (as my 0.99671 does, which is why I have adopted the team name ιχθυς). But this threshold must have been created at a time when the technology was not as advanced as it is today. I really didn't do anything particularly fishy, and it seems to me I've only scratched the surface of the problem.
\n\nThere's a couple of slightly fishy things, maybe. First, I'm technically a re-entrant, since I made 6 submissions a couple of months ago that have now rolled off the leaderboard...but my subsequent Kaggle activities have had nothing to do with image recognition, so you could hardly consider me an expert posing as a beginner. Second, my model was based on someone else's kernel that already had a high leaderboard score, so theoretically it could have been overfit to the public leaderboard by the selection process...but I don't think it was: it's not like the one I copied had a bunch of magic numbers in it; it just happened to use advanced techniques like data generation that most others didn't...and slightly different random versions of my submission get nearly equally good results.
\n\nThe main things I added to Peter Grenholm's kernel were: (1) a training run that uses the whole data set instead of reserving part for validation (since, having already run the validation, I already knew about how many epochs to run it and had no need to waste training data on validation) and (2) a GTX1080 GPU that allows me to run 72 epochs in less than half an hour, so that I could run it with a lower learning rate and get a more precise fit.
\n\nBut there's so much more to do. I could run multiple versions and take a majority vote (or, better, combine the cross-entropy values) to get a more robust result. I could combine it with other fitting techniques. (I'm kind of impressed by how far SriramanR.Krishnamurthy has gotten with a simple SVC.) I could run it overnight with a lower learning rate and more epochs. I could use annealing, like Charles Jansen. There's just so much room for improvement.
\n\nApparently this issue has come up before. I'm curious to know what a more realistic fishiness threshold would be. Looking at the leaderboard, it looks like the break between legitimate entries and truly fishy ones might be somewhere between 0.998 and 0.999 now, but who knows?
", "messages": [ { "id": 202236, "postDate": "2017-07-12T04:19:52.933Z", "content": "Kaggle thinks there is something fishy going on if your leaderboard score exceeds 0.99514 (as my 0.99671 does, which is why I have adopted the team name ιχθυς). But this threshold must have been created at a time when the technology was not as advanced as it is today. I really didn't do anything particularly fishy, and it seems to me I've only scratched the surface of the problem.
\n\nThere's a couple of slightly fishy things, maybe. First, I'm technically a re-entrant, since I made 6 submissions a couple of months ago that have now rolled off the leaderboard...but my subsequent Kaggle activities have had nothing to do with image recognition, so you could hardly consider me an expert posing as a beginner. Second, my model was based on someone else's kernel that already had a high leaderboard score, so theoretically it could have been overfit to the public leaderboard by the selection process...but I don't think it was: it's not like the one I copied had a bunch of magic numbers in it; it just happened to use advanced techniques like data generation that most others didn't...and slightly different random versions of my submission get nearly equally good results.
\n\nThe main things I added to Peter Grenholm's kernel were: (1) a training run that uses the whole data set instead of reserving part for validation (since, having already run the validation, I already knew about how many epochs to run it and had no need to waste training data on validation) and (2) a GTX1080 GPU that allows me to run 72 epochs in less than half an hour, so that I could run it with a lower learning rate and get a more precise fit.
\n\nBut there's so much more to do. I could run multiple versions and take a majority vote (or, better, combine the cross-entropy values) to get a more robust result. I could combine it with other fitting techniques. (I'm kind of impressed by how far SriramanR.Krishnamurthy has gotten with a simple SVC.) I could run it overnight with a lower learning rate and more epochs. I could use annealing, like Charles Jansen. There's just so much room for improvement.
\n\nApparently this issue has come up before. I'm curious to know what a more realistic fishiness threshold would be. Looking at the leaderboard, it looks like the break between legitimate entries and truly fishy ones might be somewhere between 0.998 and 0.999 now, but who knows?
", "rawMarkdown": "Kaggle thinks there is something fishy going on if your leaderboard score exceeds 0.99514 (as my 0.99671 does, which is why I have adopted the team name ιχθυς). But this threshold must have been created at a time when the technology was not as advanced as it is today. I really didn't do anything particularly fishy, and it seems to me I've only scratched the surface of the problem.\n\nThere's a couple of slightly fishy things, maybe. First, I'm technically a re-entrant, since I made 6 submissions a couple of months ago that have now rolled off the leaderboard...but my subsequent Kaggle activities have had nothing to do with image recognition, so you could hardly consider me an expert posing as a beginner. Second, [my model][1] was based on [someone else's kernel][2] that already had a high leaderboard score, so theoretically it could have been overfit to the public leaderboard by the selection process...but I don't think it was: it's not like the one I copied had a bunch of magic numbers in it; it just happened to use advanced techniques like data generation that most others didn't...and slightly different random versions of my submission get nearly equally good results.\n\nThe main things I added to Peter Grenholm's kernel were: (1) a training run that uses the whole data set instead of reserving part for validation (since, having already run the validation, I already knew about how many epochs to run it and had no need to waste training data on validation) and (2) a GTX1080 GPU that allows me to run 72 epochs in less than half an hour, so that I could run it with a lower learning rate and get a more precise fit.\n\nBut there's so much more to do. I could run multiple versions and take a majority vote (or, better, combine the cross-entropy values) to get a more robust result. I could combine it with other fitting techniques. (I'm kind of impressed by how far [SriramanR.Krishnamurthy][3] has gotten with a simple SVC.) I could run it overnight with a lower learning rate and more epochs. I could use annealing, like [Charles Jansen][4]. There's just so much room for improvement.\n\nApparently this issue has [come][5] up [before][6]. I'm curious to know what a more realistic fishiness threshold would be. Looking at the leaderboard, it looks like the break between legitimate entries and truly fishy ones might be somewhere between 0.998 and 0.999 now, but who knows?\n\n\n [1]: https://www.kaggle.com/aharless/fishy-keras-model/notebook\n [2]: https://www.kaggle.com/toregil/digit-recognizer/welcome-to-deep-learning-cnn-99/run/1047276\n [3]: https://www.kaggle.com/rksriram312/simple-support-vector-classifier\n [4]: https://www.kaggle.com/cjansen/grenholm-in-script-edited-lb-0-99286\n [5]: https://www.kaggle.com/c/digit-recognizer/discussion/5490\n [6]: https://www.kaggle.com/c/digit-recognizer/discussion/23999", "votes": 3 }, { "id": 217474, "postDate": "2017-08-30T17:21:05.497Z", "content": "Something else to add to the \"room for improvement\" list: test-time augmentation
", "rawMarkdown": "Something else to add to the \"room for improvement\" list: [test-time augmentation][1]\n\n\n [1]: http://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36694" }, { "id": 217421, "postDate": "2017-08-30T15:20:30.333Z", "content": "As far as I can see, the better MNIST performances are obtained with an error of 0,21%
\n\n\n\nSo I do believe that above 0.9979 is at least strange.\nThe above paper I mentioned come from 2013, does any other found something better and more recent ?
", "rawMarkdown": "As far as I can see, the better MNIST performances are obtained with an error of 0,21%\n\nhttps://www.google.it/url?sa=t&source=web&rct=j&url=https://cs.nyu.edu/~wanli/dropc/dropc.pdf&ved=0ahUKEwjEyvH6nv_VAhWlCJoKHV_cB5MQFgghMAA&usg=AFQjCNG8FyBDNLkWmdIk9FJSAOsPImBiGg\n\nSo I do believe that above 0.9979 is at least strange.\nThe above paper I mentioned come from 2013, does any other found something better and more recent ? ", "replies": [ { "id": 217468, "postDate": "2017-08-30T17:05:43Z", "content": "There is some level of error in the \"ground truth\" labels themselves, so in principle there is some theoretical limit above which higher scores are not really better and can only be obtained either by overfitting or by mimicking the mistakes of the particular human judges. I think one would want to look at variation among human judges to determine where that limit lies. Using reported machine results is a problem, though, because they are almost always out of date by the time you can find the paper. Some things that are standard practice now were still considered new in 2013. (For example, I see machine learning MOOCs from circa 2012 that don't even mention \"relu\")
", "rawMarkdown": "There is *some* level of error in the \"ground truth\" labels themselves, so in principle there is some theoretical limit above which higher scores are not really better and can only be obtained either by overfitting or by mimicking the mistakes of the particular human judges. I think one would want to look at variation among human judges to determine where that limit lies. Using reported machine results is a problem, though, because they are almost always out of date by the time you can find the paper. Some things that are standard practice now were still considered new in 2013. (For example, I see machine learning MOOCs from circa 2012 that don't even mention \"relu\")" } ] } ], "comments": [ { "id": 217474, "author_name": "Andy Harless", "author_url": "", "post_date": "2017-08-30T17:21:05.497000", "content": "Something else to add to the \"room for improvement\" list: test-time augmentation
", "votes": 0, "replies": [] }, { "id": 217421, "author_name": "AlGiLa", "author_url": "", "post_date": "2017-08-30T15:20:30.333000", "content": "As far as I can see, the better MNIST performances are obtained with an error of 0,21%
\n\n\n\nSo I do believe that above 0.9979 is at least strange.\nThe above paper I mentioned come from 2013, does any other found something better and more recent ?
", "votes": 0, "replies": [ { "id": 217468, "author_name": "Andy Harless", "author_url": "", "post_date": "2017-08-30T17:05:43", "content": "There is some level of error in the \"ground truth\" labels themselves, so in principle there is some theoretical limit above which higher scores are not really better and can only be obtained either by overfitting or by mimicking the mistakes of the particular human judges. I think one would want to look at variation among human judges to determine where that limit lies. Using reported machine results is a problem, though, because they are almost always out of date by the time you can find the paper. Some things that are standard practice now were still considered new in 2013. (For example, I see machine learning MOOCs from circa 2012 that don't even mention \"relu\")
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "202236": "Kaggle thinks there is something fishy going on if your leaderboard score exceeds 0.99514 (as my 0.99671 does, which is why I have adopted the team name ιχθυς). But this threshold must have been created at a time when the technology was not as advanced as it is today. I really didn't do anything particularly fishy, and it seems to me I've only scratched the surface of the problem.\n\nThere's a couple of slightly fishy things, maybe. First, I'm technically a re-entrant, since I made 6 submissions a couple of months ago that have now rolled off the leaderboard...but my subsequent Kaggle activities have had nothing to do with image recognition, so you could hardly consider me an expert posing as a beginner. Second, [my model][1] was based on [someone else's kernel][2] that already had a high leaderboard score, so theoretically it could have been overfit to the public leaderboard by the selection process...but I don't think it was: it's not like the one I copied had a bunch of magic numbers in it; it just happened to use advanced techniques like data generation that most others didn't...and slightly different random versions of my submission get nearly equally good results.\n\nThe main things I added to Peter Grenholm's kernel were: (1) a training run that uses the whole data set instead of reserving part for validation (since, having already run the validation, I already knew about how many epochs to run it and had no need to waste training data on validation) and (2) a GTX1080 GPU that allows me to run 72 epochs in less than half an hour, so that I could run it with a lower learning rate and get a more precise fit.\n\nBut there's so much more to do. I could run multiple versions and take a majority vote (or, better, combine the cross-entropy values) to get a more robust result. I could combine it with other fitting techniques. (I'm kind of impressed by how far [SriramanR.Krishnamurthy][3] has gotten with a simple SVC.) I could run it overnight with a lower learning rate and more epochs. I could use annealing, like [Charles Jansen][4]. There's just so much room for improvement.\n\nApparently this issue has [come][5] up [before][6]. I'm curious to know what a more realistic fishiness threshold would be. Looking at the leaderboard, it looks like the break between legitimate entries and truly fishy ones might be somewhere between 0.998 and 0.999 now, but who knows?\n\n\n [1]: https://www.kaggle.com/aharless/fishy-keras-model/notebook\n [2]: https://www.kaggle.com/toregil/digit-recognizer/welcome-to-deep-learning-cnn-99/run/1047276\n [3]: https://www.kaggle.com/rksriram312/simple-support-vector-classifier\n [4]: https://www.kaggle.com/cjansen/grenholm-in-script-edited-lb-0-99286\n [5]: https://www.kaggle.com/c/digit-recognizer/discussion/5490\n [6]: https://www.kaggle.com/c/digit-recognizer/discussion/23999", "217474": "Something else to add to the \"room for improvement\" list: [test-time augmentation][1]\n\n\n [1]: http://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36694", "217421": "As far as I can see, the better MNIST performances are obtained with an error of 0,21%\n\nhttps://www.google.it/url?sa=t&source=web&rct=j&url=https://cs.nyu.edu/~wanli/dropc/dropc.pdf&ved=0ahUKEwjEyvH6nv_VAhWlCJoKHV_cB5MQFgghMAA&usg=AFQjCNG8FyBDNLkWmdIk9FJSAOsPImBiGg\n\nSo I do believe that above 0.9979 is at least strange.\nThe above paper I mentioned come from 2013, does any other found something better and more recent ? " } }