I am seeing a lot of variations of the same problem in the top scoring notebooks:
\n(X_train_mnist, y_train_mnist), (X_val_mnist, y_val_mnist) = mnist.load_data()
kaggle uses a training/test split of 42k/28k whereas this dataset downloaded from keras has a 60k/10k split. This means you will be using some of kaggles test data to train your algorithm, which artificially improves your score because your algorithm is just memorising a lot of the right answers.
\nDon't download external datasets for this competition - you just don't need to. The point of this competition isn't to get 100% (you can just find the answers online), the point is for you to learn how to build better models by improving your accuracy score over time.
", "messages": [ { "id": 1623434, "postDate": "2021-12-19T22:09:14.253Z", "content": "I am seeing a lot of variations of the same problem in the top scoring notebooks:
\n(X_train_mnist, y_train_mnist), (X_val_mnist, y_val_mnist) = mnist.load_data()
kaggle uses a training/test split of 42k/28k whereas this dataset downloaded from keras has a 60k/10k split. This means you will be using some of kaggles test data to train your algorithm, which artificially improves your score because your algorithm is just memorising a lot of the right answers.
\nDon't download external datasets for this competition - you just don't need to. The point of this competition isn't to get 100% (you can just find the answers online), the point is for you to learn how to build better models by improving your accuracy score over time.
", "rawMarkdown": "I am seeing a lot of variations of the same problem in the top scoring notebooks:\n\n`(X_train_mnist, y_train_mnist), (X_val_mnist, y_val_mnist) = mnist.load_data()`\n\nkaggle uses a training/test split of 42k/28k whereas this dataset downloaded from keras has a 60k/10k split. This means you will be using some of kaggles test data to train your algorithm, which artificially improves your score because your algorithm is just memorising a lot of the right answers.\n\nDon't download external datasets for this competition - you just don't need to. The point of this competition isn't to get 100% (you can just find the answers online), the point is for you to learn how to build better models by improving your accuracy score over time.", "votes": 56 }, { "id": 2316417, "postDate": "2023-06-24T22:11:28.800Z", "content": "Some of the images in the dataset is so bizarre, even classification by the human eye is tough. We can't expect machines to make this guess and get the accuracy up to 100%. This is ridiculous and almost never happens
", "rawMarkdown": "Some of the images in the dataset is so bizarre, even classification by the human eye is tough. We can't expect machines to make this guess and get the accuracy up to 100%. This is ridiculous and almost never happens", "votes": 1 }, { "id": 2142407, "postDate": "2023-02-13T13:59:49.677Z", "content": "exactly, the goal is to get better accuracy, by making better models! many people just get average score and not focus on what the competition is all about !
", "rawMarkdown": "exactly, the goal is to get better accuracy, by making better models! many people just get average score and not focus on what the competition is all about !", "votes": 1 }, { "id": 1632999, "postDate": "2021-12-30T09:56:34.977Z", "content": "Actually, that is why many submissions reach an incredible score. This is very well described here, where it is explained that 0.9979 is the max achievable score (without cheating).
", "rawMarkdown": "Actually, that is why many submissions reach an incredible score. This is very well described [here](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist/notebook), where it is explained that 0.9979 is the max achievable score (without cheating). ", "votes": 3 }, { "id": 1638057, "postDate": "2022-01-04T12:09:16.540Z", "content": "I was wondering why so many people can reach 100% accuracy. I can only get 99% so far, which I have made a lot of tweak on my notebook.
", "rawMarkdown": "I was wondering why so many people can reach 100% accuracy. I can only get 99% so far, which I have made a lot of tweak on my notebook. ", "votes": 2 }, { "id": 1698479, "postDate": "2022-02-20T12:10:10.937Z", "content": "well said. People should use caution while using training and test data
", "rawMarkdown": "well said. People should use caution while using training and test data", "votes": 1 }, { "id": 1791022, "postDate": "2022-05-15T15:22:18.397Z", "content": "Good point! Interested to see how those models would perform without using external datasets π€
", "rawMarkdown": "Good point! Interested to see how those models would perform without using external datasets π€" }, { "id": 1769771, "postDate": "2022-04-27T15:00:47.260Z", "content": "Very good point and great catch on the subtle differences of the training data being used. I also agree that the point of this competition is meant for learning and improvement rather than just achieving the best accuracy possible. I must rather place where I place than get an artificially high score without understanding why or how I got there. The skill of learning how to tune and optimize a model is much more valuable in my opinion, especially in a world where we have new datasets being generated every second of every day.
", "rawMarkdown": "Very good point and great catch on the subtle differences of the training data being used. I also agree that the point of this competition is meant for learning and improvement rather than just achieving the best accuracy possible. I must rather place where I place than get an artificially high score without understanding why or how I got there. The skill of learning how to tune and optimize a model is much more valuable in my opinion, especially in a world where we have new datasets being generated every second of every day." }, { "id": 1728282, "postDate": "2022-03-18T18:26:09.003Z", "content": "Very well said π
", "rawMarkdown": "Very well said π" }, { "id": 1722946, "postDate": "2022-03-15T01:51:41.197Z", "content": "I saw lots o people using external datasets (on kaggle or outside) which feels so unfair, i guess you're supposed to use the provided dataβ¦
", "rawMarkdown": "I saw lots o people using external datasets (on kaggle or outside) which feels so unfair, i guess you're supposed to use the provided data..." }, { "id": 1718744, "postDate": "2022-03-11T06:08:54.207Z", "content": "Well pointed.
", "rawMarkdown": "Well pointed." }, { "id": 1705842, "postDate": "2022-02-26T21:54:43.873Z", "content": "This explains alot
", "rawMarkdown": "This explains alot" }, { "id": 1631717, "postDate": "2021-12-28T18:01:27.803Z", "content": "That is interesting
", "rawMarkdown": "That is interesting" }, { "id": 1625357, "postDate": "2021-12-21T17:42:23.933Z", "content": "Good point
", "rawMarkdown": "Good point" }, { "id": 1850269, "postDate": "2022-07-10T09:09:20.397Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2316417, "author_name": "Niveditha Vudayagiri", "author_url": "", "post_date": "2023-06-24T22:11:28.800000", "content": "Some of the images in the dataset is so bizarre, even classification by the human eye is tough. We can't expect machines to make this guess and get the accuracy up to 100%. This is ridiculous and almost never happens
", "votes": 1, "replies": [] }, { "id": 2142407, "author_name": "SHYAM GUPTA", "author_url": "", "post_date": "2023-02-13T13:59:49.677000", "content": "exactly, the goal is to get better accuracy, by making better models! many people just get average score and not focus on what the competition is all about !
", "votes": 1, "replies": [] }, { "id": 1632999, "author_name": "Vicente Pallares", "author_url": "", "post_date": "2021-12-30T09:56:34.977000", "content": "Actually, that is why many submissions reach an incredible score. This is very well described here, where it is explained that 0.9979 is the max achievable score (without cheating).
", "votes": 3, "replies": [] }, { "id": 1638057, "author_name": "Richardjl Jiang", "author_url": "", "post_date": "2022-01-04T12:09:16.540000", "content": "I was wondering why so many people can reach 100% accuracy. I can only get 99% so far, which I have made a lot of tweak on my notebook.
", "votes": 2, "replies": [] }, { "id": 1698479, "author_name": "Hemshanker Raval", "author_url": "", "post_date": "2022-02-20T12:10:10.937000", "content": "well said. People should use caution while using training and test data
", "votes": 1, "replies": [] }, { "id": 1791022, "author_name": "Murilo Gustineli", "author_url": "", "post_date": "2022-05-15T15:22:18.397000", "content": "Good point! Interested to see how those models would perform without using external datasets π€
", "votes": 0, "replies": [] }, { "id": 1769771, "author_name": "Kevin Kwan", "author_url": "", "post_date": "2022-04-27T15:00:47.260000", "content": "Very good point and great catch on the subtle differences of the training data being used. I also agree that the point of this competition is meant for learning and improvement rather than just achieving the best accuracy possible. I must rather place where I place than get an artificially high score without understanding why or how I got there. The skill of learning how to tune and optimize a model is much more valuable in my opinion, especially in a world where we have new datasets being generated every second of every day.
", "votes": 0, "replies": [] }, { "id": 1728282, "author_name": "Mudit Mathur", "author_url": "", "post_date": "2022-03-18T18:26:09.003000", "content": "Very well said π
", "votes": 0, "replies": [] }, { "id": 1722946, "author_name": "Daniel Moreira de Sousa", "author_url": "", "post_date": "2022-03-15T01:51:41.197000", "content": "I saw lots o people using external datasets (on kaggle or outside) which feels so unfair, i guess you're supposed to use the provided dataβ¦
", "votes": 0, "replies": [] }, { "id": 1718744, "author_name": "Harshwardhan Yadav", "author_url": "", "post_date": "2022-03-11T06:08:54.207000", "content": "Well pointed.
", "votes": 0, "replies": [] }, { "id": 1705842, "author_name": "M. A. Ali", "author_url": "", "post_date": "2022-02-26T21:54:43.873000", "content": "This explains alot
", "votes": 0, "replies": [] }, { "id": 1631717, "author_name": "NKozak", "author_url": "", "post_date": "2021-12-28T18:01:27.803000", "content": "That is interesting
", "votes": 0, "replies": [] }, { "id": 1625357, "author_name": "Mahmoud Reda", "author_url": "", "post_date": "2021-12-21T17:42:23.933000", "content": "Good point
", "votes": 0, "replies": [] }, { "id": 1850269, "author_name": "", "author_url": "", "post_date": "2022-07-10T09:09:20.397000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1623434": "I am seeing a lot of variations of the same problem in the top scoring notebooks:\n\n`(X_train_mnist, y_train_mnist), (X_val_mnist, y_val_mnist) = mnist.load_data()`\n\nkaggle uses a training/test split of 42k/28k whereas this dataset downloaded from keras has a 60k/10k split. This means you will be using some of kaggles test data to train your algorithm, which artificially improves your score because your algorithm is just memorising a lot of the right answers.\n\nDon't download external datasets for this competition - you just don't need to. The point of this competition isn't to get 100% (you can just find the answers online), the point is for you to learn how to build better models by improving your accuracy score over time.", "2316417": "Some of the images in the dataset is so bizarre, even classification by the human eye is tough. We can't expect machines to make this guess and get the accuracy up to 100%. This is ridiculous and almost never happens", "2142407": "exactly, the goal is to get better accuracy, by making better models! many people just get average score and not focus on what the competition is all about !", "1632999": "Actually, that is why many submissions reach an incredible score. This is very well described [here](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist/notebook), where it is explained that 0.9979 is the max achievable score (without cheating). ", "1638057": "I was wondering why so many people can reach 100% accuracy. I can only get 99% so far, which I have made a lot of tweak on my notebook. ", "1698479": "well said. People should use caution while using training and test data", "1791022": "Good point! Interested to see how those models would perform without using external datasets π€", "1769771": "Very good point and great catch on the subtle differences of the training data being used. I also agree that the point of this competition is meant for learning and improvement rather than just achieving the best accuracy possible. I must rather place where I place than get an artificially high score without understanding why or how I got there. The skill of learning how to tune and optimize a model is much more valuable in my opinion, especially in a world where we have new datasets being generated every second of every day.", "1728282": "Very well said π", "1722946": "I saw lots o people using external datasets (on kaggle or outside) which feels so unfair, i guess you're supposed to use the provided data...", "1718744": "Well pointed.", "1705842": "This explains alot", "1631717": "That is interesting", "1625357": "Good point", "1850269": "" } }