{"cells":[{"metadata":{"_uuid":"bd7599c54a95a7cb11c9a3ee5ec5e02018d1b87f"},"cell_type":"markdown","source":"# Accuracy=100% using kNN k=1 and MNIST 70k images\nThis kernel is an example of what NOT to do. I did not submit the result from this kernel but if I did, it would score 100% on Kaggle's leaderboard. I performed kNN k=1 with Kaggle's 28,000 \"test.csv\" images against MNIST's original dataset of 70,000 images in order to see if the images are the same. The result verifies that Kaggle's unknown \"test.csv\" images are entirely contained unaltered within MNIST's original dataset with known labels. Therefore we CANNOT train with MNIST's original data, we must train our models with Kaggle's \"train.csv\" 42,000 images, data augmentation,  and/or non-MNIST image datasets.\n\n# Load MNIST's 70,000 original images"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"a8411c264c011affb4288c66e435e7f5f6c85604","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"def load_data(path):\n    with np.load(path) as f:\n        x_train, y_train = f['x_train'], f['y_train']\n        x_test, y_test = f['x_test'], f['y_test']\n        return (x_train, y_train), (x_test, y_test)\n\n(x_train, y_train), (x_test, y_test) = load_data('../input/mnist-numpy/mnist.npz')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"24d4db9cf6ed9d45b78b65f06ad333fc4b05f8d2","collapsed":true,"_kg_hide-input":true},"cell_type":"code","source":"x_train = x_train.reshape(x_train.shape[0],784)\ny_train = y_train.reshape(y_train.shape[0],1)\nx_test = x_test.reshape(x_test.shape[0],784)\ny_test = y_test.reshape(y_test.shape[0],1)\nMNIST_image = np.vstack( (x_train,x_test) )\nMNIST_label = np.vstack( (y_train,y_test) )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4cb5833125662d3eadc2e11b82a5b6e4da97d280"},"cell_type":"markdown","source":"# Load Kaggle's 28,000 competition test images"},{"metadata":{"trusted":true,"_uuid":"7f41b99dba71d2b2f95afdfc5ae0bda5146cf733","collapsed":true},"cell_type":"code","source":"Kaggle_test_image = pd.read_csv(\"../input/digit-recognizer/test.csv\")\nKaggle_test_image = Kaggle_test_image.values.astype(\"uint8\")\nKaggle_test_label = np.empty( (28000,1), dtype=\"uint8\" )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"830b81ee7c6912aba4b6f576236a2b8720c4a2b1"},"cell_type":"markdown","source":"# Perform kNN k=1\nFor each of Kaggle's 28000 test images, we search within MNIST's original 70000 image dataset for the closest image. If the closest image has distance zero that means the images are exactly the same and the label of the unknown Kaggle test image is precisely the label of the known MNIST image."},{"metadata":{"trusted":true,"_uuid":"d123cf5e6e5b0246145a359a6accd616eab55538","collapsed":true},"cell_type":"code","source":"c1=0; c2=0;\nprint(\"Classifying Kaggle's 'test.csv' using kNN k=1 and MNIST 70k images\")\nfor i in range(0,28000): # loop over Kaggle test\n    for j in range(0,70000): # loop over MNIST images\n        if np.absolute(Kaggle_test_image[i,] - MNIST_image[j,]).sum()==0:\n            Kaggle_test_label[i] = MNIST_label[j]\n            if i%1000==0:\n                print(\"  %d images classified perfectly\" % (i))\n            if j<60000:\n                c1 += 1\n            else:\n                c2 += 1\n            break\nif c1+c2==28000:\n    print(\"  28000 images classified perfectly\")\n    print(\"Kaggle's 28000 test images are fully contained within MNIST's 70000 dataset\")\n    print(\"%d images are in MNIST-train's 60k and %d are in MNIST-test's 10k\" % (c1,c2))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f68d4f66f10a7bc2758601c3d64dbf03a5ac155"},"cell_type":"markdown","source":"# Result 100% classification accuracy !!\nEvery Kaggle \"test.csv\" image was found unaltered within MNIST's 70,000 image dataset. Therefore we CANNOT use the original 70,000 MNIST image dataset to train models for Kaggle's MNIST competition. Since MNIST's full dataset contains labels, we would know precisely what each unknown Kaggle test image's label is. We must train our models with Kaggle's \"train.csv\" 42,000 images, data augmentation,  and/or non-MNIST image datasets. The following csv file would score 100% on Kaggle's public and private leaderboard if submitted."},{"metadata":{"trusted":true,"_uuid":"c37b1ebec69aa372dbdc7338944788aee3336eab","collapsed":true},"cell_type":"code","source":"results = pd.Series(Kaggle_test_label.reshape(28000,),name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"Do_not_submit.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a1c136bf53981f9cf6305c3df9d03985f21d3be"},"cell_type":"markdown","source":"# Examples"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"47662b4f9f070b386ef096e110a4dc6c26fdbc9c","collapsed":true},"cell_type":"code","source":"# find examples\nindex = np.zeros(6,dtype='uint')\nfor i in range(0,6): # loop over Kaggle test\n    for j in range(0,70000): # loop over MNIST images\n        if np.absolute(Kaggle_test_image[i,] - MNIST_image[j,]).sum()==0:\n            index[i] = j\n            break\n            \n# plot examples\nimport matplotlib.pyplot as plt\nplt.figure(figsize=(15,5))\nfor i in range(6):  \n    plt.subplot(2, 6, 2*i+1)\n    plt.imshow(Kaggle_test_image[i].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"Kaggle test #%d\\nLabel unknown\" % i,y=0.9)\n    plt.axis('off')\n    plt.subplot(2, 6, 2*i+2)\n    plt.imshow(MNIST_image[index[i]].reshape((28,28)),cmap=plt.cm.binary)\n    plt.title(\"MNIST #%d\\nLabel = %d\" % (index[i],MNIST_label[index[i]]),y=0.9)\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.1, hspace=0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f0dc6f05d18a8c9bba404211ee47055813d0fb7"},"cell_type":"markdown","source":"Using kNN k=1, we know with 100% accuracy that Kaggle's first six test images are digits 2, 0, 9, 0, 3, 7 respectively. Similarly we know the next 27994 test images perfectly too."},{"metadata":{"_uuid":"a14f00116957322154f334d2cc6e2895c0a401ec"},"cell_type":"markdown","source":"# What accuracy is actually possible?\nHere are the best published MNIST classifiers found on the internet:\n* 99.79% [Regularization of Neural Networks using DropConnect, 2013][1]\n* 99.77% [Multi-column Deep Neural Networks for Image Classification, 2012][2]\n* 99.77% [APAC: Augmented PAttern Classification with Neural Networks, 2015][3]\n* **99.76% [Kaggle published kernel, 2018][12]**\n* 99.76% [Batch-normalized Maxout Network in Network, 2015][4]\n* 99.71% [Generalized Pooling Functions in Convolutional Neural Networks, 2016][5]\n* More examples: [here][7], [here][8], and [here][9]  \n  \nOn Kaggle's website, there is only one published kernel more accurate than 99.70%. The few other high scoring published kernels train with the full MNIST 70000 image dataset and therefore have actual accuracy under 99.70%.\n[1]:https://cs.nyu.edu/~wanli/dropc/\n[2]:http://people.idsia.ch/~ciresan/data/cvpr2012.pdf\n[3]:https://arxiv.org/abs/1505.03229\n[4]:https://arxiv.org/abs/1511.02583\n[5]:https://arxiv.org/abs/1509.08985\n[7]:http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html\n[8]:http://yann.lecun.com/exdb/mnist/\n[9]:https://en.wikipedia.org/wiki/MNIST_database\n[10]:https://www.kaggle.com/cdeotte/mnist-perfect-100-using-knn/\n[12]:https://www.kaggle.com/cdeotte/35-million-images-0-99757-mnist"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}