{"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"kaggle":{"accelerator":"nvidiaTeslaK80","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"},{"sourceId":34877,"sourceType":"datasetVersion","datasetId":27352}],"dockerImageVersionId":18199,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# introduce\nIf this kernel can help you, feel free to add some contents below to yours and waht I need is just a reference link. Thanks!\n\n## EDA \nExplore data analysis is the first step when you get an dataset.   \nAll the techniques to summarize the main characteristics, often with visual methods, are part of EDA.\n\nWith this digit dataset, what we need to do is to transform the matrix to a image.\n\n## pre-precess\n* get more data (but not MNIST)  \n    Not recommend to use MNIST because some test data is from MNIST, this makes this competition meaningless.  \n    But in fact, it's impossible to reach 1.00 or even 0.999 without using MNIST. In other words, when the acc reaches 1.00 or even 0.999, this model is already overfitting.\n    \n     Maybe you can try   \n    1. [Distorting the MNIST Image Data Set](https://msdn.microsoft.com/en-us/magazine/dn754573.aspx)  \n    2. [Optical Recognition of Handwritten Digits Data Set](http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits)  \n    3. [Pen-Based Recognition of Handwritten Digits Data Set](http://archive.ics.uci.edu/ml/datasets/Pen-Based+Recognition+of+Handwritten+Digits)  \n    4. [Semeion Handwritten Digit Data Set](http://archive.ics.uci.edu/ml/datasets/Semeion+Handwritten+Digit)  \n    5. [The EMNIST Dataset's Digit Part](https://www.nist.gov/itl/iad/image-group/emnist-dataset) It's independent of MNIST Dataset.  \n    6. [Chars74K Dataset](http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/) is also considerable, but I doubt the performance after training.\n    \n* change grey value from int to float\n* 0-1 format\n* Cropping and Resize  \n\nNote: I used MNIST and got 0,99942 before I realize that there're some test data is from MNIST, which is not recommended.\n\nIf you want to improve your model you can try more pre-precess tech like Data Augmentation and ensemble technique like stacking.    \nYou can also read the classic paper [Gradient-Based Learning Applied to Document Recognition](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf) to get more ideas.\n\n## feature selection\n* VarianceThreshold\n* ch2\n* PCA\n\n## model\n* naive bayes\n* knn\n* svm\n* decison tree\n* random forest\n* cnn\n\"LaNet5\"\"Simple CNN\" and \"Complex CNN\n\n## model ensemble\nSimply by average voting.\n\n## summary and result\nUse cnn and rank 20th   \nMy summary is at the end of this kernel.","metadata":{"_uuid":"60ec54c704e3545c61b592d66afaee32dbb2fac4","_cell_guid":"416be346-0574-4e05-b955-511e7f6aa962"}},{"cell_type":"markdown","source":"# import data","metadata":{"_uuid":"48eb3c743935d4a7da1425090fb70252e33d9896","_cell_guid":"1ac6fdfc-1682-440e-9d6a-309e9c67cdf1"}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n#\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport os\nprint(os.listdir(\"../input\"))\n\nsize_img = 28\nthreshold_color = 100 / 255\n\n# Any results you write to the current directory are saved as output.\nfile = open(\"../input/digit-recognizer/train.csv\")\ndata_train = pd.read_csv(file)\n\ny_train = np.array(data_train.iloc[:, 0])\nx_train = np.array(data_train.iloc[:, 1:])\n\nfile = open(\"../input/digit-recognizer/test.csv\")\ndata_test = pd.read_csv(file)\nx_test = np.array(data_test)\n\nn_features_train = x_train.shape[1]\nn_samples_train = x_train.shape[0]\nn_features_test = x_test.shape[1]\nn_samples_test = x_test.shape[0]\nprint(n_features_train, n_samples_train, n_features_test, n_samples_test)\nprint(x_train.shape, y_train.shape, x_test.shape)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# show the image\nMaybe we can call this EDA (haha~","metadata":{"_uuid":"cc3bb4eed5898be07e64aa94eb1e8d35033f7178","_cell_guid":"c3f5f6c9-fc52-4cbc-aa85-1a0bfbebc5fd"}},{"cell_type":"code","source":"def show_img(x):\n    plt.figure(figsize=(8,7))\n    if x.shape[0] > 100:\n        print(x.shape[0])\n        n_imgs = 16\n        n_samples = x.shape[0]\n        x = x.reshape(n_samples, size_img, size_img)\n        for i in range(20):\n            plt.subplot(5, 4, i+1) #devide figure into 5x4 and choose i+1 to draw\n            plt.imshow(x[i],cmap = 'gray')\n        plt.show()\n    else:\n        plt.imshow(x)\n        plt.show()","metadata":{"_uuid":"f1800ec26c1b2f78eaa716ebfdf9cc7ffefc8907","_cell_guid":"7513ba7b-7c3e-4102-ab67-ba391adf2736","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_img(x_train)","metadata":{"_uuid":"4a7f087b67dedba8ffe1d0f167440ec61032189c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_img(x_test)","metadata":{"_uuid":"7e0bfa80d2bf2d8517cf04763917cb4ac5bfaccd","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# pre-process","metadata":{"_uuid":"8ae4a35dd1a01e8116977a9629b20a5a5fb5835c","_cell_guid":"9cc8a7b6-82f3-4e50-9cd2-04d6dfc3224d"}},{"cell_type":"markdown","source":"## change grey value from int to float","metadata":{"_uuid":"320d4ac450efdafb5b6875ae0805c8857ca6acbd","_cell_guid":"380ea52e-552b-4402-aeb3-5ea6b5ae6650"}},{"cell_type":"code","source":"def int2float_grey(x):\n    x = x / 255\n    return x","metadata":{"_uuid":"e83760060ca6a791fbbe93920dfc6161c3ae23d4","_cell_guid":"4cbc24e8-761c-438b-84de-cc46a76ec28b","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 0-1 format","metadata":{"_uuid":"2d9cc4c3497abbbdaf131a57ea60b9b41faf1173","_cell_guid":"38266503-ebb0-4a7f-8f30-d0010d4171cc"}},{"cell_type":"code","source":"# x_train[x_train<100] = 0\n# x_train[x_train>=100] = 1\n# # print(x_train[0])\n# show_img(x_train)","metadata":{"_uuid":"bd596128072519371d8f2a7d8f87a8f5e381be11","_cell_guid":"c74300f8-a6c3-4a81-b446-3d425277d70d","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cropping and Resize","metadata":{"_uuid":"1150882975043238eee28101194e7d87da7089b8","_cell_guid":"758f0a2f-0254-4a1b-a963-af911a274c47"}},{"cell_type":"code","source":"# find the left egde\n# Note: the problem is that I don't do the parrallel part\ndef find_left_edge(x):\n    edge_left = []\n    n_samples = x.shape[0]\n    for k in range(n_samples):\n        for j in range(size_img):\n            for i in range(size_img):\n                if (x[k, size_img*i+j] >= threshold_color):\n                    edge_left.append(j)\n                    break\n            if (len(edge_left) > k):\n                break\n    return edge_left","metadata":{"_uuid":"a2baac64a9de93cee2d696cc76b7567122ba1afa","_cell_guid":"2c2d3486-8b0d-4e19-9317-ad507699cdc7","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# find the right egde\n# Note: the problem is that I don't do the parrallel part\ndef find_right_edge(x):\n    edge_right = []\n    n_samples = x.shape[0]\n    for k in range(n_samples):\n        for j in range(size_img):\n            for i in range(size_img):\n                if (x[k, size_img*i+(size_img-1-j)] >= threshold_color):\n                    edge_right.append(size_img-1-j)\n                    break\n            if (len(edge_right) > k):\n                break\n    return edge_right","metadata":{"_uuid":"1f5ab16336d5d6c07573d088532cc8e40022e7f2","_cell_guid":"846df1ee-b76c-4bf0-a677-e04d636044df","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# find the top egde\n# Note: the problem is that I don't do the parrallel part\ndef find_top_edge(x):\n    edge_top = []\n    n_samples = x.shape[0]\n    for k in range(n_samples):\n        for i in range(size_img):\n            for j in range(size_img):\n                if (x[k, size_img*i+j] >= threshold_color):\n                    edge_top.append(i)\n                    break\n            if (len(edge_top) > k):\n                break\n    return edge_top","metadata":{"_uuid":"60e0d7dbdbb479dcd3cc009d48ae7cd6fe771558","_cell_guid":"ef1fd741-5068-467a-bb7d-16180399fa27","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# find the bottom egde\n# Note: the problem is that I don't do the parrallel part\ndef find_bottom_edge(x):\n    edge_bottom = []\n    n_samples = x.shape[0]\n    for k in range(n_samples):\n        for i in range(size_img):\n            for j in range(size_img):\n                if (x[k, size_img*(size_img-1-i)+j] >= threshold_color):\n                    edge_bottom.append(size_img-1-i)\n                    break\n            if (len(edge_bottom) > k):\n                break\n    return edge_bottom","metadata":{"_uuid":"9ed9582a529205146561e8a22a4bbc603b2203f4","_cell_guid":"8d809f4c-3915-4298-bcf0-d44fad9bfbbe","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Note：when we do the stretch part by ourselves,there may be some blank cells\n# when the scale factor is more than 2\n\nfrom skimage import transform\ndef stretch_image(x):\n    #get edges\n    edge_left = find_left_edge(x)\n    edge_right = find_right_edge(x)\n    edge_top = find_top_edge(x)\n    edge_bottom = find_bottom_edge(x)\n    \n    #cropping and resize\n    n_samples = x.shape[0]\n    x = x.reshape(n_samples, size_img, size_img)\n    for i in range(n_samples):      \n        x[i] = transform.resize(x[i][edge_top[i]:edge_bottom[i]+1, edge_left[i]:edge_right[i]+1], (size_img, size_img))\n    x = x.reshape(n_samples, size_img ** 2)\n    show_img(x)\nstretch_image(x_train)","metadata":{"_uuid":"77b673c86fc179554ec3d9d767e54f24d18ceb30","_cell_guid":"e49eeb72-abcb-4ba5-badd-5acbbaea14f0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature Selection\n### VarianceThreshold","metadata":{"_uuid":"616d5567d35156ce7d384a567dfc59de2c5a6bc2","_cell_guid":"bfe0b9b2-ad85-4a15-aad4-ded9a7525c42"}},{"cell_type":"code","source":"from sklearn.feature_selection import VarianceThreshold\ndef get_threshold(x_train, x_test):\n    selector = VarianceThreshold(threshold=0).fit(x_train)\n    x_train = selector.transform(x_train)\n    x_test = selector.transform(x_test)\n    print(x_train.shape)\n    return x_train, x_test","metadata":{"_uuid":"9a0a242132087699ccd0b24b70f1790362f33426","_cell_guid":"ee9bb9c4-2544-4e66-8571-8a87f592f95e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### chi2","metadata":{"_uuid":"b518af6415ca17684e1c81d127ddd580888f4a44","_cell_guid":"f04e3069-da73-4937-a351-be03db268d6a"}},{"cell_type":"code","source":"from sklearn.feature_selection import SelectKBest\nfrom sklearn.feature_selection import chi2\n\n#选择K个最好的特征，返回选择特征后的数据\n# selector = SelectKBest(chi2, k=500).fit(x_train, y_train)\n# x_train = selector.transform(x_train)\n# x_test = selector.transform(x_test)\n# print(x_train.shape)","metadata":{"_uuid":"1f6d9d701e24fada770d69913e674531269fc28a","_cell_guid":"ca9262c0-4ddc-4b0a-be73-a6a92a5803b1","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### PCA","metadata":{"_uuid":"6ca42d16292b5dd413fb7b3ea2e618149c8bd2c5","_cell_guid":"bb73d9a9-7918-4e83-ba4e-582f28b123c8"}},{"cell_type":"code","source":"from sklearn.decomposition import PCA\ndef get_pca(x_train, x_test):\n    pca = PCA(n_components=0.95)\n    pca.fit(x_train)\n    x_train = pca.transform(x_train)\n    x_test = pca.transform(x_test)\n    print(x_train.shape, x_test.shape)\n    return x_train, x_test","metadata":{"_uuid":"b80196e83530367b41580b5fcc232646f59fccbb","_cell_guid":"9476eda4-d676-4171-a7a4-5937049d3e08","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Here you can choose to run the subpart","metadata":{"_uuid":"5b55b16d05011259c722f10089440071667867d3"}},{"cell_type":"code","source":"# do the pre-process part\n\nx_train = int2float_grey(x_train)\nx_test = int2float_grey(x_test)\n# stretch_image(x_train)\n# stretch_image(x_test)\n# x_train, x_test = get_threshold(x_train, x_test)\n# x_train, x_test = get_pca(x_train, x_test)","metadata":{"_uuid":"3b95a4b11822c53e9a4cb4766c2b566d5f1a34eb","_cell_guid":"03fa09c0-7f1e-4caa-a158-d71bf7ea0736","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# General function","metadata":{"_uuid":"7646b6ad1c53458dc2d5b2e21c47af89f8c8c848","_cell_guid":"d7ea11d4-d697-4c9d-b682-b247b1de3809"}},{"cell_type":"code","source":"def general_function(mod_name, model_name):\n    y_pred = model_train_predict(mod_name, model_name)\n    output_prediction(y_pred, model_name)","metadata":{"_uuid":"de9e648b01f3c17bc23490c917b4fc2755b79e12","_cell_guid":"187d8e4f-b06b-48e1-b9e8-d2e38002c04f","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import cross_val_score\ndef model_train_predict(mod_name, model_name):\n    import_mod = __import__(mod_name, fromlist = str(True))\n    if hasattr(import_mod, model_name):\n         f = getattr(import_mod, model_name)\n    else:\n        print(\"404\")\n        return []\n    clf = f()\n    clf.fit(x_train, y_train)\n    y_pred = clf.predict(x_train)\n    get_acc(y_pred, y_train)\n    scores = cross_val_score(clf, x_train, y_train, cv=5)\n    print(\"Accuracy: %0.2f (+/- %0.2f)\" % (scores.mean(), scores.std() * 2))\n    y_pred = clf.predict(x_test)\n    return y_pred","metadata":{"_uuid":"fb6f372ba5b9fd53342337cd809ae635b02b9d6b","_cell_guid":"ea5de3dd-8ae5-40fb-8a0a-e9b1d97e4e63","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_acc(y_pred, y_train):\n    right_num = (y_train == y_pred).sum()\n    print(\"acc: \", right_num/n_samples_train)","metadata":{"_uuid":"216c15bb29615f16e744780be6e5db2a521cd8c3","_cell_guid":"5cf3c885-a6f9-4521-ba73-d0bbef4ab026","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def output_prediction(y_pred, model_name):\n    print(y_pred)\n    data_predict = {\"ImageId\":range(1, n_samples_test+1), \"Label\":y_pred}\n    data_predict = pd.DataFrame(data_predict)\n    data_predict.to_csv(\"dr output %s.csv\" %model_name, index = False)","metadata":{"_uuid":"2596f0cade632dccd6ceb6d35f30477bbab48b14","_cell_guid":"62490f02-3c02-4609-bcc2-d1844f79b17a","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# use Naive Bayes","metadata":{"_uuid":"0c790788d14d613d611c8f409bf2df732b9d2176","_cell_guid":"01aabf6a-42a0-4fa0-bd75-d07ee82bd78e"}},{"cell_type":"code","source":"# from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB\nmod_name = \"sklearn.naive_bayes\"\nmodel_name = \"GaussianNB\"\n# model_name = \"MultinomialNB\"\n# general_function(mod_name, model_name)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# use KNN","metadata":{"_uuid":"760e9b7c93e8888d9d550cfd9c70e9a255ad1be6","_cell_guid":"e1e82a7f-4c65-4b59-91d9-2bbd1fc632af"}},{"cell_type":"code","source":"# from sklearn.neighbors import KNeighborsClassifier\nmod_name = \"sklearn.neighbors\"\nmodel_name = \"KNeighborsClassifier\"\n# general_function(mod_name, model_name)","metadata":{"_uuid":"aa1003bc6abae84cbc08f0d9ae9caf6dca40ec4c","scrolled":true,"_cell_guid":"24f6fcb6-b871-4e6c-9c12-f45e3b25c485","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# use k-means","metadata":{"_uuid":"e874bbddc7c2ef668db1aaa0b4d945e2d1919986","_cell_guid":"ad07fc07-810e-4f88-9bed-b701356399d7"}},{"cell_type":"code","source":"# from sklearn.cluster import KMeans\nmod_name = \"sklearn.cluster\"\nmodel_name = \"KMeans\"\n# general_function(mod_name, model_name)","metadata":{"_uuid":"4a427569e1cc4f3764383d190bca67604c2f2177","_cell_guid":"26453fe4-58cf-4ced-93a2-a409c7ad3499","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# use SVM","metadata":{"_uuid":"82ba9b075e560e1ea7e6ca789dfd512615bef6ae","_cell_guid":"0db70f93-60f3-4d9d-ba98-88f5b077fdd4"}},{"cell_type":"code","source":"# from sklearn.svm import SVC\nmod_name = \"sklearn.svm\"\nmodel_name = \"SVC\"\n# general_function(mod_name, model_name)","metadata":{"_uuid":"06700d5fd85b5459d6625036f91c86a783bce70e","_cell_guid":"9cd7b0a8-cb6f-4ac4-b25c-00fcda5afda7","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# use the decision tree","metadata":{"_uuid":"4825999696bdf0f3657dedf44153800653ac529e","_cell_guid":"7c81d455-ff8b-4189-80a9-ddcb46e31188"}},{"cell_type":"code","source":"from sklearn.model_selection import cross_val_score\nfrom sklearn.tree import DecisionTreeClassifier\nmod_name = \"sklearn.tree\"\nmodel_name = \"DecisionTreeClassifier\"\n# general_function(mod_name, model_name)","metadata":{"_uuid":"857cf5ee1ef655a70c376b3575592d1cb77ab42e","_cell_guid":"ab724322-9313-4d5c-9352-6322c39dd668","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# use Random Forest","metadata":{"_uuid":"47d5bed36481ca9606a25c14be3bc7c5cd09e73a","_cell_guid":"d0bf9e06-2b3e-4fd9-a8e2-0c7098531797"}},{"cell_type":"code","source":"# from sklearn.ensemble import RandomForestClassifier\nmod_name = \"sklearn.ensemble\"\nmodel_name = \"RandomForestClassifier\"\n# general_function(mod_name, model_name)","metadata":{"_uuid":"35b4289f4c190518687b32f2d00b45fdc7d9b4c7","_cell_guid":"6d30efc9-fde3-4019-9f03-6e50d4df4fe5","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# use CNN\nIn CNN part, I try three different structures, which I call \"LaNet5\"\"Simple CNN\" and \"Complex CNN\".","metadata":{"_uuid":"e001489551e5975d14df601a4746d3a23d988e8f"}},{"cell_type":"code","source":"import tensorflow as tf\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Activation, Reshape, Conv2D, AveragePooling2D, Flatten\nfrom keras.layers import MaxPooling2D\nfrom keras.optimizers import adam\n\ny_train = keras.utils.to_categorical(y_train, num_classes=10)\ny_all_pred = np.zeros((3, n_samples_test)).astype(np.int64)\nprint(y_all_pred.dtype)","metadata":{"_uuid":"5702647c61cd495fc2d635e6f42d2976c091786b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## LaNet5\n![LaNet5 architecture](https://endlesslethe.com/wordpress/wp-content/uploads/2018/06/LaNet5-architecture.jpg)\nLeNet-5 comprises 7 layers, not counting theinput, all of which contain trainable parameters (weights).  \n\nThe input is a 32x32 pixel image. This is significantly larger than the largest character in the database (at most 20x20 pixels centered in a 28x28 field).   \nThe reason is that it is desirable that potential distinctive features such as stroke end-points or corner can appear in the center of the receptive ield of the highest-level feature detectors.\n\ninput layer: 32*32  \ncov layer(3): C1 6x5x5 C3 16x5x5 C5 120x5x5  \npooling layer(2):S2 S4 size is 2x2  \nfull connection layer(2): F6'active function is \"tanh\" and output layer  uses RBF\noutput layer：10（possibilities of digit 0-9）  \n\n### Improvement\n1. Here I use 28x28 pixel image as input, and on the first conv layer I use 6x3x3 fliter.  \n2. On the output layer I use softmax fuction to get the predicted labels rather than RBF function\n3. I use two full connection layer with \"relu\" function instead of one layer with \"tanh\" fumction.","metadata":{"_uuid":"5ff3c78cc55b06596e7c6610fc64d401f0f2b829"}},{"cell_type":"code","source":"model_name = \"LaNet5\"\nmodel = Sequential()\nmodel.add(Reshape(target_shape=(1, 28, 28), input_shape=(784,)))\nmodel.add(Conv2D(kernel_size=(3, 3), filters=6, padding=\"same\", data_format=\"channels_first\", kernel_initializer=\"uniform\", use_bias=False))\nmodel.add(MaxPooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Conv2D(kernel_size=(5, 5), filters=16, padding=\"same\", data_format=\"channels_first\", kernel_initializer=\"uniform\", use_bias=False))\nmodel.add(MaxPooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Conv2D(kernel_size=(5, 5), filters=120, padding=\"same\", data_format=\"channels_first\", kernel_initializer=\"uniform\", use_bias=False))\nmodel.add(Flatten())\nmodel.add(Dense(output_dim=120, activation='relu'))\nmodel.add(Dense(output_dim=120, activation='relu'))\nmodel.add(Dense(output_dim=10, activation='softmax'))\n\nadam = keras.optimizers.Adam(lr=0.0005, beta_1=0.9, beta_2=0.999, epsilon=1e-08)\nmodel.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])\n\nmodel.fit(x_train, y_train, epochs=30, batch_size=64)\ny_pred = model.predict_classes(x_test)\noutput_prediction(y_pred, model_name)\ny_all_pred[0] = y_pred","metadata":{"_uuid":"4909c474d22bb27e0d7628cabec981c211dfdff5","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Simple CNN\nAlthough I use some technics that AlexNet uses like dropout, relu function,  the performance reaches a plateau because of the depth.","metadata":{"_uuid":"19a8488bf752b6a241c9bc7ddd5b066ee8f09639","_cell_guid":"9739b735-b781-4c66-adcd-3a217c541d6f"}},{"cell_type":"code","source":"model_name = \"CNN\"\nmodel = Sequential()\n\nmodel.add(Reshape(target_shape=(1, 28, 28), input_shape=(784,)))\nmodel.add(Conv2D(kernel_size=(3, 3), filters=32, padding=\"same\", data_format=\"channels_first\", kernel_initializer=\"uniform\", use_bias=False))\nmodel.add(AveragePooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Conv2D(kernel_size=(3, 3), filters=32, padding=\"same\", data_format=\"channels_first\", kernel_initializer=\"uniform\", use_bias=False))\nmodel.add(AveragePooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Conv2D(kernel_size=(3, 3), filters=64, padding=\"same\", data_format=\"channels_first\", kernel_initializer=\"uniform\", use_bias=False))\nmodel.add(AveragePooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Dropout(0.25))\nmodel.add(Flatten())\nmodel.add(Dense(output_dim=256, activation='relu'))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(output_dim=256, activation='relu'))\nmodel.add(Dense(output_dim=10, activation='softmax'))\n\nadam = keras.optimizers.Adam(lr=0.0005, beta_1=0.9, beta_2=0.999, epsilon=1e-08)\nmodel.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])\n\nmodel.fit(x_train, y_train, epochs=50, batch_size=128)\ny_pred = model.predict_classes(x_test)\noutput_prediction(y_pred, model_name)\n\ny_all_pred[1] = y_pred","metadata":{"_uuid":"42d165c4b73b4e6690ee6cb1f80985b463fdd86f","_cell_guid":"967fb60f-0cb6-44db-9027-b196b06aec78","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Complex CNN","metadata":{"_uuid":"242ff04ff9122980f31b6c62178f213bf884a266"}},{"cell_type":"code","source":"model_name = \"ComplexCNN\"\n\nmodel = Sequential()\nmodel.add(Reshape(target_shape=(1, 28, 28), input_shape=(784,)))\nmodel.add(Conv2D(32, (3, 3), padding='same', activation='relu'))\nmodel.add(Conv2D(32, (3, 3), activation='relu', padding='same'))\nmodel.add(MaxPooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Conv2D(32, (3, 3), padding='same', activation='relu'))\nmodel.add(Conv2D(32, (3, 3), padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Conv2D(64, (3, 3), padding='same', activation='relu'))\nmodel.add(Conv2D(64, (3, 3), padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2), data_format=\"channels_first\"))\nmodel.add(Dropout(0.25))\nmodel.add(Flatten())\nmodel.add(Dense(output_dim=256, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(output_dim=256, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(output_dim=10, activation='softmax'))\n\nadam = keras.optimizers.Adam(lr=0.0005, beta_1=0.9, beta_2=0.999, epsilon=1e-08)\nmodel.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])\n\nmodel.fit(x_train, y_train, epochs=80, batch_size=128)\ny_pred = model.predict_classes(x_test)\noutput_prediction(y_pred, model_name)\n\ny_all_pred[2] = y_pred","metadata":{"_uuid":"333677454e802c75d89da17826ad004c8223c085","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Ensemble","metadata":{"_uuid":"bb33ffa37890a3fe123f02e21dfedb8eda47f364"}},{"cell_type":"code","source":"model_name = \"Ensemble\"\nprint(y_pred.shape)\ny_ensem_pred = np.zeros((n_samples_test,))\nfor i,line in enumerate(y_all_pred.T):\n    y_ensem_pred[i] = np.argmax(np.bincount(line))\nprint(y_ensem_pred.shape, y_ensem_pred)\ny_ensem_pred = y_ensem_pred.astype(\"int64\")\noutput_prediction(y_ensem_pred, model_name)","metadata":{"_uuid":"1d84e947ccb810c5d619c8049d624b1009c60262","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# summary\n**Note: There're some posts saying that it's impossible to make acc to 1 beacuse some samples are so hard to recognize for both machines and human beings, and some of these whose model has acc \"1.0\" cheats. So I won't keep improving my model. **\n\nAnd interestingly, my last submission ranks 21th, but after one night it ranks now 20th XD. Happy.\n![dr](https://endlesslethe.com/wordpress/wp-content/uploads/2018/05/dr.png)\n\n## data processing\nThe cropping and resize technique improve models' performance slightly, and reduce the number of features significantly with pca. Before this, when we choose 90% principal component, there are 87 features. After this, there's only 37 features. So that we can increase the percentage of principal component to 95%, which stage has 58 features.\n\nPCA works well with GaussianNB. It reduce the number of features significantly. However, Its disadvantage is that we have to use GaussianNB because of the meaningless position in an new vector space.\n\nAnd chi2 doesn't improve this model because the label has the same importance. It's easily to find out that the more points we take into account, the more accurate our model will be.\n\n## model\nNaive Bayes is so simple, but it shows a decent result.\nDecision tree works so well on the train set, but on the k-cross-hold test set it works even worse than Naive Bayes.\nKNN, SVM and RF all give a good answer whose accuracy is more than 0.95.\nAnd RF performs so well based on decision tree with vote strategy.\n\nCNN is of course the best way to solve this problem.\n\n# result\n## Naive Bayes\nWhen I use **MultinomialNB only**, the acc on the train set is **0.8261190476190476**, and on the test set is **0.83114**, ranks 2180th.  \nWhen I use **BernoulliNB only**, the acc on the train set is **0.8347857142857142**.    \nWhen i use **GaussianNB only**, the acc on the train set is  **0.5571904761904762**.  \nWhen i use **GaussianNB with pca**, the acc on the train set is  **0.8739285714285714**, and on the test set is **0.87385**, ranks 2136th.\n\n## KNN\nWhen I use **KNN with PCA**, the acc on the train set is ** 0.982452380952381**, and on the test set is **0.97371**, ranks 1400th. \n\n**Note: On the dataset of the following models, we use pca and the cropping and resize technique.**\n\n## SVM\nBased on the basic dataset, the acc on the train set is ** 0.9966190476190476**, and on the test set is **0.98585**, ranks 1039th.  \nAnd based on the bigger dataset, the acc on the train set is ** 0.9966190476190476**, and on the test set is **0.99614**, ranks 214th.\n\n## the decision tree\nthe acc on the train set is **1.0**, but on the test set is only **0.83857**\n\n## RF\nBased the basic dataset, the acc on the train set is ** 0.998**, and on k-cross-hold test set, the acc is **0.98 +- 0.02**.  \nAnd based on the bigger dataset, the acc on the test set is **0.99914**, ranks 36th.\n\n## CNN\nthe acc on the train set is **1**, and on the test set is **0.99942**, ranks 21th.\n\n# screen shoots\n## knn\n![dr knn](https://endlesslethe.com/wordpress/wp-content/uploads/2018/05/dr-knn.png)\n\n## svm\nuse the basic dataset:\n![dr svm](https://endlesslethe.com/wordpress/wp-content/uploads/2018/05/dr-svm.png)\nuse the bigger dataset:\n![dr svm 2](https://endlesslethe.com/wordpress/wp-content/uploads/2018/05/dr-svm-2.png)\n\n## decision tree\n![dr dt](https://endlesslethe.com/wordpress/wp-content/uploads/2018/05/dr-dt.png)\n\n## rf\nuse the bigger dataset:\n![dr rf](https://endlesslethe.com/wordpress/wp-content/uploads/2018/05/dr-rf.png)\n\n## cnn\n![dr cnn](https://endlesslethe.com/wordpress/wp-content/uploads/2018/05/dr-cnn.png)\n","metadata":{"_uuid":"8965b8133f39b8ae993a19fc1b470cd04d385cd6","_cell_guid":"c1659388-15bc-472f-8308-cb41f324a59f"}},{"cell_type":"markdown","source":"","metadata":{"_uuid":"75ed2188f41c9c0eaa5b09dfd30386d129bc8f57","_cell_guid":"2a56567d-e145-4138-b81a-e67aa0cb34a6"}}]}