{"cells":[{"metadata":{"_uuid":"5200f73cab8850c7444a45b21fe192a0a1df5ae2"},"cell_type":"markdown","source":"## Determine baseline"},{"metadata":{"_uuid":"2960bbe1eff234aae2a00e0207ccf54fe3b18cd3"},"cell_type":"markdown","source":"Before investing time in building a neural net, we should understand what would constitute an improvement over random guessing. \n\nSince each image is one of ten digits, random guessing has a 10% chance of getting the digit right. **So if our model achieves of accuracy of >10% against the test dataset, we win.**"},{"metadata":{"_uuid":"9cbabe66ca03920c8fc7b990e5c74e9f7a04d884"},"cell_type":"markdown","source":"## Set up imports and magics"},{"metadata":{"_uuid":"ae769122576a2d6fb15d1b0244084f4fdb3b0c69"},"cell_type":"markdown","source":"We need **keras** for the neural net, **matplotlib** for charting our loss and accuracy across epochs, **numpy** for decoding one-hot predictions, and **pandas** for loading the data."},{"metadata":{"trusted":true,"_uuid":"ff40db778c9e911be0e9226445b422a27fd8066c"},"cell_type":"code","source":"from keras import callbacks\nfrom keras import layers\nfrom keras import models\nfrom keras import regularizers\nfrom keras.utils import to_categorical\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0a36fdd5d47901a5ae7342ab1e9b9811429bf77"},"cell_type":"markdown","source":"Make sure loss and accuracy charts appear directly in this notebook."},{"metadata":{"trusted":true,"_uuid":"9783added165c0088a5b6ed77f05ae257665f594"},"cell_type":"code","source":"%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56aed583f19a72fde9dbbcd364af5046c4c08001"},"cell_type":"markdown","source":"## Preprocess data"},{"metadata":{"_uuid":"b8a2abf829b16b5064d062b89e2b045f0d618cba"},"cell_type":"markdown","source":"We *could* read the Kaggle-provided data file directly into a Numpy array, but Pandas has easier-to-understand methods for handling CSVs. So instead, we'll read the data file into a Pandas DataFrame and then convert that into a Numpy array."},{"metadata":{"trusted":true,"_uuid":"8428ae121d0d9862a9d65cfee095bcdb44ca37d8"},"cell_type":"code","source":"train_data_df = pd.read_csv('../input/train.csv')\ntrain_data = train_data_df.values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1bf1a07069f7b7f02a48ccb226c9740f8f15ced7"},"cell_type":"markdown","source":"Per naming standards, store our training data in *X_train*. Exclude the labels, since we'll store that in a separate *y_train* variable. "},{"metadata":{"trusted":true,"_uuid":"17e51732fac09225cfb87fe139b7803ce669a9f5"},"cell_type":"code","source":"X_train = train_data[:, 1:]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd8cd365d164a427a66e2c5c8782976670c788b8"},"cell_type":"markdown","source":"Cast the training data as 16-bit floats, which most GPUs find faster to deal with than 32-bit floats. Also, scale the training data so it ranges from 0 to 1.0. Neural nets usually operate better with this range of input data."},{"metadata":{"trusted":true,"_uuid":"e4a06ae370b8110a634595da46c9af8b63b5604b"},"cell_type":"code","source":"X_train = X_train.astype('float16') / 255.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"620a63972f490e42e42cf7f3a12028b0a9f86309"},"cell_type":"markdown","source":"Convert the training data (which consists of images) to 28x28 matrices, representing the 28 pixel height, 28 pixel width of the images."},{"metadata":{"trusted":true,"_uuid":"f9633aef369ec30a3877a439c64154b29fd4ffa5"},"cell_type":"code","source":"X_train = X_train.reshape((42000, 28, 28, 1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84df262dc06508ffa8f750bb53a76b86a570eda8"},"cell_type":"markdown","source":"Store the labels in *y_train* (per naming conventions), and convert them to one-hot encoding."},{"metadata":{"trusted":true,"_uuid":"2301d5ae1702c39b4d6cbe5d98d46d51b6d19913"},"cell_type":"code","source":"y_train = train_data[:, 0]\ny_train = to_categorical(y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"685359a08519d867b05a251d4b33ebc6f72e0226"},"cell_type":"markdown","source":"Do most of the same preprocessing as described above (minus processing the targets), but this time on our test data."},{"metadata":{"trusted":true,"_uuid":"7d56bf857f64525cf6c4df188b112d18400ceae8"},"cell_type":"code","source":"test_data_df = pd.read_csv('../input/test.csv')\ntest_data = test_data_df.values\n\nX_test = test_data\nX_test = X_test.astype('float16') / 255.0\nX_test = X_test.reshape((28000, 28, 28, 1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"acceb1193a2c1ae013bc6d8486275027f6ce3abb"},"cell_type":"markdown","source":"## Establish Keras callbacks"},{"metadata":{"_uuid":"cfccb55da4b36bab5b425e46b4aa656f112e7a46"},"cell_type":"markdown","source":"Use a callback to halt training if it hasn't seen any improvement in validation accuracy (during exploratory training runs) or training accuracy (during \"production\" runs, when when we train on the entire dataset with nothing held aside for validation) over the last three training epochs. It also realoads the weights from whichever epoch produced the highest validation accuracy. Sweet!"},{"metadata":{"trusted":true,"_uuid":"af96b3c2bec490770f27a374f7b2ebd2622c34a4"},"cell_type":"code","source":"# use this callback during exploratory training\n# stop_early_callback = callbacks.EarlyStopping(monitor='val_acc', \n#                                               patience=3, \n#                                               restore_best_weights=True,\n#                                               verbose=1)\n\n# use this callback during \"production\" training, when we don't set aside a validation dataset\nstop_early_callback = callbacks.EarlyStopping(monitor='acc', \n                                              patience=5, \n                                              restore_best_weights=True,\n                                              verbose=1)\n\ncallbacks_list = [stop_early_callback]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcfe817f712da59325e6a1961927506d0b79d686"},"cell_type":"markdown","source":"## Build and train a convolutional neural net"},{"metadata":{"_uuid":"1f4f5648b861d1b6f99ad07206997db918d8ac97"},"cell_type":"markdown","source":"Our model has three convolution layers in the base and one dense layer in the head. Experimentation with this dataset revealed that L1 and L2 regularizers have no effect. BachNormalization layers have no effect. The dropout layer in the head does marginally reduce overfitting.\n\nAll hyperparameters were set through long trial and error. It was easy to mess up validation accuracy, but very hard to get any higher than 99.0%."},{"metadata":{"trusted":true,"_uuid":"c484b5ba548afdae531b012b5e47c4eeb78b48b8"},"cell_type":"code","source":"model = models.Sequential()\nmodel.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1))) \nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(64, (3, 3), activation='relu')) \nmodel.add(layers.MaxPooling2D((2, 2)))\nmodel.add(layers.Conv2D(64, (3, 3), activation='relu'))\n\nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(128, activation='relu'))\nmodel.add(layers.Dropout(0.4))\nmodel.add(layers.Dense(10, activation='softmax'))\n\nmodel.compile(optimizer='rmsprop', \n              loss='categorical_crossentropy', \n              metrics=['acc'])\n\nvalidation_split = 0.0 # we'll need this variable later, when deciding whether to graph our validation figures\n\nhistory = model.fit(X_train, \n                    y_train, \n                    callbacks=callbacks_list, \n                    epochs=30, \n                    batch_size=256, \n                    validation_split=validation_split, \n                    verbose=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a296b710a840674dec78746e9ad00fc2a5c77a52"},"cell_type":"markdown","source":"## Graph our progress"},{"metadata":{"_uuid":"0983b98cbbed62c983a2b58e2355b83f027cd6ad"},"cell_type":"markdown","source":"When experimenting with hyperparamter values, it's helpful to see graphs of four values after each epoch:\n* training loss\n* validation loss\n* training accuracy\n* validation accuracy "},{"metadata":{"trusted":true,"_uuid":"236d969dc32edcbceb38a7eb515e8080ae97f7c7"},"cell_type":"code","source":"epochs = range(1, len(history.history['loss']) + 1)\nplt.plot(epochs, history.history['loss'], 'ro', label='training loss')\nif validation_split > 0:\n    plt.plot(epochs, history.history['val_loss'], 'r', label='val loss')\nplt.xlabel('epochs')\nplt.ylabel('loss')\nplt.legend()\nplt.show()\n\nplt.plot(epochs, history.history['acc'], 'bo', label='training acc')\nif validation_split > 0:\n    plt.plot(epochs, history.history['val_acc'], 'b', label='val acc')\nplt.xlabel('epochs')\nplt.ylabel('acc')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98211cbc339e7f3c01e07d533cb20a04abd61dc1"},"cell_type":"markdown","source":"## Make predictions for Kaggle's test data"},{"metadata":{"trusted":true,"_uuid":"0a729ddc39909681f10d5b2c1fdc8c6652850e05"},"cell_type":"code","source":"predictions_one_hot = model.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36c21d9d79da020a37ead8653d4697035e6a6080"},"cell_type":"markdown","source":"The CNN's output is one-hot encoded, so we need to convert it from (for example) **[0, 0, 1, 0, 0, 0, 0, 0, 0, 0]** to **2** since the latter is the form that Kaggle expects in submissions."},{"metadata":{"trusted":true,"_uuid":"96ab8f6172b545a059fc4f38cc27b459bf1eba72"},"cell_type":"code","source":"predictions_ints = [np.argmax(prediction_one_hot) for prediction_one_hot in predictions_one_hot]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a75d3287dc596378a846146a7f5ab95afc5335d4"},"cell_type":"markdown","source":"Make a CSV containing our predictions, using Kaggle's required format."},{"metadata":{"trusted":true,"_uuid":"aa18a236cec56b01ab521cdc9c41ed684601df8c"},"cell_type":"code","source":"with open('predictions.csv', 'w') as predictions_file:\n    predictions_file.write('ImageId,Label' + '\\n')\n    for i in range(len(test_data)):\n        predictions_file.write(f'{i + 1},{predictions_ints[i]}' + '\\n')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}