{"cells":[{"metadata":{"trusted":true},"cell_type":"markdown","source":"We are Amar Shah and Aarjav Jain, and this project is our first ever attempt at solving a problem using neural nets. \n\nWe extensively referred to [this](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6) excellent notebook when constructing our model, as well as the Kaggle Deep Learning course. \n\nThe final score obtained was an accuracy of 99.414%, which at the time of writing is a top 21% finish on the leaderboard.\n\nIncluded in this notebook are some commented unused pieces of code where we attempted some alternative methods."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nfrom sklearn.model_selection import train_test_split\nimport keras.backend as K\nfrom keras.utils import to_categorical\nfrom keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, Dropout\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1>Preprocessing</h1> \n\nInput data, reshape from linear to array and normalise."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')\nX_train = pd.DataFrame(train.drop(['label'], axis = 1))\ny_train = pd.DataFrame(train['label'])\nX_train = (X_train.values/255).reshape(-1,28,28,1)\n\nX_test = (test.values/255).reshape(-1,28,28,1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The for loop below was used to examine the images, and then the target was encoded categorically to enable a binary classification."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nfor i in range(0,10):\n    g = plt.imshow(X_train[i][:,:,0], cmap = 'gray_r')\n    plt.title(label = 'Digit: ' + str(y_train.values[i][0]))\n    plt.show()\n\"\"\"\ny_train = pd.DataFrame(to_categorical(y_train, 10))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# detect and init the TPU\n# tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n# tf.config.experimental_connect_to_cluster(tpu)\n# tf.tpu.experimental.initialize_tpu_system(tpu)\n\n# # instantiate a distribution strategy\n# tpu_strategy = tf.distribute.experimental.TPUStrategy(tpu)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1> Data Augmentation </h1> \n\nWe used a small number of augmentations as some augmentations would lead to digits being unrecognisable. This slightly improves the score but more parameter changes could be investigated further."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_generator_with_aug = tf.python.keras.preprocessing.image.ImageDataGenerator(\n    width_shift_range = 0.1,height_shift_range = 0.1,rotation_range = 20)\ndata_generator_with_aug.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1> Model Building </h1>\n\nThe top code is where we were testing our model on the TPU without data augmentation. Unfortunately it proved to be more complicated than we thought to use data augmentation on a TPU, so instead we used a GPU accelerator when using augmented data.\n\nThe model itself was initially set to the model from [this](https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6) project (same as in the introduction above). Then we added an extra layer which improved performance, and increased the dropout as the model was starting to overfit.\n\nThe learning rate reduction was also initially set to the same as the other project, however we found that making the reduction of the learning rate less strong tended to improve performance. \n\n50 Epochs takes around 15 minutes to train but gives very good results.\n\n(It's worth noting we initially did not specify the input_shape, and based on a quick internet search the model can perform better without the shape. Hence if this model was to be run it would produce slightly worse results with input_shape compared to without)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# instantiating the model in the strategy scope creates the model on the TPU\n\"\"\"with tpu_strategy.scope():\n    model = Sequential()\n    model.add(Conv2D(20, kernel_size=5, activation = 'relu', padding = 'Same'))\n    model.add(Conv2D(20, kernel_size=5, activation = 'relu', padding = 'Same'))\n    #model.add(Dropout(0.5))\n    model.add(Conv2D(20, kernel_size=5, activation = 'relu', padding = 'Same'))\n    model.add(Conv2D(20, kernel_size=5, activation = 'relu', padding = 'Same'))\n    \n    model.add(Flatten())\n    model.add(Dense(100))\n    model.add(Dense(10, activation = 'softmax'))\n # define your model normally\n    model.compile(loss = \"categorical_crossentropy\", optimizer = \"Adam\", metrics = [\"accuracy\"])\n\"\"\"\n\n\nmodel = Sequential()\nmodel.add(Conv2D(32, kernel_size=5, activation = 'relu', padding = 'Same', input_shape = (28,28,1)))\nmodel.add(Conv2D(32, kernel_size=5, activation = 'relu', padding = 'Same'))\nmodel.add(Dropout(0.5))\n\nmodel.add(Conv2D(64, kernel_size=5, activation = 'relu', padding = 'Same'))\nmodel.add(Conv2D(64, kernel_size=5, activation = 'relu', padding = 'Same'))\nmodel.add(Dropout(0.5))\n\nmodel.add(Conv2D(64, kernel_size=5, activation = 'relu', padding = 'Same'))\nmodel.add(Conv2D(64, kernel_size=5, activation = 'relu', padding = 'Same'))\nmodel.add(Dropout(0.5))\n\n\nmodel.add(Flatten())\nmodel.add(Dense(100))\nmodel.add(Dense(10, activation = 'softmax'))\n\nmodel.compile(loss = \"categorical_crossentropy\", optimizer = \"RMSprop\", metrics = [\"accuracy\"])\nbatch_size = 500\nX_training,X_valid,y_training, y_valid = train_test_split(X_train, y_train, test_size = 0.1)\n\n\n\nreduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.2,\n                              patience=3, min_lr=0.00001)\n\n#model.fit_generator(data_generator_with_aug.flow(X_training,y_training, batch_size=batch_size), epochs = 50, steps_per_epoch = 75, validation_data = (X_valid, y_valid), callbacks = [reduce_lr])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we experimented with an alternative method found here: https://www.kaggle.com/muhammadshahzadkhan/mnist-digit-recognizer-using-cnn-keras\nWe thought this was interesting as it involved using the weights from the augmented data to initialise the model on the real training data. Ultimately it proved to give worse results than simply augmenting all the data and training the model in the way presented above, however it was an interesting approach that could be explored in the future."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\ntrained_weights = model.get_weights()\nK.clear_session()\nreduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.2,\n                              patience=5, min_lr=0.001)\nmodel.set_weights(trained_weights)\nmodel.fit(X_training, y_training,\n                      batch_size = batch_size,\n                      validation_data=(X_valid, y_valid),\n                      epochs=50,\n                      verbose=1, callbacks = [reduce_lr])\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1> Predictions </h1>\n\nPrediction in the required format for competition."},{"metadata":{"trusted":true},"cell_type":"code","source":"# preds = model.predict(X_test, verbose = 1)\n# preds = np.argmax(preds,axis = 1)\n# test_submission = pd.DataFrame({'ImageId':range(1,28001),'Label': preds})\n# test_submission.to_csv('submission_12.csv', index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}