{"cells":[{"metadata":{},"cell_type":"markdown","source":"This is experimental kernel. Here, before using more complex deep learning models, I want to test simplest possible ones to see, how they can handle with this data. I was managed to get 97% accuracy on classic MNIST data using MLP with 1 hidden layer and I'm curious about how much I can get here.\nSo, let's get started."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Importing all necessary libraries\nimport pandas as pd\nimport numpy as np\nimport cv2 as cv\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nimport os\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.utils.np_utils import to_categorical\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\n\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importing datasets\ntrain_df = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest_df = pd.read_csv('../input/Kannada-MNIST/test.csv')\n\n# I want to use Dig-MNIST for final validation\nval_df = pd.read_csv('../input/Kannada-MNIST/Dig-MNIST.csv')\nsub_df = pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\n\n# Pandas conversts data to int64 dtype, so, to reduce memory usage, I'll convert it to uint8 dtype\ntrain_df = train_df.astype(np.uint8)\ntest_df = test_df.astype(np.uint8)\nval_df = val_df.astype(np.uint8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df.info(), '\\n')\nprint(test_df.info(), '\\n')\nprint(val_df.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_df has 'id' column, we need to drop it before predicting\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look whether balanced our data or not."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (13, 4))\nfor i, dataset, name in ((1, train_df, 'train_df'), (2, val_df, 'val_df')):\n    plt.subplot(f'12{i}')\n    counts = dataset['label'].value_counts()\n    sns.barplot(x = counts.index, y = counts.values).set_title(name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both train_df and val_df val_df are ballanced.\n\nOn next step I want to plot some images from train_df and val_df."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating function for plotting\ndef plot_images(dataset, figsize = (17, 17)):\n    \n    '''\n    Plots 100 images from selected dataset in 10x10 shape.\n    '''\n    \n    fig = plt.figure(figsize = figsize)\n    for i in range(10):\n        data = dataset[dataset['label'] == i].drop('label', axis = 1) # Data, that contains only i'th labels\n\n        for j in range(10):\n            ax = fig.add_subplot(10, 10, int(f'{i}{j}') + 1)\n            img = data.sample(1) # I'm taking random sample from data to plot\n            index = img.index[0] # Index for title\n            img = np.array(img).reshape((28, 28)) # To plot our image, we need to reshape it to 28x28\n            plt.imshow(img, cmap = 'gray') # Plot image\n            plt.axis('off') # Don't show X and Y axes \n            ax.set_title(f'{i} ({index})') # Set plot title\n\n    plt.tight_layout() # Doesn't allow our plots overlap each other","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train_df images\nplot_images(train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dig-MNIST images\nplot_images(val_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I need to make important note about Dig-MNIST dataset (val_df) - it's looks like we have augmented data here, it's clear that some digits shifted or scaled, also we have vertical and horizontal lines on some images.\n\nIn this kernel I'm not going to clean this data or use data augmentation for train_df, I'll use this dataset as is to create some scores for future models comparison."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data preparation\n# Train data\nY = train_df['label']\nX = train_df.drop('label', axis = 1)\n\n# Validation data\ny_val = val_df['label']\nX_val = val_df.drop('label', axis = 1)\n\n# Dropping 'id' column in test dataset\ntest_df = test_df.drop('id', axis = 1)\n\n# Normalize the data\nX = X / 255.0\nX_val = X_val / 255.0\ntest_df = test_df / 255.0\n\n# One-hot encoding of train features\nY = to_categorical(Y, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating train and test datasets for model training\n# I'm using stratify to ensure that we have equal proportion of samples from each class in our datasets\nX_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.2, stratify = Y, random_state = 666) \nX_train.shape, X_test.shape, y_train.shape, y_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, I want to test simplest possible models - single-layer perceptron and multi-layer perceptron with 1 and 2 hidden layers.\n\nBecause this is fully connected networks, we can use next rules to find a number of layers and nodes in each layer:\n\n* The number of hidden layers equals one and the number of neurons in that layer is the mean of the neurons in the input and output layers. \n* The number of hidden neurons should be between the size of the input layer and the size of the output layer.\n* The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.\n* The number of hidden neurons should be less than twice the size of the input layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dictionaries to store our models results\nresults = {} # Accuracy and loss\npreds = {} # Model predictions from val_df\nepochs = 20","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll start with basic model - it's a single-layer perceptron, it includes only input and output layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Base model\nname = 'Base_Model'\n\n# Crating and training model\nmodel = Sequential()\nmodel.add(Dense(10, input_shape = (784, ), activation = 'softmax'))\nmodel.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\nhistory = model.fit(X_train, y_train, epochs = epochs, batch_size = 64, validation_data = [X_test, y_test], verbose = 0)\n\n# Appending results to dictionaries\nresults[name] = history.history\npreds[name] = model.predict(X_val)\n\n# Creating predictions for test_df and submission file\nsub_preds = model.predict_classes(test_df)\nid_col = np.arange(sub_preds.shape[0])\nsubmission = pd.DataFrame({'id': id_col, 'label': sub_preds})\nsubmission.to_csv(f'{name}.csv', index = False)  \n\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next - multi-layer perceptron with 1 hidden layer. I'm going to train multiple models with different number of nodes and different activation functions - relu and sigmoid, because sigmoid typically has good result on shallow networks (no more than 2 hidden layers).\n\nAlso I want to use different initializations for each activation - 'he_normal' for relu and 'glorot_uniform' for sigmoid."},{"metadata":{"trusted":true},"cell_type":"code","source":"# 1 hidden layer model\n\nnodes = (128, 256, 397, 512, 1024)\nactivations = ('relu', 'sigmoid')\n\nfor node in nodes:\n    for act in activations:\n        name = f'1_hidden_{node}_nodes_{act}_activation'\n        print(f'Training: {name}')\n        \n        model = Sequential()       \n        if act == 'relu':            \n            model.add(Dense(node, input_shape = (784, ), activation = 'relu', kernel_initializer='he_normal'))\n        else:\n            model.add(Dense(node, input_shape = (784, ), activation = 'sigmoid', kernel_initializer='glorot_uniform'))        \n        model.add(Dense(10, activation = 'softmax'))        \n        model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n        \n        history = model.fit(X_train, y_train, epochs = epochs, batch_size = 64, validation_data = [X_test, y_test], verbose = 0)\n        \n        # Appending results to dictionaries\n        results[name] = history.history\n        preds[name] = model.predict(X_val)\n        \n        # Creating predictions for test_df and submission file\n        \n        sub_preds = model.predict_classes(test_df)\n        id_col = np.arange(sub_preds.shape[0])\n        submission = pd.DataFrame({'id': id_col, 'label': sub_preds})\n        submission.to_csv(f'{name}.csv', index = False)       ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next - model with 2 hidden layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"# 2 hidden layers\nnodes = (128, 256, 397, 512, 1024)\nactivations = ('relu', 'sigmoid')\n\nfor node in nodes:\n    for act in activations:\n        name = f'2_hidden_{node}_nodes_{act}_activation'\n        print(f'Training: {name}')\n        \n        model = Sequential()        \n        if act == 'relu':\n            model.add(Dense(node, input_shape = (784, ), activation = 'relu', kernel_initializer='he_normal'))\n            model.add(Dense(node, activation = 'relu', kernel_initializer='he_normal'))\n        else:\n            model.add(Dense(node, input_shape = (784, ), activation = 'sigmoid', kernel_initializer='glorot_uniform'))\n            model.add(Dense(node, activation = 'sigmoid', kernel_initializer='glorot_uniform'))        \n        model.add(Dense(10, activation = 'softmax'))        \n        model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n        \n        history = model.fit(X_train, y_train, epochs = epochs, batch_size = 64, validation_data = [X_test, y_test], verbose = 0)\n        \n        # Appending results to dictionaries\n        results[name] = history.history\n        preds[name] = model.predict(X_val)\n        \n        # Creating predictions for test_df and submission file\n        sub_preds = model.predict_classes(test_df)\n        id_col = np.arange(sub_preds.shape[0])\n        submission = pd.DataFrame({'id': id_col, 'label': sub_preds})\n        submission.to_csv(f'{name}.csv', index = False)  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can look at our results.\n\nI'm going to plot accuracies and losses for each model:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for key in results.keys():\n    fig = plt.figure(figsize = (15, 4))\n    plt.subplot(121)\n    plt.plot(results[key]['accuracy'], label = 'acc')\n    plt.plot(results[key]['val_accuracy'], label = 'val_acc')\n    plt.legend()\n    plt.title(f'{key} accuracy')\n    \n    plt.subplot(122)\n    plt.plot(results[key]['loss'], label = 'loss')\n    plt.plot(results[key]['val_loss'], label = 'val_loss')\n    plt.legend()\n    plt.title(f'{key} loss')\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that all models have a good accuracy, but models with relu activation converges faster and tends to overfit, models with sigmoid activation is more stable, but have higher losses.\n\nLet's look at predictions on Dig-MNIST dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for key in preds.keys():\n    print(f'{key}: {accuracy_score(y_val, preds[key].argmax(axis = 1))}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's a poor results, but as I mentioned earlier - Dig-MNIST have augmented images with white lines, so it was quite obvious that we get such results.\n\nAlso, I tried some submissions and got next scores on public leaderboard:\n* Base_Model 10 epochs - 0.90740\n* 2_hidden_1024_nodes_sigmoid_activation 10 epochs - 0.94060\n* 2_hidden_1024_nodes_relu_activation 10 epochs - 0.94500\n* 1_hidden_397_nodes_sigmoid_activation 4 epochs - 0.90720\n* 1_hidden_397_nodes_relu_activation 4 epochs - 0.94100\n* 1_hidden_1024_nodes_relu_activation 4 epochs - 0.94420\n\nSo, we can see that using simple model we can get accuracy score about 95%.\n\nNext steps - data augmentation and using of convolutional neural networks, it will be in another kernel."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}