{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Solving MNIST\n\nProblem statement: Predict digit value given handwriting picture (28 x 28 pixel)\nType: Multiclass Classification (0-9 digit)\nPerformance metric: Accuracy; Since number of samples each label is quite the same.\n\nThe problem might be solved by these learning algorithms:\n1. Multiclass Classifier\n2. Ensemble Binary Classifier\n3. Multilayer Perceptron\n4. Convolutional Neural Network\n\nWhat I did in this notebook:\n1. Data pre-processing:\n    a. Flatten the data (for Multiclass Classifier, Ensemble Binary Classifier, Multilayer Perceptron).\n    b. Normalize the data so it could be faster to converge.\n2. Modelling:\n    a. Compare performance metric on cross-validation for each learning algorithm."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Library\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.models import Sequential\nimport numpy as np\nimport pandas as pd\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport datetime\n\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import train_test_split \n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ndf_train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ndf_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\n\nX_train = df_train.iloc[:,1:]\ny_train = df_train.iloc[:,0]\n\nX_train, X_val, y_train, y_val = train_test_split(X_train, y_train,\n                                                    stratify = y_train,\n                                                    test_size=0.2, random_state = 41)\n\nX_train.shape, y_train.shape, X_val.shape, y_val.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Data Pre-Processing\nIn this section, I want to see samples of training data; and prepare the dataset so it could be fed into learning algorithm."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Proportion of samples per digit in training dataset\n# I need to make sure whether the dataset is balance or imbalance\n\nunique, counts = np.unique(y_train, return_counts=True)\nproportions = counts/counts.sum()\ndict(zip(unique, proportions))\n\n# The samples are slightly well distributed. The proportion of each label between 9-10%.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# See samples of the image\ndef series_to_array(series):\n    data = np.array(series)\n    return data\n\n# Plot Digit\ndef plot_digit(series):\n    data = series_to_array(series)\n    image = data.reshape(28, 28)\n    plt.imshow(image, cmap = mpl.cm.binary,\n               interpolation=\"nearest\")\n    plt.axis(\"off\")\n\nplot_digit(X_train.iloc[10,:])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize the data\n# To get faster time to converge, I need to scale the feature value to the value between 0-1.\n# Currently the value representation is between 0-255. So we need to divide it with 255.\n\nX_train_flat = X_train/255\nX_val_flat = X_val/255\n\nX_train_flat.shape, X_val_flat.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Modelling\nIn this section, I will do following tasks:\n1. Prepare cross-validation function\n2. Define several models\n3. Compare performance metric of each models"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Stratified k-fold being used in order to get stratified split\nskfolds = StratifiedKFold(n_splits=3)\n\n# Calculate Accuracy using cross-validation\ndef calculate_cv_accuracy(model, X, y):\n    acc = cross_val_score(model, X, y, cv=skfolds, scoring = \"accuracy\")\n    return acc\n    \ndef evaluate_models(models, X_train, y_train, X_test, y_test):\n    results = []\n    for name, model in models:\n        cv_score = calculate_cv_accuracy(model, X_train, y_train)\n        model.fit(X_train, y_train)\n        y_pred = model.predict(X_test)\n        test_acc = accuracy_score(y_test, y_pred)\n        results.append((name, model, cv_score.mean(), cv_score.std(), test_acc))\n        msg = \"{}: Accuracy on CV {} ({}); Test {}\".format(name, cv_score.mean(), cv_score.std(), test_acc)\n        print(msg)\n    \n    return results","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Multiclass Classifier\nIn this section, I use Logistic Regression and Random Forest to build multiclass classifier. These two learning algorithms have different approach to solve the problem and I want to see the result out of two different algorithm."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define model\nlogistic_multi = LogisticRegression(multi_class='multinomial') # solver sage & scaled feature are used to get faster converge time\nrf_multi = RandomForestClassifier(n_estimators = 50, random_state=41)\n\n# List of models\nmodels = []\nmodels.append((\"Logistic Multiclass\", logistic_multi))\nmodels.append((\"Random Forest Multiclass\", rf_multi))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evaluate models\n# The evaluation done in cross-validation and test dataset\n# So I could make sure the model is not overfitting the training data\nmulticlass_result = evaluate_models(models, X_train_flat, y_train, X_val_flat, y_val)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Ensemble Binary Classifiers\nInstead of creating one model to classify 10 labels, I want to create 10 models for each labels and ensemble them. The ensemble learning usually have better performance if it is constructed with different models."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Before creating the ensemble model,\n# We need to adjust the target for each binary classifiers first\n\n# Let's create target for binary classification\ny_train_0 = (y_train == 0)\ny_train_1 = (y_train == 1)\ny_train_2 = (y_train == 2)\ny_train_3 = (y_train == 3)\ny_train_4 = (y_train == 4)\ny_train_5 = (y_train == 5)\ny_train_6 = (y_train == 6)\ny_train_7 = (y_train == 7)\ny_train_8 = (y_train == 8)\ny_train_9 = (y_train == 9)\n# Now we have separated target for each digit\n\n# Lets define Binary Classifier for each digit\n# I will use two type binary classifier for each digit i.e. Logistic Regression and Random Forest\nrf_0 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_1 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_2 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_3 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_4 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_5 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_6 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_7 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_8 = RandomForestClassifier(n_estimators = 2, random_state=41)\nrf_9 = RandomForestClassifier(n_estimators = 2, random_state=41)\n\nlogit_0 = LogisticRegression(l1_ratio= 0.5)\nlogit_1 = LogisticRegression(l1_ratio= 0.5)\nlogit_2 = LogisticRegression(l1_ratio= 0.5)\nlogit_3 = LogisticRegression(l1_ratio= 0.5)\nlogit_4 = LogisticRegression(l1_ratio= 0.5)\nlogit_5 = LogisticRegression(l1_ratio= 0.5)\nlogit_6 = LogisticRegression(l1_ratio= 0.5)\nlogit_7 = LogisticRegression(l1_ratio= 0.5)\nlogit_8 = LogisticRegression(l1_ratio= 0.5)\nlogit_9 = LogisticRegression(l1_ratio= 0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evaluate each of binary classifiers\nbinary_models_rf = []\nbinary_models_rf.append((\"Random Forest digit 0\", rf_0))\nbinary_models_rf.append((\"Random Forest digit 1\", rf_1))\nbinary_models_rf.append((\"Random Forest digit 2\", rf_2))\nbinary_models_rf.append((\"Random Forest digit 3\", rf_3))\nbinary_models_rf.append((\"Random Forest digit 4\", rf_4))\nbinary_models_rf.append((\"Random Forest digit 5\", rf_5))\nbinary_models_rf.append((\"Random Forest digit 6\", rf_6))\nbinary_models_rf.append((\"Random Forest digit 7\", rf_7))\nbinary_models_rf.append((\"Random Forest digit 8\", rf_8))\nbinary_models_rf.append((\"Random Forest digit 9\", rf_9))\n\nbinary_models = []\nbinary_models.append((\"Logistic Regression digit 0\", logit_0))\nbinary_models.append((\"Logistic Regression digit 1\", logit_1))\nbinary_models.append((\"Logistic Regression digit 2\", logit_2))\nbinary_models.append((\"Logistic Regression digit 3\", logit_3))\nbinary_models.append((\"Logistic Regression digit 4\", logit_4))\nbinary_models.append((\"Logistic Regression digit 5\", logit_5))\nbinary_models.append((\"Logistic Regression digit 6\", logit_6))\nbinary_models.append((\"Logistic Regression digit 7\", logit_7))\nbinary_models.append((\"Logistic Regression digit 8\", logit_8))\nbinary_models.append((\"Logistic Regression digit 9\", logit_9))\n\ndict_y_train = {'y_train_0': y_train_0,\n               'y_train_1': y_train_1,\n               'y_train_2': y_train_2,\n               'y_train_3': y_train_3,\n               'y_train_4': y_train_4,\n               'y_train_5': y_train_5,\n               'y_train_6': y_train_6,\n               'y_train_7': y_train_7,\n               'y_train_8': y_train_8,\n               'y_train_9': y_train_9\n               }\ni = 0\nfor name, model in binary_models:\n    y = (\"y_train_\" + str(i))\n    cv_score = calculate_cv_accuracy(model, X_train_flat, dict_y_train[y])\n    msg = \"{}: Accuracy on CV {} ({});\".format(name, cv_score.mean(), cv_score.std())\n    i += 1\n    print(msg)\n\ni = 0\nfor name, model in binary_models_rf:\n    y = (\"y_train_\" + str(i))\n    cv_score = calculate_cv_accuracy(model, X_train_flat, dict_y_train[y])\n    msg = \"{}: Accuracy on CV {} ({});\".format(name, cv_score.mean(), cv_score.std())\n    i += 1\n    print(msg)    \n    \n# We now have different binary classifiers for each label with different performance\n# The accuracy of each binary classifiers ranging from  95% - 99%\n# I am afraid that each classifiers is overfitting the training dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit each of binary classifiers\nrf_0.fit(X_train_flat, y_train_0)\nrf_1.fit(X_train_flat, y_train_1)\nrf_2.fit(X_train_flat, y_train_2)\nrf_3.fit(X_train_flat, y_train_3)\nrf_4.fit(X_train_flat, y_train_4)\nrf_5.fit(X_train_flat, y_train_5)\nrf_6.fit(X_train_flat, y_train_6)\nrf_7.fit(X_train_flat, y_train_7)\nrf_8.fit(X_train_flat, y_train_8)\nrf_9.fit(X_train_flat, y_train_9)\n\nlogit_0.fit(X_train_flat, y_train_0)\nlogit_1.fit(X_train_flat, y_train_1)\nlogit_2.fit(X_train_flat, y_train_2)\nlogit_3.fit(X_train_flat, y_train_3)\nlogit_4.fit(X_train_flat, y_train_4)\nlogit_5.fit(X_train_flat, y_train_5)\nlogit_6.fit(X_train_flat, y_train_6)\nlogit_7.fit(X_train_flat, y_train_7)\nlogit_8.fit(X_train_flat, y_train_8)\nlogit_9.fit(X_train_flat, y_train_9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prepare training data for the second layer model\n# The training data is the result of prediction of each models in first layer\n# I'll put probability prediciton from Random Forest and Logistic Regression\n\ndef generate_second_layer_data(X):\n\n    X2_pred_0 = rf_0.predict_proba(X)\n    X2_pred_0  = [item[1] for item in X2_pred_0]\n    X2_pred_1 = rf_1.predict_proba(X)\n    X2_pred_1  = [item[1] for item in X2_pred_1]\n    X2_pred_2 = rf_2.predict_proba(X)\n    X2_pred_2  = [item[1] for item in X2_pred_2]\n    X2_pred_3 = rf_3.predict_proba(X)\n    X2_pred_3  = [item[1] for item in X2_pred_3]\n    X2_pred_4 = rf_4.predict_proba(X)\n    X2_pred_4  = [item[1] for item in X2_pred_4]\n    X2_pred_5 = rf_5.predict_proba(X)\n    X2_pred_5  = [item[1] for item in X2_pred_5]\n    X2_pred_6 = rf_6.predict_proba(X)\n    X2_pred_6  = [item[1] for item in X2_pred_6]\n    X2_pred_7 = rf_7.predict_proba(X)\n    X2_pred_7  = [item[1] for item in X2_pred_7]\n    X2_pred_8 = rf_8.predict_proba(X)\n    X2_pred_8  = [item[1] for item in X2_pred_8]\n    X2_pred_9 = rf_9.predict_proba(X)\n    X2_pred_9  = [item[1] for item in X2_pred_9]\n\n    X2_log_pred_0 = logit_0.predict_proba(X)\n    X2_log_pred_0  = [item[1] for item in X2_log_pred_0]\n    X2_log_pred_1 = logit_1.predict_proba(X)\n    X2_log_pred_1  = [item[1] for item in X2_log_pred_1]\n    X2_log_pred_2 = logit_2.predict_proba(X)\n    X2_log_pred_2  = [item[1] for item in X2_log_pred_2]\n    X2_log_pred_3 = logit_3.predict_proba(X)\n    X2_log_pred_3  = [item[1] for item in X2_log_pred_3]\n    X2_log_pred_4 = logit_4.predict_proba(X)\n    X2_log_pred_4  = [item[1] for item in X2_log_pred_4]\n    X2_log_pred_5 = logit_5.predict_proba(X)\n    X2_log_pred_5  = [item[1] for item in X2_log_pred_5]\n    X2_log_pred_6 = logit_6.predict_proba(X)\n    X2_log_pred_6  = [item[1] for item in X2_log_pred_6]\n    X2_log_pred_7 = logit_7.predict_proba(X)\n    X2_log_pred_7  = [item[1] for item in X2_log_pred_7]\n    X2_log_pred_8 = logit_8.predict_proba(X)\n    X2_log_pred_8  = [item[1] for item in X2_log_pred_8]\n    X2_log_pred_9 = logit_9.predict_proba(X)\n    X2_log_pred_9  = [item[1] for item in X2_log_pred_9]\n\n    # Create dataframe of all the result\n    data = {'X2_zero':X2_pred_0,\n            'X2_log_zero':X2_log_pred_0,\n            'X2_one':X2_pred_1,\n            'X2_log_one':X2_log_pred_1,\n            'X2_two':X2_pred_2,\n            'X2_log_two':X2_log_pred_2,\n            'X2_three':X2_pred_3,\n            'X2_log_three':X2_log_pred_3,\n            'X2_four':X2_pred_4,\n            'X2_log_four':X2_log_pred_4,\n            'X2_five':X2_pred_5,\n            'X2_log_five':X2_log_pred_5,\n            'X2_six':X2_pred_6,\n            'X2_log_six':X2_log_pred_6,\n            'X2_seven':X2_pred_7,\n            'X2_log_seven':X2_log_pred_7,\n            'X2_eight':X2_pred_8, \n            'X2_log_eight':X2_log_pred_8, \n            'X2_nine':X2_pred_9,\n            'X2_log_nine':X2_log_pred_9\n           }\n\n    return pd.DataFrame(data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# See samples of prediction result\ndf_X2 = generate_second_layer_data(X_train_flat)\ndf_X2.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define model for second layer in stacking\n# I will use also random forest and training it using the predictions of several models in first layer\nrf_final = RandomForestClassifier(n_estimators = 10, random_state=41)\nlogistic_final = LogisticRegression(multi_class='multinomial', solver='saga')\n\nrf_final.fit(df_X2, y_train)\nlogistic_final.fit(df_X2, y_train)\n\nstacking_models = []\nstacking_models.append(('Random Forest 2nd layer', rf_final))\nstacking_models.append(('Logistic Regression 2nd layer', logistic_final))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prediction in Stacking model\n# I will use test dataset to see whether the stacking model will perform better or not\ndf_test_X2 = generate_second_layer_data(X_val_flat)\n\nfor name, model in stacking_models:\n    y_pred = model.predict(df_test_X2)\n    acc = accuracy_score(y_val, y_pred)\n    msg = \"{}: Accuracy on Test {};\".format(name, acc)\n    print(msg)\n    \n# It turns out that our stacking model do not outperform the multiclass classifiers earlier\n# The multiclass classifier has 0.9677 accuracy on test dataset\n# The ensemble binary classifier has 0.9461 accuracy on test dataset\n# Having these finding, We learn that multiclass random forest classifier perform better on the MNIST dataset\n# I believe that the stacking model is overfitting the training dataset,\n# Supporteb by the fact that each binary classifiers have really high accuracy on training dataset, but not in the test dataset.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Multilayer Perceptron\nIn this section I will approach the problem using Multilayer Perceptron (MLP) or Neural Network . The implementation will be done using keras framework."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize the X_train and X_test\n# So it could be in range of 0-1\n# It will help MLP to converge faster\n\nX_train = X_train / 255\nX_val = X_val / 255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Since the keras only take input of array, I need to convert dataframe to array\narr_X_train = X_train.to_numpy()\narr_X_val = X_val.to_numpy()\narr_y_train = y_train.to_numpy()\narr_y_val = y_val.to_numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# First I need to define the model first\n# I will use 2 hidden layers in this model and 384/256 units for each layers\n# I will use elu activation and set he initialization of each weight\n# L1 regularization is applied to get faster convergence time in sparse dataset like MNIST\n\nmlp_model = keras.Sequential([\n#     keras.layers.Flatten(input_shape = (28,28)),\n    keras.layers.Dense(384, activation= 'relu'\n                       , kernel_initializer= 'he_normal'\n                       , kernel_regularizer= keras.regularizers.l1(0.00001)\n                      ),\n    keras.layers.Dense(256, activation= 'relu'\n                       , kernel_initializer= 'he_normal'\n                       , kernel_regularizer= keras.regularizers.l1(0.0001)\n                      ),\n    keras.layers.Dense(10)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compile the model\nmlp_model.compile(optimizer= 'adam',\n                  loss= tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n                  metrics= ['accuracy']\n                 )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the model\nmlp_model.fit(arr_X_train, arr_y_train, epochs=8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evaluate model in test dataset\ntest_loss, test_accuracy = mlp_model.evaluate(arr_X_val,arr_y_val, verbose= 0)\nprint(\"MLP Accuracy on Test: {}\".format(test_accuracy))\n\n# It turns out that MLP model do outperform the multiclass classifiers earlier\n# It has 0.98 accuracy on test dataset! 0.0073 better!\n# The multiclass classifier has 0.9677 accuracy on test dataset\n# The ensemble binary classifier has 0.9461 accuracy on test dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create prediction\n# Prediction for submission\narr_X_test = df_test.to_numpy()\narr_X_test = arr_X_test / 255\narr_y_pred = mlp_model.predict(arr_X_test)\n\ny_pred_mlp = []\nfor i in range(len(arr_y_pred)):\n    y_pred_mlp.append(np.argmax(arr_y_pred[i]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create file submisssion from MLP\nsubmission_mlp = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\nsubmission_mlp.iloc[:,1] = (y_pred_mlp)\nsubmission_mlp.to_csv(\"submission_mlp.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Convolutional Neural Network (CNN)\nIn this section I will approach the problem using Convolutional Neural Network (CNN). The difference with pervious MLP is instead of connecting all feature to all units in layer, it selects only particular visual receptor. The CNN often use in visual applications such as object detection and semantic segmentation."},{"metadata":{"trusted":true},"cell_type":"code","source":"arr_X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prepare dataset for CNN\n\n# Get data from previous model\narr_X_train = X_train.to_numpy()\narr_X_val = X_val.to_numpy()\narr_y_train = y_train.to_numpy()\narr_y_val = y_val.to_numpy()\n\n# Get data from csv file and normalize it\narr_X_test = df_test.to_numpy()\narr_X_test = arr_X_test / 255\n\n# input image dimensions\nimg_rows, img_cols = 28, 28\nnum_classes = 10\n\n# Reshape the array to (28,28,1)\nX_train = arr_X_train.reshape(arr_X_train.shape[0], img_rows, img_cols, 1)\nX_val = arr_X_val.reshape(arr_X_val.shape[0], img_rows, img_cols, 1)\nX_test = arr_X_test.reshape(arr_X_test.shape[0], img_rows, img_cols, 1)\ninput_shape = (img_rows, img_cols, 1)\n\n# convert class vectors to binary class matrices\ny_train = keras.utils.to_categorical(y_train, num_classes)\ny_val = keras.utils.to_categorical(y_val, num_classes)\n\nX_train.shape, y_train.shape, X_val.shape, y_val.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define CNN architecture\n# The model is constructed by several layers\n# I use relu activation and he initializer to get faster converge time\n# The first layer is convolutional layer with 16 filter, 3x3 kernel size, 1 stride\n# The second layer is the same with the first layer. They are supposed to capture feature maps from digit image.\n# The third layer is pooling with maximum aggregation. It is used to reduce the size of feature maps.\n# The fourth layer is convolutional layer for the result from maximum pooling.\n# The next layer is dense neural network with 256 units, before feeding this layer with data I need to flatten first the data\n# The last layer / output layer is dense neural network with 10 units (the same with number of class)\n\ncnn_model = keras.models.Sequential([\n    keras.layers.Conv2D(16, kernel_size=(3, 3), activation='relu', input_shape=input_shape, kernel_initializer= 'he_normal'),\n    keras.layers.Conv2D(16, kernel_size=(3, 3), activation='relu', kernel_initializer= 'he_normal'),\n    keras.layers.MaxPooling2D(pool_size=(3, 3)),\n    keras.layers.Conv2D(16, kernel_size=(3, 3), activation='relu', kernel_initializer= 'he_normal'),\n    keras.layers.Flatten(),\n    keras.layers.Dense(256, activation='relu', kernel_initializer= 'he_normal'),\n    keras.layers.Dense(num_classes, activation='softmax')\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compile the model\ncnn_model.compile(optimizer= 'adam',\n                  loss=keras.losses.categorical_crossentropy,\n                  metrics= ['accuracy']\n                 )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the model\ncnn_model.fit(X_train, y_train, epochs=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Evaluate model in test dataset\ntest_loss, test_accuracy = cnn_model.evaluate(X_val, y_val, verbose= 0)\nprint(\"CNN Accuracy on Test: {}\".format(test_accuracy))\n\n# It turns out that CNN model do outperform previos models\n# It has 0.99 accuracy on test dataset!!! 0.01 better than MLP. It is huge improvement.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prediction for submission\narr_y_pred = cnn_model.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create list of prediction\ny_pred = []\nfor i in range(len(arr_y_pred)):\n    y_pred.append(np.argmax(arr_y_pred[i]))    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create file submisssion from CNN\nsubmission_cnn = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\nsubmission_cnn.iloc[:,1] = (y_pred)\nsubmission_cnn.to_csv(\"submission_cnn.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":4}