{"cells":[{"metadata":{"_uuid":"183ae15274b0742ffd368805d33a3e0e99ef8e1d"},"cell_type":"markdown","source":"# Digit Recognizer Kaggle competition\n\n## Introduction  \nIn order to solve the problem proposed at Kaggle's Digit Recognizer competition, a simple neural network will be trained and used for predictions.  \nThe objective is to correctly identify digits from a dataset of handwritten images.  \nThe algorithm used belongs to the class MLPClassifier from sklearn.neural_network and trains the neural network for multiclass classification using backpropagation.  \nDuring the analysis, the influence of some parameters and configurations of the network in the predictions result will be tested.  "},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"707bf1ab6539623abbe22a97f9870dd4cd955d8b"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import StratifiedShuffleSplit, cross_validate\nfrom sklearn.pipeline import Pipeline\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\n%matplotlib inline","execution_count":2,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"d7ad269f214d8090ebeface152d191361a63904c"},"cell_type":"code","source":"#Configure plots\nrc={'savefig.dpi': 75, 'figure.figsize': [12,8], 'lines.linewidth': 2.0, 'lines.markersize': 8, 'axes.labelsize': 18,\\\n   'axes.titlesize': 18, 'font.size': 18, 'legend.fontsize': 16}\n\nsns.set(style='dark',rc=rc)","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"e05ad6da707532cdda67eaeca7fe017ecbc466ad"},"cell_type":"markdown","source":"## Dataset  \nTwo datasets were provided. One with training data and the other with test data.  \nThey contain gray-scale images of hand-drawn digits, from zero through nine. Each image is 28 pixels in height and 28 pixels in width. That is a total of 784 features.\n\nLoading the data:"},{"metadata":{"trusted":false,"collapsed":true,"_uuid":"c138aa7942086296e3126abfa078fcd103f4faaf"},"cell_type":"code","source":"train_data = pd.read_csv('../input/train.csv')\ntest_data = pd.read_csv('../input/test.csv')","execution_count":3,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"6070c0f83c038bf3f720c3e6d0cdf2cbc68f031b"},"cell_type":"code","source":"train_data.keys()","execution_count":4,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"71c14cff065e5ce3f9288763fcaec40cd10d655d"},"cell_type":"code","source":"train_data.shape","execution_count":5,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"98f7db793edda5932a9ad41fd7b4b29a83d0c34e"},"cell_type":"code","source":"test_data.shape","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"a3608c48ec8be47b35ce380a2f5d864816561f1a"},"cell_type":"markdown","source":"Checking for missing values"},{"metadata":{"trusted":false,"_uuid":"9cc7331211522470809f1e4e077706f3d458b777"},"cell_type":"code","source":"train_data.columns[train_data.isnull().any()]","execution_count":7,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8446b4f52b09979b3cc68756c1b5504ede274597"},"cell_type":"code","source":"test_data.columns[test_data.isnull().any()]","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"914fb165366f3d20b4232cee36973a9efe92fec5"},"cell_type":"markdown","source":"There are no missing values in the train or test datasets.\n\nCounting the number of examples of each digit in the training dataset:"},{"metadata":{"trusted":false,"_uuid":"517c71a5414459c4f18255b7125d5a543d01a99c"},"cell_type":"code","source":"sns.countplot(x='label', data=train_data, color=\"#99c2ff\")","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"899d3cbdfe5abd30baec87fb90b724d44a9be573"},"cell_type":"markdown","source":"Split train dataset between labels and features:"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"6d4afbb30606f4c778a2e17af16f812bc1a64e41"},"cell_type":"code","source":"labels_train = train_data.iloc[:,0]\nfeatures_train = train_data.iloc[:,1:]","execution_count":10,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"cc62415cf7e07dda4a18f5552040e995d289688b"},"cell_type":"code","source":"labels_train.head()","execution_count":11,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"3b4a4c9f662f9633b1353deb38e44eb736653a07"},"cell_type":"code","source":"labels_train.shape","execution_count":12,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9dada00ab38a3fab9669b605fabf68c7f3c906c4"},"cell_type":"code","source":"features_train.shape","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"a3f6624d0f3d4c2ad9e5d2efe770c2ee0c01c626"},"cell_type":"markdown","source":"## Neural Network"},{"metadata":{"_uuid":"069f815cc4680132a63407a22a5849638f0c38ef"},"cell_type":"markdown","source":"**Scaling**  \nBefore building the model, the features will be scaled so that they have have mean 0 and variance 1.  \n\n**Model and Optimization**  \nThe Neural Network proposed for this problem has 1 hidden layer. The activation function is the logistic sigmoid function. And the optimization will be through the LBFGS solver.  \nThese configurations will remain unchanged during this analysis.  \nThe following settings will be modified and different options tested in order to verify how they influence the results:  \nThe number of units in the hidden layer.  \nThe regularization term (alpha).  \nThe maximum number of iterations for the solver.  \n\n**Validation**  \nThe stratified shuffle split cross-validation method was applied to validate the results. 3 iterations were made. In each one, the training dataset was split in two: 70% used to train the model and 30% to test and to calculate the accuracy. The parameters alpha, max_iter and number of units in the hidden layer were selected according to the results of the cross-validation. "},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"64ff8d40fb0e2c82a6c3a7bdb6e82b037b4872c4"},"cell_type":"code","source":"#cross-validation method\ncv = StratifiedShuffleSplit(n_splits = 3, random_state = 0, \n                                train_size = 0.70)","execution_count":14,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"bc86a601dc1df642012ba16506ce719cf80dd3ed"},"cell_type":"code","source":"#Function to perform cross-validation\ndef nn_cross_validation(alpha, units, max_iter):\n    mlp = MLPClassifier(solver='lbfgs', activation = 'logistic', random_state=0, \n                    max_iter = max_iter, alpha = alpha, hidden_layer_sizes = (units,))\n    \n    pipe = Pipeline([('scaling', StandardScaler()),\n                 ('clf', mlp)])\n    \n    scores = cross_validate(pipe, features_train, labels_train, cv=cv, scoring = 'accuracy')\n\n    return scores","execution_count":20,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"f2c323cc1416f4d6a89b89be57630f9f59c7782e"},"cell_type":"code","source":"#Function to test the different configurations and plot the result\ndef test_options(alpha, units, max_iter, parameter):    \n    mean_train = []\n    std_train = []\n    mean_test = []\n    std_test = []\n    option = 0\n    \n    if parameter == 'alpha':\n        options = alpha\n        var1, var2, var3 = option, units, max_iter\n    elif parameter == 'max_iter':\n        options = max_iter\n        var1, var2, var3 = alpha, units, option\n    else:\n        options = units\n        var1, var2, var3 = alpha, option, max_iter\n\n    for option in options:\n        \n        if parameter == 'alpha':\n            var1 = option\n        elif parameter == 'max_iter':\n            var3 = option\n        else:\n            var2 = option\n            \n        CV_scores = nn_cross_validation(var1, var2, var3)\n        mean_train = np.append(mean_train, CV_scores['train_score'].mean())\n        std_train = np.append(std_train, CV_scores['train_score'].std())\n        mean_test = np.append(mean_test, CV_scores['test_score'].mean())\n        std_test = np.append(std_test, CV_scores['test_score'].std())\n\n    plt.figure()\n    plt.errorbar(options, mean_train, std_train, linestyle='None', marker='*', c = 'b')\n    plt.errorbar(options, mean_test, std_test, linestyle='None', marker='*', c = 'r') \n\n    plt.ylabel('Accuracy')\n    plt.legend(['Train data', 'Test data'])","execution_count":33,"outputs":[]},{"metadata":{"_uuid":"c77264d5c293e328e703b51ab9b6e622f663ef1c"},"cell_type":"markdown","source":"The first test has different regularization terms. The following plot shows the main train accuracy, main test accuracy and the standard deviation of the three cross-validation iterations for each value of alpha.  \nThe test data in this case is the 30% of the training dataset selected to test the data during each iteration of the cross-validation."},{"metadata":{"trusted":false,"_uuid":"25036b33a938b0918faa513c350f391bd369121d"},"cell_type":"code","source":"alphas = [0.0001, 0.1, 1, 10, 100]\n\ntest_options(alphas, 50, 50, 'alpha')\n\nplt.xscale('log', basex = 10)\nplt.xlabel('Regularization term (alpha)')","execution_count":32,"outputs":[]},{"metadata":{"_uuid":"a38064ad66c6366e9110deb56c23485998fcf821"},"cell_type":"markdown","source":"Adding regularization to the cost function helped to increase the test data accuracy and reduced overfitting. However a very large value for alpha results in underfitting. The value 10 will be selected for alpha for this neural network.  \nNext, different values of the maximum number of iterations will be tested.  "},{"metadata":{"trusted":false,"_uuid":"38895f7a5a635360f12d05dc0eafcc64ea91d317"},"cell_type":"code","source":"max_iter = [50, 100, 150]\n\ntest_options(10, 50, max_iter, 'max_iter')\n\nplt.xlabel('Maximum number of iterations')","execution_count":35,"outputs":[]},{"metadata":{"_uuid":"35631385530afe3c9f4d258da59d6140d89747d6"},"cell_type":"markdown","source":"Increasing the maximum number of iterations makes the model prone to overfitting. In this way, one should be careful when increasing that number.  \nThe parameter max_iter equal to 100 resulted in a higher accuracy of the test data. However the test for the value 150 didn't improve that accuracy very much. The value 100 will be selected.  \n\nThe next test tries different numbers of hidden layer units.  "},{"metadata":{"trusted":false,"_uuid":"e0b2d1842ef2a18293225c96a2e73b2acb2f432f"},"cell_type":"code","source":"units = [50, 100, 150]\n\ntest_options(10, units, 100, 'units')\n\nplt.xlabel('Number of units in the hidden layer')","execution_count":38,"outputs":[]},{"metadata":{"_uuid":"ac9a5f573176ea60d30e67f407acb0c45ef8d436"},"cell_type":"markdown","source":"Increasing the number of units in the hidden layer to 100 improved the accuracy. In the case of 150, the gain wasn't very relevant. 100 units will be used in the model."},{"metadata":{"_uuid":"389c0fdec73bbba24f6093ef4cb2dbced729df4f"},"cell_type":"markdown","source":"## Final Model and Predictions  \n\nBased on the configurations selected through cross-validation, the final model will be trained using all the data from the training dataset. Then the test dataset will be used to make the predictions. \n\nApplying the standard scaler:"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"8ff21aae6196f3b05b86379ea5238448984f923f"},"cell_type":"code","source":"scaler = StandardScaler()\n\nfeatures_transf = scaler.fit_transform(features_train)","execution_count":39,"outputs":[]},{"metadata":{"_uuid":"a9ee68c8bd79264558b0844d9a1a86380cd597e3"},"cell_type":"markdown","source":"Training the neural network:"},{"metadata":{"trusted":false,"_uuid":"9a21bd403cd71121c103f2ed257d291980add23c"},"cell_type":"code","source":"mlp = MLPClassifier(solver='lbfgs', activation = 'logistic', random_state=0, \n                    max_iter = 100, alpha = 10, hidden_layer_sizes = (100,))\n\nmlp.fit(features_transf, labels_train)","execution_count":40,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c19adbee137bab63678df7fdc24d6dd18794996b"},"cell_type":"code","source":"mlp.score(features_transf, labels_train)","execution_count":41,"outputs":[]},{"metadata":{"_uuid":"ccd6911dbedb1a6af6cb0f1ece021831b754fcd6"},"cell_type":"markdown","source":"The accuracy of the trained data is 0.99. That is very high and it is possible that the model is overfitted. However, that accuracy was high at all the cases tested. Improving the model with a more complex neural network or with another classification algorithm may avoid this issue and result in a better accuracy for the test dataset.  \n\nNext, the predictions will be calculated for the test data and saved in an output file."},{"metadata":{"trusted":true,"_uuid":"69164258b7d84c0dfec5ce964b98faa4a5d9ee63"},"cell_type":"code","source":"test_data_transf = scaler.transform(test_data)\npred = mlp.predict(test_data_transf)","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"fc0a1d33f1f89c8109e417b2be2c7af38e02f72e"},"cell_type":"markdown","source":"Ploting the results:"},{"metadata":{"trusted":false,"_uuid":"891c9c46665f9b4a0844052c4799378951926801"},"cell_type":"code","source":"unique, counts = np.unique(pred, return_counts=True)\nresult = dict(zip(unique, counts))\n\nplt.bar(result.keys(), result.values())","execution_count":51,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"fc747cf780307153026385c8a7079d0ffc0b7a23"},"cell_type":"code","source":"output = {}\n\noutput['Label'] = pred","execution_count":77,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"9dab0e74167caea1bae82d334918535da301a149"},"cell_type":"code","source":"output = pd.DataFrame(output)\noutput.index+=1","execution_count":79,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"ad1b7beba0b84a58f2d8689ab55227c9735428b4"},"cell_type":"code","source":"output.to_csv('output.csv', index_label = 'ImageID')","execution_count":80,"outputs":[]},{"metadata":{"_uuid":"574f15c9da9bf6fd9e3760dce32417bbdfa3bf0c"},"cell_type":"markdown","source":"## Conclusion\n\nA simple neural network was created to recognize handwritten digits. The network configuration is described below:\n\n - Number of hidden layers = 1\n - Number of hidden layer units = 100\n - Activation function = logistic sigmoid function\n - Solver = LBFGS\n - Regularization term = 10\n - Maximum number of iterations = 100\n\nThe algorithm used trains the neural network for multiclass classification using backpropagation.  \nThe model built is prone to overfitting, since it presented train data accuracy above 0.99 and the test data accuracy wasn't that high during the cross-validation. In spite of that issue, the model presented very good results for the test data acurracy. The best result was above 0.96. Other methods could be applied to improve that accuracy, but depending on the application this method may offer a suitable result.  "},{"metadata":{"_uuid":"ff7e97a5da359d09dc833dc3ea3f1bc55554a737"},"cell_type":"markdown","source":"## References  \nLinks to references used in this project are listed below:  \nhttp://scikit-learn.org/stable/modules/neural_networks_supervised.html  \nhttp://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm  \nhttps://machinelearningmastery.com/train-final-machine-learning-model/  \nhttps://stackoverflow.com/questions/22481854/plot-mean-and-standard-deviation  \nhttps://stackoverflow.com/questions/15777951/how-to-suppress-pandas-future-warning  \nhttps://stackoverflow.com/questions/28663856/how-to-count-the-occurrence-of-certain-item-in-an-ndarray-in-python  \nhttps://stackoverflow.com/questions/20167930/start-index-at-1-when-writing-pandas-dataframe-to-csv  "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}