{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Goal\n### I am the new learner in data science. In this time, I'm trying to work on the digit recognition project.\n### I would like to compare different data scaling and model selection, and it can understand what is the difference accuracy among the methods"},{"metadata":{},"cell_type":"markdown","source":"# Importing Library"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\n#print(os.listdir())\n\n#Plot Graph\nimport matplotlib.pyplot as plt\n\n#--------------For Machine Learning-----------------#\n##Preprocessing\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import MinMaxScaler\n##Modeling\nfrom sklearn import svm\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import RandomForestClassifier\n#CNN\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D\nfrom keras.utils import np_utils\n##Model Saving and Reading\nfrom sklearn.externals import joblib\nfrom keras.models import load_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading Training Data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#train_data = pd.read_csv('../input/train.csv')\n#test_data = pd.read_csv('../input/test.csv')\ntrain_data = pd.read_csv('../input/digit-recognizer/train.csv')\ntest_data = pd.read_csv('../input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image Edge Extraction "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Show the training sample digit\nplt.figure(num='training_sample',figsize=(10,10)) \nfor row in range(1,26):\n    plt.subplot(5, 5, row) #row must be >0 for subplot function\n    plt.title('Digit:' + str(train_data.iloc[row-1,0]))\n    plt.axis('off')\n    plt.imshow(train_data.iloc[row-1,1:].values.reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ~~ Start ~~"},{"metadata":{},"cell_type":"markdown","source":"# Starting to Split data for Training Model\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Split tarin-test sets of 80%-20% respectively, \n#and set the random_state for consistent even I rerun the project.\nx_train, x_test, y_train, y_test = train_test_split(train_data.iloc[:, 1:], \n                                                      train_data.iloc[:, 0], test_size = 0.2, random_state = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Data Description\nx_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.value_counts().sort_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2. Scaling\n\n#### Reason of Scaling --- https://medium.com/greyatom/why-how-and-when-to-scale-your-features-4b30ab09db5e\n\nMost of the times, your dataset will contain features highly varying in magnitudes, units and range. But since, most of the machine learning algorithms use Eucledian distance between two data points in their computations, this is a problem.\n\nIf left alone, these algorithms only take in the magnitude of features neglecting the units. The results would vary greatly between different units, 5kg and 5000gms. The features with high magnitudes will weigh in a lot more in the distance calculations than features with low magnitudes.\n\nTo supress this effect, we need to bring all features to the same level of magnitudes. This can be acheived by scaling.\n\n## Scaling Method:\nI would like to try all scaling method to understand what is the difference of accuracy in one of the models.\n\nIt contains 4 scaling methods:\n1. Standarisation\n2. Mean Normalisation\n3. Min-Max Scaling\n4. Unit Vector"},{"metadata":{},"cell_type":"markdown","source":"Create the function to find the model score with different scaling methods.\n\nFurther Improvement: \n1. Add the hyperparameter tuning into the function for more efficiency and convenient.\n2. Add the training fitting score(cross validation checking score)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Function to find out all scaling result to compare:\ndef model_score(model_name , x_train, x_test, y_train, y_test):\n    model_path = '../input/digit-recognition-model-backup/' #read back the trained model\n    score_save_path = '../input/'\n    history_path = '../input/'\n    score_list = {}\n    scaling_method = {}\n    model_algo = {}\n    stdsc = StandardScaler()    \n    scaling_method = {'original':[x_train, x_test],\n                      'Standarisation':[stdsc.fit_transform(x_train), stdsc.fit_transform(x_test)],\n                      'Mean Normalisation':[x_train.apply(lambda x: (x - np.mean(x))/(255-0)), \n                                           x_test.apply(lambda x: (x - np.mean(x))/(255-0))],\n                      'Unit':[x_train.apply(lambda x: (x - 0)/(255 - 0)),\n                                     x_test.apply(lambda x: (x - 0)/(255 - 0))]}\n    for method in scaling_method:\n        _x_train, _x_test = scaling_method[method]\n        \n        ###SVM###-----------------------------------------------------------\n        if model_name.upper() == 'SVM':\n            try:\n                model = joblib.load(model_path + 'svm_model' + '_' + method + '.pkl')\n                print('Model Reading Success')\n            except: #If no exist model, we train\n                print('No existed Model, it is fitting...')\n                model = svm.SVC(gamma = 0.0001) #Since 'Auto' for orginial data is expensive\n                model.fit(_x_train, y_train)\n                print('Model is fitted')\n            #Model Saving\n            #joblib.dump(model, model_path + 'svm_model' + '_' + method + '.pkl') # Kaggle Only provide read mode\n        ###Logistics###-----------------------------------------------------    \n        elif model_name.upper() == 'LOGISTIC':\n            try:\n                model = joblib.load(model_path + 'log_model' + '_' + method + '.pkl')\n                print('Model Reading Success')\n            except:\n                print('No existed Model, it is fitting...')\n                model = LogisticRegression(random_state = 1)\n                model.fit(_x_train, y_train)\n                print('Model is fitted')\n            #Model Saving\n            #joblib.dump(model, model_path + 'log_model' + '_' + method + '.pkl')\n        \n        ###Decision Tree###-------------------------------------------------\n        elif model_name.upper() == 'DECISION TREE':\n            try:\n                model = joblib.load(model_path + 'tree_model' + '_' + method + '.pkl')\n                print('Model Reading Success')\n            except:\n                model = DecisionTreeClassifier(random_state = 1)\n                model.fit(_x_train, y_train)\n                print('Model is fitted')\n            #Model Saving\n            #joblib.dump(model, model_path + 'tree_model' + '_' + method + '.pkl')\n        \n        ###Random Forecast###-----------------------------------------------\n        elif model_name.upper() == 'RANDOM FOREST':\n            try:\n                model = joblib.load(model_path + 'RF_model' + '_' + method + '.pkl')\n                print('Model Reading Success')\n            except:\n                model = RandomForestClassifier(random_state = 1)\n                model.fit(_x_train, y_train)\n                print('Model is fitted')\n            #Model Saving\n            #joblib.dump(model, model_path + 'RF_model' + '_' + method + '.pkl')\n\n\n        ###Convolutional Neural Network (CNN) ###----------------------------\n        elif model_name.upper() == 'CNN':\n            num_class = 10 #Digit 0-9\n            #Data Reshaping\n            if isinstance(_x_train, pd.DataFrame):\n                re_x_train = _x_train.values.reshape(_x_train.shape[0], 28,28,1).astype('float32')\n                re_x_test = _x_test.values.reshape(_x_test.shape[0], 28,28,1).astype('float32')\n            else:\n                re_x_train = _x_train.reshape(_x_train.shape[0], 28,28,1).astype('float32')\n                re_x_test = _x_test.reshape(_x_test.shape[0], 28,28,1).astype('float32')\n\n            re_y_train = np_utils.to_categorical(y_train, num_class)\n            re_y_test = np_utils.to_categorical(y_test, num_class)\n            \n            try:\n                model = load_model(model_path + 'cnn_model' + '_' + method + '.h5')\n                print('Model Reading Success')\n            except:\n                ##model building\n                model = Sequential()\n                #convolutional layer with rectified linear unit activation\n                model.add(Conv2D(32, kernel_size=(3, 3),   #32 convolution filters used each of size 3x3\n                                 activation='relu',\n                                 input_shape=(28,28,1)))\n                model.add(Conv2D(64, (3, 3), activation='relu')) #64 convolution filters used each of size 3x3\n                model.add(MaxPooling2D(pool_size=(2, 2))) #choose the best features via pooling\n                #randomly turn neurons on and off to improve convergence\n                model.add(Dropout(0.25))\n                #flatten since too many dimensions, we only want a classification output\n                model.add(Flatten())\n                #fully connected to get all relevant data\n                model.add(Dense(128, activation='relu'))\n                #one more dropout for convergence' sake :) \n                model.add(Dropout(0.5))\n                #output a softmax to squash the matrix into output probabilities\n                model.add(Dense(num_class, activation='softmax'))\n                #Model Compile\n                model.compile(loss=keras.losses.categorical_crossentropy,\n                              optimizer=keras.optimizers.Adadelta(),\n                              metrics=['accuracy'])\n                #Model Fitting\n                history = model.fit(re_x_train,\n                          re_y_train,\n                          batch_size = 128,\n                          epochs = 12,\n                          verbose = 1,\n                          validation_data = (re_x_test,\n                                             re_y_test)\n                         )\n                #Model Saving\n                #model.save(model_path + 'cnn_model' + '_' + method + '.h5')  \n                hist_df = pd.DataFrame(history.history) # convert the history.history dict to a pandas DataFrame\n                # save to json:  \n                hist_json_file = history_path + 'cnn_model' + '_' + method + '_history.json' \n                with open(hist_json_file, mode='w') as f:\n                    hist_df.to_json(f)\n                \n                \n        else:\n            raise NameError('Model Name Error')\n            \n        ###Fitting into the test data to get the score###---------------------\n        print('Model Saved')\n        print('Fitting the score')\n        if model_name.upper() in ['SVM', 'LOGISTIC', 'DECISION TREE', 'RANDOM FOREST']:\n            score = model.score(_x_test, y_test)\n            print(model_name.upper() + '-' + method + \"_score: \"+ str(score))\n            score_list[model_name + '_' + method] = score\n            model_algo[method] = model\n        if model_name.upper() in ['CNN']:\n            score = model.evaluate(re_x_test, re_y_test, verbose = 0)\n            print('Test loss:', score[0])\n            print('Test accuracy:', score[1])\n            score_list[model_name + '_' + method] = score[1]\n            model_algo[method] = model\n            \n    ###To get the score DF -----------------------------------------------------\n    score_df = pd.DataFrame.from_dict(score_list, orient='index').reset_index(drop = False)\n    split = score_df[\"index\"].str.split(\"_\", n = 1, expand = True)\n    score_df['model'] = split[0]\n    score_df['scaling_method'] = split[1]\n    score_df = score_df.drop(columns = ['index'])\n    score_df.columns = ['score', 'model', 'scaling_method']\n    score_df = score_df[['model', 'scaling_method', 'score']]\n    score_df.to_excel(score_save_path +model_name + '_score.xlsx', index = False)\n    print('Score is saved as Excel')\n    return(score_df, model_algo)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model_1 - SVM"},{"metadata":{"trusted":true},"cell_type":"code","source":"svm_score, svm_algo = model_score('SVM', x_train, x_test, y_train, y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model_2 - Logistics Regression"},{"metadata":{"trusted":true},"cell_type":"code","source":"log_score, log_algo = model_score('Logistic', x_train, x_test, y_train, y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model_3 - Decision Tree"},{"metadata":{"trusted":true},"cell_type":"code","source":"tree_score, tree_algo = model_score('Decision Tree', x_train, x_test, y_train, y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model_4 - Random Forest"},{"metadata":{"trusted":true},"cell_type":"code","source":"rf_score, rf_algo = model_score('Random Forest', x_train, x_test, y_train, y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model_5 - CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"cnn_score, cnn_algo = model_score('CNN', x_train, x_test, y_train, y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#The below function is used to get back the CNN history training record,\n#and even for plotting the training and testing trend\nimport json,codecs\ndef read_cnn_hist():\n    data_scaling = ['original', 'Standarisation', 'Mean Normalisation', 'Unit']\n    dict_hist = {}\n    backup_path = '../input/digit-recognition-model-backup/'\n    for scaling in data_scaling:\n        try:\n            path = backup_path + 'cnn_model_' + scaling + '_history.json'\n            with codecs.open(path, 'r', encoding='utf-8') as f:\n                    n = json.loads(f.read())\n            dict_hist[scaling] = n\n        except:\n            print('No such ' + scaling + ' history file')\n    return(dict_hist)\ncnn_hist = read_cnn_hist()\ncnn_hist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_train_history(history_dict):\n    plt.subplots_adjust(wspace =0, hspace =0.7)\n    plt.subplot(2,1,1)\n    plt.title('Loss -- Train vs Test')\n    plt.plot(history_dict['loss'].values())\n    plt.plot(history_dict['val_loss'].values())\n    plt.ylabel('Loss')\n    plt.xlabel('Epoch')\n    plt.legend(['train', 'test'], loc='upper right')\n    \n    plt.subplot(2,1,2)\n    plt.title('Acc -- Train vs Test')\n    plt.plot(history_dict['acc'].values())\n    plt.plot(history_dict['val_acc'].values())\n    plt.ylabel('Accuracy')\n    plt.xlabel('Epoch')\n    plt.legend(['train', 'test'], loc='upper right')\n\n#Here I Just show the Unit Scaling data in training CNN history \nshow_train_history(history_dict=cnn_hist['Unit'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3. Model and Scaling Methods Comparison"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read all score excel\nimport glob\nall_data = pd.DataFrame()\ninput_path = '../input/digit-recognition-model-backup/'\nfor f in glob.glob(input_path + \"*.xlsx\"):\n    df = pd.read_excel(f)\n    all_data = all_data.append(df,ignore_index=True)\nall_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show the comparison in table:\npivot_table = pd.pivot_table(all_data, index = ['scaling_method'], columns = ['model'], values = 'score').sort_index(ascending = False)\npivot_table.style.highlight_max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above is shown the maximum value among the data scaling method among different models."},{"metadata":{},"cell_type":"markdown","source":"# Step 4. Hyperparameters tuning\nIf we seek more accuary model, we can try the hyperparameters tuning. It returns the best hyper-parameter value.(e.g. gamma, regulation_rule value etc.)\nHowever, the running cost is very expensive. For example, in SVM, I tried it and my local computer's CPU was overheat.\nTherefore, I just write down the code and method for reference."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Hyperparameter Grid Search\n#from sklearn.model_selection import GridSearchCV\n#from sklearn.pipeline import make_pipeline\n#pipe_svc = make_pipeline(svm.SVC(random_state = 1))\n#param_range = [0.0001, 0.001, 0.1, 1.0, 100.0, 1000.0]\n#param_grid = [{'svc__C' : param_range, 'svc__kernel' :['linear']},\n#            {'svc__C' : param_range, 'svc__gamma' : param_range, 'svc__kernel' :['linear']}]\n#gs = GridSearchCV(estimator = pipe_svc, param_grid= param_grid, scoring = 'accuracy', cv = 10 , n_jobs = 3)\n#gs = gs.fit(X_train_std, x_label)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The hyper-parameters tuning of Decision Tee example is shown below: "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Hyperparameter Grid Search\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.metrics import roc_auc_score\n\npipe_tree = DecisionTreeClassifier(random_state = 1)\nparam_grid = {'max_depth':np.arange(3, 10)}\ngs = GridSearchCV(estimator  = pipe_tree, param_grid = param_grid)\ngs.fit(x_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 5. Cross Validation Checking\nCross validation checking is used to prevent the overfitting problem.\n\nBelow link is shown the reason and the method of CV checking:\nhttps://towardsdatascience.com/5-reasons-why-you-should-use-cross-validation-in-your-data-science-project-8163311a1e79\n\nhttp://localhost:8888/notebooks/Desktop/My_Python/Machine%20Learning/Digit%20Recognition/Digit%20Recognition%20-%20Self_learning.ipynb#4.-Unit-Vector\n\nFor my `model_score` function have shown the acceptable testing score in testing data. Therefore, the model is not overfitting. If we would like to further confirm the model , of course, we can use the cross validation checking. I would like to show the example of cross validation checking of decision tree in below:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import cross_val_score\ncross_val_score(DecisionTreeClassifier(random_state = 1), x_train, y_train, scoring='accuracy', cv = 5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above has shown the score of the decision tree model, and it verified the model is not overfitting by specific training data."},{"metadata":{},"cell_type":"markdown","source":"At the end, I would like to use the CNN model to show the predicted digit, and try to verify it whether is correct."},{"metadata":{"trusted":true},"cell_type":"code","source":"unit_x_test = x_test.apply(lambda x: (x - 0)/(255 - 0))\nunit_x_test = unit_x_test.values.reshape(unit_x_test.shape[0], 28,28,1).astype('float32')\nprediction = cnn_algo['Unit']\nprediction = prediction.predict_classes(unit_x_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prediction","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot y-train Data\nplt.figure(num='cnn_test_fig',figsize=(10,10)) \nfor row in range(1,26):\n    plt.subplot(5, 5, row) #must be >0\n    plt.title('Predicted Digit:' + str(prediction[row-1]))\n    plt.axis('off')\n    plt.imshow(x_test.reset_index(drop = True).loc[row-1].values.reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see the picture and predicted digit is almost the same. Therefore, I would like to use this model to predict the test-data for submission."},{"metadata":{},"cell_type":"markdown","source":"# Prediction For Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predict the Test.csv for submission\n# I would like to use \"Unit\" Scaling\nx_submission = test_data.apply(lambda x: (x - 0)/(255 - 0)).values.reshape(test_data.shape[0], 28,28,1).astype('float32')\nx_submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prediction = cnn_algo['Unit']\npredicted = prediction.predict_classes(x_submission)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot Submission Test Data set\nplt.figure(num='cnn_submission_fig',figsize=(10,10)) \nfor row in range(1,26):\n    plt.subplot(5, 5, row) #must be >0\n    plt.title('Predicted Digit:' + str(predicted[row-1]))\n    plt.axis('off')\n    plt.imshow(test_data.loc[row-1].values.reshape(28,28))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Submission\nsubmission = pd.DataFrame.from_dict(dict(enumerate(predicted)), orient = 'index')\nsubmission = submission.reset_index()\nsubmission.columns = ['ImageId', 'Label']\nsubmission['ImageId'] = submission['ImageId'] + 1\nsubmission.to_csv('submission.csv', index = False)\nsubmission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Summary\nFor digit recognition, most of the model using 'Unit' Scaling method would perform better result. It is the same with most of the data scientist statement. \n\nHowever, it has observed that the SVM model using 'Standardisation' would perform better. The reason might be the SVM is focusing on the maximum of the margin, and the data scaling for maximum the margin might perform better after 'Standardisation'.\n\nDuring working on this project, I have learned and noted some following points:\n1. In Data Scaling, try it to use the self-formula, not only rely on sklearn library.\n2. My model_score function can write to be better, and it can apply on different projects too.\n3. CNN model is really very useful and powerful for digit recognition.\n4. When Training model, it really should use cloud computing rather than the local computer, it really was spending lots of resource of my PC.\n5. Learning coding from other data scientist is very important, and try to summarise them and create my function would learn a lot\n6. Data is really interesting."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}