{"cells":[{"metadata":{"_uuid":"14cbe9960af3a79e7146e93e22ac4ae5fa065c15"},"cell_type":"markdown","source":"# MNIST classification\nIn this notebook I tried to train and optimize a range of models on the MNIST dataset. The final result will be a stacking classifier of SVM, RF and XGB. I used other people's work and combined this into a more complete analysis. I am a Python beginner (more used to R) so bare with me.."},{"metadata":{"_uuid":"5f4d9bea9132f7d9ad56b3a5ddbfdf504b4b7b23"},"cell_type":"markdown","source":"### Sources (among others)\n- https://jakevdp.github.io/PythonDataScienceHandbook/05.08-random-forests.html\n- https://www.kaggle.com/archaeocharlie/a-beginner-s-approach-to-classification\n"},{"metadata":{"_uuid":"7d59d8e013453bf8481470fd342dcdca42be6fdc"},"cell_type":"markdown","source":"# Basics"},{"metadata":{"_uuid":"70f9b00352a08e42ff33940d436de07981df20c6"},"cell_type":"markdown","source":"### Imports"},{"metadata":{"trusted":true,"_uuid":"ef0b769e0e86bda1e110d39e237e2295fcb90b96"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn import svm \nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.ensemble import BaggingClassifier\nfrom xgboost import XGBClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.metrics import classification_report,confusion_matrix\nfrom sklearn.model_selection import StratifiedShuffleSplit\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.model_selection import RandomizedSearchCV\n\nimport seaborn as sns \nimport time\n#from scipy import stats\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1dda739be81823f4076ab401fef32f54d0616657"},"cell_type":"markdown","source":"### Global variables"},{"metadata":{"trusted":true,"_uuid":"f21e44b9329d39f8765c4a0e5351dab3e55bc610"},"cell_type":"code","source":"seed = 2\n\ntrain_size = 0.8\ntest_size = 1- train_size\n\nlabeled_images = pd.read_csv('../input/train.csv')\ntest_data=pd.read_csv('../input/test.csv') #The kaggle test set (to get leaderboard score). For kaggle kernel add: ../input/\nsample_size = len(labeled_images.index) #total number of instances for training and testing","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b9a9d88f4f155728ea07a383de47f82de331e9d3"},"cell_type":"markdown","source":"### Functions"},{"metadata":{"trusted":true,"_uuid":"85246989daa581f6939faa1ea5d3429a0c74d677"},"cell_type":"code","source":"def plot_as_grid(images, labels, m):\n    \"Plot N x M grid of digits and labels/predictions\"\n    n_pixels = len(images.columns)\n    dimension = int(np.sqrt(n_pixels))\n\n    # set up the figure\n    fig = plt.figure(figsize=(6, 6))  # figure size in inches\n    fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)\n\n    # plot the digits: each image is max mxm pixels\n    for i in range( min(m*m, len(images.index))):\n        ax = fig.add_subplot(m, m, i + 1, xticks=[], yticks=[])    \n    \n        img=images.iloc[i].values.reshape((dimension,dimension))\n    \n        ax.imshow(img, cmap=plt.cm.binary, interpolation='nearest')\n        ax.text(0, 7, str(labels.iloc[i,0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b4748b5e38b0e08c95ce595d118963184bc5d3b"},"cell_type":"code","source":"def param_selection(model, X, y, param_random, n_iter = 20, n_jobs = 2, n_folds = 3):\n    \"Determines the best parameters using randomSearch (slow)\"\n    search = RandomizedSearchCV(model, param_random, n_iter = n_iter, n_jobs = n_jobs ,\n                                random_state = seed, cv=n_folds, iid=True) \n    \n    search.fit(X, y)\n    return search.best_params_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"deba4876c33a2e8d864b9b3bee86bac792175d31"},"cell_type":"code","source":"def inspect_performance(model, train_images, train_labels, test_images, test_labels, ypred):\n    \"Prints training performance, test performance and a performance report\"\n    print(\"Training error: \", model.score(train_images,train_labels))\n    print(\"Test error: \", model.score(test_images,test_labels))\n    print(\"Test report: \")\n    print(metrics.classification_report(ypred, test_labels))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9a85f83fceef3032de746707a1a235ecf367a4b"},"cell_type":"code","source":"def plot_confusion_matrix(labels, predictions):\n    \"Plots a confusion matrix using a heatmap\"\n    mat = confusion_matrix(labels, predictions)\n    sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False)\n    plt.xlabel('true label')\n    plt.ylabel('predicted label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bf2742d7213e27cabce58dfea5dde6b6527e1f7"},"cell_type":"code","source":"def plot_incorrect_classifications(ypred, test_labels, test_images):\n    \"Plots incorrectly classified images and corresponding prediction\"\n    ypred = pd.DataFrame(ypred)\n    ypred = ypred.set_index(test_labels.index.values)\n    ypred.columns = ['prediction']\n    predict_df = pd.concat([ypred, test_labels], axis=1)\n    predict_df['Incorrect'] = predict_df.prediction != predict_df.label\n    idx = predict_df.index[predict_df['Incorrect']]\n\n    plot_as_grid(test_images.loc[idx], predict_df['prediction'].loc[idx].to_frame(), 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8cb300295f3e8cfb687678c7f51def725a7cce99"},"cell_type":"code","source":"def make_submission_file(model, data, name):\n    \"Makes a submission file by predicting on test_data and returning the predictions\"\n    data[data>0]=1\n    results=model.predict(data) \n\n    #Set to proper format:\n    df = pd.DataFrame(results)\n    df.index.name='ImageId'\n    df.index+=1\n    df.columns=['Label']\n    df.to_csv(name + '.csv', header=True)\n    return(results)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"345e320effdaf10d500cb4c2f634e2b462de5bf3"},"cell_type":"markdown","source":"### Pre-processing"},{"metadata":{"trusted":true,"_uuid":"e5e6d54c18a30db88ead0ed327730a2a1a53ded6"},"cell_type":"code","source":"labeled_images_sample = labeled_images.sample(n=sample_size, random_state= seed)\nimages = labeled_images_sample.iloc[:,1:] # select images (all columns except the first)\n\nmax_value = images.values.max()\nimages /= max_value #scale\n\nlabels = labeled_images_sample.iloc[:,:1] # select labels (only first column)\n\ntrain_images, test_images,train_labels, test_labels = train_test_split(images, labels, train_size=train_size, \n                                                                       test_size=test_size, random_state=seed)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f8195de421bd0ce0026fdc6b17337c115112fe8"},"cell_type":"markdown","source":"### Visualisation(s)"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"5774ea066235ee3252f2b747d7209c78718586b8"},"cell_type":"code","source":"plot_as_grid(train_images, train_labels, 8) #digit and label","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"748dd5b5baede7f99bbec56ccfb72b5f6ce7b069"},"cell_type":"markdown","source":"# Model 1: SVM\nAccording to following steps:\n- train model\n- inspect performance\n- error analysis\n- write to .csv\n\n**Note**: There is no parameter tuning for SVM because it is typically a slow algorithm. It takes too long to train. Also, I expect SVM not to be the best model for digit recognition so I won't spend too much time on it. "},{"metadata":{"_uuid":"6768224e1a89f22030a83343d53821e2757b4309"},"cell_type":"markdown","source":"### Train model"},{"metadata":{"trusted":true,"_uuid":"5ead2ce661d3a3d298a2ef495407c1f208cd72bb"},"cell_type":"code","source":"clf = svm.SVC(gamma='auto', C=1.0, kernel='rbf')\nclf.fit(train_images, train_labels.values.ravel())\nypred_svm = clf.predict(test_images) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39c88b28feee8b1ac34816b59ef0181efec3bef1"},"cell_type":"markdown","source":"### Inspect performance"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"3cb1ffd69906c6f7774122a3a915d06cd8ff79f8"},"cell_type":"code","source":"#takes long, for now commented out\n#inspect_performance(clf, train_images, train_labels, test_images, test_labels, ypred_svm)    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59fb5de583b229d07cda9d75125f7d5dce7c6e86"},"cell_type":"markdown","source":"### Error analysis"},{"metadata":{"trusted":true,"_uuid":"2494afc545ec6e3f9ef7ade50c05d8f7ed708515"},"cell_type":"code","source":"plot_confusion_matrix(test_labels, ypred_svm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"011ad503b0d605af63f3cd6b2eacfe6385155548"},"cell_type":"code","source":"plot_incorrect_classifications(ypred_svm, test_labels, test_images) #digit and classification  ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e96c6bb58154764fa6347c026eb5ac1620ce7ee2"},"cell_type":"markdown","source":"### Write to .csv"},{"metadata":{"trusted":true,"_uuid":"999b0ac4b85cc32865ec4049857b21a63dcf9db4"},"cell_type":"code","source":"ypred_svm_test = make_submission_file(clf, test_data, \"results_svm\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed6c7ce976968beda5c8c86510f6599d03ff22bd"},"cell_type":"markdown","source":"# Model 2: Random Forest\nAccording to following steps:\n- train model with optimal parameters\n- inspect performance\n- error analysis\n- write to .csv"},{"metadata":{"_uuid":"a35725e34e8f0c3193256be5c64d022b3551953a"},"cell_type":"markdown","source":"### Train model with optimal parameters"},{"metadata":{"trusted":true,"_uuid":"7e3038b4a6c10064a0872a6057cc82cea8989e30"},"cell_type":"code","source":"start_time = time.clock()\nparam_random = {\"n_estimators\": [400, 700], # you can try multiple options, as long as you update n_iter accordingly:  \"n_estimators\": [400, 500, 600, 700], then n_iter should be 4*2\n                'max_features' : ['auto', 'log2']} \nparameters = param_selection(RandomForestClassifier(), train_images, train_labels.values.ravel(), param_random, n_iter = 8, n_folds = 3 ) \nprint(parameters)\nprint (time.clock() - start_time, \"seconds\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d519294f0436de13d87f6f4a2166ac20dd747a65"},"cell_type":"code","source":"rf = RandomForestClassifier(n_estimators = parameters['n_estimators'], max_features = parameters['max_features'], \n                            random_state=seed,n_jobs=2)\nrf.fit(train_images, train_labels.values.ravel())\nypred_rf = rf.predict(test_images) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be3f0667ca1eda82d5d445feb054bfea04c92b18"},"cell_type":"markdown","source":"### Inspect performance"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"ac08c3c3117d154bab815009bc7dc9ac60319fbc"},"cell_type":"code","source":"inspect_performance(rf, train_images, train_labels, test_images, test_labels, ypred_rf)    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8659ba67f70083c70f9be5ad1b3f1d3db5290c2"},"cell_type":"markdown","source":"### Error analysis"},{"metadata":{"trusted":true,"_uuid":"e2f38cb64c67ad863eb1295ac5d00eb4558ecc2d"},"cell_type":"code","source":" plot_confusion_matrix(test_labels, ypred_rf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6bfbb0e97417ef699d1014d21bbe949b51ade5d"},"cell_type":"code","source":"plot_incorrect_classifications(ypred_rf, test_labels, test_images) #digit and classification    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3b4acbd303c3bc9c2e7603e5a631e5e666b0c90"},"cell_type":"markdown","source":"### Write to .csv"},{"metadata":{"trusted":true,"_uuid":"44e2899fc4a6c0c9b088d739e7692fa3fa3e5730"},"cell_type":"code","source":"ypred_rf_test = make_submission_file(rf, test_data, \"results_rf\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"177130ad1a3ecd8b33d50e9cd787780e189b381b"},"cell_type":"markdown","source":"# Model 3: Xtreme Gradient Boosting\nAccording to following steps:\n- train model with optimal parameters\n- inspect performance\n- error analysis\n- write to .csv"},{"metadata":{"_uuid":"ec53859924fbd8a0eae0ee1bb5f0eab0a48d0654"},"cell_type":"markdown","source":"### Train model with optimal parameters"},{"metadata":{"trusted":true,"_uuid":"34a79916089ef823277208e96b361523d53219e0"},"cell_type":"code","source":"start_time = time.clock()\nparam_random = {\"eta\": [0.3], # you can try multiple options, as long as you update n_iter accordingly:  \"eta\": [0.01, 0.02, 0.05, 0.1, 0.25, 0.5], then n_iter should be 6\n                \"early_stopping_rounds\": [50],\n                \"n_estimators\" : [100],\n                \"eval_metric\" : [\"merror\"]}\nparameters = param_selection(XGBClassifier(), train_images, train_labels.values.ravel(), param_random, n_iter = 1) \nprint(parameters)\nprint (time.clock() - start_time, \"seconds\")\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f324f5dd35d2386b3b43f3fc0f5bf190191544b7"},"cell_type":"code","source":"xgb = XGBClassifier(eta = parameters[\"eta\"], early_stopping_rounds = parameters[\"early_stopping_rounds\"], \n                    n_estimators = parameters[\"n_estimators\"],eval_metric = parameters[\"eval_metric\"],n_jobs=2)\nxgb.fit(train_images,train_labels.values.ravel())\nypred_xgb = xgb.predict(test_images)  ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"128bacf20cd05059b663a9ce004fe0f07b689cb0"},"cell_type":"markdown","source":"### Inspect performance"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"cb018ccb43054d91fbc0e08aef80408d0b40acf0"},"cell_type":"code","source":"inspect_performance(xgb, train_images, train_labels, test_images, test_labels, ypred_xgb)    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"360a47605f8eb57246707672b00baa3f31699099"},"cell_type":"markdown","source":"### Error analysis"},{"metadata":{"trusted":true,"_uuid":"06dd588291c1bf905b45635571aa96cd6352f776"},"cell_type":"code","source":"plot_confusion_matrix(test_labels, ypred_xgb)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be30737599f66f3dd0124e3db90a0c696130c361"},"cell_type":"code","source":"plot_incorrect_classifications(ypred_rf, test_labels, test_images) #digit and classification","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"85227b7c3aec70114ac49ace4b51655eecc41542"},"cell_type":"markdown","source":"### Write to .csv"},{"metadata":{"trusted":true,"_uuid":"1b9ab4fc96ebe0dfe25fe7e00e93abdf89559491"},"cell_type":"code","source":"ypred_xgb_test = make_submission_file(rf, test_data, \"results_xgb\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"baf5acc46d5fcf52c9012ae4f7a0a354b316cb74"},"cell_type":"markdown","source":"# Model 4: Stacking classifier\nStack SVM, RF and XGB together in a stacking classifier. Parameter tuning turned out to be a computational nightmare so I will try this approach instead."},{"metadata":{"trusted":true,"_uuid":"a4106234525e99b00cb0a706ed49dedd4d0bcff8"},"cell_type":"code","source":"stack_train = pd.concat([pd.DataFrame(ypred_svm) , pd.DataFrame(ypred_rf), pd.DataFrame(ypred_xgb)], axis=1)\nstack_test = pd.concat([pd.DataFrame(ypred_svm_test) , pd.DataFrame(ypred_rf_test), pd.DataFrame(ypred_xgb_test)], axis=1)\n\nlr = LogisticRegression(solver ='newton-cg', multi_class=\"auto\")\nlr.fit(stack_train,test_labels.values.ravel())\nypred_lr = lr.predict(stack_test)  \n\nresults=ypred_lr\nname = \"results_stacking\"\n#Set to proper format:\ndf = pd.DataFrame(results)\ndf.index.name='ImageId'\ndf.index+=1\ndf.columns=['Label']\ndf.to_csv(name + '.csv', header=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These different models can be submitted. Please let me know if this notebook was helpful and/or if you have feedback."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}