{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"1. [Introduction](#introduction)\n1. [Data Collection](#datacollection)\n1. [Data Analysis](#dataanalysis)\n    1. [Explore Dataset](#exploredataset)\n    1. [Explore Target Variable](#exploretargetvariable)\n    1. [Data Visualization](#datavisualization)\n1. [Data Preprocessing](#datapreprocessing)\n    1. [PCA](#dppca)\n    1. [Deskew](#dpdeskew)\n    1. [Data Augmentation](#dpdataaugmentation)\n    1. [Pipeline](#dppipeline)\n1. [Model Training and Evaluation](#trainandevaluate)\n    1. [Estimate PCA params](#estimatepcaparams)\n    1. [Estimate Scalers](#estimatescalers)\n    1. [Estimate Deskew](#estimatedeskew)\n    1. [Estimate Different Models (default params)](#estimatedifferentmodels)\n    1. [Fine-tuning](#finetuning)\n         1. [SVC](#finetuningsvc)\n         1. [Logistic Regression](#finetuninglr)\n         1. [RandomForestClassifier](#finetuningrf)\n         1. [ExtraTreesClassifier](#finetuninget)\n         1. [KNeighborsClassifier](#finetuningknn)\n         1. [HistGradientBoostingClassifier](#finetuninghb)\n         1. [AdaBoostClassifier (base: HistGradientBoostingClassifier)](#finetuningadbhb)\n         1. [AdaBoostClassifier (base: ExtraTreesClassifier)](#finetuningadbet)\n         1. [AdaBoostClassifier (base: GaussianNB)](#finetuningadbgnb)\n    1. [Estimations Results](#estimationsresults)\n    1. [Ensemble learning](#ensemblelearning)\n        1. [VotingClassifier](#votingclassifier)\n1. [Submission](#submission)\n1. [Conclusion](#conclusion)","metadata":{}},{"cell_type":"markdown","source":"<center>\n<div style=\"color:;\n           display:fill;\n           border-radius:5px;\n           background-color: lightgray;\n           font-size:110%;\n           letter-spacing:0.5px\">\n\n<h2 style=\"padding: 10px;\n              color:white;\">MNIST: Digits Recognizer\n</h2>\n</div>","metadata":{}},{"cell_type":"markdown","source":"# Introduction <a class=\"anchor\" id=\"introduction\"></a>","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we will explore the popular MNIST dataset which contains a large number of images of handwritten digits. Our goal is to train a classifier (without using of neural networks) that can accurately recognize the digits in these images.\n\nThis Notebook Covers:\n\n1. Exploratory Data Analysis\n1. Building preprocessing pipeline\n1. Data augmentation\n1. Comparison of different models' baseline scores\n1. Tuning best models\n1. Evaluation ensemble classifiers\n1. Final submission (achieved Public Score 0.992).\n\nOverall, this notebook serves as an introduction to image classification with machine learning and provides insights into the techniques and tools used for this task.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nimport itertools\nimport functools\nimport re\nimport copy\nimport imgaug.augmenters as iaa\nimport xgboost as xgb\nfrom mlxtend import classifier\nfrom mlxtend.classifier import EnsembleVoteClassifier\nfrom sklearn.experimental import enable_halving_search_cv \nfrom sklearn.model_selection import train_test_split, GridSearchCV, HalvingGridSearchCV, cross_val_predict, learning_curve\nfrom sklearn.linear_model import LogisticRegression, SGDClassifier\nfrom sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier, BaggingClassifier, VotingClassifier, ExtraTreesClassifier, StackingClassifier, HistGradientBoostingClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.neighbors import KNeighborsClassifier, RadiusNeighborsClassifier\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.svm import SVC\nfrom sklearn.preprocessing import MinMaxScaler, StandardScaler, Binarizer\nfrom sklearn.pipeline import Pipeline, make_pipeline\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.metrics import confusion_matrix, accuracy_score, classification_report, f1_score, precision_score, recall_score, make_scorer,roc_curve, precision_recall_curve\nfrom sklearn.decomposition import PCA\nfrom skopt import BayesSearchCV, space, plots\nfrom scipy.stats import randint\nfrom scipy.ndimage import interpolation\nfrom IPython.display import Markdown, display\nfrom matplotlib.colors import ListedColormap\nfrom sklearn.manifold import TSNE\n\n# to make this notebook's output stable across runs\nnp.random.seed(42)\n\nwarnings.simplefilter(action='ignore', category=(RuntimeWarning, FutureWarning))\n\nON_KAGGLE = os.getenv(\"KAGGLE_KERNEL_RUN_TYPE\") is not None\n\nverbose_tuning = 0\n\ndef download_from_kaggle(competition):\n    command = \"kaggle competitions download -c \"\n    filepath = Path(\"data/\" + competition + \".zip\")\n\n    if not filepath.is_file():\n        subprocess.run((command + competition).split())\n        Path(\"data\").mkdir(parents=True, exist_ok=True)\n        shutil.unpack_archive(competition + \".zip\", \"data\")\n        shutil.move(competition + \".zip\", \"data\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-09T20:12:26.700866Z","iopub.execute_input":"2023-04-09T20:12:26.702233Z","iopub.status.idle":"2023-04-09T20:12:26.716879Z","shell.execute_reply.started":"2023-04-09T20:12:26.702157Z","shell.execute_reply":"2023-04-09T20:12:26.715497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Collection","metadata":{}},{"cell_type":"code","source":"competition = \"digit-recognizer\"\nif not ON_KAGGLE:\n    download_from_kaggle(competition)\n    train_path = \"data/train.csv\"\n    test_path = \"data/test.csv\"\nelse:\n    train_path = f\"/kaggle/input/{competition}/train.csv\"\n    test_path = f\"/kaggle/input/{competition}/test.csv\"\n    \ntrain_data = pd.read_csv(train_path)\ntest_data = pd.read_csv(test_path)","metadata":{"execution":{"iopub.status.busy":"2023-04-09T18:39:51.346760Z","iopub.execute_input":"2023-04-09T18:39:51.347246Z","iopub.status.idle":"2023-04-09T18:39:57.052303Z","shell.execute_reply.started":"2023-04-09T18:39:51.347198Z","shell.execute_reply":"2023-04-09T18:39:57.051191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Analysis <a class=\"anchor\" id=\"dataanalysis\"></a>","metadata":{}},{"cell_type":"markdown","source":"## Explore Dataset <a class=\"anchor\" id=\"exploredataset\"></a>","metadata":{}},{"cell_type":"markdown","source":"Explore the MNIST dataset by analyzing its distribution, visualizing some of its samples.","metadata":{}},{"cell_type":"code","source":"print(f'Train dataset shape:\\t {train_data.shape}',\n    f'\\nTest dataset shape:\\t {test_data.shape}')","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:13.913367Z","iopub.execute_input":"2023-04-08T10:24:13.913619Z","iopub.status.idle":"2023-04-08T10:24:13.919079Z","shell.execute_reply.started":"2023-04-08T10:24:13.913595Z","shell.execute_reply":"2023-04-08T10:24:13.917755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:13.920070Z","iopub.execute_input":"2023-04-08T10:24:13.920369Z","iopub.status.idle":"2023-04-08T10:24:13.940703Z","shell.execute_reply.started":"2023-04-08T10:24:13.920340Z","shell.execute_reply":"2023-04-08T10:24:13.939532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:13.943168Z","iopub.execute_input":"2023-04-08T10:24:13.943452Z","iopub.status.idle":"2023-04-08T10:24:13.961371Z","shell.execute_reply.started":"2023-04-08T10:24:13.943427Z","shell.execute_reply":"2023-04-08T10:24:13.960076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check for null and missing values","metadata":{}},{"cell_type":"code","source":"print('Train dataset, missing values:', train_data.isna().sum().sum())\nprint('Test dataset, missing values:', test_data.isna().sum().sum())","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:13.962843Z","iopub.execute_input":"2023-04-08T10:24:13.964180Z","iopub.status.idle":"2023-04-08T10:24:14.012017Z","shell.execute_reply.started":"2023-04-08T10:24:13.964143Z","shell.execute_reply":"2023-04-08T10:24:14.010806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are no missing values in the train and test datasets. So there are no corrupted images","metadata":{}},{"cell_type":"markdown","source":"### Explore Target Variable <a class=\"anchor\" id=\"exploretargetvariable\"></a>","metadata":{}},{"cell_type":"code","source":"def countplot_target_var(df, label, title):\n    ax = sns.countplot(x=label, data=df)\n    ax.figure.suptitle(title)\n    for container in ax.containers:\n        ax.bar_label(container)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T14:09:41.002396Z","iopub.execute_input":"2023-04-08T14:09:41.004557Z","iopub.status.idle":"2023-04-08T14:09:41.013677Z","shell.execute_reply.started":"2023-04-08T14:09:41.004488Z","shell.execute_reply":"2023-04-08T14:09:41.012548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"countplot_target_var(train_data, 'label', 'Digits count')","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:14.013260Z","iopub.execute_input":"2023-04-08T10:24:14.013632Z","iopub.status.idle":"2023-04-08T10:24:14.231285Z","shell.execute_reply.started":"2023-04-08T10:24:14.013593Z","shell.execute_reply":"2023-04-08T10:24:14.230475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks nearly balanced.","metadata":{}},{"cell_type":"markdown","source":"### Data Visualization <a class=\"anchor\" id=\"datavisualization\"></a>","metadata":{}},{"cell_type":"markdown","source":"Create function to plot digits","metadata":{}},{"cell_type":"code","source":"def plot_digits(instances, actual_labels, predicted_labels, images_per_row=10, figsize=(8,8), **options):\n    size = 28\n    images_per_row = min(len(instances), images_per_row)\n    n_rows = (len(instances) - 1) // images_per_row + 1\n    n_empty = n_rows * images_per_row - len(instances)\n    padded_instances = np.concatenate([instances, np.zeros((n_empty, size * size))], axis=0)\n    fig = plt.figure(figsize=figsize)\n    for i in range(len(padded_instances)):\n        ax = fig.add_subplot(n_rows, images_per_row, i + 1)\n        grid_data = padded_instances[i].reshape(28,28)\n        plt.imshow(grid_data, cmap = mpl.cm.binary, interpolation=\"nearest\")\n        actual_title = f'Actual: {actual_labels[i]}' if len(actual_labels) else ''\n        predicted_title = f'\\nPredicted: {predicted_labels[i]}' if len(predicted_labels) else ''\n        ax.set_title(f'{actual_title}{predicted_title}')\n        ax.get_xaxis().set_ticks([])\n        ax.get_yaxis().set_ticks([])\n        ax.patch.set_linewidth('1')\n        ax.spines['bottom'].set_color('lightgray')\n        ax.spines['top'].set_color('lightgray') \n        ax.spines['right'].set_color('lightgray')\n        ax.spines['left'].set_color('lightgray')\n    fig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2023-04-09T20:12:55.364990Z","iopub.execute_input":"2023-04-09T20:12:55.366210Z","iopub.status.idle":"2023-04-09T20:12:55.377696Z","shell.execute_reply.started":"2023-04-09T20:12:55.366161Z","shell.execute_reply":"2023-04-09T20:12:55.376233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot one random digit.","metadata":{}},{"cell_type":"code","source":"example_index = randint.rvs(0, len(train_data)-1, size=1)[0]\nexample_image = train_data.iloc[example_index, 1:785]\nexample_label = train_data.iloc[example_index, 0]\nplot_digits([example_image], [example_label], [], images_per_row=1, figsize=(3,3))","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:14.243877Z","iopub.execute_input":"2023-04-08T10:24:14.244289Z","iopub.status.idle":"2023-04-08T10:24:14.317260Z","shell.execute_reply.started":"2023-04-08T10:24:14.244239Z","shell.execute_reply":"2023-04-08T10:24:14.316143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot multiple random digits","metadata":{}},{"cell_type":"code","source":"exmaple_indexes = randint.rvs(0, len(train_data)-1, size=49)\nexample_images = train_data.iloc[exmaple_indexes, 1:785].to_numpy()\nexample_labels = train_data.iloc[exmaple_indexes, 0].to_numpy()\nplot_digits(example_images, example_labels, [], images_per_row=7)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:14.320655Z","iopub.execute_input":"2023-04-08T10:24:14.320974Z","iopub.status.idle":"2023-04-08T10:24:16.296679Z","shell.execute_reply.started":"2023-04-08T10:24:14.320923Z","shell.execute_reply":"2023-04-08T10:24:16.295433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### t-SNE","metadata":{}},{"cell_type":"code","source":"def plot_tsne(data2D, labels, title):\n    plt.scatter(data2D[:, 0], data2D[:, 1], s=5, c=labels, cmap='Spectral')\n    plt.colorbar(boundaries=np.arange(11)-0.5).set_ticks(np.arange(10))\n    plt.title(title, fontsize=14)\n    plt.axis('off')\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-09T20:02:28.792335Z","iopub.execute_input":"2023-04-09T20:02:28.792781Z","iopub.status.idle":"2023-04-09T20:02:28.799915Z","shell.execute_reply.started":"2023-04-09T20:02:28.792737Z","shell.execute_reply":"2023-04-09T20:02:28.798699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_tsne = Pipeline([\n    (\"pca\", PCA(n_components=100, random_state=42)),\n    (\"tsne\", TSNE(n_components=2, random_state=42)),\n])\ntsne_subset_size=15000\ntrain_data_subset = train_data.loc[:(tsne_subset_size-1)].drop(\"label\", axis=1)\ntrain_data_labels_subset = train_data.loc[:(tsne_subset_size-1), 'label'].to_numpy()\ntrain_data_tsne_reduced = pca_tsne.fit_transform(train_data_subset)\nplot_tsne(train_data_tsne_reduced, train_data_labels_subset, 'Visualizing MNIST through t-SNE (train_data)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that the classes of objects are well separated from each other in the visualization, which indicates the high quality of the data and the possibility of using machine learning models for classification tasks. Some objects belonging to different classes may be close to each other, which may indicate the difficulty in separating them by machine learning models.","metadata":{}},{"cell_type":"markdown","source":"# Data Preprocessing <a class=\"anchor\" id=\"datapreprocessing\"></a>","metadata":{}},{"cell_type":"code","source":"X_full = train_data.drop(\"label\", axis=1)\ny_full = train_data[\"label\"]\nX_train, X_val, y_train, y_val = train_test_split(X_full, y_full, \n                                              test_size=0.10, stratify = y_full)\ny_train = y_train.to_numpy()\ny_val = y_val.to_numpy()\nprint(f'Train shape. \\t\\tX: {X_train.shape}, y: {y_train.shape}',\n    f'\\nValidation shape. \\tX: {X_val.shape}, y: {y_val.shape}')","metadata":{"execution":{"iopub.status.busy":"2023-04-09T19:13:29.998399Z","iopub.execute_input":"2023-04-09T19:13:29.999385Z","iopub.status.idle":"2023-04-09T19:13:30.298948Z","shell.execute_reply.started":"2023-04-09T19:13:29.999341Z","shell.execute_reply":"2023-04-09T19:13:30.297818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## PCA <a class=\"anchor\" id=\"dppca\"></a>","metadata":{}},{"cell_type":"code","source":"def plot_pca(pca, n_comp_example, ax, title):\n    ax.plot(np.cumsum(pca.explained_variance_ratio_))\n    cev_example = np.cumsum(pca.explained_variance_ratio_[:n_comp_example+1])[-1]\n    ax.hlines(y=cev_example, xmin=0, xmax=n_comp_example, color='r', ls='--', lw=1)\n    ax.vlines(x=n_comp_example, ymin=0, ymax=cev_example, color='r', ls='--', lw=1)\n    ax.plot([n_comp_example], [cev_example], \"ro\")\n    ax.set_xticks(list(ax.get_xticks()) + [n_comp_example])\n    ax.set_yticks(list(ax.get_yticks()) + [round(cev_example, 2)])\n    ax.set_xlim(0, pca.n_components+round(pca.n_components*0.05, 0))\n    ax.set_ylim(0, 1.05)\n    ax.set_xlabel('Number of components')\n    ax.set_ylabel('Cumulative explained variance')\n    ax.grid()\n    ax.set_title(title)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T10:24:16.499754Z","iopub.execute_input":"2023-04-08T10:24:16.500592Z","iopub.status.idle":"2023-04-08T10:24:16.509214Z","shell.execute_reply.started":"2023-04-08T10:24:16.500563Z","shell.execute_reply":"2023-04-08T10:24:16.508257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot the cumulative explained variance.","metadata":{}},{"cell_type":"code","source":"f, (ax0, ax1) = plt.subplots(1, 2, figsize=(14,4))\nX_train_scaled1 = MinMaxScaler().fit_transform(X_train)\nX_train_scaled2 = StandardScaler().fit_transform(X_train)\npca1 = PCA(n_components=X_train.shape[1]).fit(X_train_scaled1)\npca2 = PCA(n_components=X_train.shape[1]).fit(X_train_scaled2)\nplot_pca(pca1, 100, ax0, \"PCA with MinMaxScaler\")\nplot_pca(pca2, 100, ax1, \"PCA with StandardScaler\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:16.510767Z","iopub.execute_input":"2023-04-08T10:24:16.511177Z","iopub.status.idle":"2023-04-08T10:24:23.881246Z","shell.execute_reply.started":"2023-04-08T10:24:16.511140Z","shell.execute_reply":"2023-04-08T10:24:23.880210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that 100 components describe about 92% of the variance with MinMaxScaler. PCA with MinMaxScaler also performs better than StandardScaler. We will check the performance with the following models.","metadata":{}},{"cell_type":"markdown","source":"Let's see how the digits look like after the reconstruction.","metadata":{}},{"cell_type":"code","source":"random_indexes = randint.rvs(0, len(X_train)-1, size=8)\nexample_images = X_train.iloc[random_indexes]\npca = PCA(n_components=100)\nX_train_pca = pca.fit_transform(X_train)\nX_train_pca_inverted = pca.inverse_transform(X_train_pca)\nexample_images_pca_inverted = X_train_pca_inverted[random_indexes]\nplot_digits(np.concatenate([example_images, example_images_pca_inverted]), \n            [], [], images_per_row=8, figsize=(8,2))","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:23.882433Z","iopub.execute_input":"2023-04-08T10:24:23.882712Z","iopub.status.idle":"2023-04-08T10:24:26.291925Z","shell.execute_reply.started":"2023-04-08T10:24:23.882683Z","shell.execute_reply":"2023-04-08T10:24:26.290598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The quality of the reconstructed images has decreased, but the numbers are still recognizable. Of course, it depends on the number of components.","metadata":{}},{"cell_type":"markdown","source":"## Deskew <a class=\"anchor\" id=\"dpdeskew\"></a>","metadata":{}},{"cell_type":"markdown","source":"Handwritten digits are skewed. So, we will try to deskew them and check the performance in the training section.","metadata":{}},{"cell_type":"code","source":"def moments(image):\n    c0,c1 = np.mgrid[:image.shape[0],:image.shape[1]] # A trick in numPy to create a mesh grid\n    totalImage = np.sum(image) #sum of pixels\n    m0 = np.sum(c0*image)/totalImage #mu_x\n    m1 = np.sum(c1*image)/totalImage #mu_y\n    m00 = np.sum((c0-m0)**2*image)/totalImage #var(x)\n    m11 = np.sum((c1-m1)**2*image)/totalImage #var(y)\n    m01 = np.sum((c0-m0)*(c1-m1)*image)/totalImage #covariance(x,y)\n    mu_vector = np.array([m0,m1]) # Notice that these are \\mu_x, \\mu_y respectively\n    covariance_matrix = np.array([[m00,m01],[m01,m11]]) # Do you see a similarity between the covariance matrix\n    return mu_vector, covariance_matrix\n\ndef deskew(image):\n    c,v = moments(image)\n    alpha = v[0,1]/v[0,0]\n    affine = np.array([[1,0],[alpha,1]])\n    ocenter = np.array(image.shape)/2.0\n    offset = c-np.dot(affine,ocenter)\n    return interpolation.affine_transform(image,affine,offset=offset)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-04-09T19:13:37.958191Z","iopub.execute_input":"2023-04-09T19:13:37.958656Z","iopub.status.idle":"2023-04-09T19:13:37.969934Z","shell.execute_reply.started":"2023-04-09T19:13:37.958609Z","shell.execute_reply":"2023-04-09T19:13:37.968575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_deskewed = np.apply_along_axis(lambda x: deskew(x.reshape(28,28)).flatten(), axis=1, arr=X_train.astype(np.uint8).to_numpy())\nX_val_deskewed = np.apply_along_axis(lambda x: deskew(x.reshape(28,28)).flatten(), axis=1, arr=X_val.astype(np.uint8).to_numpy())\nX_full_deskewed = np.apply_along_axis(lambda x: deskew(x.reshape(28,28)).flatten(), axis=1, arr=X_full.astype(np.uint8).to_numpy())\ntest_data_deskewed = np.apply_along_axis(lambda x: deskew(x.reshape(28,28)).flatten(), axis=1, arr=test_data.astype(np.uint8).to_numpy())","metadata":{"execution":{"iopub.status.busy":"2023-04-09T19:13:39.161768Z","iopub.execute_input":"2023-04-09T19:13:39.162240Z","iopub.status.idle":"2023-04-09T19:14:20.816444Z","shell.execute_reply.started":"2023-04-09T19:13:39.162193Z","shell.execute_reply":"2023-04-09T19:14:20.815259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_deskewed_images(df, df_deskewed, size=8):\n    example_indexes = randint.rvs(0, len(df)-1, size=size)\n    original_images = df.iloc[example_indexes]\n    deskewed_images = df_deskewed[example_indexes]\n    plot_digits(original_images, [], [], images_per_row=8, figsize=(10,2))\n    plt.suptitle('Original images')\n    plt.show()\n    plot_digits(deskewed_images, [], [], images_per_row=8, figsize=(10,2))\n    plt.suptitle('Deskewed images')\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T19:43:09.493679Z","iopub.execute_input":"2023-04-08T19:43:09.494187Z","iopub.status.idle":"2023-04-08T19:43:09.502075Z","shell.execute_reply.started":"2023-04-08T19:43:09.494131Z","shell.execute_reply":"2023-04-08T19:43:09.500746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_deskewed_images(X_train, X_train_deskewed, 8)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T19:43:25.790273Z","iopub.execute_input":"2023-04-08T19:43:25.791596Z","iopub.status.idle":"2023-04-08T19:43:26.688782Z","shell.execute_reply.started":"2023-04-08T19:43:25.791496Z","shell.execute_reply":"2023-04-08T19:43:26.687303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize the deskewed data using t-SNE.","metadata":{}},{"cell_type":"code","source":"X_train_deskewed_tsne_reduced = pca_tsne.fit_transform(X_train_deskewed[:tsne_subset_size])\nplot_tsne(X_train_deskewed_tsne_reduced, y_train[:tsne_subset_size], 'Visualizing MNIST through t-SNE (deskewed data)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clusters became more visually separable after applying deskew.","metadata":{}},{"cell_type":"markdown","source":"### Data augmentation <a class=\"anchor\" id=\"dpdataaugmentation\"></a>","metadata":{}},{"cell_type":"markdown","source":"Create functions to augment data.","metadata":{}},{"cell_type":"code","source":"def shift_image(image, dx, dy):\n    shifted_image = interpolation.shift(image, [dy, dx], cval=0, mode=\"constant\")\n    return shifted_image.reshape([-1])\n\ndef rotate_image(image, angle):\n    rotate_augmenter = iaa.Rotate(angle)\n    return rotate_augmenter.augment_image(image).flatten()\n\ndef scale_image(image, scale):\n    h, w = image.shape\n    seq = iaa.Sequential([\n        iaa.Affine(scale=(scale)),\n        iaa.CropToFixedSize(width=w, height=h, position='center')\n    ])\n    augmented_image = seq(image=image)\n    return augmented_image.flatten()\n\ndef get_augmented_data(X_data, y_data, shift=True, rotate=True, scale=True):\n    if isinstance(X_data, pd.DataFrame) or isinstance(X_data, pd.Series):\n        X_data = X_data.astype(np.uint8).to_numpy()\n    if isinstance(y_data, pd.DataFrame) or isinstance(y_data, pd.Series):\n        y_data = y_data.to_numpy()\n    X_augmented = [X_data]\n    y_augmented = [y_data]\n    if shift:\n        for dx, dy in ((0, 1), (1, 0), (-1, 0), (0, -1)):\n            shifted_images = np.apply_along_axis(lambda x, dx, dy: shift_image(x.reshape(28, 28), dx, dy), \n                                                 axis=1, arr=X_data, dx=dx, dy=dy)\n            X_augmented.append(shifted_images)\n            y_augmented.append(y_train)\n    if rotate:\n        for angle in [10, -10]:\n            rotated_images = np.apply_along_axis(lambda x, angle: rotate_image(x.reshape(28, 28), angle), \n                                                 axis=1, arr=X_data, angle=angle)\n            X_augmented.append(rotated_images)\n            y_augmented.append(y_train)\n    if scale:\n        for scale in [1.08, 0.92]:\n            scaled_images = np.apply_along_axis(lambda x, scale: scale_image(x.reshape(28, 28), scale),\n                                                axis=1, arr=X_data, scale=scale)\n            X_augmented.append(scaled_images)\n            y_augmented.append(y_train)\n    X_augmented_all = np.concatenate(X_augmented)\n    y_augmented_all = np.concatenate(y_augmented)\n    return (X_augmented_all, y_augmented_all)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-09T19:58:08.861187Z","iopub.execute_input":"2023-04-09T19:58:08.862402Z","iopub.status.idle":"2023-04-09T19:58:08.877612Z","shell.execute_reply.started":"2023-04-09T19:58:08.862351Z","shell.execute_reply":"2023-04-09T19:58:08.876569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize random augmented digit.","metadata":{}},{"cell_type":"code","source":"example_indexes = randint.rvs(0, len(X_train_deskewed)-1, size=1)\nexample_images = X_train_deskewed[example_indexes]\nexample_labels = y_train[example_indexes]\naugmented_images, _ = get_augmented_data(example_images, example_labels)\ndisplay(Markdown(f'#### <div style=\"color: black;\">Augmentation: Shifting, Rotation, Scale</div>'))\nplot_digits(augmented_images, [], [], images_per_row=9, figsize=(10,2))\nplt.suptitle(f'Digit: {y_train[example_indexes]}')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:24:55.109583Z","iopub.execute_input":"2023-04-08T10:24:55.110397Z","iopub.status.idle":"2023-04-08T10:24:55.611033Z","shell.execute_reply.started":"2023-04-08T10:24:55.110354Z","shell.execute_reply":"2023-04-08T10:24:55.608973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Creation of augmented data for deskewed images.","metadata":{}},{"cell_type":"code","source":"X_train_des_aug0, y_train_aug0 = get_augmented_data(X_train_deskewed, y_train, \n                                                    shift=True, rotate=False, scale=True)\nprint(f'Train shape. \\tX: {X_train_deskewed.shape}, y: {y_train.shape}')\nprint(f'Train (augmented) shape. \\tX: {X_train_des_aug0.shape}, y: {y_train_aug0.shape}')\nprint('Proportion: \\t{:.0f}:1'.format(X_train_des_aug0.shape[0]/X_train_deskewed.shape[0]))","metadata":{"execution":{"iopub.status.busy":"2023-04-09T19:58:13.373518Z","iopub.execute_input":"2023-04-09T19:58:13.374141Z","iopub.status.idle":"2023-04-09T19:59:46.975932Z","shell.execute_reply.started":"2023-04-09T19:58:13.374069Z","shell.execute_reply":"2023-04-09T19:59:46.974749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Shuffle of the augmented training data.","metadata":{}},{"cell_type":"code","source":"shuffle_idx = np.random.permutation(len(X_train_des_aug0))\nX_train_des_aug = X_train_des_aug0[shuffle_idx]\ny_train_aug = y_train_aug0[shuffle_idx]","metadata":{"execution":{"iopub.status.busy":"2023-04-09T20:00:11.027673Z","iopub.execute_input":"2023-04-09T20:00:11.028093Z","iopub.status.idle":"2023-04-09T20:00:11.131712Z","shell.execute_reply.started":"2023-04-09T20:00:11.028057Z","shell.execute_reply":"2023-04-09T20:00:11.130754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize some digits to make sure everything is correct after the augmentation and shuffling.","metadata":{}},{"cell_type":"code","source":"exmaple_indexes = randint.rvs(0, len(X_train_des_aug)-1, size=8)\nexample_images = X_train_des_aug[exmaple_indexes]\nexample_labels = y_train_aug[exmaple_indexes]\nplot_digits(example_images, example_labels, [], images_per_row=8, figsize=(8,2))","metadata":{"execution":{"iopub.status.busy":"2023-04-09T20:13:00.440501Z","iopub.execute_input":"2023-04-09T20:13:00.441670Z","iopub.status.idle":"2023-04-09T20:13:00.942233Z","shell.execute_reply.started":"2023-04-09T20:13:00.441609Z","shell.execute_reply":"2023-04-09T20:13:00.940569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the augmented data using t-SNE.","metadata":{}},{"cell_type":"code","source":"X_train_des_aug_tsne_reduced = pca_tsne.fit_transform(X_train_des_aug[:tsne_subset_size])\nplot_tsne(X_train_des_aug_tsne_reduced, y_train_aug[:tsne_subset_size], 'Visualizing MNIST through t-SNE (des_aug data)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pipeline <a class=\"anchor\" id=\"dppipeline\"></a>","metadata":{}},{"cell_type":"markdown","source":"This is the final pipeline, given the performance estimates below.","metadata":{}},{"cell_type":"code","source":"preprocessor = Pipeline([\n    ('scaler', MinMaxScaler()),\n    ('pca', PCA(n_components=75))\n])","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:26:06.627290Z","iopub.execute_input":"2023-04-08T10:26:06.627604Z","iopub.status.idle":"2023-04-08T10:26:06.632591Z","shell.execute_reply.started":"2023-04-08T10:26:06.627569Z","shell.execute_reply":"2023-04-08T10:26:06.631391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_pipeline = Pipeline([\n    ('preprocessor', preprocessor),\n    ('model', None)\n], verbose=False)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:26:06.633905Z","iopub.execute_input":"2023-04-08T10:26:06.634691Z","iopub.status.idle":"2023-04-08T10:26:06.645779Z","shell.execute_reply.started":"2023-04-08T10:26:06.634660Z","shell.execute_reply":"2023-04-08T10:26:06.644805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Deskewing data is a time-consuming process, so we will not create a separate transformer for it and just use the X_train_deskewed, X_val_deskewed, X_full_deskewed, test_data_deskewed and X_train_des_aug variables.","metadata":{}},{"cell_type":"markdown","source":"# Model Training and Evaluation <a class=\"anchor\" id=\"trainandevaluate\"></a>","metadata":{}},{"cell_type":"markdown","source":"Creating classes for reporting. Instead of using separate functions, I also created EstimatorReport and GridSearchReport classes for reporting functionality.","metadata":{}},{"cell_type":"code","source":"import scikitplot as skplt\n\nclass EstimatorReport():\n    \"\"\"\n    A class used for generating a report on an estimator's performance \n\n    Attributes\n    ----------\n    X_val : np.ndarray\n        Validation dataset features.\n    y_val : np.ndarray\n        Validation dataset labels.\n    X_train : np.ndarray\n        Training dataset features.\n    y_train : np.ndarray\n        Training dataset labels.\n    estimator : estimator object\n        Scikit-learn estimator object.\n    estimator_name : str\n        Name of the estimator.\n    plot_micro : bool\n        Whether micro average ROC curve should be plotted.\n    plot_macro : bool\n        Whether macro average ROC curve should be plotted.\n    metrics_calculated : bool\n        Whether the performance metrics have been calculated.\n\n    Methods\n    -------\n    calculate_metrics()\n        Calculates accuracy, confusion matrix, classification report and predicted probabilities\n        based on the validation set and sets metrics_calculated flag to True.\n    plot_curves(figsize=(18, 4))\n        Plots ROC and PR curves and learning curve based on the validation set.\n    display_report(figsize=(10, 5))\n        Displays a report on estimator's performance based on validation set.\n    __plot_confusion_matrix(ax)\n        Plots confusion matrix.\n    __plot_classification_report(ax)\n        Plots classification report.\n    \"\"\"\n    def __init__(\n        self,\n        estimator,\n        X_val: np.ndarray,\n        y_val: np.ndarray,\n        X_train: np.ndarray,\n        y_train: np.ndarray,\n        estimator_name: str,\n    ) -> None:\n        self.X_val = X_val\n        self.y_val = y_val\n        self.X_train = X_train\n        self.y_train = y_train\n        self.estimator = estimator\n        self.estimator_name = estimator_name\n        self.plot_micro = False\n        self.plot_macro = False\n        self.metrics_calculated = False\n    \n    def calculate_metrics(self) -> None:\n        self.y_val_predicted = self.estimator.predict(self.X_val)\n        self.accuracy = accuracy_score(self.y_val, self.y_val_predicted)\n        self.conf_mx = confusion_matrix(self.y_val, self.y_val_predicted)\n        self.classif_report = classification_report(\n            self.y_val, self.y_val_predicted, output_dict=True\n        )\n        try:\n            self.y_val_probas = self.estimator.predict_proba(self.X_val)\n        except:\n            self.y_val_probas = self.estimator.decision_function(self.X_val)\n        self.metrics_calculated = True\n        \n    def plot_curves(self, figsize=(18, 4)) -> None:\n        if not self.metrics_calculated:\n            self.calculate_metrics()\n        fig, axes = plt.subplots( \n            1, 3, figsize=figsize, sharex=False, \n            gridspec_kw={'width_ratios': [1, 1, 1]}, dpi=100\n        )\n        skplt.metrics.plot_roc(\n            y_true=self.y_val, y_probas=self.y_val_probas, ax=axes[0], \n            title='ROC curve', plot_micro=self.plot_micro, plot_macro=self.plot_macro\n        )\n        skplt.metrics.plot_precision_recall(\n            y_true=self.y_val, y_probas=self.y_val_probas, ax=axes[1], \n            title='PR curve', plot_micro=self.plot_micro\n        )\n        skplt.estimators.plot_learning_curve(\n            self.estimator, self.X_train, self.y_train, ax=axes[2]\n        )\n        axes[0].grid()\n        axes[1].grid()\n        \n    def display_report(self, figsize=(10, 5)) -> None:\n        if not self.metrics_calculated:\n            self.calculate_metrics()\n        display(Markdown(f'### <span style=\"color: black;\">Evaluation on the Validation Set (estimator: {self.estimator_name})</span>'))\n        display(Markdown('Accuracy: {:.4f}%'.format(self.accuracy*100)))\n        fig, (ax0, ax1) = plt.subplots(1, 2, figsize=figsize, sharex=False, \n                                   gridspec_kw={'width_ratios': [1, 2.2]}, dpi=100)\n        self.__plot_classification_report(ax0)\n        self.__plot_confusion_matrix(ax1)\n        fig.tight_layout()\n        plt.show()\n        \n    def __plot_confusion_matrix(self, ax) -> None:\n        norm_conf_mx = self.conf_mx.astype('float') / self.conf_mx.sum(axis=1, keepdims=True)\n        np.fill_diagonal(norm_conf_mx, 0)\n        sns.heatmap(\n            norm_conf_mx, ax=ax, \n            annot=False, cmap=plt.cm.gray, cbar=False\n        )\n        plt.title('Confusion matrix')\n        thresh = norm_conf_mx.max() / 2\n        for i, j in itertools.product(range(self.conf_mx.shape[0]), range(self.conf_mx.shape[1])):\n            plt.text(j+0.5, i+0.6, self.conf_mx[i, j],\n                ha=\"center\",\n                va='baseline',\n                color=\"white\" if norm_conf_mx[i, j] < thresh else \"black\")\n        plt.tight_layout()\n        plt.ylabel('True label')\n        plt.xlabel('Predicted label')\n    \n    def __plot_classification_report(self, ax) -> None:\n        clf_report_df = pd.DataFrame(self.classif_report ).iloc[:-1, :].T\n        classes_count = len(self.estimator.classes_)\n        mask_classes = np.zeros_like(clf_report_df, dtype=bool)\n        mask_classes[classes_count:, :] = True\n        mask_metrics = np.zeros_like(clf_report_df, dtype=bool)\n        mask_metrics[:classes_count, :] = True\n        cmap_white = ListedColormap(['white'])\n        sns.heatmap(\n            clf_report_df, ax=ax, mask=mask_classes, \n            annot=True, cmap='viridis', fmt='.6f', cbar=False\n        )\n        sns.heatmap(\n            clf_report_df, ax=ax, mask=mask_metrics, \n            annot=True, cmap=cmap_white, fmt='.6f', cbar=False\n        )\n        ax.set_title(\"Classification report\", fontsize=10, pad=10)\n        ax.xaxis.tick_top()\n        ax.xaxis.set_label_position('top')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T19:38:43.616682Z","iopub.execute_input":"2023-04-08T19:38:43.617171Z","iopub.status.idle":"2023-04-08T19:38:43.681567Z","shell.execute_reply.started":"2023-04-08T19:38:43.617108Z","shell.execute_reply":"2023-04-08T19:38:43.680104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas.io.formats.style\n\nclass GridSearchReport:\n    \"\"\"\n    A class for generating GridSearchCV reports with a summary of the results\n\n    Attributes:\n        grid_search (sklearn.model_selection.GridSearchCV): GridSearchCV object\n        param_model (str): Name of the model parameter in the hyperparameter grid\n\n    Methods:\n        get_summary(filter_models=[], columns=[], sortby='mean_test_f1')\n            Returns a summary of the GridSearchCV results in a styled Pandas DataFrame\n        display_report()\n            Displays the raw GridSearchCV results and the best parameters found\n    \"\"\"\n    def __init__(self, grid_search, param_model='model') -> None:\n        \"\"\"\n        Initializes a new instance of the GridSearchReport class\n\n        Args:\n            grid_search (sklearn.model_selection.GridSearchCV): GridSearchCV object\n            param_model (str, optional): Name of the model parameter in the hyperparameter grid. Default is 'model'\n        \"\"\"\n        self.grid_search = grid_search\n        self.param_model = param_model\n\n    def get_summary(self, filter_models=[], columns=[], sortby='mean_test_score') -> pd.io.formats.style.Styler:\n        \"\"\"\n        Returns a summary of the GridSearchCV results in a styled Pandas DataFrame\n\n        Args:\n            filter_models (list, optional): List of models (as strings) to filter the results by. Default is an empty list\n            columns (list, optional): List of additional columns of interest to include in the summary. Default is an empty list\n            sortby (str, optional): Column to sort the results by. Default is 'mean_test_f1'\n\n        Returns:\n            pandas.io.formats.style.Styler: A styled Pandas DataFrame with the summary of the GridSearchCV results\n        \"\"\"\n        cv_results = self.grid_search.cv_results_\n        cv_results_df = pd.DataFrame(cv_results)\n        try:\n            regex_pattern = f'(?<=\\'{self.param_model}\\'. )\\w+(?=\\()'\n            cv_results_df['model'] = cv_results_df['params'].astype('str').str.findall(regex_pattern).apply(', '.join)\n        except:\n            print(\"Error: 'model' parameter not found in cv_results\")\n            return\n        if len(filter_models) > 0:\n            if len(set(filter_models) - set(cv_results_df['model'].unique())) > 0:\n                print('Error: filter_models contains models not present in cv_results_df')\n                return\n            cv_results_df = cv_results_df[cv_results_df['model'].isin(filter_models)]\n        mean_columns = cv_results_df.filter(regex='^mean_test').columns.tolist()\n        mean_columns = mean_columns + columns if len(columns) > 0 else mean_columns\n        if len(mean_columns) == 0:\n            print('Error: mean_columns is empty')\n            return\n        filtered_columns = np.concatenate((['model'], mean_columns))\n        return (cv_results_df.sort_values(by=[sortby], ascending=False)\n            .groupby('model')\n            .first()\n            .reindex(columns=filtered_columns)\n            .loc[:, mean_columns]\n            .sort_values(by=[sortby], ascending=False)\n            .style\n            .background_gradient(cmap='viridis')\n        )\n\n    def display_report(self, sortby='rank_test_score', ascending=True) -> None:\n        \"\"\"\n        Displays the raw GridSearchCV results and the best parameters found\n        \"\"\"\n        display(pd.DataFrame(self.grid_search.cv_results_)\n            .sort_values(by=[sortby], ascending=ascending)\n            .reset_index(drop=True)\n            .head()\n        )\n        if hasattr(self.grid_search, 'best_score_'):\n            display(f\"Best score: {self.grid_search.best_score_}\")\n        if hasattr(self.grid_search, 'best_params_'):\n            display(\"Best params:\")\n            display(self.grid_search.best_params_)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T13:42:07.182169Z","iopub.execute_input":"2023-04-08T13:42:07.182636Z","iopub.status.idle":"2023-04-08T13:42:07.206844Z","shell.execute_reply.started":"2023-04-08T13:42:07.182590Z","shell.execute_reply":"2023-04-08T13:42:07.205063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def add_to_results(report, results_df):\n    row = {\n        'model': report.estimator_name,\n        'accuracy': report.accuracy\n    }\n    model_name = report.estimator_name\n    if model_name in results_df['model'].values:\n        row_index = results_df.loc[results_df['model'] == model_name].index[0]\n        results_df.loc[row_index, 'accuracy'] = row['accuracy']\n    else:\n        results_df = results_df.append(row, ignore_index=True)\n    return results_df","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:26:06.692861Z","iopub.execute_input":"2023-04-08T10:26:06.693442Z","iopub.status.idle":"2023-04-08T10:26:06.704171Z","shell.execute_reply.started":"2023-04-08T10:26:06.693409Z","shell.execute_reply":"2023-04-08T10:26:06.703136Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_errors(estimator, X, y, size, per_row, figsize=(8,5)):\n    y_predicted = estimator.predict(X)\n    errors = (y_predicted - y != 0)\n    X_errors = X_val_deskewed[errors]\n    y_errors = y[errors]\n    y_errors_predicted = y_predicted[errors]\n    exmaple_indexes = randint.rvs(0, len(X_errors)-1, size=size)\n    example_images = X_errors[exmaple_indexes]\n    example_labels = y_errors[exmaple_indexes]\n    example_labels_pred = y_errors_predicted[exmaple_indexes]\n    plot_digits(example_images, example_labels, example_labels_pred, images_per_row=per_row, figsize=figsize)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T10:26:06.710449Z","iopub.execute_input":"2023-04-08T10:26:06.710771Z","iopub.status.idle":"2023-04-08T10:26:06.719688Z","shell.execute_reply.started":"2023-04-08T10:26:06.710742Z","shell.execute_reply":"2023-04-08T10:26:06.718333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define DataFrame for estimation results.","metadata":{}},{"cell_type":"code","source":"results_df = pd.DataFrame(columns=['model', 'accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:26:06.721127Z","iopub.execute_input":"2023-04-08T10:26:06.721625Z","iopub.status.idle":"2023-04-08T10:26:06.731311Z","shell.execute_reply.started":"2023-04-08T10:26:06.721595Z","shell.execute_reply":"2023-04-08T10:26:06.730254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Estimate PCA params <a class=\"anchor\" id=\"estimatepcaparams\"></a>","metadata":{}},{"cell_type":"markdown","source":"In the following, we will estimate PCA, scalers, and deskewing (using the SVC model).","metadata":{}},{"cell_type":"code","source":"pipeline_pca = Pipeline([\n    ('pca', None),\n    ('model', SVC()),\n])\nparams_pca = [\n    {\n        'pca': [PCA()],\n        'pca__n_components': np.arange(25, 784, 50)\n    },\n    {\n        'pca': [None],\n    }\n]\ngrid_search_pca = HalvingGridSearchCV(pipeline_pca, params_pca, cv=3, \n                                      scoring='accuracy', verbose=verbose_tuning, \n                                      n_jobs=-1)\ngrid_search_pca.fit(X_train[:20000], y_train[:20000])\nGridSearchReport(grid_search_pca).display_report()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid_search_pca_df = pd.DataFrame(grid_search_pca.cv_results_)\nmean_test_score_no_pca = grid_search_pca_df[grid_search_pca_df[\"param_pca\"].isna()]['mean_test_score'].values[0]\ndisplay(f\"Best score without pca: {mean_test_score_no_pca}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-08T14:12:24.196283Z","iopub.execute_input":"2023-04-08T14:12:24.197796Z","iopub.status.idle":"2023-04-08T14:12:24.211462Z","shell.execute_reply.started":"2023-04-08T14:12:24.197736Z","shell.execute_reply":"2023-04-08T14:12:24.210258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The accuracy increased by about 4% after applying PCA.  So we see that not all the information of the digits is important for accurate classifications.","metadata":{}},{"cell_type":"markdown","source":"### Estimate scalers <a class=\"anchor\" id=\"estimatescalers\"></a>","metadata":{}},{"cell_type":"markdown","source":"Determine which scaler to use.","metadata":{}},{"cell_type":"code","source":"pipeline_scaler = Pipeline([\n    ('pca', PCA(n_components=75)),\n    ('scaler', None),\n    ('model', SVC()),\n])\nparams_scaler = [\n    {\n        'scaler': [MinMaxScaler(), StandardScaler(), Binarizer(threshold=50)]\n    }\n]\ngrid_search_scaler = GridSearchCV(pipeline_scaler, params_scaler, cv=3, \n                                  scoring='accuracy', n_jobs=-1, verbose=verbose_tuning)\ngrid_search_scaler.fit(X_train[:20000], y_train[:20000])\nGridSearchReport(grid_search_scaler).display_report()\nreport_scaler = EstimatorReport(grid_search_scaler, X_val, y_val, [], [], \"SVC (with PCA and scaler)\")\nreport_scaler.display_report()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that the MinMaxScaler is the best one (with SVC classifier).","metadata":{}},{"cell_type":"markdown","source":"### Estimate Deskew <a class=\"anchor\" id=\"estimatedeskew\"></a>","metadata":{}},{"cell_type":"code","source":"pipe_no_deskewed = copy.copy(full_pipeline).set_params(model=SVC())\npipe_no_deskewed.fit(X_train, y_train)\nreport_nodes = EstimatorReport(pipe_no_deskewed, X_val, y_val, [], [], \"SVC (PCA, Scaler, no deskew)\")\nreport_nodes.display_report()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:28:55.619098Z","iopub.execute_input":"2023-04-08T10:28:55.619364Z","iopub.status.idle":"2023-04-08T10:29:18.643413Z","shell.execute_reply.started":"2023-04-08T10:28:55.619335Z","shell.execute_reply":"2023-04-08T10:29:18.642626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipe_deskewed = copy.copy(full_pipeline).set_params(model=SVC())\npipe_deskewed.fit(X_train_deskewed, y_train)\nreport_des = EstimatorReport(pipe_deskewed, X_val_deskewed, y_val, [], [], \"SVC (PCA, Scaler, deskew)\")\nreport_des.display_report()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:29:18.644751Z","iopub.execute_input":"2023-04-08T10:29:18.645282Z","iopub.status.idle":"2023-04-08T10:29:36.105579Z","shell.execute_reply.started":"2023-04-08T10:29:18.645251Z","shell.execute_reply":"2023-04-08T10:29:36.104570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model performance with deskewed data is better, so we will continue to use it.","metadata":{}},{"cell_type":"markdown","source":"### Estimate different models (default params) <a class=\"anchor\" id=\"estimatedifferentmodels\"></a>","metadata":{}},{"cell_type":"markdown","source":"Estimate different models using default hyperparams.","metadata":{}},{"cell_type":"code","source":"params = [\n    {'model': [\n        SGDClassifier(max_iter=4000),\n        LogisticRegression(max_iter=8000),\n        SVC(),\n        KNeighborsClassifier(),\n        GaussianNB(),\n        DecisionTreeClassifier(),\n        RandomForestClassifier(),\n        ExtraTreesClassifier(),\n        xgb.XGBClassifier(),\n        GradientBoostingClassifier(n_estimators=20),\n        HistGradientBoostingClassifier(),\n        AdaBoostClassifier(),\n        BaggingClassifier(),\n    ]}]\ngrid_search_allmodels = GridSearchCV(full_pipeline, params, cv=3, \n                                     scoring=['accuracy', 'f1_macro'], n_jobs=1, \n                                     verbose=10, refit=False)\ngrid_search_allmodels.fit(X_train_deskewed, y_train)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-08T10:29:36.106753Z","iopub.execute_input":"2023-04-08T10:29:36.107017Z","iopub.status.idle":"2023-04-08T10:48:07.749198Z","shell.execute_reply.started":"2023-04-08T10:29:36.106992Z","shell.execute_reply":"2023-04-08T10:48:07.748206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"report_gridsearch_all = GridSearchReport(grid_search_allmodels)\nreport_gridsearch_all.display_report(sortby='mean_test_accuracy', ascending=False)\ndisplay(report_gridsearch_all.get_summary(columns=['mean_fit_time'], sortby='mean_test_accuracy'))","metadata":{"execution":{"iopub.status.busy":"2023-04-08T13:40:07.047377Z","iopub.execute_input":"2023-04-08T13:40:07.048197Z","iopub.status.idle":"2023-04-08T13:40:07.105778Z","shell.execute_reply.started":"2023-04-08T13:40:07.048159Z","shell.execute_reply":"2023-04-08T13:40:07.104720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The best classifier with default parameters is SVC, the fastest are KNeighborsClassifier and GaussianNB.","metadata":{}},{"cell_type":"markdown","source":"### Fine-Tuning <a class=\"anchor\" id=\"finetuning\"></a>","metadata":{}},{"cell_type":"markdown","source":"Below I use different tools for hyperparameteres search like GridSearcCV, HalvingGridSearchCV, BayesSearchCV. It depends on fitting speed, search complexity.","metadata":{}},{"cell_type":"markdown","source":"#### SVC <a class=\"anchor\" id=\"finetuningsvc\"></a>","metadata":{}},{"cell_type":"markdown","source":"I also tested the poly kernel, but it showed poor performance, so I removed it from the final evaluation.","metadata":{}},{"cell_type":"code","source":"params_svc = [\n    {\n        'model': [SVC()],\n        'model__kernel': ['rbf'],\n        'model__C': np.arange(3, 15, 1),\n        'model__gamma': np.arange(0.001, 0.03, 0.003),\n        'model__probability': [True],\n    }\n]\ngrid_search_svc = HalvingGridSearchCV(full_pipeline, params_svc, cv=3, \n                                      scoring='accuracy', verbose=verbose_tuning, \n                                      n_jobs=-1)\ngrid_search_svc.fit(X_train_deskewed, y_train)\nbest_svc = grid_search_svc.best_estimator_\nGridSearchReport(grid_search_svc).display_report()\nreport_svc = EstimatorReport(best_svc, X_val_deskewed, y_val, [], [], \"SVC\")\nreport_svc.display_report()\nresults_df = add_to_results(report_svc, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:48:07.807709Z","iopub.execute_input":"2023-04-08T10:48:07.808495Z","iopub.status.idle":"2023-04-08T10:52:45.467118Z","shell.execute_reply.started":"2023-04-08T10:48:07.808436Z","shell.execute_reply":"2023-04-08T10:52:45.465931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that the tuning also slightly increased the metrics.","metadata":{}},{"cell_type":"markdown","source":"### Errors visualization","metadata":{}},{"cell_type":"markdown","source":"Let's look at some errors after fine-tuning the SVC classifier.","metadata":{}},{"cell_type":"code","source":"plot_errors(best_svc, X_val_deskewed, y_val, 10, 5)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:52:45.468627Z","iopub.execute_input":"2023-04-08T10:52:45.468906Z","iopub.status.idle":"2023-04-08T10:52:48.300894Z","shell.execute_reply.started":"2023-04-08T10:52:45.468878Z","shell.execute_reply":"2023-04-08T10:52:48.299737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### LogisticRegression <a class=\"anchor\" id=\"finetuninglr\"></a>","metadata":{}},{"cell_type":"code","source":"params_lr = [\n    {\n        'model': [LogisticRegression()],\n        'model__solver': ['lbfgs'],\n        'model__C': np.arange(0.1, 1, 0.05),\n        'model__max_iter': [6000],\n    },\n]\ngrid_search_lr = HalvingGridSearchCV(full_pipeline, params_lr, cv=3, \n                                     scoring='accuracy', verbose=verbose_tuning, n_jobs=-1)\ngrid_search_lr.fit(X_train_deskewed, y_train)\nbest_lr = grid_search_lr.best_estimator_\nGridSearchReport(grid_search_lr).display_report()\nreport_lr = EstimatorReport(best_lr, X_val_deskewed, y_val, [], [], \"LogisticRegression\")\nreport_lr.display_report()\nresults_df = add_to_results(report_lr, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:52:48.302989Z","iopub.execute_input":"2023-04-08T10:52:48.303357Z","iopub.status.idle":"2023-04-08T10:53:31.035944Z","shell.execute_reply.started":"2023-04-08T10:52:48.303310Z","shell.execute_reply":"2023-04-08T10:53:31.034978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### RandomForestClassifier <a class=\"anchor\" id=\"finetuningrf\"></a>","metadata":{}},{"cell_type":"code","source":"params_rf = [\n    {\n        'model': [RandomForestClassifier()],\n        'model__n_estimators': np.arange(150, 800, 100),\n        'model__min_samples_leaf': [2, 3],\n        'model__n_jobs': [-1],\n    }\n]\ngrid_search_rf = HalvingGridSearchCV(full_pipeline, params_rf, cv=3, \n                                     scoring='accuracy', verbose=verbose_tuning, \n                                     n_jobs=-1)\ngrid_search_rf.fit(X_train_deskewed, y_train)\nbest_rf = grid_search_rf.best_estimator_\nGridSearchReport(grid_search_rf).display_report()\nreport_rf = EstimatorReport(best_rf, X_val_deskewed, y_val, [], [], \"RandomForestClassifier\")\nreport_rf.display_report()\nresults_df = add_to_results(report_rf, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T10:53:31.037484Z","iopub.execute_input":"2023-04-08T10:53:31.038066Z","iopub.status.idle":"2023-04-08T11:02:58.886217Z","shell.execute_reply.started":"2023-04-08T10:53:31.038031Z","shell.execute_reply":"2023-04-08T11:02:58.885411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ExtraTreesClassifier <a class=\"anchor\" id=\"finetuninget\"></a>","metadata":{}},{"cell_type":"code","source":"params_et = [\n    {\n        'model': [ExtraTreesClassifier()],\n        'model__n_estimators': np.arange(150, 550, 100),\n        'model__min_samples_leaf': np.arange(2, 5, 1),\n#         'model__n_jobs': [-1],\n    }\n]\ngrid_search_et = HalvingGridSearchCV(full_pipeline, params_et, cv=3, \n                                     scoring='accuracy', verbose=verbose_tuning, \n                                     n_jobs=-1)\ngrid_search_et.fit(X_train_deskewed, y_train)\nbest_et = grid_search_et.best_estimator_\nGridSearchReport(grid_search_et).display_report()\nreport_et = EstimatorReport(best_et, X_val_deskewed, y_val, [], [], \"ExtraTreesClassifier\")\nreport_et.display_report()\nresults_df = add_to_results(report_et, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:02:58.887707Z","iopub.execute_input":"2023-04-08T11:02:58.888515Z","iopub.status.idle":"2023-04-08T11:05:29.441844Z","shell.execute_reply.started":"2023-04-08T11:02:58.888450Z","shell.execute_reply":"2023-04-08T11:05:29.441004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### KNeighborsClassifier <a class=\"anchor\" id=\"finetuningknn\"></a>","metadata":{}},{"cell_type":"code","source":"params_knn = [\n    {\n        'model': [KNeighborsClassifier()],\n        'model__n_neighbors': np.arange(2, 20, 3),\n        'model__n_jobs': [-1],\n    }\n]\ngrid_search_knn = HalvingGridSearchCV(full_pipeline, params_knn, cv=3, \n                                      scoring='accuracy', verbose=verbose_tuning, \n                                      n_jobs=-1)\ngrid_search_knn.fit(X_train_deskewed, y_train)\nbest_knn = grid_search_knn.best_estimator_\nGridSearchReport(grid_search_knn).display_report()\nreport_knn = EstimatorReport(best_knn, X_val_deskewed, y_val, [], [], \"KNeighborsClassifier\")\nreport_knn.display_report()\nresults_df = add_to_results(report_knn, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:05:29.443716Z","iopub.execute_input":"2023-04-08T11:05:29.444110Z","iopub.status.idle":"2023-04-08T11:06:58.936029Z","shell.execute_reply.started":"2023-04-08T11:05:29.444071Z","shell.execute_reply":"2023-04-08T11:06:58.934945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### HistGradientBoostingClassifier <a class=\"anchor\" id=\"finetuninghb\"></a>","metadata":{}},{"cell_type":"code","source":"params_hb = [\n    {\n        'model': [HistGradientBoostingClassifier()],\n        'model__learning_rate': space.Real(0.01, 0.25),\n        'model__min_samples_leaf': space.Integer(1, 14),\n    }\n]\nbayes_search_hb = BayesSearchCV(full_pipeline, params_hb, cv=3, n_iter = 12, \n                                scoring='accuracy', refit=True, verbose=verbose_tuning, n_points = 1, \n                                return_train_score=True, random_state=42)\nbayes_search_hb.fit(X_train_deskewed, y_train)\nbest_hb = bayes_search_hb.best_estimator_\nGridSearchReport(bayes_search_hb).display_report()\nreport_hb = EstimatorReport(best_hb, X_val_deskewed, y_val, [], [], \"HistGradientBoostingClassifier\")\nreport_hb.display_report()\nresults_df = add_to_results(report_hb, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:06:58.937654Z","iopub.execute_input":"2023-04-08T11:06:58.938039Z","iopub.status.idle":"2023-04-08T11:12:53.523362Z","shell.execute_reply.started":"2023-04-08T11:06:58.938008Z","shell.execute_reply":"2023-04-08T11:12:53.522270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plots.plot_objective(bayes_search_hb.optimizer_results_[0],\n                     size = 2.5, levels = 25)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:12:53.525249Z","iopub.execute_input":"2023-04-08T11:12:53.525740Z","iopub.status.idle":"2023-04-08T11:12:54.213207Z","shell.execute_reply.started":"2023-04-08T11:12:53.525693Z","shell.execute_reply":"2023-04-08T11:12:54.212215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### AdaBoostClassifier (base: HistGradientBoostingClassifier) <a class=\"anchor\" id=\"finetuningadbhb\"></a>","metadata":{}},{"cell_type":"code","source":"params_adb_hb = [\n    {\n        'model': [AdaBoostClassifier()],\n        'model__base_estimator': [best_hb['model']],\n        'model__n_estimators': np.arange(5, 40, 10),\n        'model__learning_rate': np.arange(0.1, 0.9, 0.1),\n        'model__algorithm': ['SAMME.R'],\n    }\n]\ngrid_search_adb_hb = HalvingGridSearchCV(full_pipeline, params_adb_hb, cv=3, \n                                     scoring='accuracy', verbose=verbose_tuning, \n                                     n_jobs=-1)\ngrid_search_adb_hb.fit(X_train_deskewed, y_train)\nbest_adb_hb = grid_search_adb_hb.best_estimator_\nGridSearchReport(grid_search_adb_hb).display_report()\nreport_adb_hb = EstimatorReport(best_adb_hb, X_val_deskewed, y_val, [], [], \"AdaBoostClassifier (HistGB)\")\nreport_adb_hb.display_report()\nresults_df = add_to_results(report_adb_hb, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:12:54.215229Z","iopub.execute_input":"2023-04-08T11:12:54.216202Z","iopub.status.idle":"2023-04-08T11:28:52.990237Z","shell.execute_reply.started":"2023-04-08T11:12:54.216146Z","shell.execute_reply":"2023-04-08T11:28:52.988569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### AdaBoostClassifier (base: ExtraTreesClassifier) <a class=\"anchor\" id=\"finetuningadbet\"></a>","metadata":{}},{"cell_type":"code","source":"params_adb_et = [\n    {\n        'model': [AdaBoostClassifier()],\n        'model__base_estimator': [best_et['model']],\n        'model__n_estimators': [10, 20],\n        'model__learning_rate': [0.02, 0.2],\n        'model__algorithm': ['SAMME.R'],\n    }\n]\ngrid_search_adb_et = HalvingGridSearchCV(full_pipeline, params_adb_et, cv=3, \n                                        scoring='accuracy', verbose=verbose_tuning, \n                                        n_jobs=-1)\ngrid_search_adb_et.fit(X_train_deskewed, y_train)\nbest_adb_et = grid_search_adb_et.best_estimator_\nGridSearchReport(grid_search_adb_et).display_report()\nreport_adb_et = EstimatorReport(best_adb_et, X_val_deskewed, y_val, [], [], \"AdaBoostClassifier (ExtTr)\")\nreport_adb_et.display_report()\nresults_df = add_to_results(report_adb_et, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:28:52.992270Z","iopub.execute_input":"2023-04-08T11:28:52.992592Z","iopub.status.idle":"2023-04-08T11:32:03.569786Z","shell.execute_reply.started":"2023-04-08T11:28:52.992561Z","shell.execute_reply":"2023-04-08T11:32:03.568388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### AdaBoostClassifier (base: GaussianNB) <a class=\"anchor\" id=\"finetuningadbgnb\"></a>","metadata":{}},{"cell_type":"code","source":"params_adb_gnb = [\n    {\n        'model': [AdaBoostClassifier()],\n        'model__base_estimator': [GaussianNB()],\n        'model__n_estimators': space.Integer(5, 50),\n        'model__learning_rate': space.Real(0.005, 0.5),\n        'model__algorithm': ['SAMME.R'],\n    }\n]\nbayes_search_adb_gnb = BayesSearchCV(full_pipeline, params_adb_gnb, cv=3, n_iter = 10, \n                                     scoring='accuracy', refit=True, verbose=verbose_tuning, n_points = 1, \n                                     return_train_score=True, random_state=42)\nbayes_search_adb_gnb.fit(X_train_deskewed, y_train)\nbest_adb_gnb = bayes_search_adb_gnb.best_estimator_\nGridSearchReport(bayes_search_adb_gnb).display_report()\nreport_adb_gnb = EstimatorReport(best_adb_gnb, X_val_deskewed, y_val, [], [], \"AdaBoostClassifier (GNB)\")\nreport_adb_gnb.display_report()\nresults_df = add_to_results(report_adb_gnb, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:32:03.572285Z","iopub.execute_input":"2023-04-08T11:32:03.572960Z","iopub.status.idle":"2023-04-08T11:36:17.163268Z","shell.execute_reply.started":"2023-04-08T11:32:03.572891Z","shell.execute_reply":"2023-04-08T11:36:17.161776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plots.plot_objective(bayes_search_adb_gnb.optimizer_results_[0],\n                     size = 2.5, levels = 25)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:36:17.165031Z","iopub.execute_input":"2023-04-08T11:36:17.165433Z","iopub.status.idle":"2023-04-08T11:36:17.911464Z","shell.execute_reply.started":"2023-04-08T11:36:17.165392Z","shell.execute_reply":"2023-04-08T11:36:17.910119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Estimations results <a class=\"anchor\" id=\"estimationsresults\"></a>","metadata":{}},{"cell_type":"markdown","source":"Compare the performance of models before and after tuning.","metadata":{}},{"cell_type":"code","source":"def display_merged_results(init_df, tuned_df):\n    merged_df = pd.merge(init_df, tuned_df, on='model', how='outer')\n    merged_df = merged_df.rename(columns={'accuracy_x': 'accuracy_init', 'accuracy_y': 'accuracy_tuned'})\n    display(merged_df\n            .sort_values(by='accuracy_tuned', ascending=False)\n            .reset_index(drop=True)\n            .style\n            .background_gradient(cmap='viridis')\n            .set_properties(**{'text-align': 'left'})\n           )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T11:40:06.890442Z","iopub.execute_input":"2023-04-08T11:40:06.890859Z","iopub.status.idle":"2023-04-08T11:40:06.898824Z","shell.execute_reply.started":"2023-04-08T11:40:06.890820Z","shell.execute_reply":"2023-04-08T11:40:06.897586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"report_alldefaults_df = (report_gridsearch_all\n                         .get_summary(sortby='mean_test_accuracy')\n                         .data\n                         .reset_index()\n                         .rename(columns={'mean_test_accuracy': 'accuracy'})\n                         .drop(['mean_test_f1_macro'], axis=1)\n                        )\ndisplay_merged_results(report_alldefaults_df, results_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-08T11:40:10.704900Z","iopub.execute_input":"2023-04-08T11:40:10.705304Z","iopub.status.idle":"2023-04-08T11:40:10.748627Z","shell.execute_reply.started":"2023-04-08T11:40:10.705271Z","shell.execute_reply":"2023-04-08T11:40:10.747661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ensemble learning <a class=\"anchor\" id=\"ensemblelearning\"></a>","metadata":{}},{"cell_type":"code","source":"# Create models without fine-tuning\nif True == False:\n    print('Create models')\n    best_svc = copy.copy(full_pipeline)\n    best_svc.set_params(model=SVC(C=10, gamma=0.013, kernel='rbf', probability=True))\n\n    best_rf = copy.copy(full_pipeline)\n    best_rf.set_params(model=RandomForestClassifier(min_samples_leaf=2, n_estimators=550, n_jobs=-1))\n\n    best_et = copy.copy(full_pipeline)\n    best_et.set_params(model=ExtraTreesClassifier(min_samples_leaf=2, n_estimators=350))\n\n    best_knn = copy.copy(full_pipeline)\n    best_knn.set_params(model=KNeighborsClassifier(n_neighbors=5, n_jobs=-1))\n\n    best_adb_et = copy.copy(full_pipeline)\n    best_adb_et.set_params(model=AdaBoostClassifier(base_estimator=best_et['model'], n_estimators=20, \n                                      learning_rate=0.02, algorithm='SAMME.R'))\n\n    best_hb = copy.copy(full_pipeline)\n    best_hb.set_params(model=HistGradientBoostingClassifier(learning_rate=0.13, min_samples_leaf=8))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-08T14:20:52.639429Z","iopub.execute_input":"2023-04-08T14:20:52.639973Z","iopub.status.idle":"2023-04-08T14:20:52.650315Z","shell.execute_reply.started":"2023-04-08T14:20:52.639911Z","shell.execute_reply":"2023-04-08T14:20:52.648835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### VotingClassifier <a class=\"anchor\" id=\"votingclassifier\"></a>","metadata":{}},{"cell_type":"markdown","source":"Create an Ensemble Voting Classifier with the best classifiers and fit it using the augmented data.\n\nI also tried EnsembleVoteClassifier from mlxtend, but performance with VotingClassifier from sklearn seemed slightly better. The advantage of EnsembleVoteClassifier is that it has a fit_base_estimators option, unlike sklearn.VotingClassifier, which doesn't exist in the current version of Kaggle. Keeping this in mind, we can test different weights with EnsembleVoteClassifier without having to fit base estimators each time and then use values with VotingClassifier. In addition, I tried StackingClassifier, but it didn't show good enough results.","metadata":{}},{"cell_type":"code","source":"estimators_vt = [\n    ('best_svc', best_svc['model']),\n    ('best_knn', best_knn['model']),\n    ('best_adb_hb', best_adb_hb['model']),\n    ('best_hb', best_hb['model']),\n    ('best_et', best_et['model']),\n    ('best_rf', best_rf['model']),\n]\npipe_vt = copy.copy(full_pipeline)\npipe_vt.set_params(model=VotingClassifier(estimators=estimators_vt, voting='soft', verbose=1, \n                                          weights=[10, 4, 3, 1, 1, 1]\n                                         ))\npipe_vt.fit(X_train_des_aug, y_train_aug)\nreport_vt = EstimatorReport(pipe_vt, X_val_deskewed, y_val, [], [], \"VotingClassifier\")\nreport_vt.display_report()","metadata":{"execution":{"iopub.status.busy":"2023-04-08T12:01:40.786099Z","iopub.execute_input":"2023-04-08T12:01:40.786652Z","iopub.status.idle":"2023-04-08T13:39:25.213879Z","shell.execute_reply.started":"2023-04-08T12:01:40.786604Z","shell.execute_reply":"2023-04-08T13:39:25.212536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission <a class=\"anchor\" id=\"submission\"></a>","metadata":{}},{"cell_type":"markdown","source":"Let's evaluate our results on the test set and save it in submission dataframe.","metadata":{}},{"cell_type":"code","source":"def create_submissions(estimator, test_data, index_data, index_col, target_col):\n    sumbission = pd.DataFrame(\n        {\n            index_col: index_data,\n            target_col: estimator.predict(test_data),\n        }\n    ).set_index(index_col)\n    sumbission.to_csv(\"submission.csv\")\n#     print(\"Submission file head:\")\n#     display(sumbission.head())","metadata":{"execution":{"iopub.status.busy":"2023-04-08T13:57:20.157610Z","iopub.execute_input":"2023-04-08T13:57:20.158116Z","iopub.status.idle":"2023-04-08T13:57:20.165503Z","shell.execute_reply.started":"2023-04-08T13:57:20.158065Z","shell.execute_reply":"2023-04-08T13:57:20.164525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"create_submissions(pipe_vt, test_data_deskewed, np.arange(1, len(test_data_deskewed)+1), 'ImageId', 'Label')","metadata":{"execution":{"iopub.status.busy":"2023-04-08T13:57:23.233165Z","iopub.execute_input":"2023-04-08T13:57:23.233535Z","iopub.status.idle":"2023-04-08T14:02:05.368687Z","shell.execute_reply.started":"2023-04-08T13:57:23.233502Z","shell.execute_reply":"2023-04-08T14:02:05.367418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion <a class=\"anchor\" id=\"conclusion\"></a>\nSolving the task without using neural networks showed relatively good performance. Preprocessing techniques and ensemble learning of the best classifiers allowed to reach a public score of 99.2%.","metadata":{}}]}