{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit recognizer project","metadata":{}},{"cell_type":"markdown","source":"According to the documentation of this competition, we have:\nthe data files train.csv and test.csv which contain gray-scale images of hand-drawn digits, from zero through nine.\n\nEach image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\nWe will build an approach to solve this problem by using a dimentionality reduction using NCA (Neighborhood Component Analysis) and KNN (K Nearest Neighbors) classifier and see that is a powerful algorithm. And finally, to tune our model, we will use the Optuna library which is a hyperparameter optimization framework.","metadata":{}},{"cell_type":"code","source":"# Imports\nimport os\nimport random\nimport warnings\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport optuna\nimport pandas as pd\nimport seaborn as sb\n\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import cross_validate\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.neighbors import NeighborhoodComponentsAnalysis","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:23:04.946923Z","iopub.execute_input":"2023-04-30T08:23:04.947695Z","iopub.status.idle":"2023-04-30T08:23:06.861310Z","shell.execute_reply.started":"2023-04-30T08:23:04.947653Z","shell.execute_reply":"2023-04-30T08:23:06.859997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filter warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:24:09.264328Z","iopub.execute_input":"2023-04-30T08:24:09.264802Z","iopub.status.idle":"2023-04-30T08:24:09.270144Z","shell.execute_reply.started":"2023-04-30T08:24:09.264762Z","shell.execute_reply":"2023-04-30T08:24:09.268799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Functions\ndef show_image(data: np.array, target: np.array) -> None:\n    \"\"\"\n    This function take the a random image from the \n    the array that contains images, reshape it and\n    plot it, with as title the truth value of the image.\n    \"\"\"\n    \n    # Reshape the data, to make 28x28 images\n    images = data.reshape(-1, 28, 28) \n    # -1 tells numpy that the value must be inferred \n    # from the length of the array and remaining dimensions.\n    \n    idx = random.randint(0, len(images))\n                   \n    # Plot a random image\n    plt.figure(figsize=(3,3))\n    plt.imshow(images[idx])\n    plt.title(f\"Real number: {target[idx]}\")\n    plt.show()\n    \ndef predict_image(data: np.array, target: np.array) -> None:\n    \"\"\"\n    This function take the a random image from the \n    the array that contains test images, reshape it \n    and plot it, with as title the predicted value \n    of the image.\n    \"\"\"\n    \n    # Reshape the data, to make 28x28 images\n    images = data.reshape(-1, 28, 28) \n    # -1 tells numpy that the value must be inferred \n    # from the length of the array and remaining dimensions.\n    \n    idx = random.randint(0, len(images))\n                   \n    # Plot a random image\n    plt.figure(figsize=(3,3))\n    plt.imshow(images[idx])\n    plt.title(f\"Predicted number: {target[idx]}\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:24:11.633621Z","iopub.execute_input":"2023-04-30T08:24:11.633995Z","iopub.status.idle":"2023-04-30T08:24:11.644303Z","shell.execute_reply.started":"2023-04-30T08:24:11.633965Z","shell.execute_reply":"2023-04-30T08:24:11.643122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"# Check files avaliable\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:33:11.549970Z","iopub.execute_input":"2023-04-30T08:33:11.550392Z","iopub.status.idle":"2023-04-30T08:33:11.558243Z","shell.execute_reply.started":"2023-04-30T08:33:11.550359Z","shell.execute_reply":"2023-04-30T08:33:11.557116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read train, test and submission csv\ndf_train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ndf_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\ndf_submission = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\n\n# Display dimensions of each dataframe\nprint(f\"The dimension of the train df is: {df_train.shape}\")\nprint(f\"The dimension of the test df is: {df_test.shape}\")\nprint(f\"The dimension of the submission df is: {df_submission.shape}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:33:14.422547Z","iopub.execute_input":"2023-04-30T08:33:14.422957Z","iopub.status.idle":"2023-04-30T08:33:20.481874Z","shell.execute_reply.started":"2023-04-30T08:33:14.422925Z","shell.execute_reply":"2023-04-30T08:33:20.480715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we have the 28k images in the test sample that need to be predict. The result will be store in the submission dataframe, and return as csv to kaggle for evaluation.","metadata":{}},{"cell_type":"code","source":"# Analyze the train dataframe\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:33:20.483935Z","iopub.execute_input":"2023-04-30T08:33:20.484334Z","iopub.status.idle":"2023-04-30T08:33:20.524547Z","shell.execute_reply.started":"2023-04-30T08:33:20.484295Z","shell.execute_reply":"2023-04-30T08:33:20.523055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The label, your target, is located in the first column of the dataframe.\nSo we can already seperate the X_train and y_train using our df_train data.","metadata":{}},{"cell_type":"code","source":"# Extract the target from the data of the train dataframe\n# And convert into numpy array\nX_train = df_train.drop(labels='label', axis=1).values\ny_train = df_train['label'].values\nX_test = df_test.values","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:33:20.526256Z","iopub.execute_input":"2023-04-30T08:33:20.526700Z","iopub.status.idle":"2023-04-30T08:33:20.703636Z","shell.execute_reply.started":"2023-04-30T08:33:20.526659Z","shell.execute_reply":"2023-04-30T08:33:20.702364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Repartition of the classes across the dataset\nplt.figure(figsize=(12,5))\nsb.countplot(data=df_train, x='label')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-30T08:33:23.259766Z","iopub.execute_input":"2023-04-30T08:33:23.260714Z","iopub.status.idle":"2023-04-30T08:33:23.534175Z","shell.execute_reply.started":"2023-04-30T08:33:23.260672Z","shell.execute_reply":"2023-04-30T08:33:23.533128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have a fairly balanced dataset here, no need to apply some methods to deal with imbalanced dataset.","metadata":{}},{"cell_type":"code","source":"# Display an image as example\nshow_image(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:01:34.594412Z","iopub.execute_input":"2023-04-30T09:01:34.594798Z","iopub.status.idle":"2023-04-30T09:01:34.773255Z","shell.execute_reply.started":"2023-04-30T09:01:34.594766Z","shell.execute_reply":"2023-04-30T09:01:34.772341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will use NCA for Neighborhood Component Analysis, which can be used as dimensionality reduction method. It learns a linear transformation in a supervised fashion to improve the classification accuracy of a stochastic nearest neighbors rule in the transformed space. In the first time we will use it for data visualization.\n\nBut before using dimensionality reduction techniques, we will normalize our data, that help to run faster the differents algorithm, and ensure more precision. Since each pixel is between 0 to 255, we can just divide all the array by 255.","metadata":{}},{"cell_type":"code","source":"# Normalize the data\nX_train = X_train / 255.0\nX_test = X_test / 255.0","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:01:37.684909Z","iopub.execute_input":"2023-04-30T09:01:37.685873Z","iopub.status.idle":"2023-04-30T09:01:37.907474Z","shell.execute_reply.started":"2023-04-30T09:01:37.685819Z","shell.execute_reply":"2023-04-30T09:01:37.905031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"AS this is only for data visualization (and observation are randomly sampled), we can use the only 500 first observations to train, transform and visualise our data points in 2D.","metadata":{}},{"cell_type":"code","source":"# Invoking the NCA method\nNCA = NeighborhoodComponentsAnalysis(n_components=2, random_state=123)\nX_nca = NCA.fit_transform(X_train[:500], y_train[:500])","metadata":{"execution":{"iopub.status.busy":"2023-04-28T12:52:28.743126Z","iopub.execute_input":"2023-04-28T12:52:28.743613Z","iopub.status.idle":"2023-04-28T12:52:29.628526Z","shell.execute_reply.started":"2023-04-28T12:52:28.743574Z","shell.execute_reply":"2023-04-28T12:52:29.626640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can plot in 2D our data and look how it is distributed across axis.","metadata":{}},{"cell_type":"code","source":"# Plot the result of TSNE reduction\nplt.figure(figsize=(12,5))\nplt.scatter(X_nca[:, 0], X_nca[:, 1], c=y_train[:500], alpha=0.7)\nfor i in range(500):\n    plt.text(X_nca[i, 0], X_nca[i, 1], str(y_train[i]))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T12:52:32.623145Z","iopub.execute_input":"2023-04-28T12:52:32.623596Z","iopub.status.idle":"2023-04-28T12:52:34.408530Z","shell.execute_reply.started":"2023-04-28T12:52:32.623556Z","shell.execute_reply":"2023-04-28T12:52:34.407195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can see that the result is pretty good. Number like 9 and 6 are close to each other and far from number like 1.\nEven if we have taken only the 500 first images, there is almost no overlaping between the different groups. It shows how powerfull is this method (NCA).\n\nMoreover, we can see that it looks like possible to reduce the dimensionality of our dataset, to ensure good and fast performance with classical machine learning methods, and avoid overfitting. Especially with KNN, since it looks for neighbors.","metadata":{}},{"cell_type":"markdown","source":"# Modeling","metadata":{}},{"cell_type":"markdown","source":"In this section, we will transform our dataset using NCA, and build a model using a simple KNN classifier on the reduced dataset. And optimize these hyperparameters with Optuna to see if we can have better result.\n\nNCA take some time to run, and moreover, Kaggle has some memory limitation, so we will use only 40% of the data to train the algorithm. To be sure to have the same representation of the classes distribution in our training set (for NCA), we use the stratify argument to ensure that each set contains approximately the same percentage of samples of each target class as the complete set.\n\nFurthemore, we set the number of component to 154 based on the result for PCA analysis in my previous notebook [PCA & SVM on MNIST](https://www.kaggle.com/code/adrienriaux/pca-svm-on-mnist).","metadata":{}},{"cell_type":"code","source":"# Invoking the NCA method\nNCA = NeighborhoodComponentsAnalysis(n_components=154, random_state=123)\n\n# Get stratified training sample for NCA \nX_train_nca, _, y_train_nca, _ = train_test_split(\n    X_train, \n    y_train, \n    train_size=0.4, \n    stratify=y_train\n)\n\n# Fit and transform the dataset the algorithm\nNCA.fit(X_train_nca, y_train_nca)\nX_train_reduced = NCA.transform(X_train)\nX_test_reduced = NCA.transform(X_test)\n\n# Display the shape of the reduced data\nX_train_reduced.shape","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:02:15.324549Z","iopub.execute_input":"2023-04-30T09:02:15.325689Z","iopub.status.idle":"2023-04-30T09:07:52.627794Z","shell.execute_reply.started":"2023-04-30T09:02:15.325646Z","shell.execute_reply":"2023-04-30T09:07:52.626240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The last cell take some time to run, but at the end, we can see that our training set has a shape of 42000*154!\n\nWe can now pass to the modeling part. We will train a KNN algorithm on the initial train set and on the reduced train set to compare both models. To avoid overfitting, we use cross validation process with 3 folds.\n\nFirst the KNN on the initial train set:","metadata":{}},{"cell_type":"code","source":"# Use cross validation to avoid overfitting\nresult_knn = pd.DataFrame(cross_validate(\n    KNeighborsClassifier(),\n    X_train,\n    y_train,\n    return_train_score=True,\n    cv=3,\n    scoring=['accuracy']\n))\n\n# Display result\nresult_knn","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:09:18.793147Z","iopub.execute_input":"2023-04-30T09:09:18.793573Z","iopub.status.idle":"2023-04-30T09:11:46.778126Z","shell.execute_reply.started":"2023-04-30T09:09:18.793540Z","shell.execute_reply":"2023-04-30T09:11:46.776916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we achieve pretty good result on the train set. But the most important is to look at the validation set. Again, we have a very good score: 96%, which means that we are not in overfitting (and not in underfitting). We can calculate the mean cross validation score on the validation set (here called the \"test_accuracy\").","metadata":{}},{"cell_type":"code","source":"# Calculate mean validation score\nprint(f\"CV validation score: {result_knn['test_accuracy'].mean():.3f}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:11:52.202866Z","iopub.execute_input":"2023-04-30T09:11:52.203626Z","iopub.status.idle":"2023-04-30T09:11:52.209737Z","shell.execute_reply.started":"2023-04-30T09:11:52.203564Z","shell.execute_reply":"2023-04-30T09:11:52.208332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can look at the result of the KNN on the reduced train set:","metadata":{}},{"cell_type":"code","source":"# Use cross validation to avoid overfitting\nresult_nca_knn = pd.DataFrame(cross_validate(\n    KNeighborsClassifier(),\n    X_train_reduced,\n    y_train,\n    return_train_score=True,\n    cv=3,\n    scoring=['accuracy']\n))\n\n# Display result\nresult_nca_knn","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:11:54.990206Z","iopub.execute_input":"2023-04-30T09:11:54.991263Z","iopub.status.idle":"2023-04-30T09:13:17.579772Z","shell.execute_reply.started":"2023-04-30T09:11:54.991219Z","shell.execute_reply":"2023-04-30T09:13:17.578634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we achieve better result on the validation set that the previous modeling, we have a very good score: 97%. And we can see that each fold take approximately twice less time to run. We can calculate the mean cross validation score on the validation set (here called the \"test_accuracy\").","metadata":{}},{"cell_type":"code","source":"# Calculate mean validation score\nprint(f\"CV validation score: {result_nca_knn['test_accuracy'].mean():.3f}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:13:56.329316Z","iopub.execute_input":"2023-04-30T09:13:56.330456Z","iopub.status.idle":"2023-04-30T09:13:56.337216Z","shell.execute_reply.started":"2023-04-30T09:13:56.330392Z","shell.execute_reply":"2023-04-30T09:13:56.336076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The result is already very good ! But may be we can see if we can improve by adding/changing some hyperparameter of the model.\n\nTo optimize our model, we will use randomized search based on cross validation using Optuna which allow to parallelize searches over multiple threads or processes without modifying code. In contrast to GridSearchCV, not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions. \n\nWe add some timeout to limit our search, and avoid too long optimization.","metadata":{}},{"cell_type":"code","source":"# Set the verbosity of optuna to 0\noptuna.logging.set_verbosity(optuna.logging.WARNING)\n\n# Create a study to optimize the parameters\nstudy = optuna.create_study(\n    study_name=\"knn_opti\", \n    direction=\"maximize\", \n    sampler=optuna.samplers.RandomSampler()\n)\n\n# Define the dictionary of parameter to optimize\nparam_grid = {\n    'n_neighbors' : optuna.distributions.IntDistribution(2, 10),\n    'metric' : optuna.distributions.CategoricalDistribution(['minkowski', 'cosine'])\n}\n\n# Define the bayesian search optimizer \noptuna_search = optuna.integration.OptunaSearchCV(\n    KNeighborsClassifier(),\n    param_grid,\n    cv=3,\n    return_train_score=True,\n    study=study,\n    timeout=600\n)\n# Fit\noptuna_search.fit(X_train_reduced, y_train)\n\nprint(f\"Number of finished trials: {optuna_search.n_trials_}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:14:53.416926Z","iopub.execute_input":"2023-04-30T09:14:53.418047Z","iopub.status.idle":"2023-04-30T09:25:51.599466Z","shell.execute_reply.started":"2023-04-30T09:14:53.417999Z","shell.execute_reply":"2023-04-30T09:25:51.597947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can check the best validation score and with which parameters this score is reached.","metadata":{}},{"cell_type":"code","source":"# Print best score and best parameters\nprint(f\"Best parameter are: {optuna_search.best_params_}\")\nprint(f\"Validation score: {optuna_search.best_score_:.3f}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:26:44.590512Z","iopub.execute_input":"2023-04-30T09:26:44.590942Z","iopub.status.idle":"2023-04-30T09:26:44.599022Z","shell.execute_reply.started":"2023-04-30T09:26:44.590909Z","shell.execute_reply":"2023-04-30T09:26:44.597467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we did not really improve our model but we can define it anyway using these hyperparameter to predict the value on the test set. ","metadata":{}},{"cell_type":"code","source":"# Define our model with the best one\nmodel = optuna_search.best_estimator_\n\n# Fit the model\nmodel.fit(X_train_reduced, y_train)\n\n# Get prediction on the test set\ny_pred = model.predict(X_test_reduced)","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:28:54.568667Z","iopub.execute_input":"2023-04-30T09:28:54.569056Z","iopub.status.idle":"2023-04-30T09:29:19.373697Z","shell.execute_reply.started":"2023-04-30T09:28:54.569023Z","shell.execute_reply":"2023-04-30T09:29:19.372397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have predict value for the test set, we can plot one to see if the result seems good.","metadata":{}},{"cell_type":"code","source":"# Example of prediction\npredict_image(X_test, y_pred)","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:29:47.569682Z","iopub.execute_input":"2023-04-30T09:29:47.570648Z","iopub.status.idle":"2023-04-30T09:29:47.760553Z","shell.execute_reply.started":"2023-04-30T09:29:47.570593Z","shell.execute_reply":"2023-04-30T09:29:47.759653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the predicted number is the number on the image!\n\nSo our work is ended. We just have to save our result in a csv file and submit this file to be analyzed and scored by Kaggle.","metadata":{}},{"cell_type":"code","source":"# Replace the label (all 0) with the prediction\ndf_submission['Label'] = y_pred\n\n# Save as csv the result\ndf_submission.to_csv('submission.csv', index=False)\n!head submission.csv # Use Linux command to show the first row of our file","metadata":{"execution":{"iopub.status.busy":"2023-04-30T09:29:52.540718Z","iopub.execute_input":"2023-04-30T09:29:52.541146Z","iopub.status.idle":"2023-04-30T09:29:53.742545Z","shell.execute_reply.started":"2023-04-30T09:29:52.541108Z","shell.execute_reply":"2023-04-30T09:29:53.741367Z"},"trusted":true},"execution_count":null,"outputs":[]}]}