{"metadata":{"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.12.5"},"colab":{"machine_shape":"hm","provenance":[],"toc_visible":true},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Frame the Problem","metadata":{"id":"fED5xCKMSi3v"}},{"cell_type":"markdown","source":"The objective is to develop a model capable of accurately identifying handwritten digits (0-9) from images, supporting automation in tasks like processing bank checks or digitizing handwritten forms. The primary business goal is to achieve high accuracy in identifying digits, ensuring reliability in various applications.\n\n\n\nThis is a **supervised learning problem** because the dataset includes labeled training data (images with their corresponding digit labels). The task is **offline** and follows a **batch learning** approach, where the model is trained once and evaluated later without continuous updates during prediction. After training, it will be deployed to make predictions without further updates.\n\n\n\nPerformance will be measured using **accuracy**, which represents the proportion of correctly classified digits out of all predictions. A higher accuracy ensures the model can reliably identify digits for practical applications like data entry and automation.\n\n\n\nThe minimum performance required will depend on the application:\n\n- In highly sensitive applications (e.g., financial systems), near-perfect accuracy (>99%) may be required.\n\n- For general educational tools, an accuracy of 90%+ might be sufficient.\n\n\n","metadata":{"id":"GrEoii1f3VhK"}},{"cell_type":"markdown","source":"# Get  the Data and Quick Look at Data Structure\n","metadata":{"id":"5ubXKGnsowqS"}},{"cell_type":"markdown","source":"To begin, we load the data from CSV files using Pandas. This is provied by kaggle. The training data includes both images and labels, while the test data only contains the images.","metadata":{"id":"Oh03ik0r4VPU"}},{"cell_type":"code","source":"import numpy as np\n\nimport pandas as pd\n\nimport matplotlib.pyplot as plt","metadata":{"id":"Pdj3zKReo6xc"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df = pd.read_csv(\"digit-recognizer/train.csv\",)\n\ntest_features_df = pd.read_csv(\"digit-recognizer/test.csv\",)","metadata":{"id":"CGt9ImPho2DS"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":" Next, we inspect the structure of the dataset by displaying basic information such as the number of entries and data types","metadata":{"id":"XRezBoGZ56mg"}},{"cell_type":"code","source":"train_df.info(show_counts=True)","metadata":{"executionInfo":{"elapsed":8,"status":"ok","timestamp":1733517803398,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"0i4t5FyDvGDt","outputId":"f2b44be6-26db-4412-f89f-c5c1272cf7e9"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.shape","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1733517803398,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"rJnO-QDMV1eX","outputId":"4541edd1-2d11-4daf-dbdb-9a3a5de37466"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Our dataset consists of 42,000 images, each represented by 784 features corresponding to a 28x28 pixel grid (all `int64`), along with one column representing the label.\n","metadata":{"id":"8xSE1HlG3TgS"}},{"cell_type":"markdown","source":"# Explore the Data","metadata":{"id":"HIpW_UHVvkyz"}},{"cell_type":"markdown","source":"## Descriptive Analysis","metadata":{"id":"iiD3pUatY8Wo"}},{"cell_type":"code","source":"train_df.describe()","metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1733517803398,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"X0Eh6JNpWhAf","outputId":"7256695b-041a-44e9-ff1e-3f5332c76561"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_pixel_value = train_df.iloc[:, 1:].max().max()\n\nprint(f\"Maximum pixel value in the dataset: {max_pixel_value}\")\n\n\n\nmin_pixel_value = train_df.iloc[:, 1:].min().min()\n\nprint(f\"Minimum pixel value in the dataset: {min_pixel_value}\")","metadata":{"executionInfo":{"elapsed":4423,"status":"ok","timestamp":1733517807818,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"x7AzpuKw3sb0","outputId":"d356a136-11fd-4819-c488-695dd37cc03b"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The dataset consists of 42,000 images, each with 784 pixel values (28x28 grid). The **label** ranges from 0 to 9. Pixel values range from 0 to 255, with a mean near 0 and minimal variation, indicating a large amount of empty space (black pixels).\n","metadata":{"id":"KGIqKWwHasNT"}},{"cell_type":"markdown","source":"## Missing Values","metadata":{"id":"Xx3qBqnWZBCX"}},{"cell_type":"code","source":"missing_values = train_df.isnull().sum()\n\nprint(\"Missing values in each column:\")\n\nprint(missing_values)\n\n\n\n\n\nif missing_values.sum() == 0:\n\n    print(\"\\nNo missing data detected.\")\n\nelse:\n\n    print(\"\\nMissing data detected. Consider imputation or removal.\")\n","metadata":{"executionInfo":{"elapsed":9,"status":"ok","timestamp":1733517807818,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"tsUmgXuYZCQ3","outputId":"e22412c7-a209-43dc-c0e2-1d40127ed2ec"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"There are no missing values in the dataset, as indicated by the output where each column has 0 missing values. This means the dataset is complete, and no imputation or removal of missing data is required.\n","metadata":{"id":"ehKqRBMtbHK2"}},{"cell_type":"markdown","source":"## Data Visualisation","metadata":{"id":"PjgZAq4JZIf_"}},{"cell_type":"markdown","source":"Here, we want to visualize the first 20 images along with their corresponfing labels.\n","metadata":{"id":"6n06CAvq6Qba"}},{"cell_type":"code","source":"def plot_digit(image_data):\n\n    image = image_data.reshape(28, 28)\n\n    plt.imshow(image, cmap=\"binary\")\n\n    plt.axis(\"off\")\n\n\n\nplt.figure(figsize=(10, 6))\n\n\n\nfor index in range(20):\n\n    plt.subplot(4, 5, index + 1)\n\n    image_data = train_df.iloc[index, 1:].values\n\n    label = train_df.iloc[index, 0]\n\n    plot_digit(image_data)\n\n    plt.title(f\"Label: {label}\")\n\n\n\nplt.tight_layout()\n\nplt.show()","metadata":{"executionInfo":{"elapsed":8,"status":"ok","timestamp":1733517807818,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"VBW-r59LXDTq","outputId":"24f25aeb-678c-4368-d711-d8008d813129"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Class Distribution","metadata":{"id":"AkbsFk--ZMSI"}},{"cell_type":"markdown","source":"Next, we examine the distribution of each label to determine if special handling is required. By analyzing the label distribution, we can assess if the data is balanced or if any class imbalance exists that might affect model performance.","metadata":{"id":"kdXXlDv4-lkF"}},{"cell_type":"code","source":"label_distribution = train_df['label'].value_counts()\n\nprint(\"Label Distribution:\")\n\nprint(label_distribution)\n\n\n\nlabel_distribution.plot(kind='bar', color='blue')\n\nplt.title(\"Label Distribution\")\n\nplt.xlabel(\"Class Labels\")\n\nplt.ylabel(\"Frequency\")\n\nplt.show()\n","metadata":{"executionInfo":{"elapsed":7,"status":"ok","timestamp":1733517807818,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"aYYOGxCiaRTW","outputId":"e0a215e8-1720-4ba1-99dd-d5326a3475c3"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As we can see, the data distribution across each class appears balanced, meaning no special handling for class imbalance is required.\n","metadata":{"id":"Qxcmjhp4bfH7"}},{"cell_type":"markdown","source":"## Correlations Between Features","metadata":{"id":"uD6kbUQoZT2C"}},{"cell_type":"code","source":"\n\ncorrelation_matrix = train_df.iloc[:, 1:].corr()\n\nimport seaborn as sns\n\n\n\nplt.figure(figsize=(12, 8))\n\nsns.heatmap(correlation_matrix, cmap='coolwarm', xticklabels=False, yticklabels=False)\n\nplt.title(\"Pixel-to-Pixel Correlation Matrix\")\n\nplt.show()\n\n\n\n\n","metadata":{"executionInfo":{"elapsed":69886,"status":"ok","timestamp":1733517877699,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"MoMttY-qZWs2","outputId":"af0f5d7c-6c44-429e-afd6-8c4ba0e76ad2"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As expected, we can observe that neighboring pixels tend to have higher correlation values, especially along the diagonal, which reflects the structure of the images. The weak correlation between distant pixels suggests that they have less direct relationship with each other. This is typical for image data, where local features (like edges or patterns) are more strongly correlated than pixels that are far apart.\n","metadata":{"id":"aaXT-JpYrdgL"}},{"cell_type":"markdown","source":"## Correlation Between Features and Target","metadata":{"id":"Px_tAOzLZ_gn"}},{"cell_type":"code","source":"import pandas as pd\n\nimport numpy as np\n\nimport matplotlib.pyplot as plt\n\n\n\ntarget_columns = pd.get_dummies(train_df.iloc[:, 0])\n\n\n\n\n\ncorrelation_per_digit = {}\n\nfor digit in range(10):\n\n    correlation_per_digit[digit] = train_df.iloc[:, 1:].corrwith(target_columns[digit])","metadata":{"executionInfo":{"elapsed":6540,"status":"ok","timestamp":1733517884235,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"RU7U66yjaCtO","outputId":"263963c5-de0c-41e2-ed9d-f425ed341733"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\n\nfor digit in range(10):\n\n    correlation_image = correlation_per_digit[digit].values.reshape(28, 28)  # Reshape to 28x28 grid\n\n\n\n    plt.figure(figsize=(8, 8))\n\n    plt.imshow(correlation_image, cmap='coolwarm', interpolation='nearest')\n\n    plt.colorbar(label=\"Correlation Coefficient\")\n\n    plt.title(f\"Pixel Correlations with Target Digit: {digit}\")\n\n    plt.xlabel(\"Pixel Position (X)\")\n\n    plt.ylabel(\"Pixel Position (Y)\")\n\n    plt.show()\n","metadata":{"executionInfo":{"elapsed":3421,"status":"ok","timestamp":1733517887652,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"pOxifjzOzcqT","outputId":"9e6d8ab8-cf6c-44e8-9922-ef1605d7cfeb"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The heatmap displays the correlation between each pixel and the target digit (0-9), with positive correlations in red and negative correlations in blue. For example, the correlation for digit \"5\" highlights key pixels that are strongly correlated and resemble the shape of the digit, showing their significance in digit identification. This visualization helps focus on relevant features for model training.\n","metadata":{"id":"qCrk963bhi7o"}},{"cell_type":"markdown","source":"# Prepating the Data for ML","metadata":{"id":"m5uWmdzT0XWC"}},{"cell_type":"markdown","source":"## Clean the Data","metadata":{"id":"GIMnE4F_0wzr"}},{"cell_type":"markdown","source":"Since we are working with the MNIST dataset, it consists of standardized grayscale images, meaning there are no outliers. Additionally, all 784 pixel values for each image are complete, means no missing values.\n","metadata":{"id":"tglg0yUL02a6"}},{"cell_type":"markdown","source":"## Perform Feature Selection","metadata":{"id":"eG37S53k1caI"}},{"cell_type":"markdown","source":"Here, we will perform feature selection by thresholding the correlation of each pixel with a certain value. This process eliminates pixels or areas that are rarely touched by handwriting, or removing inactive pixels. This process is akin to dropping attributes that provide no useful information for the task.","metadata":{"id":"iq65XlOs3eKl"}},{"cell_type":"code","source":"correlations = train_df.iloc[:, 1:].corrwith(train_df.iloc[:, 0])\n\n\n\ncorrelation_threshold = 0.003\n\nimportant_features = correlations[correlations.abs() > correlation_threshold].index\n\n\n\n\n\nX_reduced = train_df.loc[:, important_features]\n\nprint(f\"Original number of features: {train_df.shape[1] - 1}\")\n\nprint(f\"Reduced number of features: {X_reduced.shape[1]}\")\n","metadata":{"executionInfo":{"elapsed":498,"status":"ok","timestamp":1733517888145,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"fdq5jRAB2cW0","outputId":"671103db-4b66-41e4-c0ac-d06f597cac7e"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We successfully reduced the features from 784 to 676. Next, we will visualize which pixels are commonly active and which pixels are inactive (eliminated due to the low correlation threshold).\n\n\n","metadata":{"id":"eoEYet4s4vZU"}},{"cell_type":"code","source":"\n\nimport matplotlib.pyplot as plt\n\n\n\n\n\nselected_features_grid = np.zeros((28, 28))\n\n\n\nfor feature in important_features:\n\n    pixel_index = int(feature.replace(\"pixel\", \"\"))\n\n    row, col = divmod(pixel_index, 28)\n\n    selected_features_grid[row, col] = 1  # Mark the selected pixel\n\n\n\nplt.figure(figsize=(8, 8))\n\nplt.imshow(selected_features_grid, cmap='Greys', interpolation='nearest')\n\nplt.colorbar(label=\"Selected Features (1 = Selected, 0 = Not Selected)\")\n\nplt.title(\"Visualization of Selected Features\")\n\nplt.xlabel(\"Pixel Position (X)\")\n\nplt.ylabel(\"Pixel Position (Y)\")\n\nplt.show()\n","metadata":{"executionInfo":{"elapsed":643,"status":"ok","timestamp":1733517888785,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"fmn3VG5P3Yui","outputId":"ec2016d6-9463-49bd-e6cb-52cd577cf9b4"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Standardize Features","metadata":{"id":"Eo4GXtvN54cV"}},{"cell_type":"markdown","source":"As we know, the pixel values range from 0 to 255. We will later implement standardization to center the values around 0 and scale them to unit variance, incorporating this step into the pipeline.\n","metadata":{"id":"V3QQTSci6OsM"}},{"cell_type":"markdown","source":"## Create Validation and Smaller Training Dataset\n\n\n","metadata":{"id":"RRBqu4nuZ8kb"}},{"cell_type":"markdown","source":"To improve model evaluation and training efficiency, we will create the following datasets:\n\n1. **Validation Dataset**: A subset of the data to assess model performance during training and tuning.\n\n2. **Smaller Training Dataset**: A reduced version of the training data for faster experimentation.\n\n\n\nNow, we will split the training set into 80% for training and 20% for validation, as no special handling is needed due to (almost) balanced class distribution as we saw earlier.","metadata":{"id":"y6ulALrHaMzs"}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n\n\nX = train_df.drop(columns=['label'])\n\ny = train_df['label']\n\n\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)","metadata":{"id":"LkvMY0c1aBFR"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Training set class distribution:\")\n\nprint(y_train.value_counts(normalize=True))\n\nprint(\"Number of images in training set:\", len(y_train))\n\n\n\nprint(\"\\nValidation set class distribution:\")\n\nprint(y_val.value_counts(normalize=True))\n\nprint(\"Number of images in validation set:\", len(y_val))\n","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1733517888785,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"mQ49Uf9Bb4NI","outputId":"2141e554-fd13-4a3c-f892-cc169a301674"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"To simplify the model selection process and speed up training, we first create a smaller subset of the dataset. This reduced set allows for faster model evaluation and selection.","metadata":{"id":"HiP3XGVqzNAe"}},{"cell_type":"code","source":"X_small_train, _, y_small_train, _ = train_test_split(X_train, y_train, test_size=0.8, random_state=42, stratify=y_train)\n\n\n\nprint(f\"Size of the smaller training set: {X_small_train.shape[0]} images\")\n","metadata":{"executionInfo":{"elapsed":334,"status":"ok","timestamp":1733517889115,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"omchJ8Yvu5xk","outputId":"34bd3ecb-5093-4629-ab5f-8d77d9c0fe2d"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Making The Pipeline","metadata":{"id":"-_u4yAGb_y9-"}},{"cell_type":"code","source":"from sklearn.base import BaseEstimator, TransformerMixin\n\nfrom sklearn.pipeline import Pipeline\n\nfrom sklearn.preprocessing import StandardScaler\n\nimport pandas as pd\n\n\n\nclass CorrelationSelector(BaseEstimator, TransformerMixin):\n\n    def __init__(self, threshold=0.01):\n\n        self.threshold = threshold\n\n        self.selected_features = None\n\n\n\n    def fit(self, X, y):\n\n        correlations = pd.DataFrame(X).corrwith(pd.Series(y))\n\n        self.selected_features = correlations[correlations.abs() > self.threshold].index\n\n        return self\n\n\n\n    def transform(self, X):\n\n        return pd.DataFrame(X).iloc[:, self.selected_features].values\n\n\n\npipeline = Pipeline([\n\n    ('correlation_selector', CorrelationSelector(threshold=0.003)),\n\n    ('standard_scaler', StandardScaler()),\n\n])\n\n\n\nX_train_transformed = pipeline.fit_transform(X_train, y_train)\n\n\n\ndef process_dataset(X, pipeline):\n\n    return pipeline.transform(X)\n\n\n\nX_val_transformed = process_dataset(X_val, pipeline)\n\nX_small_train, _, y_small_train, _ = train_test_split(X_train, y_train, test_size=0.8, random_state=42, stratify=y_train)\n\nX_small_train_transformed = process_dataset(X_small_train, pipeline)\n\n\n\nprint(f\"Original number of features: {X_train.shape[1]}\")\n\nprint(f\"Reduced number of features: {X_train_transformed.shape[1]}\")\n\nprint(f\"Validation set shape after transformation: {X_val_transformed.shape}\")\n\nprint(f\"Small training set shape after transformation: {X_small_train_transformed.shape}\")\n","metadata":{"executionInfo":{"elapsed":1084,"status":"ok","timestamp":1733517908795,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"XfcBsqXy_1k5","outputId":"0c85150d-fc79-4431-e1cb-fae45a6a1c47"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Training and Selecting a Model","metadata":{"id":"N1NdlA5rYVAa"}},{"cell_type":"markdown","source":"## Train Dirty Models","metadata":{"id":"ScoiVzeB-uW-"}},{"cell_type":"markdown","source":"Here, we will train 6 models from different categories with standard parameters","metadata":{"id":"tNTxW6dZpFiq"}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression, SGDClassifier\n\nfrom sklearn.ensemble import RandomForestClassifier\n\nfrom sklearn.tree import DecisionTreeClassifier\n\nfrom sklearn.svm import SVC\n\nfrom sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier\n\nfrom sklearn.model_selection import cross_val_score\n\nimport joblib\n\nimport os\n\n\n\n# Define classification models\n\nmodels = {\n\n    \"Logistic Regression\": LogisticRegression(max_iter=1000, random_state=42),\n\n    \"SGD OvR\": OneVsRestClassifier(SGDClassifier(max_iter=1000, random_state=42)),\n\n    \"SGD OvO\": OneVsOneClassifier(SGDClassifier(max_iter=1000, random_state=42)),\n\n    \"Decision Tree Classifier\": DecisionTreeClassifier(max_depth=10, random_state=42),\n\n    \"Random Forest Classifier\": RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),\n\n    \"Support Vector Classifier\": SVC()\n\n}\n\n\n\n# Placeholder to store model scores\n\nmodel_scores = {}\n\n\n\n# Iterate over models\n\nfor name, model in models.items():\n\n    filename = f\"{name.replace(' ', '_').lower()}_model_crosval.joblib\"\n\n\n\n    if os.path.exists(filename):\n\n        model = joblib.load(filename)\n\n        accuracy_loaded = cross_val_score(model, X_small_train_transformed, y_small_train, cv=5, scoring='accuracy').mean()\n\n        print(f\"Loaded model accuracy for {name}: {accuracy_loaded}\")\n\n    else:\n\n        accuracy_score = cross_val_score(model, X_small_train_transformed, y_small_train, cv=5, scoring='accuracy').mean()\n\n        model_scores[name] = accuracy_score\n\n        print(f\"Cross-validated accuracy for {name}: {accuracy_score}\")\n\n\n\n        model.fit(X_small_train_transformed, y_small_train)\n\n\n\n        joblib.dump(model, filename)\n\n\n\n# Print results\n\nprint(\"\\nModel scores:\")\n\nfor name, score in model_scores.items():\n\n    print(f\"{name}: {score}\")\n","metadata":{"executionInfo":{"elapsed":153033,"status":"ok","timestamp":1733518061826,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"zy8kWYsc-wvR","outputId":"42fb81a5-46d5-4aa1-da8b-234ff5bc5f64"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now, we want to evaluate whether using a pipeline for data preprocessing improves accuracy. To do this, we will compare the performance of each training model with and without data preprocessing through a pipeline.\n","metadata":{"id":"GdTF8hLnikjW"}},{"cell_type":"code","source":"\n\nmodel_scores_raw = {}\n\n\n\nfor name, model in models.items():\n\n    filename = f\"{name.replace(' ', '_').lower()}_raw_model_crosval.joblib\"\n\n\n\n    if os.path.exists(filename):\n\n        model = joblib.load(filename)\n\n        accuracy_loaded = cross_val_score(model, X_small_train, y_small_train, cv=5, scoring='accuracy').mean()\n\n        print(f\"Loaded model accuracy (raw data) for {name}: {accuracy_loaded}\")\n\n    else:\n\n        accuracy_score = cross_val_score(model, X_small_train, y_small_train, cv=5, scoring='accuracy').mean()\n\n        model_scores_raw[name] = accuracy_score\n\n        print(f\"Cross-validated accuracy (raw data) for {name}: {accuracy_score}\")\n\n\n\n        model.fit(X_small_train, y_small_train)\n\n\n\n        joblib.dump(model, filename)\n\n\n\nprint(\"\\nModel scores (raw data):\")\n\nfor name, score in model_scores_raw.items():\n\n    print(f\"{name}: {score}\")\n","metadata":{"executionInfo":{"elapsed":74347,"status":"ok","timestamp":1733518136169,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"NS-Ibwl5zfAZ","outputId":"1000a49f-787b-4df9-f0c2-8dfd29424dcb"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As we can see, all models showed improved accuracy with the pipeline, except for Random Forest, which slightly decreased, and SVR. For SVR, we will avoid using the pipeline during training if that model is selected for fine-tuning.\n","metadata":{"id":"bmx4Jyx2qIo1"}},{"cell_type":"code","source":"X_small_train.shape","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1733518136169,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"FW5RCwb5vLNC","outputId":"3df0a2e4-c242-4e15-be0a-b843478e6a92"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Error Analysis","metadata":{"id":"cRWDGV24I3Om"}},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay, classification_report, accuracy_score\n\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\n\nimport joblib\n\n\n\nmodel_filenames = {\n\n    \"Logistic Regression\": \"logistic_regression_model_retrain.joblib\",\n\n    \"SGD OvR\": \"sgd_ovr_model_retrain.joblib\",\n\n    \"SGD OvO\": \"sgd_ovo_model_retrain.joblib\",\n\n    \"Random Forest Classifier\": \"random_forest_classifier_model_retrain.joblib\",\n\n    \"Support Vector Classifier\": \"support_vector_classifier_raw_model_retrain.joblib\"\n\n}\n\n\n\nerror_analysis = {}\n\n\n\nfor name, filename in model_filenames.items():\n\n    if name == \"Support Vector Classifier\":\n\n        X_train_final = X_train\n\n        X_val_final = X_val\n\n    else:\n\n        X_train_final = X_train_transformed\n\n        X_val_final = X_val_transformed\n\n\n\n    model = joblib.load(filename) if os.path.exists(filename) else models[name]\n\n    model.fit(X_train_final, y_train)\n\n\n\n    joblib.dump(model, filename)\n\n    print(f\"Retrained and saved model: {name}\")\n\n\n\n    y_train_pred = model.predict(X_train_final)\n\n    train_acc = accuracy_score(y_train, y_train_pred)\n\n    print(f\"Training Accuracy for {name}: {train_acc}\")\n\n\n\n    y_val_pred = model.predict(X_val_final)\n\n    val_acc = accuracy_score(y_val, y_val_pred)\n\n    print(f\"Validation Accuracy for {name}: {val_acc}\")\n\n\n\n    report = classification_report(y_val, y_val_pred, output_dict=True)\n\n\n\n    error_analysis[name] = {\n\n        \"Training Accuracy\": train_acc,\n\n        \"Validation Accuracy\": val_acc,\n\n        \"Precision\": report[\"weighted avg\"][\"precision\"],\n\n        \"Recall\": report[\"weighted avg\"][\"recall\"],\n\n        \"F1-Score\": report[\"weighted avg\"][\"f1-score\"],\n\n    }\n\n\n\n    cm = confusion_matrix(y_val, y_val_pred)\n\n    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=model.classes_)\n\n    disp.plot(cmap=\"Blues\", xticks_rotation=\"vertical\")\n\n    plt.title(f\"Confusion Matrix for {name} (Validation Set)\")\n\n    plt.show()\n\n\n\nerror_df = pd.DataFrame(error_analysis).T\n\n\n\nprint(\"\\nError Analysis Metrics (Training vs Validation):\")\n\nprint(error_df)\n\n\n\n\n","metadata":{"id":"Reid6c3RUNoK","outputId":"64bd8de1-2833-4469-c0f6-2587f6eddfcc","executionInfo":{"status":"ok","timestamp":1733518773937,"user_tz":-180,"elapsed":124291,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Logistic Regression:  \n\nThe training accuracy is relatively high (95.6%), but there is a noticeable drop in validation accuracy (90.2%). This drop suggests that the model has learned the training data well but struggles to generalize to unseen data. While this might hint at overfitting, the simplicity of Logistic Regression and its limited capacity to capture complex patterns in high-dimensional data like MNIST indicate potential underfitting. The confusion matrix highlights consistent misclassifications, especially between digits 4 and 9, and 8 and 5.\n\n\n\nSGD OvR:  \n\nSGD OvR achieves a training accuracy of 92.2% and a validation accuracy of 90.5%, indicating good generalization with minimal overfitting. The confusion matrix shows higher confusion compared to SGD OvO and SVC, especially for digits 3 and 5, and digits 8 and 9. The precision, recall, and F1-score reflect this performance. Although it performs better than Logistic Regression, it is not as robust as other shortlisted models.\n\n\n\nSGD OvO:  \n\nSGD OvO has a training accuracy of 96.7% and validation accuracy of 91.5%. The slightly higher training accuracy suggests slight overfitting compared to SGD OvR. The confusion matrix reveals higher misclassification rates between digits 6 and 8, and digits 5 and 3. Despite these limitations, it hass balanced precision, recall, and F1-score make it competitive to another models.\n\n\n\nRandom Forest Classifier:  \n\nRandom Forest exhibits perfect training accuracy (100%), indicating overfitting. However, its validation accuracy remains high at 96.5%, demonstrating strong performance on unseen data. The confusion matrix reveals slightly higher misclassification rates for classes 2, 3, and 5, but overall, it handles class differentiation well. Regularization could further reduce overfitting, making this model a strong candidate.\n\n\n\nSupport Vector Classifier (SVC):  \n\nSVC achieves training accuracy of 98.9% and validation accuracy of 97.4%, demonstrating excellent generalization (no overfitting). The confusion matrix indicates the most balanced performance among all models, with minimal confusion across all classes. Metrics such as precision, recall, and F1-score validate its superior performance. SVC is the best-performing model, with no significant signs of overfitting or underfitting.\n\n\n\n---\n\n\n\nShortlisted Models:  \n\n1. Support Vector Classifier (SVC):  \n\n   SVC outperforms all models in validation accuracy (97.4%), generalization, and balanced metrics. It is the most robust and reliable choice for deployment.\n\n\n\n2. Random Forest Classifier:  \n\n   Despite slight overfitting, Random Forest delivers strong validation accuracy (96.5%) and robust metrics. With regularization, it can compete closely with SVC.\n\n\n\n3. SGD OvO:  \n\n   While slightly prone to overfitting, SGD OvO provides competitive validation accuracy (91.5%) and balanced metrics, making it a viable option when we care about computational efficiency.\n\n\n","metadata":{"id":"p57XLjiD3eky"}},{"cell_type":"markdown","source":"# Fine Tuning The Model","metadata":{"id":"5DTscpNe3THH"}},{"cell_type":"markdown","source":"As previously mentioned, we will use the preprocessing pipeline for most models, except for the SVC, where we will use the raw data directly.\n","metadata":{"id":"XI1nOCeepasV"}},{"cell_type":"markdown","source":"## SVC","metadata":{"id":"0gXHLY3J7xd_"}},{"cell_type":"code","source":"from sklearn.svm import SVC\n\nfrom sklearn.model_selection import GridSearchCV\n\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix, ConfusionMatrixDisplay\n\nimport matplotlib.pyplot as plt\n\n\n\nparam_grid = {\n\n    'C': [0.1, 1, 10],\n\n    'gamma': ['scale', 'auto', 0.1, 1, 10],\n\n    'kernel': ['rbf', 'linear']\n\n}\n\n\n\n\n\nsvc = SVC()\n\ngrid_search = GridSearchCV(estimator=svc, param_grid=param_grid, cv=3, scoring='accuracy', n_jobs=-1, verbose=2)\n\ngrid_search.fit(X_small_train, y_small_train)\n\n\n\nbest_params = grid_search.best_params_\n\nprint(\"Best Hyperparameters from Small Dataset:\", best_params)\n\n\n\nbest_model = SVC(**best_params)\n\nbest_model.fit(X_train, y_train)\n\n\n\ntrain_predictions = best_model.predict(X_train)\n\ntrain_accuracy = accuracy_score(y_train, train_predictions)\n\nprint(\"\\nTraining Accuracy:\", train_accuracy)\n\n\n\nval_predictions = best_model.predict(X_val)\n\nval_accuracy = accuracy_score(y_val, val_predictions)\n\nprint(\"Validation Accuracy:\", val_accuracy)\n\n\n\nprint(\"\\nClassification Report (Validation Set):\")\n\nprint(classification_report(y_val, val_predictions))\n\n\n\ncm = confusion_matrix(y_val, val_predictions)\n\ndisp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=best_model.classes_)\n\ndisp.plot(cmap=\"Blues\", xticks_rotation=\"vertical\")\n\nplt.title(\"Confusion Matrix (Validation Set)\")\n\nplt.show()\n","metadata":{"id":"FkCkcWEh7yN9","executionInfo":{"status":"ok","timestamp":1733519513794,"user_tz":-180,"elapsed":739867,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}},"outputId":"e3ddba7b-3fec-4ad2-adca-75e64fe1cd4b"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As we can see, the accuracy for both training and validation has increased. Before tuning, it was 0.989 and 0.974, respectively. After tuning, it improved to 0.999 and 0.979.\n","metadata":{"id":"PtD4tTjerjcq"}},{"cell_type":"markdown","source":"## Random Forest Classifier","metadata":{"id":"yJR0KnXRFRcb"}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nfrom sklearn.model_selection import GridSearchCV\n\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix, ConfusionMatrixDisplay\n\nimport matplotlib.pyplot as plt\n\n\n\nparam_grid = {\n\n    'n_estimators': [50, 100, 200],\n\n    'max_depth': [10, 15, 20, None],\n\n    'min_samples_split': [2, 10, 20],\n\n    'min_samples_leaf': [1, 2, 5],\n\n    'max_features': ['sqrt', 'log2']\n\n}\n\n\n\nrf = RandomForestClassifier(random_state=42)\n\n\n\ngrid_search_rf = GridSearchCV(estimator=rf, param_grid=param_grid, cv=3, scoring='accuracy', n_jobs=-1, verbose=2)\n\ngrid_search_rf.fit(X_small_train_transformed, y_small_train)\n\n\n\nbest_params_rf = grid_search_rf.best_params_\n\nprint(\"Best Hyperparameters from Small Dataset:\", best_params_rf)\n\n\n\nbest_rf_model = RandomForestClassifier(**best_params_rf, random_state=42)\n\nbest_rf_model.fit(X_train_transformed, y_train)\n\n\n\ntrain_predictions_rf = best_rf_model.predict(X_train_transformed)\n\ntrain_accuracy_rf = accuracy_score(y_train, train_predictions_rf)\n\nprint(\"\\nTraining Accuracy:\", train_accuracy_rf)\n\n\n\nval_predictions_rf = best_rf_model.predict(X_val_transformed)\n\nval_accuracy_rf = accuracy_score(y_val, val_predictions_rf)\n\nprint(\"Validation Accuracy:\", val_accuracy_rf)\n\n\n\nprint(\"\\nClassification Report (Validation Set):\")\n\nprint(classification_report(y_val, val_predictions_rf))\n\n\n\ncm_rf = confusion_matrix(y_val, val_predictions_rf)\n\ndisp_rf = ConfusionMatrixDisplay(confusion_matrix=cm_rf, display_labels=best_rf_model.classes_)\n\ndisp_rf.plot(cmap=\"Blues\", xticks_rotation=\"vertical\")\n\nplt.title(\"Confusion Matrix (Validation Set)\")\n\nplt.show()\n","metadata":{"id":"Y7nBbpd2FW3q","executionInfo":{"status":"ok","timestamp":1733519853144,"user_tz":-180,"elapsed":339355,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}},"outputId":"e4d1e23a-2d02-4d37-a196-e5031cacd496"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As we can see, the accuracy for both training and validation remains almost the same before and after tuning.\n","metadata":{"id":"Hs8QJtevqtMZ"}},{"cell_type":"markdown","source":"## SGD OvO","metadata":{"id":"Lh50PzXEMMFv"}},{"cell_type":"code","source":"from sklearn.linear_model import SGDClassifier\n\nfrom sklearn.multiclass import OneVsOneClassifier\n\nfrom sklearn.model_selection import GridSearchCV\n\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix, ConfusionMatrixDisplay\n\nimport matplotlib.pyplot as plt\n\n\n\nparam_grid = {\n\n    'estimator__alpha': [0.0001, 0.001, 0.01],\n\n    'estimator__penalty': ['l2', 'l1', 'elasticnet'],\n\n    'estimator__max_iter': [1000, 2000],\n\n    'estimator__tol': [1e-3, 1e-4]\n\n}\n\n\n\nsgd_ovo = OneVsOneClassifier(SGDClassifier(random_state=42))\n\n\n\ngrid_search_sgd_ovo = GridSearchCV(estimator=sgd_ovo, param_grid=param_grid, cv=3, scoring='accuracy', n_jobs=-1, verbose=2)\n\ngrid_search_sgd_ovo.fit(X_small_train_transformed, y_small_train)\n\n\n\nbest_params_sgd_ovo = grid_search_sgd_ovo.best_params_\n\nprint(\"Best Hyperparameters from Small Dataset:\", best_params_sgd_ovo)\n\n\n\nbest_sgd_ovo_model = OneVsOneClassifier(SGDClassifier(**{k.split('__')[1]: v for k, v in best_params_sgd_ovo.items()}, random_state=42))\n\nbest_sgd_ovo_model.fit(X_train_transformed, y_train)\n\n\n\ntrain_predictions_sgd_ovo = best_sgd_ovo_model.predict(X_train_transformed)\n\ntrain_accuracy_sgd_ovo = accuracy_score(y_train, train_predictions_sgd_ovo)\n\nprint(\"\\nTraining Accuracy:\", train_accuracy_sgd_ovo)\n\n\n\nval_predictions_sgd_ovo = best_sgd_ovo_model.predict(X_val_transformed)\n\nval_accuracy_sgd_ovo = accuracy_score(y_val, val_predictions_sgd_ovo)\n\nprint(\"Validation Accuracy:\", val_accuracy_sgd_ovo)\n\n\n\nprint(\"\\nClassification Report (Validation Set):\")\n\nprint(classification_report(y_val, val_predictions_sgd_ovo))\n\n\n\ncm_sgd_ovo = confusion_matrix(y_val, val_predictions_sgd_ovo)\n\ndisp_sgd_ovo = ConfusionMatrixDisplay(confusion_matrix=cm_sgd_ovo, display_labels=best_sgd_ovo_model.classes_)\n\ndisp_sgd_ovo.plot(cmap=\"Blues\", xticks_rotation=\"vertical\")\n\nplt.title(\"Confusion Matrix (Validation Set)\")\n\nplt.show()\n","metadata":{"id":"7wHH6X3WMNcY","executionInfo":{"status":"ok","timestamp":1733520059721,"user_tz":-180,"elapsed":206577,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}},"outputId":"170cc7ed-b03a-4f20-cbfc-6976232a91ce"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As we can see, the accuracy for both training and validation has increased. Before tuning, it was 0.96 and 0.91, respectively. After tuning, it improved to 0.95 and 0.936.\n","metadata":{"id":"i5wemmy2rGEO"}},{"cell_type":"markdown","source":"## Ensemble Method: XGBoost","metadata":{"id":"xlONB4_bS217"}},{"cell_type":"markdown","source":"Now, we will begin using the ensemble method, starting with XGBoost.\n\n\n\n**Note**: The default parameters for XGBoost were unavailable, so the values were chosen arbitrarily.\n","metadata":{"id":"LvRISfZeYxSU"}},{"cell_type":"code","source":"from xgboost import XGBClassifier\n\nfrom sklearn.model_selection import GridSearchCV\n\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix, ConfusionMatrixDisplay\n\nimport matplotlib.pyplot as plt\n\n\n\nparam_grid = {\n\n    'n_estimators': [100],\n\n    'max_depth': [6, 10],\n\n    'learning_rate': [0.1],\n\n    'subsample': [0.8],\n\n    'colsample_bytree': [0.8, 1.0],\n\n}\n\n\n\nxgb = XGBClassifier(use_label_encoder=False, eval_metric='logloss', random_state=42)\n\n\n\ngrid_search_xgb = GridSearchCV(estimator=xgb, param_grid=param_grid, cv=3, scoring='accuracy', n_jobs=-1, verbose=2)\n\ngrid_search_xgb.fit(X_small_train_transformed, y_small_train)\n\n\n\nbest_params_xgb = grid_search_xgb.best_params_\n\nprint(\"Best Hyperparameters from Small Dataset:\", best_params_xgb)\n\n\n\nbest_xgb_model = XGBClassifier(**best_params_xgb, use_label_encoder=False, eval_metric='logloss', random_state=42)\n\nbest_xgb_model.fit(X_train_transformed, y_train)\n\n\n\ntrain_predictions_xgb = best_xgb_model.predict(X_train_transformed)\n\ntrain_accuracy_xgb = accuracy_score(y_train, train_predictions_xgb)\n\nprint(\"\\nTraining Accuracy:\", train_accuracy_xgb)\n\n\n\nval_predictions_xgb = best_xgb_model.predict(X_val_transformed)\n\nval_accuracy_xgb = accuracy_score(y_val, val_predictions_xgb)\n\nprint(\"Validation Accuracy:\", val_accuracy_xgb)\n\n\n\nprint(\"\\nClassification Report (Validation Set):\")\n\nprint(classification_report(y_val, val_predictions_xgb))\n\n\n\ncm_xgb = confusion_matrix(y_val, val_predictions_xgb)\n\ndisp_xgb = ConfusionMatrixDisplay(confusion_matrix=cm_xgb, display_labels=best_xgb_model.classes_)\n\ndisp_xgb.plot(cmap=\"Blues\", xticks_rotation=\"vertical\")\n\nplt.title(\"Confusion Matrix (Validation Set)\")\n\nplt.show()\n","metadata":{"id":"yl0E6zJAS5GL","executionInfo":{"status":"ok","timestamp":1733520568631,"user_tz":-180,"elapsed":508912,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}},"outputId":"1f9c3ee0-0eb3-4885-ab7b-1f6bee9db017"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Ensemble Method: LightGBM","metadata":{"id":"Tb2pOQRdYDhK"}},{"cell_type":"code","source":"from lightgbm import LGBMClassifier\n\nfrom sklearn.model_selection import GridSearchCV\n\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix, ConfusionMatrixDisplay\n\nimport matplotlib.pyplot as plt\n\n\n\nparam_grid = {\n\n    'n_estimators': [100],\n\n    'max_depth': [-1, 6],\n\n    'learning_rate': [0.1],\n\n    'subsample': [1.0]\n\n}\n\nlgbm = LGBMClassifier(random_state=42)\n\n\n\ngrid_search_lgbm = GridSearchCV(estimator=lgbm, param_grid=param_grid, cv=2, scoring='accuracy', n_jobs=-1, verbose=2)\n\ngrid_search_lgbm.fit(X_small_train_transformed, y_small_train)\n\n\n\nbest_params_lgbm = grid_search_lgbm.best_params_\n\nprint(\"Best Hyperparameters from Small Dataset:\", best_params_lgbm)\n\n\n\nbest_lgbm_model = LGBMClassifier(**best_params_lgbm, random_state=42)\n\nbest_lgbm_model.fit(X_train_transformed, y_train)\n\n\n\ntrain_predictions_lgbm = best_lgbm_model.predict(X_train_transformed)\n\ntrain_accuracy_lgbm = accuracy_score(y_train, train_predictions_lgbm)\n\nprint(\"\\nTraining Accuracy:\", train_accuracy_lgbm)\n\n\n\nval_predictions_lgbm = best_lgbm_model.predict(X_val_transformed)\n\nval_accuracy_lgbm = accuracy_score(y_val, val_predictions_lgbm)\n\nprint(\"Validation Accuracy:\", val_accuracy_lgbm)\n\n\n\nprint(\"\\nClassification Report (Validation Set):\")\n\nprint(classification_report(y_val, val_predictions_lgbm))\n\n\n\ncm_lgbm = confusion_matrix(y_val, val_predictions_lgbm)\n\ndisp_lgbm = ConfusionMatrixDisplay(confusion_matrix=cm_lgbm, display_labels=best_lgbm_model.classes_)\n\ndisp_lgbm.plot(cmap=\"Blues\", xticks_rotation=\"vertical\")\n\nplt.title(\"Confusion Matrix (Validation Set)\")\n\nplt.show()\n","metadata":{"id":"x5M49qI3YHag","executionInfo":{"status":"ok","timestamp":1733521144464,"user_tz":-180,"elapsed":575835,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}},"outputId":"7d349383-58dd-4247-9704-28e4ab985127"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Ensemble Method: Voting Classifier","metadata":{"id":"5eKeXfw9jAWD"}},{"cell_type":"markdown","source":"We will build a voting classifier using our top-performing models as voters. Random Forest is excluded due to its tendency to overfit, with perfect training accuracy but lower validation accuracy and higher misclassification rates. SGD OvO is also excluded as it performed reasonably well but had lower overall accuracy and significant misclassifications for certain digits, making it less robust than SVC, XGBoost, and LightGBM.","metadata":{"id":"RZA8ZVh7r0AF"}},{"cell_type":"code","source":"from sklearn.ensemble import VotingClassifier\n\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix, ConfusionMatrixDisplay\n\nimport matplotlib.pyplot as plt\n\n\n\nvoters = [\n\n    ('svc', best_model),\n\n    ('xgboost', best_xgb_model),\n\n    ('lightgbm', best_lgbm_model)\n\n]\n\n\n\n\n\nvoting_clf = VotingClassifier(estimators=voters, voting='hard')\n\n\n\nvoting_clf.fit(X_train_transformed, y_train)\n\n\n\ntrain_predictions = voting_clf.predict(X_train_transformed)\n\ntrain_accuracy = accuracy_score(y_train, train_predictions)\n\nprint(\"\\nTraining Accuracy:\", train_accuracy)\n\n\n\nval_predictions = voting_clf.predict(X_val_transformed)\n\nval_accuracy = accuracy_score(y_val, val_predictions)\n\nprint(\"Validation Accuracy:\", val_accuracy)\n\n\n\nprint(\"\\nClassification Report (Validation Set):\")\n\nprint(classification_report(y_val, val_predictions))\n\n\n\ncm = confusion_matrix(y_val, val_predictions)\n\ndisp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=voting_clf.classes_)\n\ndisp.plot(cmap=\"Blues\", xticks_rotation=\"vertical\")\n\nplt.title(\"Confusion Matrix (Validation Set)\")\n\nplt.show()\n","metadata":{"id":"HejGNn0WqFzj","executionInfo":{"status":"ok","timestamp":1733525975083,"user_tz":-180,"elapsed":727613,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}},"outputId":"01c033b2-c2f6-4f0f-c9fc-48033f3dee96"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now, we will try the voting classifier using the soft voting method.","metadata":{"id":"2WijIxc4YEVC"}},{"cell_type":"code","source":"from sklearn.svm import SVC\n\n\n\nsvc_params = best_model.get_params()\n\nsvc_params['probability'] = True\n\nsvc_with_proba = SVC(**svc_params)\n\n\n\nvoters = [\n\n    ('svc', svc_with_proba),\n\n    ('xgboost', best_xgb_model),\n\n    ('lightgbm', best_lgbm_model)\n\n]\n\n\n\nvoting_clf_soft = VotingClassifier(estimators=voters, voting='soft')\n\n\n\nvoting_clf_soft.fit(X_train_transformed, y_train)\n\n\n\ntrain_predictions = voting_clf_soft.predict(X_train_transformed)\n\ntrain_accuracy = accuracy_score(y_train, train_predictions)\n\nprint(\"\\nTraining Accuracy (Soft Voting):\", train_accuracy)\n\n\n\nval_predictions = voting_clf_soft.predict(X_val_transformed)\n\nval_accuracy = accuracy_score(y_val, val_predictions)\n\nprint(\"Validation Accuracy (Soft Voting):\", val_accuracy)\n\n\n","metadata":{"id":"TXpCPSlhPqez","executionInfo":{"status":"ok","timestamp":1733528606741,"user_tz":-180,"elapsed":1413824,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"}},"outputId":"79aaaa33-920f-4e32-b61a-56c5e62f47d2"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As observed, even though we used a voting classifier with our top three models, it does not outperform the SVC model. Therefore, our best model for this problem is SVC.\n","metadata":{"id":"jm6-JPPdJ4t_"}},{"cell_type":"markdown","source":"# Solution Summary","metadata":{"id":"Bp1W22JaptXT"}},{"cell_type":"markdown","source":"## Non Technical Summary\n\n\n\nWe’re building a machine learning model to solve classification tasks, like recognizing patterns or making better data-driven decisions. The goal is to create a reliable model that works well during training and can also handle unseen data effectively, making it practical for real-world use.\n\n\n\nWe tested various algorithms, including SVC, Random Forest, XGBoost, and LightGBM, and explored combining these models to get the best results. By fine-tuning their settings and validating their performance using metrics like accuracy and confusion matrices, we ensured the models weren’t overfitting or underperforming on new data.\n\n\n\nOur best model so far is the SVC, which achieved an exceptional 99.999% accuracy on the training dataset and maintained a high 97.9% accuracy on the validation dataset. This consistency shows the model’s ability to generalize well to unseen data, making it a robust and dependable solution for practical applications.","metadata":{"id":"MKe4uff057t8"}},{"cell_type":"markdown","source":"## Technical Summary  \n\n\n\nThis project focuses on developing a robust machine learning classification model to handle a challenging multi-class classification task. The approach involved systematic experimentation, hyperparameter tuning, and model evaluation to ensure high accuracy, generalization, and reliability.  \n\n\n\n### Data Preparation  \n\n- **Dataset Processing**: The dataset underwent preprocessing to remove unnecessary features, handle missing values, and standardize the data for optimal model performance. A smaller subset of the training data was utilized for quick prototyping and hyperparameter tuning.  \n\n- **Data Splitting**: The dataset was divided into training and validation sets to evaluate the model's generalization on unseen data effectively.  \n\n\n\n### Model Development  \n\n- **Algorithms Evaluated**: Multiple machine learning models were tested, including:  \n\n  - Logistic Regression  \n\n  - Decision Tree  \n\n  - SGD OvR (One-vs-Rest strategy with Stochastic Gradient Descent)  \n\n  - SGD OvO (One-vs-One strategy with Stochastic Gradient Descent)  \n\n  - Support Vector Classifier (SVC)  \n\n  - Random Forest Classifier  \n\n  - XGBoost Classifier  \n\n  - LightGBM Classifier  \n\n- **Ensemble Methods**:  \n\n  - **Voting Classifier**: Combined predictions using both hard voting and soft voting to leverage the strengths of individual models.  \n\n\n\n### Model Tuning and Evaluation  \n\n- **Hyperparameter Tuning**: Extensive grid search was conducted for each algorithm, including default values in the search space, to optimize their hyperparameters.  \n\n- **Metrics Used**: Accuracy and confusion matrices were employed to measure model performance, focusing on both overall accuracy and class-specific errors.  \n\n- **Validation Performance**: SVC emerged as the top performer, achieving the highest validation accuracy of **97.9%**, indicating excellent generalization on unseen data.  \n\n\n\n### Discussion and Next Steps  \n\nThe confusion matrices revealed areas where misclassifications occurred, offering insights for further improvement. A strategic focus on refining ensemble techniques, such as weighted voting or enhanced stacking methods, could further boost performance. Future iterations could explore advanced ensemble tuning or investigate data augmentation to address specific class imbalances.  \n\n\n\nThis systematic approach highlights the potential of machine learning for creating reliable and accurate classification models applicable to real-world tasks.  \n","metadata":{"id":"zyG6MAHJ6VpH"}},{"cell_type":"markdown","source":"# Testing on the Test Set","metadata":{"id":"RP51Wzrg38PB"}},{"cell_type":"markdown","source":"Now, we will test the code in the test set","metadata":{"id":"Ahq0lh2k7Xg_"}},{"cell_type":"code","source":"test_predictions = best_model.predict(test_features_df)\n\n\n\nsubmission = pd.DataFrame({\n\n    \"ImageId\": range(1, len(test_predictions) + 1),\n\n    \"Label\": test_predictions\n\n})\n\n\n\nsubmission.to_csv(\"submission.csv\", index=False)\n\nprint(\"Test predictions saved to 'submission.csv'.\")","metadata":{"executionInfo":{"elapsed":200191,"status":"ok","timestamp":1733529509387,"user":{"displayName":"Axel Dawne","userId":"13665121235676681634"},"user_tz":-180},"id":"MRaOaDF87Zu_","outputId":"d727b782-1ec7-482f-e8b4-d5ffbedd5615"},"outputs":[],"execution_count":null}]}