{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30626,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognizer Kaggle Competition: Machine Learning Notebook","metadata":{}},{"cell_type":"markdown","source":"## Overview\n\nThis Jupyter notebook is part of my participation in the Kaggle Digit Recognizer competition. The goal of the competition is to develop a machine learning model capable of accurately classifying handwritten digits (0 through 9) based on pixel values.\n\n## Contents\n\n1. **Loading Data:** The notebook begins by importing the necessary libraries and loading the training and test datasets from the provided CSV files.\n\n2. **Data Preprocessing:** The training dataset is split into training and validation sets to evaluate model performance. The features (X) and labels (y) are separated accordingly.\n\n3. **Model Training:** An initial Stochastic Gradient Descent (SGD) classifier is trained on the training set, and its accuracy is evaluated on the validation set.\n\n4. **Final Model Training:** The final model is trained on the entire training set using the chosen classifier.\n\n5. **Making Predictions:** The trained model is used to make predictions on the test set provided by Kaggle.\n\n6. **Submission File:** The predictions are formatted into a submission file (CSV) to be submitted to the Kaggle competition.\n\n## Model Details\n\nThe notebook uses the Stochastic Gradient Descent (SGD) classifier for its simplicity and efficiency in training large datasets. Model evaluation is performed using accuracy as the metric.\n\nFeel free to explore the code and comments for a detailed understanding of each step in the machine learning pipeline.\n\nHappy coding!\n","metadata":{}},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk(\"/kaggle/input\"):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-12-31T19:20:16.890660Z","iopub.execute_input":"2023-12-31T19:20:16.891174Z","iopub.status.idle":"2023-12-31T19:20:16.929152Z","shell.execute_reply.started":"2023-12-31T19:20:16.891128Z","shell.execute_reply":"2023-12-31T19:20:16.928001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install scikit-learn","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:16.931691Z","iopub.execute_input":"2023-12-31T19:20:16.932399Z","iopub.status.idle":"2023-12-31T19:20:30.521850Z","shell.execute_reply.started":"2023-12-31T19:20:16.932365Z","shell.execute_reply":"2023-12-31T19:20:30.520427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split, GridSearchCV\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.metrics import accuracy_score, confusion_matrix\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:30.523699Z","iopub.execute_input":"2023-12-31T19:20:30.524034Z","iopub.status.idle":"2023-12-31T19:20:31.869157Z","shell.execute_reply.started":"2023-12-31T19:20:30.524004Z","shell.execute_reply":"2023-12-31T19:20:31.868255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read the csv's Data with Pandas\ntrain_data = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest_data = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:31.870548Z","iopub.execute_input":"2023-12-31T19:20:31.871475Z","iopub.status.idle":"2023-12-31T19:20:37.298194Z","shell.execute_reply.started":"2023-12-31T19:20:31.871434Z","shell.execute_reply":"2023-12-31T19:20:37.297144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use part of the provided training set for training and part for validation\nX_train, X_val, y_train, y_val = train_test_split(\n    train_data.iloc[:, 1:].values,\n    train_data[\"label\"].values,\n    test_size=0.2,\n    random_state=42,\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:37.301534Z","iopub.execute_input":"2023-12-31T19:20:37.302487Z","iopub.status.idle":"2023-12-31T19:20:37.761149Z","shell.execute_reply.started":"2023-12-31T19:20:37.302441Z","shell.execute_reply":"2023-12-31T19:20:37.760098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import Normalizer\nfrom keras.preprocessing.image import ImageDataGenerator\nimport numpy as np\n\n# Normalize X_train only\nnormalizer = Normalizer()\nnormalized_pixels = normalizer.fit_transform(X_train)\n\n# Reshape\nnormalized_pixels = normalized_pixels.reshape(-1, 28, 28, 1)\n\n# Augment\ndatagen = ImageDataGenerator(\n    rotation_range=10, zoom_range=0.1, width_shift_range=0.1, height_shift_range=0.1\n)\naugmented_data = datagen.flow(normalized_pixels, y_train)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:37.762377Z","iopub.execute_input":"2023-12-31T19:20:37.763281Z","iopub.status.idle":"2023-12-31T19:20:50.996590Z","shell.execute_reply.started":"2023-12-31T19:20:37.763249Z","shell.execute_reply":"2023-12-31T19:20:50.995469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Print the first 5 rows of the training data\nprint(\"First 5 rows of the training data:\")\nprint(pd.DataFrame(X_train, columns=train_data.columns[1:]).head(5))","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:50.998186Z","iopub.execute_input":"2023-12-31T19:20:50.999012Z","iopub.status.idle":"2023-12-31T19:20:51.017153Z","shell.execute_reply.started":"2023-12-31T19:20:50.998967Z","shell.execute_reply":"2023-12-31T19:20:51.016236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check class distribution\nclass_distribution = train_data[\"label\"].value_counts()\nprint(\"Class Distribution:\")\nprint(class_distribution)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:51.018522Z","iopub.execute_input":"2023-12-31T19:20:51.019139Z","iopub.status.idle":"2023-12-31T19:20:51.046206Z","shell.execute_reply.started":"2023-12-31T19:20:51.019093Z","shell.execute_reply":"2023-12-31T19:20:51.045164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the range of pixel values\nprint(\"Minimum pixel value:\", train_data.iloc[:, 1:].values.min())\nprint(\"Maximum pixel value:\", train_data.iloc[:, 1:].values.max())","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:51.047547Z","iopub.execute_input":"2023-12-31T19:20:51.048229Z","iopub.status.idle":"2023-12-31T19:20:51.108129Z","shell.execute_reply.started":"2023-12-31T19:20:51.048194Z","shell.execute_reply":"2023-12-31T19:20:51.107166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize some sample digits from the training set\nplt.figure(figsize=(10, 3))\nfor i in range(10):\n    plt.subplot(2, 5, i + 1)\n    digit_pixels = X_train[i].reshape(28, 28)\n    plt.imshow(digit_pixels, cmap=\"hot_r\", interpolation=\"nearest\")\n    plt.title(f\"\\nLabel: {y_train[i]}\", color=\"grey\", fontsize=10)\n    plt.axis(\"off\")\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:51.109307Z","iopub.execute_input":"2023-12-31T19:20:51.109653Z","iopub.status.idle":"2023-12-31T19:20:51.776311Z","shell.execute_reply.started":"2023-12-31T19:20:51.109623Z","shell.execute_reply":"2023-12-31T19:20:51.775106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the distribution of digit labels in the training set\nplt.figure(figsize=(8, 4))\nsns.countplot(x=\"label\", data=pd.DataFrame({\"label\": y_train}), palette=\"viridis\")\nplt.title(\"\\nDistribution of Digit Labels in Training Set\\n\", color=\"grey\", size=12)\nplt.xlabel(\"\\nDigit Label\\n\", color=\"grey\", size=10)\nplt.ylabel(\"\\nCount\\n\", color=\"grey\", size=10)\n\n# Change the color of the x and y ticks\nplt.tick_params(axis=\"x\", colors=\"grey\")\nplt.tick_params(axis=\"y\", colors=\"grey\")\n\n# Erase ticks from the left and bottom spines\nplt.tick_params(axis=\"both\", which=\"both\", length=0)\n\n# Erase top and right spines\nplt.gca().spines[\"top\"].set_visible(False)\nplt.gca().spines[\"right\"].set_visible(False)\n\n# Change the color of the bottom and left spines\nplt.gca().spines[\"bottom\"].set_color(\"grey\")\nplt.gca().spines[\"left\"].set_color(\"grey\")\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:51.777571Z","iopub.execute_input":"2023-12-31T19:20:51.777922Z","iopub.status.idle":"2023-12-31T19:20:52.070028Z","shell.execute_reply.started":"2023-12-31T19:20:51.777891Z","shell.execute_reply":"2023-12-31T19:20:52.069198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract augmented data\nX_augmented, y_augmented = augmented_data.next()\n\n# Combine normalized and augmented data\nX_train_combined = np.concatenate((normalized_pixels, X_augmented), axis=0)\ny_train_combined = np.concatenate((y_train, y_augmented), axis=0)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:52.071242Z","iopub.execute_input":"2023-12-31T19:20:52.071848Z","iopub.status.idle":"2023-12-31T19:20:52.204132Z","shell.execute_reply.started":"2023-12-31T19:20:52.071810Z","shell.execute_reply":"2023-12-31T19:20:52.203289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create and train an SGDClassifier\nsgd_clf = SGDClassifier(random_state=42)\nsgd_clf.fit(X_train_combined.reshape(X_train_combined.shape[0], -1), y_train_combined)\n\n# Validate the model on the validation set\nnormalized_pixels_val = normalizer.transform(X_val)\nnormalized_pixels_val = normalized_pixels_val.reshape(-1, 28, 28, 1)\ny_val_pred = sgd_clf.predict(normalized_pixels_val.reshape(X_val.shape[0], -1))\naccuracy = accuracy_score(y_val, y_val_pred)\nprint(f\"Model Validation Accuracy: {accuracy:.4f}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:52.205426Z","iopub.execute_input":"2023-12-31T19:20:52.206312Z","iopub.status.idle":"2023-12-31T19:20:57.286821Z","shell.execute_reply.started":"2023-12-31T19:20:52.206274Z","shell.execute_reply":"2023-12-31T19:20:57.285632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.ensemble import RandomForestClassifier\n\n# Example with Random Forest\nparam_dist = {\n    \"n_estimators\": [50, 100, 200],\n    \"max_depth\": [None, 10, 20],\n    \"min_samples_split\": [2, 5, 10],\n}\n\nrf_model = RandomForestClassifier()\nrandom_search = RandomizedSearchCV(\n    rf_model,\n    param_distributions=param_dist,\n    n_iter=10,\n    cv=5,\n    scoring=\"accuracy\",\n    random_state=42,\n)\nrandom_search.fit(X_train, y_train)\n\nprint(f\"Best Parameters: {random_search.best_params_}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:20:57.292175Z","iopub.execute_input":"2023-12-31T19:20:57.293491Z","iopub.status.idle":"2023-12-31T19:35:57.607184Z","shell.execute_reply.started":"2023-12-31T19:20:57.293430Z","shell.execute_reply":"2023-12-31T19:35:57.605834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\n\n# Example with Stratified k-Fold\nstratified_kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nfor train_index, test_index in stratified_kfold.split(X_train, y_train):\n    X_train_fold, X_test_fold = X_train[train_index], X_train[test_index]\n    y_train_fold, y_test_fold = y_train[train_index], y_train[test_index]","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:35:57.608793Z","iopub.execute_input":"2023-12-31T19:35:57.609295Z","iopub.status.idle":"2023-12-31T19:35:57.999627Z","shell.execute_reply.started":"2023-12-31T19:35:57.609250Z","shell.execute_reply":"2023-12-31T19:35:57.998243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a Random Forest model with the best parameters\nbest_rf_model = RandomForestClassifier(\n    n_estimators=200, min_samples_split=10, max_depth=None\n)\n\n# Train the model on the entire training set\nbest_rf_model.fit(X_train, y_train)\n\n# Validate the model on the validation set\ny_val_pred = best_rf_model.predict(X_val)\naccuracy = accuracy_score(y_val, y_val_pred)\nprint(f\"Validation Accuracy with Best Hyperparameters: {accuracy:.4f}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:35:58.001350Z","iopub.execute_input":"2023-12-31T19:35:58.001864Z","iopub.status.idle":"2023-12-31T19:36:48.970191Z","shell.execute_reply.started":"2023-12-31T19:35:58.001819Z","shell.execute_reply":"2023-12-31T19:36:48.968980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.svm import SVC\nfrom xgboost import XGBClassifier\nfrom sklearn.model_selection import cross_val_score\n\n# Example with Support Vector Machine\nsvm_model = SVC()\nsvm_scores = cross_val_score(svm_model, X_train, y_train, cv=5, scoring=\"accuracy\")\nprint(f\"SVM Cross-Validation Scores: {svm_scores}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:36:48.972064Z","iopub.execute_input":"2023-12-31T19:36:48.972546Z","iopub.status.idle":"2023-12-31T19:46:45.180031Z","shell.execute_reply.started":"2023-12-31T19:36:48.972503Z","shell.execute_reply":"2023-12-31T19:46:45.178847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mean_accuracy = np.mean(svm_scores)\nstd_accuracy = np.std(svm_scores)\n\nprint(f\"Mean Accuracy: {mean_accuracy:.4f}\")\nprint(f\"Standard Deviation: {std_accuracy:.4f}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:46:45.181709Z","iopub.execute_input":"2023-12-31T19:46:45.182159Z","iopub.status.idle":"2023-12-31T19:46:45.188561Z","shell.execute_reply.started":"2023-12-31T19:46:45.182097Z","shell.execute_reply":"2023-12-31T19:46:45.187403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import learning_curve\n\ndef plot_learning_curve(estimator, X, y, title=None, ylim=None, cv=None,\n                        n_jobs=None, train_sizes=np.linspace(0.1, 1.0, 5)):\n    plt.figure(figsize=(8, 6))\n    plt.title(title, color=\"grey\", size=12)\n    if ylim is not None:\n        plt.ylim(*ylim)\n    plt.xlabel(\"\\nTraining examples\\n\", color=\"grey\")\n    plt.ylabel(\"\\nScore\\n\", color=\"grey\")\n\n    train_sizes, train_scores, test_scores = learning_curve(\n        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)\n\n    train_scores_mean = np.mean(train_scores, axis=1)\n    test_scores_mean = np.mean(test_scores, axis=1)\n    train_scores_std = np.std(train_scores, axis=1)\n    test_scores_std = np.std(test_scores, axis=1)\n    \n    # Change the color of the x and y ticks\n    plt.tick_params(axis=\"x\", colors=\"grey\")\n    plt.tick_params(axis=\"y\", colors=\"grey\")\n    \n    # Erase ticks from the left and bottom spines\n    plt.tick_params(axis=\"both\", which=\"both\", length=0)\n    \n    # Change the color of the spines\n    for spine in [\"top\", \"right\", \"bottom\", \"left\"]:\n        plt.gca().spines[spine].set_color(\"grey\")\n    \n    plt.grid(alpha=0.5)\n\n    plt.plot(train_sizes, train_scores_mean, 'o-', color=\"#117a65\", label=\"Training score\")\n    plt.plot(train_sizes, test_scores_mean, 'o-', color=\"#6d5eac\", label=\"Cross-validation score\")\n\n    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n                     train_scores_mean + train_scores_std, alpha=0.2, color=\"#117a65\")\n    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n                     test_scores_mean + test_scores_std, alpha=0.2, color=\"#6d5eac\")\n\n    plt.legend(loc=\"best\")\n\n    # Print numeric scores (rounded to 2 decimal places)\n    last_train_score = round(train_scores_mean[-1], 2)\n    last_test_score = round(test_scores_mean[-1], 2)\n    plt.text(train_sizes[-1], last_train_score, f'{last_train_score}', color=\"#117a65\", fontsize=10, va='center')\n    plt.text(train_sizes[-1], last_test_score, f'{last_test_score}', color=\"#6d5eac\", fontsize=10, va='center')\n\n    return plt\n","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:46:45.190064Z","iopub.execute_input":"2023-12-31T19:46:45.190425Z","iopub.status.idle":"2023-12-31T19:46:45.204085Z","shell.execute_reply.started":"2023-12-31T19:46:45.190397Z","shell.execute_reply":"2023-12-31T19:46:45.202900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Call the function to plot the learning curve for the SVM model\nplot_learning_curve(svm_model, X_train, y_train, title=\"\\nLearning Curve\\n\", cv=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-31T19:46:45.205641Z","iopub.execute_input":"2023-12-31T19:46:45.206103Z","iopub.status.idle":"2023-12-31T20:44:43.526862Z","shell.execute_reply.started":"2023-12-31T19:46:45.206062Z","shell.execute_reply":"2023-12-31T20:44:43.525564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_model = SVC()\nfinal_model.fit(train_data.iloc[:, 1:].values, train_data[\"label\"].values)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T20:44:43.528705Z","iopub.execute_input":"2023-12-31T20:44:43.529427Z","iopub.status.idle":"2023-12-31T20:47:17.509834Z","shell.execute_reply.started":"2023-12-31T20:44:43.529391Z","shell.execute_reply":"2023-12-31T20:47:17.508990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_competition_test = test_data.values\ncompetition_test_predictions = final_model.predict(X_competition_test)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T20:47:17.511616Z","iopub.execute_input":"2023-12-31T20:47:17.512496Z","iopub.status.idle":"2023-12-31T20:51:28.998455Z","shell.execute_reply.started":"2023-12-31T20:47:17.512452Z","shell.execute_reply":"2023-12-31T20:51:28.997237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a submission file\nsubmission_df = pd.DataFrame(\n    {\n        \"ImageId\": range(1, len(competition_test_predictions) + 1),\n        \"Label\": competition_test_predictions,\n    }\n)\nsubmission_df.to_csv(\"submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T20:51:28.999902Z","iopub.execute_input":"2023-12-31T20:51:29.000291Z","iopub.status.idle":"2023-12-31T20:51:29.044013Z","shell.execute_reply.started":"2023-12-31T20:51:29.000257Z","shell.execute_reply":"2023-12-31T20:51:29.043188Z"},"trusted":true},"execution_count":null,"outputs":[]}]}