{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-12-06T14:45:36.263603Z","iopub.execute_input":"2024-12-06T14:45:36.264066Z","iopub.status.idle":"2024-12-06T14:45:36.273562Z","shell.execute_reply.started":"2024-12-06T14:45:36.264027Z","shell.execute_reply":"2024-12-06T14:45:36.272401Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 1. Frame the Problem\n","metadata":{}},{"cell_type":"markdown","source":"**1. What is the objective?**\n\nThe goal is to build a model that can correctly recognize handwritten digits (0–9) from grayscale images, each represented as pixel values. This is a classification problem, where each image corresponds to one of ten possible labels (0–9).\n\n**2. How will the solution be used?**\n\nThe solution can be applied in real-world use cases, such as handwritten digit recognition (e.g., exam correction, bank check processing).\n\n**3. What are the performance measures?**\n\nAccuracy: The proportion of correctly classified images. This will be the primary metric.\n\n**4. What type of problem is it?**\n\nSupervised Learning: The dataset contains labeled examples (each image has a corresponding digit label).\nMulti-class Classification: There are 10 possible categories (digits 0–9).\n\n**5. What are the inputs and outputs?**\n\nInput: 28x28 grayscale images (total 784 pixel values per image, flattened into a 1D array).\nOutput: A digit label (0, 1, ..., 9) indicating the recognized digit.\n\n**6. What would be the minimum performance needed to reach the business objective?**\n\nAny random guessing would have a 10% accuracy. We should aim for higher. \n\n\n**7. What are the assumptions?**\n\nThe dataset contains no noise or mislabels.\nNo explicit need for handling class imbalance (digits are assumed to be equally distributed).\n\n","metadata":{}},{"cell_type":"markdown","source":"# 2. Get the Data","metadata":{}},{"cell_type":"code","source":"%%bash\n\ncat /kaggle/input/digit-recognizer/train.csv | head -n 5","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:38.328448Z","iopub.execute_input":"2024-12-06T14:45:38.328841Z","iopub.status.idle":"2024-12-06T14:45:38.346829Z","shell.execute_reply.started":"2024-12-06T14:45:38.328806Z","shell.execute_reply":"2024-12-06T14:45:38.345834Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df = pd.read_csv(\n    \"/kaggle/input/digit-recognizer/train.csv\",\n)\n\ntest_features_df = pd.read_csv(\n    \"/kaggle/input/digit-recognizer/test.csv\",\n)\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:38.797998Z","iopub.execute_input":"2024-12-06T14:45:38.798398Z","iopub.status.idle":"2024-12-06T14:45:44.046590Z","shell.execute_reply.started":"2024-12-06T14:45:38.798367Z","shell.execute_reply":"2024-12-06T14:45:44.045069Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:44.048558Z","iopub.execute_input":"2024-12-06T14:45:44.048879Z","iopub.status.idle":"2024-12-06T14:45:44.102304Z","shell.execute_reply.started":"2024-12-06T14:45:44.048846Z","shell.execute_reply":"2024-12-06T14:45:44.101189Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:44.104092Z","iopub.execute_input":"2024-12-06T14:45:44.104539Z","iopub.status.idle":"2024-12-06T14:45:44.126534Z","shell.execute_reply.started":"2024-12-06T14:45:44.104502Z","shell.execute_reply":"2024-12-06T14:45:44.125357Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Explore the Data","metadata":{}},{"cell_type":"code","source":"## Plot sample digit images. I want to check a sample and see if the data is correctly labled. Indeed it is. This code with prompted with \n#\"Visualize a few sample images to ensure the data is correctly labled.\"\n\nimport matplotlib.pyplot as plt\n\n\ntrain_features_df = train_df.drop(\"label\", axis=1)\ntrain_target = train_df.loc[:, \"label\"]\nfig, axes = plt.subplots(2, 5, figsize=(10, 5))\nfor i, ax in enumerate(axes.flat):\n    # Extract pixel values only (drop the label column)\n    image = train_features_df.iloc[i].values.reshape(28, 28)\n    \n    ax.imshow(image, cmap='gray')\n    ax.set_title(f\"Label: {train_target.iloc[i]}\")\n    ax.axis('off')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:44.129495Z","iopub.execute_input":"2024-12-06T14:45:44.129991Z","iopub.status.idle":"2024-12-06T14:45:44.818339Z","shell.execute_reply.started":"2024-12-06T14:45:44.129942Z","shell.execute_reply":"2024-12-06T14:45:44.817292Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#see how the data is distributed to find if there are any class imbalanced. \n#Prompt: \"See how the labels (digits) are distributed in the training dataset. \"\n\nimport seaborn as sns\n\n# Plot the distribution of labels\nplt.figure(figsize=(10, 5))\nsns.countplot(x=train_target, palette='viridis')\nplt.title('Distribution of Digit Labels')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:44.819748Z","iopub.execute_input":"2024-12-06T14:45:44.820161Z","iopub.status.idle":"2024-12-06T14:45:46.755503Z","shell.execute_reply.started":"2024-12-06T14:45:44.820128Z","shell.execute_reply":"2024-12-06T14:45:46.754215Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Visualize the pixel intensity (0-255) distribution for a specific digit to understand how bright/dark the images are.\nplt.figure(figsize=(10, 5))\nsample_image = train_features_df.iloc[0].values  # Example: first digit\nplt.hist(sample_image, bins=30, color='purple')\nplt.title('Pixel Intensity Distribution for a Sample Digit')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:46.757009Z","iopub.execute_input":"2024-12-06T14:45:46.757491Z","iopub.status.idle":"2024-12-06T14:45:47.053903Z","shell.execute_reply.started":"2024-12-06T14:45:46.757448Z","shell.execute_reply":"2024-12-06T14:45:47.052313Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#plot pexil intisity as a heatmap\nimport numpy as np\n\n# Compute mean image for each digit\nmean_images = train_features_df.groupby(train_target).mean()\n\n# Plot mean images\nfig, axes = plt.subplots(2, 5, figsize=(10, 5))\nfor i, ax in enumerate(axes.flat):\n    ax.imshow(mean_images.iloc[i].values.reshape(28, 28), cmap='hot')\n    ax.set_title(f\"Digit: {i}\")\n    ax.axis('off')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:47.055276Z","iopub.execute_input":"2024-12-06T14:45:47.055596Z","iopub.status.idle":"2024-12-06T14:45:47.915949Z","shell.execute_reply.started":"2024-12-06T14:45:47.055564Z","shell.execute_reply":"2024-12-06T14:45:47.914859Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# 4. Prepare the Data for ML ","metadata":{}},{"cell_type":"code","source":"train_features_df = train_df.drop(\"label\", axis=1)\ntrain_target = train_df.loc[:, \"label\"]","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:47.992420Z","iopub.execute_input":"2024-12-06T14:45:47.993433Z","iopub.status.idle":"2024-12-06T14:45:48.081245Z","shell.execute_reply.started":"2024-12-06T14:45:47.993376Z","shell.execute_reply":"2024-12-06T14:45:48.080119Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Split data into training set and validation set.\n\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\n# Split into training and validation sets\nX_train, X_val, y_train, y_val = train_test_split(\n    train_features_df, train_target, test_size=0.2, random_state=42\n)\n\nprint(X_train.shape, X_val.shape)  # Confirm shapes\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:48.399071Z","iopub.execute_input":"2024-12-06T14:45:48.400189Z","iopub.status.idle":"2024-12-06T14:45:48.930495Z","shell.execute_reply.started":"2024-12-06T14:45:48.400150Z","shell.execute_reply":"2024-12-06T14:45:48.929268Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# The dataset has no missing values\n# The data set probably needs to be scaled. So just one trasnformer. \n\n#prompt: scale the training and Val sets.\nfrom sklearn.preprocessing import StandardScaler\n\n# Initialize the scaler\nscaler = StandardScaler()\n\n# Fit only on the training data, to avoid data leakage\nX_train = scaler.fit_transform(X_train)\nX_val = scaler.transform(X_val)\ntest_features_df = scaler.transform(test_features_df)\n\nprint(X_train[:5])  # Check the first few normalized samples\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:48.932639Z","iopub.execute_input":"2024-12-06T14:45:48.933110Z","iopub.status.idle":"2024-12-06T14:45:49.552388Z","shell.execute_reply.started":"2024-12-06T14:45:48.933063Z","shell.execute_reply":"2024-12-06T14:45:49.551302Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Explore Different Models","metadata":{}},{"cell_type":"code","source":"#1 Dummy model\n#Prompt:  Can you try a dummy classifier\n\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.metrics import accuracy_score\n\n# Initialize the dummy classifier (strategy: 'most_frequent' predicts the most common class)\ndummy_clf = DummyClassifier(strategy=\"most_frequent\")\n\n# Train the dummy classifier on the training data\ndummy_clf.fit(X_train, y_train)\n\n# Predict on the validation set\ny_val_pred = dummy_clf.predict(X_val)\n\n# Calculate and print the accuracy\ndummy_accuracy = accuracy_score(y_val, y_val_pred)\nprint(f\"Dummy Classifier Accuracy: {dummy_accuracy * 100:.2f}%\")\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:49.950366Z","iopub.execute_input":"2024-12-06T14:45:49.950739Z","iopub.status.idle":"2024-12-06T14:45:49.967959Z","shell.execute_reply.started":"2024-12-06T14:45:49.950707Z","shell.execute_reply":"2024-12-06T14:45:49.966781Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"***So a dummy classifier as expected has an accuracy of 10%.***\n\n","metadata":{}},{"cell_type":"code","source":"#2Logistic regression model.\nfrom sklearn.linear_model import LogisticRegression\n#encode lables\ny_train = np.array(y_train) \ny_val = np.array(y_val)\n\n\n# Initialize the model\nmodel = LogisticRegression(multi_class='multinomial', solver='lbfgs', max_iter=500)\n\n# Train the model\nmodel.fit(X_train, y_train)\n\n# Evaluate on validation data\nval_accuracy = model.score(X_val, y_val)\nprint(f\"Validation Accuracy: {val_accuracy * 100:.2f}%\")\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:45:50.536524Z","iopub.execute_input":"2024-12-06T14:45:50.537671Z","iopub.status.idle":"2024-12-06T14:46:35.775235Z","shell.execute_reply.started":"2024-12-06T14:45:50.537629Z","shell.execute_reply":"2024-12-06T14:46:35.774032Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#3 SGD Classifier.\n\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n\n# Initialize the SGD Classifier\nsgd_clf = SGDClassifier(loss='log_loss', max_iter=1000, tol=1e-3, random_state=42)\n\n# Train the classifier on the training data\nsgd_clf.fit(X_train, y_train)\n\n# Predict on the validation data\ny_val_pred = sgd_clf.predict(X_val)\n\n# Calculate accuracy\nsgd_accuracy = accuracy_score(y_val, y_val_pred)\nprint(f\"SGD Classifier Accuracy: {sgd_accuracy * 100:.2f}%\")\n\n# Display classification report and confusion matrix for detailed metrics\nprint(\"\\nClassification Report:\\n\", classification_report(y_val, y_val_pred))\nprint(\"\\nConfusion Matrix:\\n\", confusion_matrix(y_val, y_val_pred))\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:46:35.777317Z","iopub.execute_input":"2024-12-06T14:46:35.777772Z","iopub.status.idle":"2024-12-06T14:53:10.743362Z","shell.execute_reply.started":"2024-12-06T14:46:35.777714Z","shell.execute_reply":"2024-12-06T14:53:10.740943Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 6. Fine-tune the Models","metadata":{}},{"cell_type":"code","source":"\n#SGD took forever, so I proceeded with Logistic regression. \n\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import GridSearchCV\n\n# Define the parameter grid for tuning\nparam_grid = {\n    'C': [0.01, 0.1, 1, 10, 100]  # Regularization strength\n    #'penalty': ['l2'],  # 'l1' works only with certain solvers\n    #'solver': ['lbfgs', 'liblinear', 'saga'],  # Different solvers for optimization\n    #'max_iter': [100, 200, 500]  # Increase iterations to ensure convergence\n}\n\n# Initialize the logistic regression model\nlog_reg = LogisticRegression(multi_class='multinomial', class_weight='balanced')\n\n# Set up GridSearchCV to search over the parameter grid with 5-fold cross-validation\ngrid_search = GridSearchCV(\n    estimator=log_reg,\n    param_grid=param_grid,\n    cv=5,  # 5-fold cross-validation\n    n_jobs=-1,  # Utilize all cores for parallel processing\n    scoring='accuracy',\n    verbose=1\n)\n\n# Train the model with GridSearchCV\ngrid_search.fit(X_train, y_train)\n\n# Best hyperparameters and corresponding accuracy\nprint(\"Best Hyperparameters:\", grid_search.best_params_)\nprint(\"Best Cross-Validation Accuracy:\", grid_search.best_score_)\n\n# Evaluate the model with the best parameters on the validation set\nbest_model = grid_search.best_estimator_\ny_val_pred = best_model.predict(X_val)\nval_accuracy = accuracy_score(y_val, y_val_pred)\n\nprint(f\"Validation Accuracy with Best Model: {val_accuracy * 100:.2f}%\")\n","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:55:05.706227Z","iopub.execute_input":"2024-12-06T14:55:05.706704Z","iopub.status.idle":"2024-12-06T14:57:27.762388Z","shell.execute_reply.started":"2024-12-06T14:55:05.706661Z","shell.execute_reply":"2024-12-06T14:57:27.759309Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(f\"Validation Accuracy with Best Model: {val_accuracy * 100:.2f}%\")","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:57:27.766596Z","iopub.execute_input":"2024-12-06T14:57:27.768526Z","iopub.status.idle":"2024-12-06T14:57:27.780928Z","shell.execute_reply.started":"2024-12-06T14:57:27.768450Z","shell.execute_reply":"2024-12-06T14:57:27.776436Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Identify the best model","metadata":{}},{"cell_type":"markdown","source":"The best mode is LogisticRegression with hyper paramerter C = 0.01. ","metadata":{}},{"cell_type":"markdown","source":"# 7. Summarize your results","metadata":{}},{"cell_type":"markdown","source":"# I evaluated based on the accuracy score and scored 92% With a logistic regression model and hyper parameter C = 0.01\n","metadata":{}},{"cell_type":"markdown","source":"# 8. Submit your predictions to the competition","metadata":{}},{"cell_type":"code","source":"# Make predictions on the test set using the trained Logistic Regression model\npredictions = best_model.predict(test_features_df)\n\n# Prepare the submission DataFrame\nsubmission = pd.DataFrame({\n    \"ImageId\": list(range(1, len(logistic_predictions) + 1)),\n    \"Label\": logistic_predictions\n})\n\n# Save the submission file as 'submission.csv'\nsubmission.to_csv(\"submission.csv\", index=False)\n\nprint(\"Submission file saved!\")\n\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#predictions = estimator.predict(test_features_df)","metadata":{"execution":{"iopub.status.busy":"2024-12-06T14:57:27.971900Z","iopub.status.idle":"2024-12-06T14:57:27.973058Z","shell.execute_reply.started":"2024-12-06T14:57:27.972607Z","shell.execute_reply":"2024-12-06T14:57:27.972653Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"predictions[:5]","metadata":{"execution":{"iopub.status.busy":"2024-12-06T15:18:39.183238Z","iopub.execute_input":"2024-12-06T15:18:39.183612Z","iopub.status.idle":"2024-12-06T15:18:39.192524Z","shell.execute_reply.started":"2024-12-06T15:18:39.183575Z","shell.execute_reply":"2024-12-06T15:18:39.191331Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%bash\n\ncat /kaggle/input/digit-recognizer/sample_submission.csv | head -n 5","metadata":{"execution":{"iopub.status.busy":"2024-12-06T15:18:43.941181Z","iopub.execute_input":"2024-12-06T15:18:43.941590Z","iopub.status.idle":"2024-12-06T15:18:43.973734Z","shell.execute_reply.started":"2024-12-06T15:18:43.941549Z","shell.execute_reply":"2024-12-06T15:18:43.972668Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submission_df = (\n    pd.read_csv(\n        \"/kaggle/input/digit-recognizer/sample_submission.csv\",\n    ).assign(\n        Label=predictions\n    )\n)\n\nsubmission_df.to_csv(\"submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2024-12-06T15:18:47.250039Z","iopub.execute_input":"2024-12-06T15:18:47.250419Z","iopub.status.idle":"2024-12-06T15:18:47.288709Z","shell.execute_reply.started":"2024-12-06T15:18:47.250390Z","shell.execute_reply":"2024-12-06T15:18:47.287645Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%bash\n\ncat /kaggle/working/submission.csv | head -n 5","metadata":{"execution":{"iopub.status.busy":"2024-12-06T15:18:52.795674Z","iopub.execute_input":"2024-12-06T15:18:52.796090Z","iopub.status.idle":"2024-12-06T15:18:52.812582Z","shell.execute_reply.started":"2024-12-06T15:18:52.796055Z","shell.execute_reply":"2024-12-06T15:18:52.811535Z"},"trusted":true},"outputs":[],"execution_count":null}]}