{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-11-12T16:39:11.191506Z","iopub.execute_input":"2024-11-12T16:39:11.192202Z","iopub.status.idle":"2024-11-12T16:39:11.624424Z","shell.execute_reply.started":"2024-11-12T16:39:11.192127Z","shell.execute_reply":"2024-11-12T16:39:11.623192Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 1. Frame the Problem\n","metadata":{}},{"cell_type":"markdown","source":"We are given a data set of pixels as features and each row corresponds to one number. The goal in this project is to use the pixel intensity to predict what number this row corresponds to. The input data used are handwritten digits and the expected output is what digit they represent. This is a supervised multiclass classification problem where the goal is to predict the correct digit label for each image.The models that will be used in the prediction are a logistic regression and stochastic gradient descent and will be graded using accuracy as the performance metric.","metadata":{}},{"cell_type":"markdown","source":"# 2. Get the Data","metadata":{}},{"cell_type":"code","source":"%%bash\n\ncat /kaggle/input/digit-recognizer/train.csv | head -n 5","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:12.739780Z","iopub.execute_input":"2024-11-12T16:39:12.740354Z","iopub.status.idle":"2024-11-12T16:39:12.783246Z","shell.execute_reply.started":"2024-11-12T16:39:12.740298Z","shell.execute_reply":"2024-11-12T16:39:12.781909Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df = pd.read_csv(\n    \"/kaggle/input/digit-recognizer/train.csv\",\n)\n\ntest_features_df = pd.read_csv(\n    \"/kaggle/input/digit-recognizer/test.csv\",\n)\n","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:13.067609Z","iopub.execute_input":"2024-11-12T16:39:13.068622Z","iopub.status.idle":"2024-11-12T16:39:18.656914Z","shell.execute_reply.started":"2024-11-12T16:39:13.068553Z","shell.execute_reply":"2024-11-12T16:39:18.655734Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#check the entries of the dataset\ntrain_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:18.658756Z","iopub.execute_input":"2024-11-12T16:39:18.659163Z","iopub.status.idle":"2024-11-12T16:39:18.723480Z","shell.execute_reply.started":"2024-11-12T16:39:18.659121Z","shell.execute_reply":"2024-11-12T16:39:18.722234Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The dataset contains grayscale images of handwritten digits, each represented as a 28x28 pixel image. The data provided in the dataset is all numeric with no apparent missing entries. Each image in the training set is labeled with a digit from 0 to 9. The training set provides labeled examples to train the model. From the data info summary we can infer that there are no categorical features.","metadata":{}},{"cell_type":"markdown","source":"# 3. Explore the Data","metadata":{}},{"cell_type":"code","source":"#get a feel for the organization of the data\ndisplay(train_df.head())\ndisplay(train_df.tail())","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:18.724696Z","iopub.execute_input":"2024-11-12T16:39:18.725181Z","iopub.status.idle":"2024-11-12T16:39:18.764709Z","shell.execute_reply.started":"2024-11-12T16:39:18.725140Z","shell.execute_reply":"2024-11-12T16:39:18.763426Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#shows some useful statistics about the dataset\ntrain_df.describe()","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:18.767574Z","iopub.execute_input":"2024-11-12T16:39:18.768095Z","iopub.status.idle":"2024-11-12T16:39:20.929637Z","shell.execute_reply.started":"2024-11-12T16:39:18.768039Z","shell.execute_reply":"2024-11-12T16:39:20.928378Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"A lot of the features are redundant and hold the same data as others. There are mulyiple columns where all entries are 0, therefore will be dropped in the pipeline to increase model speed.","metadata":{}},{"cell_type":"code","source":"#plot the number of times each number appears in the dataset\nimport matplotlib.pyplot as plt\n\ntrain_target = train_df['label']\n\nlabel_instance = (train_target.value_counts().sort_index().plot(kind=\"bar\"))\nlabel_instance = plt.xticks(rotation=-45)\nlabel_instance = plt.ylim(0, 5000)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:20.930970Z","iopub.execute_input":"2024-11-12T16:39:20.931313Z","iopub.status.idle":"2024-11-12T16:39:21.308688Z","shell.execute_reply.started":"2024-11-12T16:39:20.931278Z","shell.execute_reply":"2024-11-12T16:39:21.307199Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Prepare the Data for ML ","metadata":{}},{"cell_type":"code","source":"#create a dataframe for features and one for labels\ntrain_features_df = train_df.drop(\"label\", axis=1)\ntrain_target = train_df.loc[:, \"label\"]\n","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:21.310387Z","iopub.execute_input":"2024-11-12T16:39:21.310849Z","iopub.status.idle":"2024-11-12T16:39:21.404195Z","shell.execute_reply.started":"2024-11-12T16:39:21.310793Z","shell.execute_reply":"2024-11-12T16:39:21.402503Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n# For a 8% sample with stratification to decrease data size and increase speed\ntrain_features_sampled, _, train_target_sampled, _ = train_test_split(\n    train_features_df, train_target, test_size=0.8, stratify=train_target, random_state=42\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-12T18:18:08.220984Z","iopub.execute_input":"2024-11-12T18:18:08.221501Z","iopub.status.idle":"2024-11-12T18:18:09.074577Z","shell.execute_reply.started":"2024-11-12T18:18:08.221455Z","shell.execute_reply":"2024-11-12T18:18:09.073345Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Shows that stratification kept similar distributions\nprint(train_target.describe())\nprint(train_target_sampled.describe())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-12T18:18:09.076501Z","iopub.execute_input":"2024-11-12T18:18:09.076914Z","iopub.status.idle":"2024-11-12T18:18:09.092156Z","shell.execute_reply.started":"2024-11-12T18:18:09.076871Z","shell.execute_reply":"2024-11-12T18:18:09.090550Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Logistic Regression**","metadata":{}},{"cell_type":"code","source":"from sklearn import linear_model, feature_selection, metrics\nfrom sklearn import model_selection, pipeline, preprocessing, impute\n\n#Create a pipeline for logistic regression which incorporates a standard scaler, drops every \n#feature that has all 0 values, imputes missing values with mean, and applies the model\nLR_pipeline = pipeline.make_pipeline(\n    impute.SimpleImputer(strategy='mean'), \n    feature_selection.VarianceThreshold(threshold=0.0),\n    preprocessing.StandardScaler(),\n    linear_model.LogisticRegression(max_iter=1000, n_jobs=-1),\n    verbose=True\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:18:11.010369Z","iopub.execute_input":"2024-11-12T18:18:11.010807Z","iopub.status.idle":"2024-11-12T18:18:11.017734Z","shell.execute_reply.started":"2024-11-12T18:18:11.010763Z","shell.execute_reply":"2024-11-12T18:18:11.016241Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Stochastic Gradient Descent**","metadata":{}},{"cell_type":"code","source":"#Create a pipeline for stochastic grradient descent which incorporates a standard scaler, drops\n#every feature that has all 0 values, imputes missing values with mean,and applies the model\nSGD_pipeline = pipeline.make_pipeline(\n    impute.SimpleImputer(strategy='mean'),\n    feature_selection.VarianceThreshold(threshold=0.0),\n    preprocessing.StandardScaler(),\n    linear_model.SGDClassifier(\n        max_iter=10000, n_jobs=-1\n    ),\n    verbose=True\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:18:13.143782Z","iopub.execute_input":"2024-11-12T18:18:13.144277Z","iopub.status.idle":"2024-11-12T18:18:13.150661Z","shell.execute_reply.started":"2024-11-12T18:18:13.144234Z","shell.execute_reply":"2024-11-12T18:18:13.149432Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Explore Different Models","metadata":{}},{"cell_type":"markdown","source":"Now we create pipelines for both models, one for logistic regression and another for stochastic gradient descent.","metadata":{}},{"cell_type":"code","source":"from sklearn import dummy\n\n\nestimator = dummy.DummyClassifier()\n_ = estimator.fit(train_features_df, train_target)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T16:39:22.592809Z","iopub.execute_input":"2024-11-12T16:39:22.593412Z","iopub.status.idle":"2024-11-12T16:39:22.609937Z","shell.execute_reply.started":"2024-11-12T16:39:22.593356Z","shell.execute_reply":"2024-11-12T16:39:22.608539Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Logistic Regression**","metadata":{}},{"cell_type":"code","source":"#Define features and target sets for logistic regression for clearness\ntrain_features_LR = train_features_sampled\ntrain_target_LR = train_target_sampled\n\n#fit the data to the pipeline\nLR_pipeline.fit(train_features_LR, train_target_LR) ","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:18:17.682669Z","iopub.execute_input":"2024-11-12T18:18:17.683115Z","iopub.status.idle":"2024-11-12T18:18:36.360480Z","shell.execute_reply.started":"2024-11-12T18:18:17.683069Z","shell.execute_reply":"2024-11-12T18:18:36.358844Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#perform predictions and show report with performance parameters\ntrain_predictions_LR = LR_pipeline.predict(train_features_LR)\nLR_report = metrics.classification_report(\n    train_target_LR,\n    train_predictions_LR,\n)\n\nprint(LR_report)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:19:27.821401Z","iopub.execute_input":"2024-11-12T18:19:27.821893Z","iopub.status.idle":"2024-11-12T18:19:28.013514Z","shell.execute_reply.started":"2024-11-12T18:19:27.821831Z","shell.execute_reply":"2024-11-12T18:19:28.012304Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Stochastic Gradient Descent**","metadata":{}},{"cell_type":"code","source":"#Define features and target sets for stochastic gradient descent for clearness\ntrain_features_SGD = train_features_sampled\ntrain_target_SGD = train_target_sampled\n\n#fit the data to the pipeline\nSGD_pipeline.fit(train_features_SGD, train_target_SGD) ","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:19:32.050938Z","iopub.execute_input":"2024-11-12T18:19:32.051381Z","iopub.status.idle":"2024-11-12T18:19:51.507320Z","shell.execute_reply.started":"2024-11-12T18:19:32.051336Z","shell.execute_reply":"2024-11-12T18:19:51.506028Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#perform predictions and show report with performance parameters\ntrain_predictions_SGD = SGD_pipeline.predict(train_features_SGD)\nSGD_report = metrics.classification_report(\n    train_target_SGD,\n    train_predictions_SGD,\n)\n\nprint(SGD_report)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:19:51.509605Z","iopub.execute_input":"2024-11-12T18:19:51.510029Z","iopub.status.idle":"2024-11-12T18:19:51.691638Z","shell.execute_reply.started":"2024-11-12T18:19:51.509987Z","shell.execute_reply":"2024-11-12T18:19:51.688400Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 6. Fine-tune the Models","metadata":{}},{"cell_type":"markdown","source":"**Logistic Regression**","metadata":{}},{"cell_type":"code","source":"#Check the general confusion matrix\n_= (\n    metrics.ConfusionMatrixDisplay\n           .from_predictions(\n                train_target_LR,\n                train_predictions_LR\n            )\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:20:29.321914Z","iopub.execute_input":"2024-11-12T18:20:29.322324Z","iopub.status.idle":"2024-11-12T18:20:29.848203Z","shell.execute_reply.started":"2024-11-12T18:20:29.322285Z","shell.execute_reply":"2024-11-12T18:20:29.847054Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#check the confusion matrix with normalization and show true labels as a percent of predictions\n_ = (\n    metrics.ConfusionMatrixDisplay\n           .from_predictions(\n                train_target_LR,\n                train_predictions_LR,\n                normalize=\"true\",\n                values_format=\".0%\"\n            )\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:20:30.914667Z","iopub.execute_input":"2024-11-12T18:20:30.915162Z","iopub.status.idle":"2024-11-12T18:20:31.496484Z","shell.execute_reply.started":"2024-11-12T18:20:30.915115Z","shell.execute_reply":"2024-11-12T18:20:31.495183Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#In the following confusion matrix there is an emphasis on incorrect prediction by setting the \n#sample weight to show incorrect values, percent of correct predictions is not accounted for.\n#Moreover, the normalization is done with respect to the predictions, so the values are divided by \n#the sum of columns.\n_ = (\n    metrics.ConfusionMatrixDisplay\n           .from_predictions(\n                train_target_LR,\n                train_predictions_LR,\n                normalize=\"pred\",\n                sample_weight=(train_target_LR != train_predictions_LR),\n                values_format=\".0%\"\n            )\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:20:31.498238Z","iopub.execute_input":"2024-11-12T18:20:31.498623Z","iopub.status.idle":"2024-11-12T18:20:32.042550Z","shell.execute_reply.started":"2024-11-12T18:20:31.498583Z","shell.execute_reply":"2024-11-12T18:20:32.041273Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(LR_pipeline.steps)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:20:32.105798Z","iopub.execute_input":"2024-11-12T18:20:32.106943Z","iopub.status.idle":"2024-11-12T18:20:32.114334Z","shell.execute_reply.started":"2024-11-12T18:20:32.106891Z","shell.execute_reply":"2024-11-12T18:20:32.112867Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Import grid search and randomized search\nfrom sklearn.model_selection import GridSearchCV,RandomizedSearchCV\n\n#Set up a hyperparameter grid\nparam_grid_LR = {\n    'logisticregression__C': [0.01, 0.1, 1, 10, 100],  #Strength of regularization\n    'logisticregression__penalty': ['l2'],             #Type of penalty\n    'logisticregression__solver': ['lbfgs'],           #Algorithm for solving\n    'logisticregression__max_iter': [1000, 10000],     # Lower iterations for quicker tuning\n}\n\n# Set up RandomizedSearchCV\ngrid_search_LR = GridSearchCV(\n    param_grid=param_grid_LR,\n    estimator=LR_pipeline, \n    cv=5,                                 # 5-fold cross-validation\n    verbose=10,                           # Verbose output for progress\n    n_jobs=-1                              # Use all available cores\n)\n\n# Fit the model\ngrid_search_LR.fit(train_features_LR, train_target_LR)\n\n# Get the best parameters\nprint(\"Best Parameters:\", grid_search_LR.best_params_)\nprint(\"Best Cross-validation Score:\", grid_search_LR.best_score_)\n\n# Get the best estimator (the model with the best parameters)\nbest_model_LR = grid_search_LR.best_estimator_\n\n# Show accuracy score for best estimator\nbest_accuracy_LR = best_model_LR.score(train_features_LR, train_target_LR)\nprint(\"Test Set Accuracy of Best Model:\", best_accuracy_LR)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T18:20:35.008319Z","iopub.execute_input":"2024-11-12T18:20:35.008915Z","iopub.status.idle":"2024-11-12T18:24:58.741354Z","shell.execute_reply.started":"2024-11-12T18:20:35.008845Z","shell.execute_reply":"2024-11-12T18:24:58.737992Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Print tune model report\ntrain_predictions_tuned_LR = grid_search_LR.predict(train_features_LR)\nLR_tuned_report = metrics.classification_report(\n    train_target_LR,\n    train_predictions_tuned_LR,\n)\n\nprint(LR_tuned_report)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:15:39.697503Z","iopub.execute_input":"2024-11-12T17:15:39.697952Z","iopub.status.idle":"2024-11-12T17:15:39.844416Z","shell.execute_reply.started":"2024-11-12T17:15:39.697910Z","shell.execute_reply":"2024-11-12T17:15:39.842879Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The tuned code produces a model with lower performance in all measures compared to the original logistic regression model. However the initial logistic regression model doesn't go through cross validation and technically just memorizes the data.\n","metadata":{}},{"cell_type":"markdown","source":"**Stochastic Gradient Descent**","metadata":{}},{"cell_type":"code","source":"#Check the confusion matrix\n_= (\n    metrics.ConfusionMatrixDisplay\n           .from_predictions(\n                train_target_SGD,\n                train_predictions_SGD\n            )\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:16:15.357666Z","iopub.execute_input":"2024-11-12T17:16:15.358159Z","iopub.status.idle":"2024-11-12T17:16:15.897310Z","shell.execute_reply.started":"2024-11-12T17:16:15.358113Z","shell.execute_reply":"2024-11-12T17:16:15.895722Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#check the confusion matrix with normalization and show true labels as a percent of predictions\n#This normalization is done with respect to the sum of the row\n_ = (\n    metrics.ConfusionMatrixDisplay\n           .from_predictions(\n                train_target_SGD,\n                train_predictions_SGD,\n                normalize=\"true\",\n                values_format=\".0%\"\n            )\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:16:16.021387Z","iopub.execute_input":"2024-11-12T17:16:16.021815Z","iopub.status.idle":"2024-11-12T17:16:16.547576Z","shell.execute_reply.started":"2024-11-12T17:16:16.021773Z","shell.execute_reply":"2024-11-12T17:16:16.546067Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#In the following confusion matrix there is an emphasis on incorrect prediction by setting the \n#sample weight to show incorrect values, percent of correct predictions is not accounted for.\n#Moreover, the normalization is done with respect to the predictions, so the values are divided by \n#the sum of columns.\n_ = (\n    metrics.ConfusionMatrixDisplay\n           .from_predictions(\n                train_target_SGD,\n                train_predictions_SGD,\n                normalize=\"pred\",\n                sample_weight=(train_target_SGD != train_predictions_SGD),\n                values_format=\".0%\"\n            )\n)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:16:20.807228Z","iopub.execute_input":"2024-11-12T17:16:20.807652Z","iopub.status.idle":"2024-11-12T17:16:21.366124Z","shell.execute_reply.started":"2024-11-12T17:16:20.807610Z","shell.execute_reply":"2024-11-12T17:16:21.364430Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(SGD_pipeline.steps)\nlinear_model.SGDClassifier?","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:45:50.023001Z","iopub.execute_input":"2024-11-12T17:45:50.023545Z","iopub.status.idle":"2024-11-12T17:45:50.040877Z","shell.execute_reply.started":"2024-11-12T17:45:50.023492Z","shell.execute_reply":"2024-11-12T17:45:50.039243Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Import grid search and randomized search\nfrom sklearn.model_selection import GridSearchCV,RandomizedSearchCV\n\n#Set up a hyperparameter grid\nparam_grid_SGD = {\n    'sgdclassifier__loss': ['hinge', 'log_loss'],               # Different loss functions\n    'sgdclassifier__penalty': ['l2', 'l1'],                     # Regularization types\n    'sgdclassifier__learning_rate': ['optimal', 'adaptive'],    # learning rate\n    'sgdclassifier__max_iter': [1000, 10000],                   # Number of iterations (epochs)\n    'sgdclassifier__early_stopping': [True], \n    'sgdclassifier__n_iter_no_change': [1, 5, 10, 15]\n}\n\n# Set up RandomizedSearchCV\ngrid_search_SGD = GridSearchCV(\n    estimator=SGD_pipeline, \n    param_grid=param_grid_SGD,            # Parameter grid\n    cv=3,                                 # 3-fold cross-validation\n    verbose=10,                           # Verbose output for progress\n    n_jobs=4                              # Use all available cores\n)\n\n# Fit the model\ngrid_search_SGD.fit(train_features_SGD, train_target_SGD)\n\n# Get the best parameters\nprint(\"Best Parameters:\", grid_search_SGD.best_params_)\nprint(\"Best Cross-validation Score:\", grid_search_SGD.best_score_)\n\n# Get the best estimator (the model with the best parameters)\nbest_model_SGD = grid_search_SGD.best_estimator_\n\n# Show accuracy score for best estimator\nbest_accuracy_SGD = best_model_SGD.score(train_features_SGD, train_target_SGD)\nprint(\"Test Set Accuracy of Best Model:\", best_accuracy_SGD)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:51:36.419591Z","iopub.execute_input":"2024-11-12T17:51:36.420651Z","iopub.status.idle":"2024-11-12T17:53:49.871836Z","shell.execute_reply.started":"2024-11-12T17:51:36.420598Z","shell.execute_reply":"2024-11-12T17:53:49.870397Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Print tune model report\ntrain_predictions_tuned_SGD = grid_search_SGD.predict(train_features_SGD)\nSGD_tuned_report = metrics.classification_report(\n    train_target_SGD,\n    train_predictions_tuned_SGD,\n)\n\nprint(SGD_tuned_report)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:54:56.528157Z","iopub.execute_input":"2024-11-12T17:54:56.528626Z","iopub.status.idle":"2024-11-12T17:54:56.672141Z","shell.execute_reply.started":"2024-11-12T17:54:56.528585Z","shell.execute_reply":"2024-11-12T17:54:56.668245Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The tuned code produces a model with the same performance in all measures compared to the original stochastic gradient descent model.","metadata":{}},{"cell_type":"markdown","source":"# Identify the best model","metadata":{}},{"cell_type":"markdown","source":"**The best model is the tuned logistic regression model.**","metadata":{}},{"cell_type":"markdown","source":"# 7. Summarize your results","metadata":{}},{"cell_type":"markdown","source":"In this notebook two models are produced and tuned to try and better their performance. These models are logistic regression and stochastic gradient descent. The best model produced is the logistic regression with tuning. It preformed the best in all measures and was of comparable speed to the stochastic gradient descent models. Tuning hyperparameters is of high importance in this case, this is because both models are being trained on a stratified section of the total training data and fine tuning to increase the performance has been optimized to give an apparent increase in performance.","metadata":{}},{"cell_type":"markdown","source":"# 8. Submit your predictions to the competition","metadata":{}},{"cell_type":"code","source":"predictions = best_model_LR.predict(test_features_df)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:56:58.056976Z","iopub.execute_input":"2024-11-12T17:56:58.057451Z","iopub.status.idle":"2024-11-12T17:56:58.476910Z","shell.execute_reply.started":"2024-11-12T17:56:58.057406Z","shell.execute_reply":"2024-11-12T17:56:58.475425Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"predictions[:5]","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:57:01.043358Z","iopub.execute_input":"2024-11-12T17:57:01.043806Z","iopub.status.idle":"2024-11-12T17:57:01.052142Z","shell.execute_reply.started":"2024-11-12T17:57:01.043750Z","shell.execute_reply":"2024-11-12T17:57:01.050940Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%bash\n\ncat /kaggle/input/digit-recognizer/sample_submission.csv | head -n 5","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:57:03.598561Z","iopub.execute_input":"2024-11-12T17:57:03.599566Z","iopub.status.idle":"2024-11-12T17:57:03.625453Z","shell.execute_reply.started":"2024-11-12T17:57:03.599516Z","shell.execute_reply":"2024-11-12T17:57:03.624198Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submission_df = (\n    pd.read_csv(\n        \"/kaggle/input/digit-recognizer/sample_submission.csv\",\n    ).assign(\n        Label=predictions\n    )\n)\n\nsubmission_df.to_csv(\"submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2024-11-12T17:57:05.008021Z","iopub.execute_input":"2024-11-12T17:57:05.008427Z","iopub.status.idle":"2024-11-12T17:57:05.052845Z","shell.execute_reply.started":"2024-11-12T17:57:05.008388Z","shell.execute_reply":"2024-11-12T17:57:05.051359Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null}]}