{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Homework 04\n\n**Aysha Mohammed Asif**","metadata":{}},{"cell_type":"markdown","source":"# Preparing the Environment\n\nUse the cell below to import NumPy, Pandas, and the Pyplot library of Matplotlib, using the standard aliases. \n\nAlso import the following tools from Scikit-Learn: `LogisticRegresion`, `DecisionTreeClassifier`, `RandomForestClassifier`, `KNeighborsClassifier`, `GridSearchCV`, `cross_val_score`, `cross_val_predict`, `classification_report`, and `confusion_matrix`.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.model_selection import cross_val_score, cross_val_predict\nfrom sklearn.metrics import classification_report, confusion_matrix","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-30T21:51:27.570758Z","iopub.execute_input":"2023-03-30T21:51:27.571745Z","iopub.status.idle":"2023-03-30T21:51:29.223971Z","shell.execute_reply.started":"2023-03-30T21:51:27.571700Z","shell.execute_reply":"2023-03-30T21:51:29.222893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Run the cell below to suppress scientific notation for NumPy arrays. ","metadata":{}},{"cell_type":"code","source":"np.set_printoptions(suppress=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:51:29.231371Z","iopub.execute_input":"2023-03-30T21:51:29.231795Z","iopub.status.idle":"2023-03-30T21:51:29.244183Z","shell.execute_reply.started":"2023-03-30T21:51:29.231740Z","shell.execute_reply":"2023-03-30T21:51:29.241921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Part 1: Load and Explore Data\n\nIn this assignment, you will be working with the MNIST dataset. This dataset consists of several thousand black-and-white images of handwritten digits from 0 - 9. Your goal will be to create a model capable of identifying the digit displayed in an image based on the values of pixels within the image. \n\nYou can learn more about this dataset here: https://www.kaggle.com/c/digit-recognizer\n\nThe images in the dataset are 28x28 black-and-white images. As such, each image contains 784 pixel values. These pixel values are represented as integers ranging from 0 to 255. Rather than working with image files, you will be provided with CSV files that contain the pixel values. Each row in the CSV files will contain the pixel values for a single image. \n\n**1A.** Use the cell below to load the dataset `digit-recognizer/train.csv` into a DataFrame. Shuffle the DataFrame (setting `random_state=1`), print its shape, and then display the head of the DataFrame. ","metadata":{}},{"cell_type":"code","source":"# 1A\ndigit_r = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndigit_r = digit_r.sample(frac=1, random_state=1)\nprint(digit_r.shape)\ndigit_r.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:51:29.246136Z","iopub.execute_input":"2023-03-30T21:51:29.247349Z","iopub.status.idle":"2023-03-30T21:51:33.731935Z","shell.execute_reply.started":"2023-03-30T21:51:29.247294Z","shell.execute_reply":"2023-03-30T21:51:33.730062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This dataset is relatively large. For the sake of reducing runtime of this notebook, we will use only 10,000 observations when performing hyperparameter tuning and model selection. \n\n**1B**. Create a DataFrame named `df_sample` that contains 10,000 rows sampled at random from the DataFrame created in the previous cell. Set `random_state=1` when performing the sampling. Print the shape of the new DataFrame. ","metadata":{}},{"cell_type":"code","source":"# 1B \ndf_sample = digit_r.sample( n= 10000, random_state = 1)\nprint(df_sample.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:51:33.735610Z","iopub.execute_input":"2023-03-30T21:51:33.736361Z","iopub.status.idle":"2023-03-30T21:51:33.823998Z","shell.execute_reply.started":"2023-03-30T21:51:33.736321Z","shell.execute_reply":"2023-03-30T21:51:33.822873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**1C.** Create feature and label arrays named `X_sample` and `y_sample`. Use only the 10,000 observations from the sample. Print the shape of both arrays. ","metadata":{}},{"cell_type":"code","source":"# 1C\nX_sample = df_sample.iloc[:,1:].values\ny_sample = df_sample.iloc[:,0].values\nprint(X_sample.shape)\nprint(y_sample.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:51:33.825579Z","iopub.execute_input":"2023-03-30T21:51:33.825918Z","iopub.status.idle":"2023-03-30T21:51:33.833664Z","shell.execute_reply.started":"2023-03-30T21:51:33.825886Z","shell.execute_reply":"2023-03-30T21:51:33.832220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**1D.** Complete the code below to create a 2x4 grid of subplots displaying the first 8 digits in the sample, and to also print the labels for these observations. Note that the rows need to be reshaped into 28x28 arrays in order to be displayed as images. ","metadata":{}},{"cell_type":"code","source":"# 1D\nfor i in range(8):\n    plt.subplot(2,4, i+1)\n    digit = X_sample[i, :].reshape(28 , 28)\n    plt.imshow(digit, cmap='Greys')\n    plt.axis('off')\nplt.show()\n\nprint(y_sample[:8])","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:51:33.835449Z","iopub.execute_input":"2023-03-30T21:51:33.836175Z","iopub.status.idle":"2023-03-30T21:51:34.167042Z","shell.execute_reply.started":"2023-03-30T21:51:33.835928Z","shell.execute_reply":"2023-03-30T21:51:34.165386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**1E.** Some of our algorithms will have better performance if we scale the pixel values. Since we know the pixel values all range from 0 to 255, we will simply divide the feature array by 255 to get a scaled feature array with values between 0 and 1. Perform this task in the next cell and name the scaled feature array `X_scaled`. ","metadata":{}},{"cell_type":"code","source":"# 1E\nX_scaled = X_sample/255","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:51:34.169217Z","iopub.execute_input":"2023-03-30T21:51:34.170121Z","iopub.status.idle":"2023-03-30T21:51:34.206437Z","shell.execute_reply.started":"2023-03-30T21:51:34.170037Z","shell.execute_reply":"2023-03-30T21:51:34.205243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Part 2: Model Selection\n\nWe will now perform model selection and hyperparameter tuning. For this assignment, you will consider logistic regression, decision tree, random forest, and KNN models. \n\n**IMPORTANT NOTE:** For the sake of time, we will be using **5-fold** cross-validation throughout Part 2, rather than 10-fold cross-validation. ","metadata":{}},{"cell_type":"markdown","source":"### Logistic Regression\n\n**2A.** In the cell below, create a logistic regression model, setting `penalty='none'`, `solver='saga'`, and `tol=0.01`. The last two parameters are set to help with convergence. Estimate the model's out-of-sample accuracy using **5-fold** cross validation. Print the result rounded to 4 decimal places in a message of the following form: \n\n    CV Score: ____\n\nYou might need to adjust the max_iter parameter of your model to avoid convergence warnings. Note that you are not asked to actually fit a model on the full dataset at this point.","metadata":{}},{"cell_type":"code","source":"# 2A\ndigit_lrmod = LogisticRegression(penalty='none', solver = 'saga', tol = 0.01 , max_iter=1000)\ncvs = cross_val_score(digit_lrmod, X_scaled, y_sample, cv = 5).mean()\nprint(f'CV Score:       {cvs:.4f}')","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:51:34.208434Z","iopub.execute_input":"2023-03-30T21:51:34.209420Z","iopub.status.idle":"2023-03-30T21:53:04.812756Z","shell.execute_reply.started":"2023-03-30T21:51:34.209380Z","shell.execute_reply":"2023-03-30T21:53:04.811047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### K-Nearest Neighbors\n\n**2B.** Use `GridSearchCV` to perform hyperparameter tuning for KNN models, according to the following specifications:\n* Tune over the `n_neighbors` parameter, considering at least 8 values for this parameter. \n* Adjust the list of hyperparameter values as needed in order to find the best possible value. \n* Name your `GridSearchCV` object `knn_grid` to be consistent with code provided in the next step.\n* Use the scaled sample data. \n* Use **5-fold** cross_validation. \n* Use the `best_estimator_` attribute to extract the best model found. \n* Summarize your results by printing messages of the type shown below, with numerical values rounded to four decimal places.\n\n```\nBest Parameters: ____\nBest CV Score:   ____\nTraining Acc:    ____\n```","metadata":{}},{"cell_type":"code","source":"# 2B\nknn_clf = KNeighborsClassifier()\n\nknn_parameters = {\n    'n_neighbors': [2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 20 , 22],\n}\n\nknn_grid = GridSearchCV(knn_clf, knn_parameters, cv=5, refit='True', n_jobs=-1)\nknn_grid.fit(X_scaled, y_sample)\n\nknn_mod = knn_grid.best_estimator_\n\nprint(f'Best Parameters: {knn_grid.best_params_}')\nprint(f'Best CV Score:   {knn_grid.best_score_:.4f}')\nprint(f'Training Acc:    {knn_mod.score(X_scaled, y_sample):.4f}')","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:53:04.815257Z","iopub.execute_input":"2023-03-30T21:53:04.816311Z","iopub.status.idle":"2023-03-30T21:53:44.028510Z","shell.execute_reply.started":"2023-03-30T21:53:04.816242Z","shell.execute_reply":"2023-03-30T21:53:44.027475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Run the cell below to visualize your grid search results. Based on these results, you might want to adjust the hyperparameter values considered above. Your plot should provide strong support for the claim that you have found the best hyperparameter value. ","metadata":{}},{"cell_type":"code","source":"knn_summary = pd.DataFrame(knn_grid.cv_results_['params'])\nknn_summary['cv_score'] = knn_grid.cv_results_['mean_test_score']\n\nplt.plot(knn_summary.n_neighbors, knn_summary.cv_score)\nplt.xlabel('n_neighbors')\nplt.ylabel('CV Score')\nplt.xticks(knn_summary.n_neighbors)\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:53:44.032721Z","iopub.execute_input":"2023-03-30T21:53:44.033126Z","iopub.status.idle":"2023-03-30T21:53:44.300013Z","shell.execute_reply.started":"2023-03-30T21:53:44.033085Z","shell.execute_reply":"2023-03-30T21:53:44.298805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Decision Trees\n\n**2C.** Use `GridSearchCV` to perform hyperparameter tuning for decision tree models, according to the following specifications:\n* Set `random_state=1` for your model.\n* Tune over the `max_depth` and `min_samples_leaf` parameters. \n* Considering at least 6 values for `max_depth` and at least 4 values for `min_samples_leaf`. \n* Adjust the list of hyperparameter values as needed in order to find a parameter set that is at least close to the best possible.\n* Name your `GridSearchCV` object `dt_grid` to be consistent with code provided in the next step.\n* Use the scaled sample data. \n* Use **5-fold** cross_validation. \n* Use the `best_estimator_` attribute to extract the best model found.\n* Summarize your results by printing messages of the type shown below, with numerical values rounded to four decimal places.\n\n```\nBest Parameters: ____\nBest CV Score:   ____\nTraining Acc:    ____\n```","metadata":{}},{"cell_type":"code","source":"# 2C\ndigit_dtc = DecisionTreeClassifier(random_state=1)\n\ndt_parameters = {\n    'max_depth': [10, 12, 14, 15, 16, 17, 18, 20],\n    'min_samples_leaf': [1, 2, 3, 4, 5],\n}\n\ndt_grid = GridSearchCV(digit_dtc, dt_parameters, cv=5, refit='True', n_jobs=-1)\ndt_grid.fit(X_scaled, y_sample)\n\ndt_mod = dt_grid.best_estimator_\n\nprint(f'Best Parameters: {dt_grid.best_params_}')\nprint(f'Best CV Score:   {dt_grid.best_score_:.4f}')\nprint(f'Training Acc:    {dt_mod.score(X_scaled, y_sample):.4f}')","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:53:44.301458Z","iopub.execute_input":"2023-03-30T21:53:44.301905Z","iopub.status.idle":"2023-03-30T21:55:37.352059Z","shell.execute_reply.started":"2023-03-30T21:53:44.301869Z","shell.execute_reply":"2023-03-30T21:55:37.350696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Run the cell below to visualize your grid search results. Based on these results, you might want to adjust the hyperparameter values considered above. Your plot should provide strong support for the claim that you have found the best set of hyperparameter values (or at least something close to it).","metadata":{}},{"cell_type":"code","source":"dt_summary = pd.DataFrame(dt_grid.cv_results_['params'])\ndt_summary['cv_score'] = dt_grid.cv_results_['mean_test_score']\n\nfor ms in dt_parameters['min_samples_leaf']:\n    temp = dt_summary.query(f'min_samples_leaf == {ms}')\n    plt.plot(temp.max_depth, temp.cv_score, label=ms)\nplt.xlabel('Maximum Depth')\nplt.ylabel('CV Score')\nplt.legend(title='Min Samples', bbox_to_anchor=[1,1])\nplt.xticks(dt_parameters['max_depth'])\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:55:37.353670Z","iopub.execute_input":"2023-03-30T21:55:37.354132Z","iopub.status.idle":"2023-03-30T21:55:37.655333Z","shell.execute_reply.started":"2023-03-30T21:55:37.354084Z","shell.execute_reply":"2023-03-30T21:55:37.653922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Random Forests\n\n**2D.** Use `GridSearchCV` to perform hyperparameter tuning for random forest models, according to the following specifications:\n* Set `random_state=1` for your model.\n* Use 100 trees in your forest. \n* Tune over the `max_depth` and `min_samples_leaf` parameters. \n* Considering at least 6 values for `max_depth` and at least 3 values for `min_samples_leaf`. \n* Adjust the list of hyperparameter values as needed in order to find a parameter set that is at least close to the best possible.\n* Name your `GridSearchCV` object `rf_grid` to be consistent with code provided in the next step.\n* Use the scaled sample data. \n* Use **5-fold** cross_validation. \n* Use the `best_estimator_` attribute to extract the best model found.\n* Summarize your results by printing messages of the type shown below, with numerical values rounded to four decimal places.\n\n```\nBest Parameters: ____\nBest CV Score:   ____\nTraining Acc:    ____","metadata":{}},{"cell_type":"code","source":"# 2D\nrf_clf = RandomForestClassifier(n_estimators=100, random_state=1)\n\nrf_parameters = {\n    'max_depth': [6, 8, 12, 14, 15, 16, 18, 20],\n    'min_samples_leaf': [1, 2, 3, 4]\n}\n\nrf_grid = GridSearchCV(rf_clf, rf_parameters, cv=5, refit='True', n_jobs=-1)\nrf_grid.fit(X_scaled, y_sample)\n\nrf_mod = rf_grid.best_estimator_\n\nprint(f'Best Parameters: {rf_grid.best_params_}')\nprint(f'Best CV Score:   {rf_grid.best_score_:.4f}')\nprint(f'Training Acc:    {rf_mod.score(X_scaled, y_sample):.4f}')","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:55:37.656717Z","iopub.execute_input":"2023-03-30T21:55:37.657811Z","iopub.status.idle":"2023-03-30T21:59:34.746140Z","shell.execute_reply.started":"2023-03-30T21:55:37.657757Z","shell.execute_reply":"2023-03-30T21:59:34.744913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rf_summary = pd.DataFrame(rf_grid.cv_results_['params'])\nrf_summary['cv_score'] = rf_grid.cv_results_['mean_test_score']\n\nfor ms in rf_parameters['min_samples_leaf']:\n    temp = rf_summary.query(f'min_samples_leaf == {ms}')\n    plt.plot(temp.max_depth, temp.cv_score, label=ms)\nplt.xlabel('Maximum Depth')\nplt.ylabel('CV Score')\nplt.legend(title='Min Samples', bbox_to_anchor=[1,1])\nplt.xticks(rf_parameters['max_depth'])\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:59:34.747592Z","iopub.execute_input":"2023-03-30T21:59:34.747933Z","iopub.status.idle":"2023-03-30T21:59:35.019701Z","shell.execute_reply.started":"2023-03-30T21:59:34.747902Z","shell.execute_reply":"2023-03-30T21:59:35.018534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Part 3: Final Model\n\nIn this part, you will identify, train, and evaluate your final model. For the sake of time, we only used 10,000 observations for hyperparameter tuning. However, in order to improve our model, we will train the our final model on the entire set of 42,000 observations. \n\n**3A.** In the cell below, create feature and label arrays using all of the observations from the original DataFrame. Scale the feature array by dividing the pixel values by 255. Then print the shape of the scaled feature array and the label array. ","metadata":{}},{"cell_type":"code","source":"# 3A\nX = digit_r.iloc[:,1:].values\ny = digit_r.iloc[:,0].values\nX_s = X / 255\nprint(X_s.shape)\nprint(y.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:59:35.021068Z","iopub.execute_input":"2023-03-30T21:59:35.021377Z","iopub.status.idle":"2023-03-30T21:59:35.129485Z","shell.execute_reply.started":"2023-03-30T21:59:35.021347Z","shell.execute_reply":"2023-03-30T21:59:35.128294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Review the cross-validation scores for the models considered in Part 2 and identify the model with the highest CV score. If your best-performing model is a KNN model, and it is only slighlty better than another model, then select your second-best model instead. KNN models are are problematic with this dataset, for reasons discussed in class.  \n\n**3B.** After identifying your final model, train a new version of this model using the best set of hyperparameter values found previously. If creating a decision tree or random forest model, set `random_state=1`. Then fit the model to the full dataset. Next, evaluate the final model by performing **10-fold** cross-validation. Print the result rounded to 4 decimal places in a message of the following form: \n\n    CV Score: ____\n    \n **Note:** You are asked to use 10-fold CV here. ","metadata":{}},{"cell_type":"code","source":"# 3B\nfinal_model = RandomForestClassifier(max_depth = 16, min_samples_leaf = 1, random_state = 1)\nfinal_model.fit(X_s, y)\ncvs_final = cross_val_score(final_model, X_s, y, cv=10).mean()\nprint(f'CV Score:       {cvs_final:.4f}')","metadata":{"execution":{"iopub.status.busy":"2023-03-30T21:59:35.131754Z","iopub.execute_input":"2023-03-30T21:59:35.132319Z","iopub.status.idle":"2023-03-30T22:04:18.302003Z","shell.execute_reply.started":"2023-03-30T21:59:35.132282Z","shell.execute_reply":"2023-03-30T22:04:18.300909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**3C.** Use 10-fold cross-validation prediction with your final model to generate out-of-sample predictions for the dataset. Use these predictions to generate a classification report. Display the report as a DataFrame. ","metadata":{}},{"cell_type":"code","source":"# 3C\ncv_pred = cross_val_predict(final_model, X_s, y, cv=10)\ncm = confusion_matrix(y, cv_pred)\npd.DataFrame(cm, index=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], columns=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9])","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:04:18.303597Z","iopub.execute_input":"2023-03-30T22:04:18.304138Z","iopub.status.idle":"2023-03-30T22:08:34.700989Z","shell.execute_reply.started":"2023-03-30T22:04:18.304090Z","shell.execute_reply":"2023-03-30T22:08:34.699776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**3D.** Edit this markdown cell to fill complete the statements below. Please remove the underscores replacing them with the correct values. \n\n* The 1st most common misclassification mode by the model was to classify the digit 4 as the digit 9. \n* The 2nd most common misclassification mode by the model was to classify the digit 3 as the digit 5. ","metadata":{}},{"cell_type":"markdown","source":"**3E.** Use the out-of-sample predictions to create and print a classification report. ","metadata":{}},{"cell_type":"code","source":"# 3E\n#cv_pred = cross_val_predict(final_model, X_s, y, cv=10)\nprint(classification_report(y, cv_pred))","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:08:34.702423Z","iopub.execute_input":"2023-03-30T22:08:34.702787Z","iopub.status.idle":"2023-03-30T22:08:34.801758Z","shell.execute_reply.started":"2023-03-30T22:08:34.702750Z","shell.execute_reply":"2023-03-30T22:08:34.800540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Part 4: New Observations\n\nThe csv file `digit-recognizer/test.csv` contains pixel values for 28,000 unlabeled images. \n\n**4A.** Load this csv file into a DataFrame. Then use the `sample()` method to select 16 rows at random, setting `random_state=4`. Create a feature array by extract the pixel values and dividing them by 255. Print the shape of the new feature array. ","metadata":{}},{"cell_type":"code","source":"# 4A\nd = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\nd = d.sample(n = 16, random_state=4)\nX_n = d.iloc[:,:].values\nX_new = X_n / 255\nprint(X_new.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:08:34.803260Z","iopub.execute_input":"2023-03-30T22:08:34.803894Z","iopub.status.idle":"2023-03-30T22:08:36.881060Z","shell.execute_reply.started":"2023-03-30T22:08:34.803855Z","shell.execute_reply":"2023-03-30T22:08:36.879859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4B**. Adapt the code form Part 1D to create a 4x4 grid of subplots displaying the 16 images in the sample of new observations. ","metadata":{}},{"cell_type":"code","source":"# 4B\nfor i in range(16):\n    plt.subplot(4,4, i+1)\n    digit = X_new[i, :].reshape(28 , 28)\n    plt.imshow(digit, cmap='Greys')\n    plt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:08:36.882373Z","iopub.execute_input":"2023-03-30T22:08:36.882684Z","iopub.status.idle":"2023-03-30T22:08:37.542610Z","shell.execute_reply.started":"2023-03-30T22:08:36.882652Z","shell.execute_reply":"2023-03-30T22:08:37.541234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4C.** Generate class predictions for the 16 digits shown above. Print the predictions. To better compare the predictions with the digits, reshape the prediction array to a 4x4 array and then print this reshaped array. ","metadata":{}},{"cell_type":"code","source":"# 4C\npred = final_model.predict(X_new).reshape(4, 4)\nprint(pred)","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:08:37.543967Z","iopub.execute_input":"2023-03-30T22:08:37.544298Z","iopub.status.idle":"2023-03-30T22:08:37.566355Z","shell.execute_reply.started":"2023-03-30T22:08:37.544265Z","shell.execute_reply":"2023-03-30T22:08:37.565048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4D.** For which, if any, of the 16 new observations do you suspect that the model generated an incorrect prediction? Provide your answer in this mark-down cell by listing the indices (0 - 15) for the misclassified observations. \n\nAnswer: **6**","metadata":{}},{"cell_type":"markdown","source":"**4E.** Use your final model to generate class probability estimates for the 16 new observations. Print the array of probability estimates rounded to 2 decimal places.","metadata":{}},{"cell_type":"code","source":"# 4E\nnew_cv_prob = final_model.predict_proba(X_new)\nprint(new_cv_prob.round(2))","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:08:37.567669Z","iopub.execute_input":"2023-03-30T22:08:37.567992Z","iopub.status.idle":"2023-03-30T22:08:37.590299Z","shell.execute_reply.started":"2023-03-30T22:08:37.567960Z","shell.execute_reply":"2023-03-30T22:08:37.589134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4F.** Use the `max()` method with `axis=1` to determine the largest value in each row of the probability estimate array. Print these values rounded to 2 decimal places.","metadata":{}},{"cell_type":"code","source":"# 4F\ncv_prob = new_cv_prob.max(axis = 1)\nprint(cv_prob.round(2))","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:08:37.591775Z","iopub.execute_input":"2023-03-30T22:08:37.592128Z","iopub.status.idle":"2023-03-30T22:08:37.598538Z","shell.execute_reply.started":"2023-03-30T22:08:37.592095Z","shell.execute_reply":"2023-03-30T22:08:37.597140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4G.** Identify the three observations for which the model is the least confident in its prediction. Provide the indices (0-15) for these observations below, in increasing order of confidence. \n\nAnswer: 8->6->10","metadata":{}},{"cell_type":"markdown","source":"## HTML Export\n\nFollow these instructions to create an HTML render of your notebook: \n1. Replace the text `\"Username\"` in the cell below with your Maryville username. \n2. Replace 408 with 508, if appropriate. \n3. Save and Commit your notebook. \n4. When the commit is completed, open the new version in the viewer. \n5. Navigate to the \"Data\" tab in the view. \n6. Click the \"Download\" button next to your HTML file. \n7. This might open the HTML file in your browser. If so, then save that file to your local machine. \n8. Upload the HTML export to Canvas. ","metadata":{}},{"cell_type":"code","source":"!jupyter nbconvert --to html __notebook__.ipynb --output DSCI_508_HW_04_aasif1.html","metadata":{"execution":{"iopub.status.busy":"2023-03-30T22:08:37.599976Z","iopub.execute_input":"2023-03-30T22:08:37.600329Z","iopub.status.idle":"2023-03-30T22:08:53.283633Z","shell.execute_reply.started":"2023-03-30T22:08:37.600276Z","shell.execute_reply":"2023-03-30T22:08:53.282450Z"},"trusted":true},"execution_count":null,"outputs":[]}]}