{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognition using Linear Models\n- by Ayush Mandowara","metadata":{}},{"cell_type":"markdown","source":"### Models Used\n- Logistic Regression with Multi Class solver\n- Support Vector Machine\n\n### Techniques\n- Principal Component Analysis\n- Cross Validation","metadata":{}},{"cell_type":"markdown","source":"### Why Linear Models\nAlthough this problem can be solved much more effectively using Deep Learning, I wanted to show that linear models are also peforming at a very good level. As you will see in the results below, iteratively we are able to achieve better and better results by trying different types of linear models. \n\nThe best model we got was by SVM with Radias Bias Kernel whose training input was standardarized and transformed using PCA. The benefit of PCA is that we can reduce the dimensionality while keeping most of the information. This helps in reducing the overall run time. \n\nThe best accuracy score on the test set was around 96%. A similar result is seen in the final submission as well.","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"### Standard Imports\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n### Model Building\nfrom sklearn.model_selection import train_test_split, GridSearchCV, KFold, cross_val_score\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.svm import SVC\n\n### Model Evaluation\nfrom sklearn.metrics import classification_report\n\n### Saving model for future use\nfrom joblib import dump, load","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:53:15.381331Z","iopub.execute_input":"2021-11-03T08:53:15.381724Z","iopub.status.idle":"2021-11-03T08:53:16.607827Z","shell.execute_reply.started":"2021-11-03T08:53:15.381620Z","shell.execute_reply":"2021-11-03T08:53:16.606939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading and Understanding the Data","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/digit-recognizer/train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:40.953875Z","iopub.execute_input":"2021-11-03T08:55:40.954519Z","iopub.status.idle":"2021-11-03T08:55:44.621846Z","shell.execute_reply.started":"2021-11-03T08:55:40.954481Z","shell.execute_reply":"2021-11-03T08:55:44.620954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:46.702127Z","iopub.execute_input":"2021-11-03T08:55:46.702951Z","iopub.status.idle":"2021-11-03T08:55:46.731072Z","shell.execute_reply.started":"2021-11-03T08:55:46.702908Z","shell.execute_reply":"2021-11-03T08:55:46.730349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:46.878848Z","iopub.execute_input":"2021-11-03T08:55:46.879650Z","iopub.status.idle":"2021-11-03T08:55:46.936648Z","shell.execute_reply.started":"2021-11-03T08:55:46.879616Z","shell.execute_reply":"2021-11-03T08:55:46.935703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### Checking missing values\nfor i in df.columns:\n    if df[i].isna().sum():\n        print(i)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:48.714811Z","iopub.execute_input":"2021-11-03T08:55:48.715073Z","iopub.status.idle":"2021-11-03T08:55:48.852443Z","shell.execute_reply.started":"2021-11-03T08:55:48.715045Z","shell.execute_reply":"2021-11-03T08:55:48.851709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Observation: there are no missing values in the dataset","metadata":{}},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:49.029686Z","iopub.execute_input":"2021-11-03T08:55:49.029956Z","iopub.status.idle":"2021-11-03T08:55:51.107322Z","shell.execute_reply.started":"2021-11-03T08:55:49.029929Z","shell.execute_reply":"2021-11-03T08:55:51.106404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.label.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:51.109354Z","iopub.execute_input":"2021-11-03T08:55:51.109678Z","iopub.status.idle":"2021-11-03T08:55:51.117104Z","shell.execute_reply.started":"2021-11-03T08:55:51.109617Z","shell.execute_reply":"2021-11-03T08:55:51.116308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x='label', data=df)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:51.118562Z","iopub.execute_input":"2021-11-03T08:55:51.118860Z","iopub.status.idle":"2021-11-03T08:55:51.385372Z","shell.execute_reply.started":"2021-11-03T08:55:51.118832Z","shell.execute_reply":"2021-11-03T08:55:51.384415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Observation: The data is relatively equally distributed among various labels","metadata":{}},{"cell_type":"markdown","source":"## Model Building","metadata":{}},{"cell_type":"markdown","source":"### Train Test Split","metadata":{}},{"cell_type":"code","source":"X = df.drop('label', axis=1)\ny = df['label']","metadata":{"execution":{"iopub.status.busy":"2021-11-03T08:55:54.915097Z","iopub.execute_input":"2021-11-03T08:55:54.915355Z","iopub.status.idle":"2021-11-03T08:55:54.998558Z","shell.execute_reply.started":"2021-11-03T08:55:54.915328Z","shell.execute_reply":"2021-11-03T08:55:54.997641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# since the number of data points is large, we will be using only 10% of data for training\nX_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:05:47.364236Z","iopub.execute_input":"2021-11-03T09:05:47.364537Z","iopub.status.idle":"2021-11-03T09:05:47.635979Z","shell.execute_reply.started":"2021-11-03T09:05:47.364506Z","shell.execute_reply":"2021-11-03T09:05:47.635088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.shape, X_test.shape, y_train.shape, y_test.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:05:48.451805Z","iopub.execute_input":"2021-11-03T09:05:48.452094Z","iopub.status.idle":"2021-11-03T09:05:48.457832Z","shell.execute_reply.started":"2021-11-03T09:05:48.452062Z","shell.execute_reply":"2021-11-03T09:05:48.457201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Scaling","metadata":{}},{"cell_type":"code","source":"scaler = StandardScaler()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:05:50.098971Z","iopub.execute_input":"2021-11-03T09:05:50.099651Z","iopub.status.idle":"2021-11-03T09:05:50.103432Z","shell.execute_reply.started":"2021-11-03T09:05:50.099615Z","shell.execute_reply":"2021-11-03T09:05:50.102682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_scaled = scaler.fit_transform(X_train)\nX_test_scaled = scaler.transform(X_test)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:05:50.251281Z","iopub.execute_input":"2021-11-03T09:05:50.251801Z","iopub.status.idle":"2021-11-03T09:05:50.982222Z","shell.execute_reply.started":"2021-11-03T09:05:50.251753Z","shell.execute_reply":"2021-11-03T09:05:50.981043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Logistic Regression MultiNomial","metadata":{}},{"cell_type":"code","source":"model = LogisticRegression(multi_class='multinomial', solver='sag', max_iter=1000, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:07:18.962515Z","iopub.execute_input":"2021-11-03T09:07:18.962795Z","iopub.status.idle":"2021-11-03T09:07:18.966983Z","shell.execute_reply.started":"2021-11-03T09:07:18.962765Z","shell.execute_reply":"2021-11-03T09:07:18.966144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in X_train.columns:\n    if not X_train[i].dtype == 'int64':\n        print(X_train[i].dtype)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:07:58.169257Z","iopub.execute_input":"2021-11-03T09:07:58.169920Z","iopub.status.idle":"2021-11-03T09:07:58.178500Z","shell.execute_reply.started":"2021-11-03T09:07:58.169883Z","shell.execute_reply":"2021-11-03T09:07:58.177707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:08:01.272031Z","iopub.execute_input":"2021-11-03T09:08:01.272306Z","iopub.status.idle":"2021-11-03T09:26:56.160987Z","shell.execute_reply.started":"2021-11-03T09:08:01.272277Z","shell.execute_reply":"2021-11-03T09:26:56.160140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = model.predict(X_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:26:56.163046Z","iopub.execute_input":"2021-11-03T09:26:56.163529Z","iopub.status.idle":"2021-11-03T09:26:56.394703Z","shell.execute_reply.started":"2021-11-03T09:26:56.163486Z","shell.execute_reply":"2021-11-03T09:26:56.393821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:26:56.396423Z","iopub.execute_input":"2021-11-03T09:26:56.396974Z","iopub.status.idle":"2021-11-03T09:26:56.410601Z","shell.execute_reply.started":"2021-11-03T09:26:56.396923Z","shell.execute_reply":"2021-11-03T09:26:56.409707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:26:56.413936Z","iopub.execute_input":"2021-11-03T09:26:56.414717Z","iopub.status.idle":"2021-11-03T09:26:56.421171Z","shell.execute_reply.started":"2021-11-03T09:26:56.414673Z","shell.execute_reply":"2021-11-03T09:26:56.420173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_train, y_train_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:26:56.423332Z","iopub.execute_input":"2021-11-03T09:26:56.424081Z","iopub.status.idle":"2021-11-03T09:26:56.550938Z","shell.execute_reply.started":"2021-11-03T09:26:56.424014Z","shell.execute_reply":"2021-11-03T09:26:56.550044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test_pred = model.predict(X_test)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:26:56.552584Z","iopub.execute_input":"2021-11-03T09:26:56.553179Z","iopub.status.idle":"2021-11-03T09:26:56.617709Z","shell.execute_reply.started":"2021-11-03T09:26:56.553134Z","shell.execute_reply":"2021-11-03T09:26:56.616721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_test, y_test_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:26:56.619589Z","iopub.execute_input":"2021-11-03T09:26:56.620198Z","iopub.status.idle":"2021-11-03T09:26:56.683585Z","shell.execute_reply.started":"2021-11-03T09:26:56.620155Z","shell.execute_reply":"2021-11-03T09:26:56.682749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Overall Accuracy on test set is around 90%","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### Logistic Regression - Scaled","metadata":{}},{"cell_type":"code","source":"model.fit(X_train_scaled, y_train)\ny_train_pred = model.predict(X_train_scaled)\ny_test_pred = model.predict(X_test_scaled)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:26:56.685407Z","iopub.execute_input":"2021-11-03T09:26:56.686001Z","iopub.status.idle":"2021-11-03T09:45:54.775092Z","shell.execute_reply.started":"2021-11-03T09:26:56.685956Z","shell.execute_reply":"2021-11-03T09:45:54.774005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_train, y_train_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:45:54.782373Z","iopub.execute_input":"2021-11-03T09:45:54.786584Z","iopub.status.idle":"2021-11-03T09:45:54.953754Z","shell.execute_reply.started":"2021-11-03T09:45:54.786513Z","shell.execute_reply":"2021-11-03T09:45:54.952909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_test, y_test_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:45:54.956797Z","iopub.execute_input":"2021-11-03T09:45:54.957045Z","iopub.status.idle":"2021-11-03T09:45:54.999691Z","shell.execute_reply.started":"2021-11-03T09:45:54.957015Z","shell.execute_reply":"2021-11-03T09:45:54.998842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Overall Accuracy on test set increased by 0.5% to 90.6%","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### Logistic Regression - Scaled + PCA","metadata":{}},{"cell_type":"code","source":"pca = PCA(0.95, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:45:55.001143Z","iopub.execute_input":"2021-11-03T09:45:55.001610Z","iopub.status.idle":"2021-11-03T09:45:55.006182Z","shell.execute_reply.started":"2021-11-03T09:45:55.001566Z","shell.execute_reply":"2021-11-03T09:45:55.005378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_scaled_pca = pca.fit_transform(X_train_scaled)\nX_test_scaled_pca = pca.transform(X_test_scaled)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:45:55.007941Z","iopub.execute_input":"2021-11-03T09:45:55.008546Z","iopub.status.idle":"2021-11-03T09:45:59.499800Z","shell.execute_reply.started":"2021-11-03T09:45:55.008500Z","shell.execute_reply":"2021-11-03T09:45:59.498630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:45:59.501750Z","iopub.execute_input":"2021-11-03T09:45:59.502414Z","iopub.status.idle":"2021-11-03T09:45:59.510058Z","shell.execute_reply.started":"2021-11-03T09:45:59.502363Z","shell.execute_reply":"2021-11-03T09:45:59.509105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_scaled_pca.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:45:59.511628Z","iopub.execute_input":"2021-11-03T09:45:59.512039Z","iopub.status.idle":"2021-11-03T09:45:59.524910Z","shell.execute_reply.started":"2021-11-03T09:45:59.511993Z","shell.execute_reply":"2021-11-03T09:45:59.523964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(X_train_scaled_pca, y_train)\ny_train_pred = model.predict(X_train_scaled_pca)\ny_test_pred = model.predict(X_test_scaled_pca)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:45:59.529779Z","iopub.execute_input":"2021-11-03T09:45:59.530641Z","iopub.status.idle":"2021-11-03T09:53:56.372159Z","shell.execute_reply.started":"2021-11-03T09:45:59.530594Z","shell.execute_reply":"2021-11-03T09:53:56.371063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_train, y_train_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:53:56.378292Z","iopub.execute_input":"2021-11-03T09:53:56.381218Z","iopub.status.idle":"2021-11-03T09:53:56.515307Z","shell.execute_reply.started":"2021-11-03T09:53:56.381154Z","shell.execute_reply":"2021-11-03T09:53:56.514262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_test, y_test_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:53:56.516694Z","iopub.execute_input":"2021-11-03T09:53:56.517220Z","iopub.status.idle":"2021-11-03T09:53:56.557697Z","shell.execute_reply.started":"2021-11-03T09:53:56.517175Z","shell.execute_reply":"2021-11-03T09:53:56.556906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Overall Accuracy on test set increased by 1% to 91.6%","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"## SVM - Linear Kernel","metadata":{}},{"cell_type":"code","source":"svc = SVC(C=1, kernel='linear', random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:53:56.559062Z","iopub.execute_input":"2021-11-03T09:53:56.559823Z","iopub.status.idle":"2021-11-03T09:53:56.564410Z","shell.execute_reply.started":"2021-11-03T09:53:56.559779Z","shell.execute_reply":"2021-11-03T09:53:56.563733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = svc.fit(X_train_scaled, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:53:56.566517Z","iopub.execute_input":"2021-11-03T09:53:56.567191Z","iopub.status.idle":"2021-11-03T09:57:11.207102Z","shell.execute_reply.started":"2021-11-03T09:53:56.567146Z","shell.execute_reply":"2021-11-03T09:57:11.206211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = model.predict(X_train_scaled)\ny_test_pred = model.predict(X_test_scaled)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:57:11.208318Z","iopub.execute_input":"2021-11-03T09:57:11.208624Z","iopub.status.idle":"2021-11-03T10:02:43.671768Z","shell.execute_reply.started":"2021-11-03T09:57:11.208592Z","shell.execute_reply":"2021-11-03T10:02:43.670932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_train, y_train_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:02:43.673002Z","iopub.execute_input":"2021-11-03T10:02:43.673418Z","iopub.status.idle":"2021-11-03T10:02:43.762350Z","shell.execute_reply.started":"2021-11-03T10:02:43.673380Z","shell.execute_reply":"2021-11-03T10:02:43.761726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_test, y_test_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:02:43.763385Z","iopub.execute_input":"2021-11-03T10:02:43.763725Z","iopub.status.idle":"2021-11-03T10:02:43.805486Z","shell.execute_reply.started":"2021-11-03T10:02:43.763698Z","shell.execute_reply":"2021-11-03T10:02:43.804710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Overall Accuracy on test set increased by 0.5% to 92%","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### SVM - RBF Kernel","metadata":{}},{"cell_type":"code","source":"svc = SVC(C=1, kernel='rbf', random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:02:43.808642Z","iopub.execute_input":"2021-11-03T10:02:43.809030Z","iopub.status.idle":"2021-11-03T10:02:43.812152Z","shell.execute_reply.started":"2021-11-03T10:02:43.809000Z","shell.execute_reply":"2021-11-03T10:02:43.811675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = svc.fit(X_train_scaled, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:02:43.813143Z","iopub.execute_input":"2021-11-03T10:02:43.813454Z","iopub.status.idle":"2021-11-03T10:08:41.328159Z","shell.execute_reply.started":"2021-11-03T10:02:43.813427Z","shell.execute_reply":"2021-11-03T10:08:41.327339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = model.predict(X_train_scaled)\ny_test_pred = model.predict(X_test_scaled)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:08:41.329334Z","iopub.execute_input":"2021-11-03T10:08:41.329545Z","iopub.status.idle":"2021-11-03T10:18:08.731676Z","shell.execute_reply.started":"2021-11-03T10:08:41.329519Z","shell.execute_reply":"2021-11-03T10:18:08.730910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_train, y_train_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:18:08.733820Z","iopub.execute_input":"2021-11-03T10:18:08.734478Z","iopub.status.idle":"2021-11-03T10:18:08.832245Z","shell.execute_reply.started":"2021-11-03T10:18:08.734432Z","shell.execute_reply":"2021-11-03T10:18:08.831397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_test, y_test_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:18:08.833913Z","iopub.execute_input":"2021-11-03T10:18:08.834460Z","iopub.status.idle":"2021-11-03T10:18:08.874220Z","shell.execute_reply.started":"2021-11-03T10:18:08.834416Z","shell.execute_reply":"2021-11-03T10:18:08.873358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Overall Accuracy on test set increased by 3.8% to 95.8%","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### SVM - RBF Kernel + PCA","metadata":{}},{"cell_type":"code","source":"svc = SVC(C=1, kernel='rbf', random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:18:08.879504Z","iopub.execute_input":"2021-11-03T10:18:08.879754Z","iopub.status.idle":"2021-11-03T10:18:08.883689Z","shell.execute_reply.started":"2021-11-03T10:18:08.879723Z","shell.execute_reply":"2021-11-03T10:18:08.882858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = svc.fit(X_train_scaled_pca, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:18:08.884715Z","iopub.execute_input":"2021-11-03T10:18:08.884910Z","iopub.status.idle":"2021-11-03T10:20:32.422856Z","shell.execute_reply.started":"2021-11-03T10:18:08.884886Z","shell.execute_reply":"2021-11-03T10:20:32.421963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = model.predict(X_train_scaled_pca)\ny_test_pred = model.predict(X_test_scaled_pca)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:20:32.424121Z","iopub.execute_input":"2021-11-03T10:20:32.424361Z","iopub.status.idle":"2021-11-03T10:24:11.841395Z","shell.execute_reply.started":"2021-11-03T10:20:32.424331Z","shell.execute_reply":"2021-11-03T10:24:11.840445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_train, y_train_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:11.843296Z","iopub.execute_input":"2021-11-03T10:24:11.843516Z","iopub.status.idle":"2021-11-03T10:24:11.933709Z","shell.execute_reply.started":"2021-11-03T10:24:11.843488Z","shell.execute_reply":"2021-11-03T10:24:11.932953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(classification_report(y_test, y_test_pred, output_dict=True))","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:11.934957Z","iopub.execute_input":"2021-11-03T10:24:11.935178Z","iopub.status.idle":"2021-11-03T10:24:11.974606Z","shell.execute_reply.started":"2021-11-03T10:24:11.935155Z","shell.execute_reply":"2021-11-03T10:24:11.973819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Overall Accuracy on test set increased by 0.1% to `95.9`%\n- pca had not much effect on accuracy, but it definitely reduced dimensions and hence the computation time","metadata":{}},{"cell_type":"markdown","source":"##### Saving the model for future use due to extremely high test accurary","metadata":{}},{"cell_type":"code","source":"dump(model, 'svm_rbf_pca.joblib') ","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:11.975948Z","iopub.execute_input":"2021-11-03T10:24:11.976182Z","iopub.status.idle":"2021-11-03T10:24:12.027444Z","shell.execute_reply.started":"2021-11-03T10:24:11.976157Z","shell.execute_reply":"2021-11-03T10:24:12.026645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"## Submissions","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv('../input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:42:15.194024Z","iopub.execute_input":"2021-11-03T10:42:15.194433Z","iopub.status.idle":"2021-11-03T10:42:17.545245Z","shell.execute_reply.started":"2021-11-03T10:42:15.194403Z","shell.execute_reply":"2021-11-03T10:42:17.544527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:42:23.039533Z","iopub.execute_input":"2021-11-03T10:42:23.039811Z","iopub.status.idle":"2021-11-03T10:42:23.059960Z","shell.execute_reply.started":"2021-11-03T10:42:23.039782Z","shell.execute_reply":"2021-11-03T10:42:23.059164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_t = test_df.copy()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:42:23.191637Z","iopub.execute_input":"2021-11-03T10:42:23.192272Z","iopub.status.idle":"2021-11-03T10:42:23.323451Z","shell.execute_reply.started":"2021-11-03T10:42:23.192234Z","shell.execute_reply":"2021-11-03T10:42:23.322829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_t_scaled = scaler.transform(X_t)\nX_t_scaled_pca = pca.transform(X_t_scaled)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:42:23.329629Z","iopub.execute_input":"2021-11-03T10:42:23.330407Z","iopub.status.idle":"2021-11-03T10:42:23.814486Z","shell.execute_reply.started":"2021-11-03T10:42:23.330362Z","shell.execute_reply":"2021-11-03T10:42:23.813326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = model.predict(X_t_scaled_pca)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:42:40.537347Z","iopub.execute_input":"2021-11-03T10:42:40.538045Z","iopub.status.idle":"2021-11-03T10:45:06.767553Z","shell.execute_reply.started":"2021-11-03T10:42:40.538009Z","shell.execute_reply":"2021-11-03T10:45:06.766612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv('../input/digit-recognizer/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:45:06.772855Z","iopub.execute_input":"2021-11-03T10:45:06.773337Z","iopub.status.idle":"2021-11-03T10:45:06.796755Z","shell.execute_reply.started":"2021-11-03T10:45:06.773306Z","shell.execute_reply":"2021-11-03T10:45:06.796095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:45:06.798150Z","iopub.execute_input":"2021-11-03T10:45:06.798599Z","iopub.status.idle":"2021-11-03T10:45:06.806100Z","shell.execute_reply.started":"2021-11-03T10:45:06.798568Z","shell.execute_reply":"2021-11-03T10:45:06.805563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:45:06.807049Z","iopub.execute_input":"2021-11-03T10:45:06.807737Z","iopub.status.idle":"2021-11-03T10:45:06.820013Z","shell.execute_reply.started":"2021-11-03T10:45:06.807708Z","shell.execute_reply":"2021-11-03T10:45:06.819212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['Label'] = pred","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:45:06.820977Z","iopub.execute_input":"2021-11-03T10:45:06.821572Z","iopub.status.idle":"2021-11-03T10:45:06.829201Z","shell.execute_reply.started":"2021-11-03T10:45:06.821542Z","shell.execute_reply":"2021-11-03T10:45:06.828602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:45:06.830288Z","iopub.execute_input":"2021-11-03T10:45:06.830599Z","iopub.status.idle":"2021-11-03T10:45:06.842908Z","shell.execute_reply.started":"2021-11-03T10:45:06.830572Z","shell.execute_reply":"2021-11-03T10:45:06.841840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:45:06.844209Z","iopub.execute_input":"2021-11-03T10:45:06.845018Z","iopub.status.idle":"2021-11-03T10:45:06.851043Z","shell.execute_reply.started":"2021-11-03T10:45:06.844984Z","shell.execute_reply":"2021-11-03T10:45:06.850179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:52:31.549169Z","iopub.execute_input":"2021-11-03T10:52:31.549609Z","iopub.status.idle":"2021-11-03T10:52:31.605830Z","shell.execute_reply.started":"2021-11-03T10:52:31.549560Z","shell.execute_reply":"2021-11-03T10:52:31.604948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### SVM - RBF + Hyper Parameter Tuning + PCA","metadata":{}},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.1, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.092378Z","iopub.status.idle":"2021-11-03T10:24:12.092964Z","shell.execute_reply.started":"2021-11-03T10:24:12.092789Z","shell.execute_reply":"2021-11-03T10:24:12.092808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_scaled = scaler.fit_transform(X_train)\nX_test_scaled = scaler.transform(X_test)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.093854Z","iopub.status.idle":"2021-11-03T10:24:12.094491Z","shell.execute_reply.started":"2021-11-03T10:24:12.094301Z","shell.execute_reply":"2021-11-03T10:24:12.094322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_scaled_pca = pca.fit_transform(X_train_scaled)\nX_test_scaled_pca = pca.transform(X_test)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.095723Z","iopub.status.idle":"2021-11-03T10:24:12.096008Z","shell.execute_reply.started":"2021-11-03T10:24:12.095861Z","shell.execute_reply":"2021-11-03T10:24:12.095876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"folds = KFold(n_splits = 5, shuffle = True, random_state = 42)\n\n# specify range of hyperparameters\n# Set the parameters by cross-validation\nhyper_params = [ {'gamma': [1e-2, 1e-3, 1e-4],\n                     'C': [1, 10, 100, 1000]}]\n\n\n# specify model\nmodel = SVC(kernel=\"rbf\")\n\n# set up GridSearchCV()\nmodel_cv = GridSearchCV(estimator = model, \n                        param_grid = hyper_params, \n                        scoring= 'accuracy', \n                        cv = folds, \n                        verbose = 1,\n                        return_train_score=True)      \n\n# fit the model\nmodel_cv.fit(X_train_scaled_pca, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.097032Z","iopub.status.idle":"2021-11-03T10:24:12.097348Z","shell.execute_reply.started":"2021-11-03T10:24:12.097172Z","shell.execute_reply":"2021-11-03T10:24:12.097187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_cv.best_score_","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.098281Z","iopub.status.idle":"2021-11-03T10:24:12.098590Z","shell.execute_reply.started":"2021-11-03T10:24:12.098428Z","shell.execute_reply":"2021-11-03T10:24:12.098443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_cv.best_params_","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.099739Z","iopub.status.idle":"2021-11-03T10:24:12.100033Z","shell.execute_reply.started":"2021-11-03T10:24:12.099878Z","shell.execute_reply":"2021-11-03T10:24:12.099894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv_results = pd.DataFrame(model_cv.cv_results_)\ncv_results","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.100894Z","iopub.status.idle":"2021-11-03T10:24:12.101188Z","shell.execute_reply.started":"2021-11-03T10:24:12.101018Z","shell.execute_reply":"2021-11-03T10:24:12.101032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# converting C to numeric type for plotting on x-axis\ncv_results['param_C'] = cv_results['param_C'].astype('int')\n\n# # plotting\nplt.figure(figsize=(16,6))\n\n# subplot 1/3\nplt.subplot(131)\ngamma_01 = cv_results[cv_results['param_gamma']==0.01]\n\nplt.plot(gamma_01[\"param_C\"], gamma_01[\"mean_test_score\"])\nplt.plot(gamma_01[\"param_C\"], gamma_01[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.01\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='lower right')\nplt.xscale('log')\n\n# subplot 2/3\nplt.subplot(132)\ngamma_001 = cv_results[cv_results['param_gamma']==0.001]\n\nplt.plot(gamma_001[\"param_C\"], gamma_001[\"mean_test_score\"])\nplt.plot(gamma_001[\"param_C\"], gamma_001[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.001\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='lower right')\nplt.xscale('log')\n\n\n# subplot 3/3\nplt.subplot(133)\ngamma_0001 = cv_results[cv_results['param_gamma']==0.0001]\n\nplt.plot(gamma_0001[\"param_C\"], gamma_0001[\"mean_test_score\"])\nplt.plot(gamma_0001[\"param_C\"], gamma_0001[\"mean_train_score\"])\nplt.xlabel('C')\nplt.ylabel('Accuracy')\nplt.title(\"Gamma=0.0001\")\nplt.ylim([0.60, 1])\nplt.legend(['test accuracy', 'train accuracy'], loc='lower right')\nplt.xscale('log')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T10:24:12.101921Z","iopub.status.idle":"2021-11-03T10:24:12.102181Z","shell.execute_reply.started":"2021-11-03T10:24:12.102041Z","shell.execute_reply":"2021-11-03T10:24:12.102055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Observation\n- Hyper Parameter tuning takes a really long time to complete due to which we have taken only 10% of the data for training. This has resulted in poor test score compared to the previous models which had exposure to the full training data. Hence, I won't be using this model for predictions.\n- I actually tried cross_val_score training on the whole data with default parameters as well, but that was also taking an unreasonably long amount of time, so I killed it.","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}}]}