{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Week 7 Supervised Learning - Neural Networkds","metadata":{"tags":[]}},{"cell_type":"code","source":"# Requirements\n# -Conduct your analysis using a cross-validation design.\n# -Conduct / refine EDA.\n# -Conduct Design of Experiments to evaluate the performance of various neural networks by changing the layers and nodes. Tested neural network structures should be explored within a benchmark experiment, a 2x2 completely crossed design.  \n# -An example of a completely crossed designed with {2, 5} layers and {10,20} nodes follows.\n\n\n\n# -Due to the time required to fit each neural network, we will observe only one trial for each cell in the design. \n# -You will build your models on csv and submit your forecasts for test.csv to Kaggle.com, providing your name and user ID for each experimental trial. \n# -Evaluate goodness of fit metrics on the training and validation sets.\n# -Provide a multi-class confusion matrix.\n# -Discuss how your models performed.\n\n# In summary, this assignment asks you to fit a number of neural networks, comparing processing time and performance across experimental treatments. \n# Processing time will be recorded for the fitting on the train.csv. Kaggle.com accuracy scores will be reported for all benchmarks.  ","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:29.328217Z","iopub.execute_input":"2021-08-09T04:35:29.328607Z","iopub.status.idle":"2021-08-09T04:35:29.332461Z","shell.execute_reply.started":"2021-08-09T04:35:29.328559Z","shell.execute_reply":"2021-08-09T04:35:29.331452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#28x28 pixel images in gray-scale. 0 being white and 255 black\n#784 pixels total\n\n#train.csv and test.csv given. test is unlabeled as expected\n\n#train has 785 columns. one for each pixel value and then the label\n#first column is called 'label', which is the digit the user drew\n\n#'pixel{x}' where x is the location from 0 to 783 inclusive\n#i and j are integers between 0 and 27 inclusively. i for pixel row index and j for pixel column index\n#pixel31 is the pixel in the fourth column from left and second row from the top\n\n#proportion of test images that are correctly classified is the contest metric\n","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:29.333811Z","iopub.execute_input":"2021-08-09T04:35:29.334150Z","iopub.status.idle":"2021-08-09T04:35:29.344890Z","shell.execute_reply.started":"2021-08-09T04:35:29.334117Z","shell.execute_reply":"2021-08-09T04:35:29.344187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#basic os stuff\nimport os, gc, random\n\n#dataframe, array and math\nimport pandas as pd\nimport math\nimport numpy as np\n\nimport datetime\n\n#sklearn\nimport sklearn as sk\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split as tts\nfrom sklearn.metrics import confusion_matrix, roc_curve, classification_report\nfrom sklearn.utils import resample, shuffle, compute_class_weight\nfrom sklearn.preprocessing import MinMaxScaler, LabelBinarizer\n\n#xgboost\n#from xgboost import XGBClassifer\n\n#tensorflow\nimport tensorflow as tf\nfrom tensorflow.python.client import device_lib #see if gpu is connected\nimport tensorflow.keras\nfrom tensorflow.keras import Sequential, Input, Model\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint #for early stopping and reduction...\nfrom tensorflow.keras.initializers import GlorotUniform #to initialize random weights\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Input, Add, Activation, ZeroPadding2D, GlobalAveragePooling2D\nfrom tensorflow.keras.layers import BatchNormalization, Conv2D, AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D, LeakyReLU\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.preprocessing import image as image_utils\nfrom tensorflow.keras.applications.imagenet_utils import preprocess_input, decode_predictions, preprocess_input\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras import utils\nfrom tensorflow.keras.utils import get_file, plot_model, to_categorical, model_to_dot\nfrom tensorflow.keras.metrics import AUC\n\nfrom tensorflow.keras.applications.inception_v3 import InceptionV3\nimport tensorflow.keras.backend as K\n#https://www.kaggle.com/bustam/cnn-in-keras-for-kannada-digits\nfrom tensorflow.keras.optimizers import RMSprop\nfrom tensorflow.keras.callbacks import LearningRateScheduler\n\n\n#graphing\nimport pydot\nfrom IPython.display import SVG\nfrom scipy import misc\nimport cv2  # I should have cv2 but there's an issue with OpenCV\n#import opencv\nimport matplotlib.pyplot\n\nimport seaborn as sns\n%matplotlib inline\n\npd.set_option('display.max_rows',800)\npd.set_option('display.max_columns',250)\n","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:29.346190Z","iopub.execute_input":"2021-08-09T04:35:29.346575Z","iopub.status.idle":"2021-08-09T04:35:29.375750Z","shell.execute_reply.started":"2021-08-09T04:35:29.346540Z","shell.execute_reply":"2021-08-09T04:35:29.374991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check if GPU is connected","metadata":{}},{"cell_type":"code","source":"print(device_lib.list_local_devices())","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:29.376963Z","iopub.execute_input":"2021-08-09T04:35:29.377369Z","iopub.status.idle":"2021-08-09T04:35:29.382205Z","shell.execute_reply.started":"2021-08-09T04:35:29.377330Z","shell.execute_reply":"2021-08-09T04:35:29.381523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Def Functions to reset GPU","metadata":{}},{"cell_type":"code","source":"# def reset_keras():\n#     tensorflow.keras.backend.clear_session\n# reset_keras()\n\n# def store_CPU(x):\n#     with tf.device('/CPU:0'):\n#         x=x\n#         reset_keras()\n#         gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:29.383163Z","iopub.execute_input":"2021-08-09T04:35:29.383452Z","iopub.status.idle":"2021-08-09T04:35:29.393462Z","shell.execute_reply.started":"2021-08-09T04:35:29.383425Z","shell.execute_reply":"2021-08-09T04:35:29.392585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/Kannada-MNIST/train.csv\")\ntest_df = pd.read_csv(\"../input/Kannada-MNIST/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:29.394885Z","iopub.execute_input":"2021-08-09T04:35:29.395353Z","iopub.status.idle":"2021-08-09T04:35:32.658539Z","shell.execute_reply.started":"2021-08-09T04:35:29.395311Z","shell.execute_reply":"2021-08-09T04:35:32.657673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df.shape)\nprint(test_df.shape)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:32.661196Z","iopub.execute_input":"2021-08-09T04:35:32.661590Z","iopub.status.idle":"2021-08-09T04:35:32.666844Z","shell.execute_reply.started":"2021-08-09T04:35:32.661549Z","shell.execute_reply":"2021-08-09T04:35:32.665918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.concat([train_df, test_df], sort=False)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:32.669110Z","iopub.execute_input":"2021-08-09T04:35:32.669686Z","iopub.status.idle":"2021-08-09T04:35:32.818531Z","shell.execute_reply.started":"2021-08-09T04:35:32.669615Z","shell.execute_reply":"2021-08-09T04:35:32.817711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Keras Friendly Arrays","metadata":{}},{"cell_type":"code","source":"# X_train_df = pd.DataFrame(train_df.iloc[:,1:])\n# y_train_df = pd.DataFrame(train_df.iloc[:,0])\n# X_test_df = pd.DataFrame(test_df.iloc[:,1:])\n\n#array-like for keras over dataframes for sklearn\n\n\n\nX=train_df.iloc[:,1:].values \nY=train_df.iloc[:,0].values\nX_test=test_df.iloc[:,1:].values\n","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:32.819471Z","iopub.execute_input":"2021-08-09T04:35:32.819843Z","iopub.status.idle":"2021-08-09T04:35:32.826881Z","shell.execute_reply.started":"2021-08-09T04:35:32.819815Z","shell.execute_reply":"2021-08-09T04:35:32.826015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\n\nminmax = MinMaxScaler()\nX = minmax.fit_transform(X)\n\nX_test = minmax.transform(X_test)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:32.827828Z","iopub.execute_input":"2021-08-09T04:35:32.828213Z","iopub.status.idle":"2021-08-09T04:35:33.363130Z","shell.execute_reply.started":"2021-08-09T04:35:32.828185Z","shell.execute_reply":"2021-08-09T04:35:33.362389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.ndim","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.364035Z","iopub.execute_input":"2021-08-09T04:35:33.364420Z","iopub.status.idle":"2021-08-09T04:35:33.369107Z","shell.execute_reply.started":"2021-08-09T04:35:33.364392Z","shell.execute_reply":"2021-08-09T04:35:33.368419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.unique(Y)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.370241Z","iopub.execute_input":"2021-08-09T04:35:33.370666Z","iopub.status.idle":"2021-08-09T04:35:33.382262Z","shell.execute_reply.started":"2021-08-09T04:35:33.370632Z","shell.execute_reply":"2021-08-09T04:35:33.381390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y.ndim","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.383423Z","iopub.execute_input":"2021-08-09T04:35:33.383680Z","iopub.status.idle":"2021-08-09T04:35:33.390210Z","shell.execute_reply.started":"2021-08-09T04:35:33.383655Z","shell.execute_reply":"2021-08-09T04:35:33.389243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reshape for Keras CNN","metadata":{"tags":[]}},{"cell_type":"code","source":"#https://www.kaggle.com/bustam/cnn-in-keras-for-kannada-digits\n\n#60000 images, reshaped to 28 x 28 pixels and 1 channel for black and white.\nX = X.reshape(X.shape[0], 28, 28) #or another 1\nprint(X.shape)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.391615Z","iopub.execute_input":"2021-08-09T04:35:33.391994Z","iopub.status.idle":"2021-08-09T04:35:33.400160Z","shell.execute_reply.started":"2021-08-09T04:35:33.391955Z","shell.execute_reply":"2021-08-09T04:35:33.399499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test = X_test.reshape(X_test.shape[0], 28, 28)\nprint(X_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.401226Z","iopub.execute_input":"2021-08-09T04:35:33.401654Z","iopub.status.idle":"2021-08-09T04:35:33.409893Z","shell.execute_reply.started":"2021-08-09T04:35:33.401615Z","shell.execute_reply":"2021-08-09T04:35:33.409267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Make the target categorical","metadata":{}},{"cell_type":"code","source":"#for cnn\n\n# Y = tensorflow.keras.utils.to_categorical(Y, 10) \n# print(Y.shape)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.410917Z","iopub.execute_input":"2021-08-09T04:35:33.411332Z","iopub.status.idle":"2021-08-09T04:35:33.422827Z","shell.execute_reply.started":"2021-08-09T04:35:33.411293Z","shell.execute_reply":"2021-08-09T04:35:33.422134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y.shape","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.423802Z","iopub.execute_input":"2021-08-09T04:35:33.424224Z","iopub.status.idle":"2021-08-09T04:35:33.434498Z","shell.execute_reply.started":"2021-08-09T04:35:33.424188Z","shell.execute_reply":"2021-08-09T04:35:33.433856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#for cnn\n\n# X_test=test_df.drop('id', axis=1).iloc[:,:].values\n# X_test = X_test.reshape(X_test.shape[0], 28, 28) # or another 1\n","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.435385Z","iopub.execute_input":"2021-08-09T04:35:33.435772Z","iopub.status.idle":"2021-08-09T04:35:33.444937Z","shell.execute_reply.started":"2021-08-09T04:35:33.435744Z","shell.execute_reply":"2021-08-09T04:35:33.443921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/bustam/cnn-in-keras-for-kannada-digits\n#using the tts methodology for a Train Validate split. \n\n#X_train, X_valid, y_train, y_valid = tts(X, Y, test_size = 0.10, random_state=10)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.448031Z","iopub.execute_input":"2021-08-09T04:35:33.448335Z","iopub.status.idle":"2021-08-09T04:35:33.456373Z","shell.execute_reply.started":"2021-08-09T04:35:33.448297Z","shell.execute_reply":"2021-08-09T04:35:33.455426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Artificially Increasing the dataset size","metadata":{}},{"cell_type":"code","source":"#https://www.kaggle.com/bustam/cnn-in-keras-for-kannada-digits\n#save for CNN\n\n# train_datagen = ImageDataGenerator(rescale = 1./255.,\n#                                    rotation_range = 10,\n#                                    width_shift_range = 0.25,\n#                                    height_shift_range = 0.25,\n#                                    shear_range = 0.1,\n#                                    zoom_range = 0.25,\n#                                    horizontal_flip = False)\n\n# valid_datagen = ImageDataGenerator(rescale=1./255) ","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.457896Z","iopub.execute_input":"2021-08-09T04:35:33.458370Z","iopub.status.idle":"2021-08-09T04:35:33.466652Z","shell.execute_reply.started":"2021-08-09T04:35:33.458323Z","shell.execute_reply":"2021-08-09T04:35:33.465756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/bustam/cnn-in-keras-for-kannada-digits\n#learning rate decay for CNN\n\n# def lr_decay(epoch):#lrv\n#     return initial_learningrate * 0.99 ** epoch","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.468092Z","iopub.execute_input":"2021-08-09T04:35:33.468525Z","iopub.status.idle":"2021-08-09T04:35:33.476874Z","shell.execute_reply.started":"2021-08-09T04:35:33.468486Z","shell.execute_reply":"2021-08-09T04:35:33.476134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TensorFlow - 3 layers 300 x 300 x 100 units 50 epochs","metadata":{}},{"cell_type":"code","source":"\ntf.random.set_seed(10)\nmodel = tf.keras.models.Sequential()\nmodel.add(tf.keras.layers.Flatten(input_shape=[28, 28]))\nmodel.add(tf.keras.layers.Dense(300, activation=\"relu\"))\nmodel.add(tf.keras.layers.Dense(300, activation=\"relu\"))\nmodel.add(tf.keras.layers.Dense(100, activation=\"relu\"))\nmodel.add(tf.keras.layers.Dense(10, activation=\"softmax\"))","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.478412Z","iopub.execute_input":"2021-08-09T04:35:33.478943Z","iopub.status.idle":"2021-08-09T04:35:33.533470Z","shell.execute_reply.started":"2021-08-09T04:35:33.478903Z","shell.execute_reply":"2021-08-09T04:35:33.532498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.537300Z","iopub.execute_input":"2021-08-09T04:35:33.537581Z","iopub.status.idle":"2021-08-09T04:35:33.546312Z","shell.execute_reply.started":"2021-08-09T04:35:33.537549Z","shell.execute_reply":"2021-08-09T04:35:33.545198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#non-convolutional\n\n# tf.random.set_seed(10)\n# model = tf.keras.Sequential()\n# model.add(Flatten(input_shape=[28, 28]))\n# model.add(Dense(50,activation='relu', kernel_initializer='he_normal'))\n# model.add(Dense(50,activation='relu', kernel_initializer='he_normal'))\n# model.add(Dense(50,activation='relu', kernel_initializer='he_normal'))\n# model.add(Dense(10,activation='softmax'))","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.547582Z","iopub.execute_input":"2021-08-09T04:35:33.547843Z","iopub.status.idle":"2021-08-09T04:35:33.556659Z","shell.execute_reply.started":"2021-08-09T04:35:33.547819Z","shell.execute_reply":"2021-08-09T04:35:33.556012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compile and Fit","metadata":{"tags":[]}},{"cell_type":"code","source":"#compile\nmybatch = 128\nmyopt='sgd'\nmyepoch=50\nmycalls=tf.keras.callbacks.ModelCheckpoint('./model_checkpoints/', monitor='val_loss', verbose=0, save_best_only=True,save_freq='epoch')\nmymod=model.compile(optimizer=myopt, loss='sparse_categorical_crossentropy', metrics=['accuracy'])#, 'Recall','Precision']) #you could also specify loss=f1_loss and add   f1_metric to the metrics without any quotes\n\n#we don't need to do any validation set since this does the split for us\n#define the layers\n\n\nstart = datetime.datetime.now()\nhistory = model.fit(X, Y, epochs=myepoch, batch_size=mybatch, validation_split=0.2, callbacks=[mycalls])\nend=datetime.datetime.now()\n\nprint('Start Time for 5 Fold CV Grid Search:')\nprint(start)\nprint('\\n')\nprint('End Time for 5 Fold CV Grid Search:')\nprint(end)\nprint('\\n')\nprint('Total Elapsed Time to fit all of the models:')\nprint(end-start)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:35:33.557666Z","iopub.execute_input":"2021-08-09T04:35:33.557991Z","iopub.status.idle":"2021-08-09T04:37:30.568118Z","shell.execute_reply.started":"2021-08-09T04:35:33.557962Z","shell.execute_reply":"2021-08-09T04:37:30.567126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Results","metadata":{"tags":[]}},{"cell_type":"code","source":"#https://www.kaggle.com/benanakca/kannada-mnist-cnn-tutorial-with-app-top-2/notebook\nimport matplotlib.pyplot as plt\n\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:30.571683Z","iopub.execute_input":"2021-08-09T04:37:30.571980Z","iopub.status.idle":"2021-08-09T04:37:30.871790Z","shell.execute_reply.started":"2021-08-09T04:37:30.571951Z","shell.execute_reply":"2021-08-09T04:37:30.871037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.evaluate(X, Y)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:30.872719Z","iopub.execute_input":"2021-08-09T04:37:30.872968Z","iopub.status.idle":"2021-08-09T04:37:34.615547Z","shell.execute_reply.started":"2021-08-09T04:37:30.872943Z","shell.execute_reply":"2021-08-09T04:37:34.614483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\npd.DataFrame(history.history).plot(figsize=(10, 10))\nplt.grid(True)\nplt.gca().set_ylim(0, 1) # set the vertical range to [0-1]\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:34.616829Z","iopub.execute_input":"2021-08-09T04:37:34.617120Z","iopub.status.idle":"2021-08-09T04:37:34.815886Z","shell.execute_reply.started":"2021-08-09T04:37:34.617084Z","shell.execute_reply":"2021-08-09T04:37:34.814669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predict and Submit","metadata":{"tags":[]}},{"cell_type":"code","source":"#return model_cv, y_train_pred, test_pred","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:34.817194Z","iopub.execute_input":"2021-08-09T04:37:34.817460Z","iopub.status.idle":"2021-08-09T04:37:34.821366Z","shell.execute_reply.started":"2021-08-09T04:37:34.817432Z","shell.execute_reply":"2021-08-09T04:37:34.820503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = model.predict(X)\ny_train_pred = np.argmax(y_train_pred, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:34.822579Z","iopub.execute_input":"2021-08-09T04:37:34.822871Z","iopub.status.idle":"2021-08-09T04:37:37.181591Z","shell.execute_reply.started":"2021-08-09T04:37:34.822842Z","shell.execute_reply":"2021-08-09T04:37:37.180581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_pred = model.predict(X_test)\ntest_pred = np.argmax(test_pred, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:37.182832Z","iopub.execute_input":"2021-08-09T04:37:37.183143Z","iopub.status.idle":"2021-08-09T04:37:37.487920Z","shell.execute_reply.started":"2021-08-09T04:37:37.183112Z","shell.execute_reply":"2021-08-09T04:37:37.487115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_pred","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:37.489082Z","iopub.execute_input":"2021-08-09T04:37:37.489341Z","iopub.status.idle":"2021-08-09T04:37:37.495871Z","shell.execute_reply.started":"2021-08-09T04:37:37.489315Z","shell.execute_reply":"2021-08-09T04:37:37.494902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\ncf = confusion_matrix(Y, y_train_pred)\ncf","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:37.497320Z","iopub.execute_input":"2021-08-09T04:37:37.497693Z","iopub.status.idle":"2021-08-09T04:37:37.558480Z","shell.execute_reply.started":"2021-08-09T04:37:37.497652Z","shell.execute_reply":"2021-08-09T04:37:37.557479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_ids = pd.DataFrame(test_df.iloc[:,0])\n\ntest_pred_df = pd.DataFrame(test_pred)\n\ntest_pred_df_id = pd.concat([test_pred_df, test_ids], axis=1)\n\n\ntest_pred_df_id = test_pred_df_id.rename(columns={0: \"label\",\"id\":\"id\"})\n\ncols = test_pred_df_id.columns.tolist()\ncols = cols[-1:] + cols[:-1]\n\ntest_pred_df_id = test_pred_df_id[cols]\n\ntest_pred_df_id.to_csv(\"submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2021-08-09T04:37:37.559986Z","iopub.execute_input":"2021-08-09T04:37:37.560390Z","iopub.status.idle":"2021-08-09T04:37:37.578800Z","shell.execute_reply.started":"2021-08-09T04:37:37.560346Z","shell.execute_reply":"2021-08-09T04:37:37.577948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n\n\n\n\n\n\n\n\n\nyhat = np.round(model.predict(X_test))\nNNResults=pd.DataFrame(CR(y_test, yhat, output_dict=True))\nNNResults\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nplot_confusion_matrix(model,X_test,y_test)\nmypred=model.predict(X_test)\n\n### Doc Larry's Custom Functions.  Don't need to use these but it shows that you can optimize on different metrics not just the defaults.\n\ndef f1(y_true, y_pred):\n    y_pred = K.round(y_pred)\n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis = 0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n    \n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n    \n    f1 = 2*p*r / (p+r+K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return K.mean(f1)\n\ndef f1_loss(y_true, y_pred):\n    \n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis = 0)\n    tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n    \n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n    \n    f1 = 2*p*r / (p+r+K.epsilon())\n    f1 = tf.where(tf.math.is_nan(f1), tf.zeros_like(f1), f1)\n    return 1 - K.mean(f1)\n    \n    \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"import matplotlib as mpl\nimport matplotlib.pyplot as plt\n\n\nsome_digit = X_train_df.iloc[0]\n\nsome_digit_image = some_digit.values.reshape(28, 28)\n\nplt.imshow(some_digit_image, cmap = \"binary\")\nplt.axis(\"off\")\nplt.show()\n","metadata":{}},{"cell_type":"markdown","source":"train_df.iloc[0:1]","metadata":{}},{"cell_type":"markdown","source":"import matplotlib as mpl\nimport matplotlib.pyplot as plt\n\n\n\nsome_digit = X_train_df.iloc[1]\n\nsome_digit_image = some_digit.values.reshape(28, 28)\n\nplt.imshow(some_digit_image, cmap = \"binary\")\nplt.axis(\"off\")\nplt.show()\n","metadata":{}},{"cell_type":"markdown","source":"train_df.iloc[0:3]","metadata":{}},{"cell_type":"markdown","source":"X_train = pd.DataFrame(train_df.iloc[:,1:])\ny_train = pd.DataFrame(train_df.iloc[:,0])\nX_test = pd.DataFrame(test_df.iloc[:,1:])","metadata":{}},{"cell_type":"markdown","source":"print(X_train.shape)\nprint(y_train.shape)\nprint(X_test.shape)","metadata":{}},{"cell_type":"markdown","source":"from sklearn.preprocessing import MinMaxScaler\n\nminmax = MinMaxScaler()\nX_train_scale = minmax.fit_transform(X_train)\n\nX_test_scale = minmax.transform(X_test)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"import sklearn\n\nfrom sklearn import metrics\n\ndef create_model(X_train, X_test, y_train, model, model_type, param_grid, out_file_sufx, test_ids):\n    #dropping y_test\n    \"\"\"\n    From Priscilla: Creates the best model given a basic model, such as Ridge, Lasso, or ENet using GridSearchCV. All have the same parameter grid, making reproduceability easy.\n    \n    Inputs:\n        X: our feature matrix from our training set. should already be cleaned and feature engineered\n        y: our predictor vector, the log of \"SalePrice\"\n        test_set: our test matrix that we wish to predict on, containing the same features as X\n        model: an sklearn model, set with random_state=10\n        model_type: str, one of [\"ridge\",\"lasso\",\"lasso-unselected-feats\",\"elastic-net\",\"random-forest\"]\n    \n    \"\"\"\n    \n    ## SETUP\n    ## Only allowing 30% of the data to go into testing\n    #X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) #reproduceability\n    \n    print(\"Train shapes:\")\n    print(f\"X_train: {X_train.shape}\")\n    print(f\"y_train: {y_train.shape}\")\n    \n#     print(\"\\nTest shapes:\")\n#     print(f\"X_test: {X_test.shape}\")\n#     print(f\"y_test: {y_test.shape}\")\n    \n    #Instantiate GridSearchCV object: model_cv\n    model_cv = GridSearchCV(model, param_grid=param_grid, cv = 5)\n    \n    #Fit it to the data\n    start = datetime.datetime.now()\n    model_cv.fit(X_train, y_train.values.ravel())\n    end=datetime.datetime.now()\n    \n    \n    #Print the tuned parameter and score\n    \n    print(f\"\\nTuned {model_type} Regression Parameters: {model_cv.best_params_}\")\n    print(\"Best score is {}\".format(model_cv.best_score_))\n    \n    # EVALUATION\n    print(\"\\n\")\n    y_train_pred = model_cv.predict(X_train)\n    #y_test_pred = model_cv.predict(X_test)\n    \n    fig, axes = plt.subplots(2, 1, figsize=(15,15))\n    #visualizing how well we predicted on training\n    axes[0].scatter(y_train, y_train_pred)\n    axes[0].set_title(f\"{model_type} Train: True vs. Predicted\")\n    axes[0].set_xlabel(\"Train Values\")\n    axes[0].set_ylabel(\"Predicted Train Values\")\n    \n    #axes[1].scatter(y_test, y_test_pred)\n    #axes[1].set_title(f\"{model_type} Validation: True vs. Predicted\")\n    #axes[1].set_xlabel(\"Test Values\")\n    #axes[1].set_ylabel(\"Predicted Test Values\")\n    \n    plt.show();\n    \n    print(\"printing validation metrics info:\")\n    print(f\"\\n{model_type} Mean Absolute Error: {metrics.mean_absolute_error(y_train, y_train_pred)}\")\n    print(f\"{model_type} Mean Squared Error: {metrics.mean_squared_error(y_train, y_train_pred)}\")\n    print(f\"{model_type} Root Mean Squared Error: {np.sqrt(metrics.mean_squared_error(y_train, y_train_pred))}\")\n    print((f\"** {model_type} Root Mean Squared Logarithmic \\\n    Error **: {np.sqrt(metrics.mean_squared_log_error(y_train, y_train_pred))}\"))\n    \n    print(f'accuracy score is: {metrics.accuracy_score(y_train, y_train_pred)}')\n    print(f'balanced accuracy score is: {metrics.balanced_accuracy_score(y_train, y_train_pred)}')\n    print('Micro F1 score is:')\n    print(metrics.f1_score(y_train, y_train_pred, average=\"micro\"))\n    print('Macro F1 score is:')\n    print(metrics.f1_score(y_train, y_train_pred, average=\"macro\"))\n    print(f'classification report:')\n    print(metrics.classification_report(y_train, y_train_pred, digits=10))\n    \n#     from sklearn.metrics import roc_curve\n#     #model_cv_logistic1\n#     fpr, tpr, thresholds = roc_curve(y_train, y_train_pred)\n#     print(\"FPR is...\")\n#     print(fpr)\n#     print(\"TPR is...\")\n#     print(tpr)\n#     print(\"ROC Curve\")\n#     plt.plot(fpr, tpr, linewidth=2)\n#     plt.plot([0,1],[0,1], 'k--')\n#     #[...]\n#     plt.show()\n    \n    from sklearn.metrics import confusion_matrix\n    cf = confusion_matrix(y_train, y_train_pred)\n    print(\"--------------      ----Predicted----------\")\n    print(\"------------       Negative------Positive----\")\n    print(\"---------Negative------------------------------\")\n    print(\"---------------------------------------\")\n    print(\"-Actual |--------------------------------------\")\n    print(\"---------Positive------------------------------\")\n    print(cf)\n\n    \n    from sklearn.metrics import precision_score, recall_score\n    \n    precision = precision_score(y_train, y_train_pred, average='micro')\n    print(\"Micro Precision:\")\n    print(precision)\n    print(\"\\n\")\n    recall = recall_score(y_train, y_train_pred, average='micro')\n    print(\"Recall:\")\n    print(recall)\n    \n#     from sklearn.metrics import precision_recall_curve\n#     precisions, recalls, thresholds = precision_recall_curve(y_train, y_train_pred)\n#     print(\"Precision Recall Curve\")\n#     plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\")\n#     plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\")\n#     plt.show()\n    \n    #from sklearn\n    \n#     from sklearn.metrics import f1_score\n#     f1 = f1_score(y_train, y_train_pred)\n#     print(\"F1 Score:\")\n#     print(f1)\n    \n    \n\n    test_pred = model_cv.predict(X_test)\n\n\n    print('Start Time for 5 Fold CV Grid Search:')\n    print(start)\n    print('\\n')\n    print('End Time for 5 Fold CV Grid Search:')\n    print(end)\n    print('\\n')\n    print('Total Elapsed Time to fit all of the models:')\n    print(end-start)\n\n    return model_cv, y_train_pred, test_pred","metadata":{}},{"cell_type":"markdown","source":"from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn import metrics\n\n\n#no kernel trick here\n\nn_estimators = np.array([100])\ncriterion = np.array(['gini'])\nmin_samples_leaf = np.array([5])\nmax_features = np.array(['sqrt'])\nmin_impurity_decrease = np.array([0.000001])\noob_score = np.array([True])\nn_jobs = np.array([-1])\nverbose = np.array([1])\n\n#alpha = np.array([0,0.5, 1.0, 10, 100]) #hinge loss doesn't support l1\n#C = np.array([0.01, 1.0, 100, 1000])\n#fit_prior=np.array([True])\n#kernel = np.array(['linear','poly','rbf','sigmoid'])\n#loss = np.array([\"hinge\"])\n#dual = np.array([True]) #our n is greater than our number of features\n#degree = np.array([2,5,10])\n#gamma = np.array(['scale','auto'])\n\n# class_weight = {'None','Balanced'}\n#l1_ratio = np.array([0.5])\nparam_grid = {\"class_weight\": ['balanced', None], \"n_estimators\": n_estimators, \"criterion\": criterion, \"min_samples_leaf\": min_samples_leaf, \"max_features\":  max_features, \"min_impurity_decrease\": min_impurity_decrease, \"oob_score\": oob_score, \"n_jobs\": n_jobs} \n\n#y_train input is log transformed\nmodel_cv_PCA_rfc_train, pca_rfc_train_pred_train, pca_rfc_test_submission_train = create_model(X_train = X_train_reduced, y_train=y_train, X_test=X_test_pca, model=RandomForestClassifier(random_state = 10), model_type=\"RandomForestClassifier\", param_grid=param_grid, out_file_sufx='PCA_kMNIST', test_ids=test_ids)\n\n#wow... this is way better","metadata":{}},{"cell_type":"markdown","source":"\n\ntest_ids = pd.DataFrame(test_df.iloc[:,0])\n\npca_safe_RFC_Test_Submit = pd.DataFrame(pca_rfc_test_submission_train)\n\npca_safe_RFC_Test_Submit_id = pd.concat([pca_safe_RFC_Test_Submit, test_ids], axis=1)\n\n\npca_safe_RFC_Test_Submit_id = pca_safe_RFC_Test_Submit_id.rename(columns={0: \"label\",\"id\":\"id\"})\n\ncols = pca_safe_RFC_Test_Submit_id.columns.tolist()\ncols = cols[-1:] + cols[:-1]\n\npca_safe_RFC_Test_Submit_id = pca_safe_RFC_Test_Submit_id[cols]\n\npca_safe_RFC_Test_Submit_id.to_csv(\"./pca_safe_RFC_Kmnist_submit.csv\",index=False)","metadata":{}},{"cell_type":"markdown","source":"start=datetime.now()\nrf2.fit(trainimages,labels)\nend=datetime.now()\nprint(end-start)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# That’s a lot of numbers. It’s often more convenient to look at an image representation of the confusion matrix, using Matplotlib’s matshow() function:\n\nplt.matshow(conf_mx, cmap=plt.cm.gray)\nplt.show()","metadata":{}},{"cell_type":"markdown","source":"# Let’s focus the plot on the errors. First, you need to divide each value in the confusion matrix by the number of images in the corresponding class so that you can compare error rates instead of absolute numbers of errors (which would make abundant classes look unfairly bad):\n\nrow_sums = conf_mx.sum(axis=1, keepdims=True)\nnorm_conf_mx = conf_mx / row_sums\nFill the diagonal with zeros to keep only the errors, and plot the result:\n\nnp.fill_diagonal(norm_conf_mx, 0)\nplt.matshow(norm_conf_mx, cmap=plt.cm.gray)\nplt.show()","metadata":{}},{"cell_type":"markdown","source":"cl_a, cl_b = 3, 5\nX_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)]\nX_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)]\nX_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)]\nX_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]\n\nplt.figure(figsize=(8,8))\nplt.subplot(221); plot_digits(X_aa[:25], images_per_row=5)\nplt.subplot(222); plot_digits(X_ab[:25], images_per_row=5)\nplt.subplot(223); plot_digits(X_ba[:25], images_per_row=5)\nplt.subplot(224); plot_digits(X_bb[:25], images_per_row=5)\nplt.show()","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}