{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Auto-tuned CNN for image classification (Digit Recognizer)\nAuthor: Brad Magnetta\n\nThings we'll cover ...\n\n- Data collection and manipulation\n- CNN model structure yeilding **Top 6%** performance **(0.99871)** as of 12/16/19\n- Custom auto-tuning method for optimizing model paramters\n\nReferences ...\n\n- [1] Coursera Deeplearning.io Course II\n- [2] https://www.kaggle.com/amneves/top-5-with-keras-auto-hypertuning\n\n## Libraries"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tensorflow.keras.datasets import mnist\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom keras.preprocessing.image import ImageDataGenerator\nimport math\nimport pickle\nimport os.path\nfrom os import path\nimport random\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data collection\n\nHere we show how to merge the kaggle and tensorflow.keras MNIST datasets. The more data you have the deeper your network can be which leads to better performance.\n\n### Import datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the data on local machine\n#train_kaggle = pd.read_csv(\"train.csv\")\n#test_kaggle = pd.read_csv(\"test.csv\")\n# Load the data on kaggle machine\ntrain_kaggle = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest_kaggle = pd.read_csv(\"../input/digit-recognizer/test.csv\")\n\nX_train_kaggle = train_kaggle.drop(labels = [\"label\"],axis = 1).to_numpy()\nY_train_kaggle = train_kaggle[\"label\"]\n\n# Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)\nX_train_kaggle = X_train_kaggle.reshape(-1,28,28,1)\ntest_kaggle = test_kaggle.to_numpy().reshape(-1,28,28,1)\n\nprint('--- check for X_train_kaggle nan:\\n',np.any(np.isnan(X_train_kaggle)))\n#visualize the data\nplt.imshow(X_train_kaggle[1,:,:,0])\nplt.show()\nplt.hist(Y_train_kaggle,np.unique(Y_train_kaggle).shape[0],rwidth=0.9)\nplt.show()\n\nprint('--- check for test_kaggle nan:\\n',np.any(np.isnan(test_kaggle)))\n#visualize the data\nplt.imshow(test_kaggle[1,:,:,0])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(X_train_keras, Y_train_keras), (X_val_keras, Y_val_keras) = mnist.load_data()\n\n# Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)\nX_train_keras = X_train_keras.reshape(-1,28,28,1)\nX_val_keras = X_val_keras.reshape(-1,28,28,1)\n\nprint('--- check for X_train_keras nan:\\n',np.any(np.isnan(X_train_keras)))\n#visualize the data\nplt.imshow(X_train_keras[1,:,:,0])\nplt.show()\n#sns.countplot(Y_train_keras)\nplt.hist(Y_train_keras,np.unique(Y_train_keras).shape[0],rwidth=0.9)\nplt.show()\n\nprint('--- check for X_val_keras nan:\\n',np.any(np.isnan(X_val_keras)))\n#visualize the data\nplt.imshow(X_val_keras[1,:,:,0])\nplt.show()\nplt.hist(Y_val_keras,np.unique(Y_val_keras).shape[0],rwidth=0.9)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can now make a few important observations from our previous print outs ...\n\n- Our images do not contain any NAN values.\n- Our data is evenly spread about each label value.\n- The spot check of our image data shows we indeed have digits.\n\nBecause of our above conclusions, we can not safely ... \n\n- Concatenate the MINST data for kaggle and tensorflow.keras in X and Y\n- Normalize X and test_kaggle\n- Split X and Y into train and validation subsets. This will allow us to observe the bias/variance tendencies of our models [1]. It is recommended that validation split be ~5% [1]. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#If tf.keras has error in download (sometimes happens when running on kaggle) comment out concatenate\n#X = np.concatenate((X_train_kaggle,X_train_keras,X_val_keras), axis=0)\n#Y = np.concatenate((Y_train_kaggle,Y_train_keras,Y_val_keras), axis=0)\nX = X_train_kaggle\nY = Y_train_kaggle\n\n# Normalize the data\nX = X / 255.0\ntest_kaggle = test_kaggle / 255.0\n\n# Split the train and the validation set for the fitting\nX_train, X_val, Y_train, Y_val = train_test_split(X, Y, test_size = 0.05, random_state=2)\n\nprint('X.shape: ',X.shape)\nprint('X_train.shape = ',X_train.shape)\nprint('X_val.shape = ',X_val.shape)\n\nprint('Y.shape: ',Y.shape)#Note we cold have don't one-hot encoding which would yeild diff. shape\nprint('Y_train.shape = ',Y_train.shape)\nprint('Y_val.shape = ',Y_val.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Generating more data\n\nWe want to build a model so that it will predict well against future data not used in training. If our model performs poorly against future data we say that the model is baised towards the training data. We can combat this in a few ways ...\n\n- Alter our model structure or parameters to produce less biased results.\n- Feed more data through our model.\n\nHowever, even if we choose the correct un-biased model we may still reach a bottle neck that can only be solved by feeding more data into our network. While we've combined two datasets, it still may not be enough to achieve desired accuracy in the future. At this point we have no left out authentic data to use in training. Fortunately we can generate 'new data' from existing data by rotating/translating/scaling our digit images. We'll show how you can do this in keras."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n featurewise_center=False, # set input mean to 0 over the dataset\n samplewise_center=False, # set each sample mean to 0\n featurewise_std_normalization=False, # divide inputs by std of the dataset\n samplewise_std_normalization=False, # divide each input by its std\n zca_whitening=False, # apply ZCA whitening\n rotation_range=10, # randomly rotate images in the range (degrees, 0 to 180)\n zoom_range = 0.1, # Randomly zoom image \n width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)\n height_shift_range=0.1, # randomly shift images vertically (fraction of total height)\n horizontal_flip=False, # randomly flip images\n vertical_flip=False) # randomly flip images\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"NOTE: We need to be careful that we do not generate new images where digits are spilling to the edges. Everythime we call datagen in the future we will get data the size and shape of X_train, but randomly altered within the ranges of our initialization above. So for every epoch, our data will be slightly different than the data at the previous epoch."},{"metadata":{},"cell_type":"markdown","source":"## Model search\n\nIdeally, we would like a way to automatically search the model structure space (number of cnn layers and channel size, number of dense layers and neurons, pooling type and size, batch normalization, activation functions, drop out, etc ...) and parameter space (loss function, optimizer, learning rate, momentum, l1/l2 regularization, etc ...). For simplicity, in this notebook we assume that our model structure is static and only need to optimize over the parameter space. However, this code could be expanded to include both structure and paramter optimization.\n\nHere are the best practices for uniformly sampling a parameter [1] ...\n\n- Using random numbers instead of a discrete mesh allow us to explore more configurations and waste less time on bad configurations. We must randomly/uniformly sample a parameter $p$ in the following way; $p=[0.001,0.1]=[10^{-3},10^{-1}]$ so $r=rand(1.,3.)$ and $p=10^{-r}$\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rand_tunning(lr,opt_name,m,r,l2,l1,num_epochs,d):\n \n if lr == None:\n lr = 10**(random.uniform(-4.,-2.))#uniformly sample \n if opt_name == None:\n opt_name = random.choice(['sgd','adadelta','adagrad','adam','adamax','ftrl','nadam','rmsprop'])\n if m == None:\n m = 10**(random.uniform(-4.,-2))#uniformly sample \n if r == None:\n r = 10**(random.uniform(-1.,1.))#uniformly sample \n if l1 == None:\n l1 = 10**(random.uniform(-5.,1.))#uniformly sample \n if l2 == None:\n l2 = 10**(random.uniform(-5.,1.))#uniformly sample \n if d == None:\n d = random.uniform(0.4,0.6)#uniformly sample \n \n if opt_name == 'sgd':\n opt = tf.keras.optimizers.SGD(learning_rate=lr,momentum=m)\n if opt_name == 'adadelta':\n opt = tf.keras.optimizers.Adadelta(learning_rate=lr,rho=r)\n if opt_name == 'adagrad':\n opt = tf.keras.optimizers.Adagrad(learning_rate=lr)\n if opt_name == 'adam':\n opt = tf.keras.optimizers.Adam(learning_rate=lr)\n if opt_name == 'adamax':\n opt = tf.keras.optimizers.Adamax(learning_rate=lr)\n if opt_name == 'ftrl':\n opt = tf.keras.optimizers.Ftrl(learning_rate=lr)\n if opt_name == 'nadam':\n opt = tf.keras.optimizers.Nadam(learning_rate=lr)\n if opt_name == 'rmsprop':\n opt = tf.keras.optimizers.RMSprop(learning_rate=lr,momentum=m,rho=r)\n \n \n tunning_parameters = {'lr':lr,'opt_name':opt_name,'m':m,'r':r,'l2':l2,'l1':l1,'num_epochs':num_epochs,'d':d}\n print('tunning_parameters: ',tunning_parameters)\n \n IMG_HEIGHT=X_train.shape[1]\n IMG_WIDTH=X_train.shape[2]\n \n\n model = tf.keras.models.Sequential([\n tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH,1)),\n tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),\n tf.keras.layers.Conv2D(64, 3, padding='same', activation='relu'),\n tf.keras.layers.AveragePooling2D(pool_size=(2, 2)),\n tf.keras.layers.Conv2D(128, 3, padding='same', activation='relu'),\n tf.keras.layers.Flatten(),\n tf.keras.layers.BatchNormalization(),\n tf.keras.layers.Dense(128, activation='relu'),\n tf.keras.layers.BatchNormalization(),\n tf.keras.layers.Dropout(d),\n tf.keras.layers.Dense(128, activation='relu'),\n tf.keras.layers.BatchNormalization(),\n tf.keras.layers.Dropout(d),\n tf.keras.layers.Dense(10, activation='softmax')\n ])\n \n \n model.compile(optimizer=opt,\n loss='sparse_categorical_crossentropy',\n metrics=['accuracy'])\n callbacks = [ tf.keras.callbacks.ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', save_best_only=True)]\n \n fit = model.fit_generator(datagen.flow(X_train,Y_train),validation_data=(np.asarray(X_val), np.asarray(Y_val)), epochs=num_epochs, callbacks=callbacks)\n\n return tunning_parameters,model,fit\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model selection\n\nWe're most interested in the accuracy of our model on data outside our training set. This is why we created our validation set, so we can keep track of how we expect our model to perform on the entire population dataset. We need to make sure that our model is not biased towards our training dataset (acc. of training set is much higher than validation set) and that the accuracy of our validation set is not lower than desired. We can create 2D plots where the x-axis is the difference between training and validation accuracy and the y-axis is $1 -$ validation accuracy. In this way, the best models will be around the origin. \n\nHere's how we'll do selection ...\n\n- Set our model and number of epochs.\n- Sample the parameter space randomly/uniformly until we have enough models near the origin.\n- For a certain amount of iterations, randomly choose two models within some radius about the origin, randomly interpolate between their parameter values, and calculate the new model.\n- Finally, run the best parameter configuration again for a large number of epochs. If you didn't explore the space well enough before you may also now manually tweak some parameters (carefully) to optimize performance.\n\nWe store the x-axis and y-axis value using pickle allong with the parameter configuration to keep track of the explored paramter space. This occurs in the def bias_variance_visualization(). This means that when we change our model structure, we must empty our pickle files."},{"metadata":{"trusted":true},"cell_type":"code","source":"def bias_variance_visualization(parameters,historys):\n #need to comment if file doesn't exist yet, then uncomment\n if path.exists(\"test_tune.pkl\")==True:\n with open('test_tune.pkl', 'rb') as f:\n points = pickle.load(f)\n else:\n points=[]\n\n \n for i,history in enumerate(historys):\n train_acc_final = history['accuracy'][-1]\n test_acc_final = history['val_accuracy'][-1]\n bv_balance = [(train_acc_final-test_acc_final),1-(train_acc_final+test_acc_final)/2,parameters[i]]\n points.append(bv_balance)\n \n with open('test_tune.pkl', 'wb') as f:\n pickle.dump(points, f)\n \n radius = [math.sqrt(points[i][0]**2+points[i][1]**2) for i in range(len(points))]\n min_rad_index = [i for i,r in enumerate(radius) if r == min(radius)][0]\n best_point = points[min_rad_index][2]\n print('\\n\\nbest parameter config = ', best_point)\n \n\n \n points=list(zip(*points))\n\n plt.scatter(list(points[0]),list(points[1]))\n plt.xlim([-1.5*max(np.abs(list(points[0]))),1.5*max(np.abs(list(points[0])))])\n plt.ylim([0,1])\n plt.show()\n \n return best_point\n\n\n\ndef get_child(rad_th):\n\n with open('test_tune.pkl', 'rb') as f:\n points = pickle.load(f)\n\n rads = [[math.sqrt(p[0]**2+p[1]**2),p[2]] for p in points if math.sqrt(p[0]**2+p[1]**2)