{"cells":[{"metadata":{"colab_type":"text","id":"lYUHk6sxO2pO"},"cell_type":"markdown","source":"# DSCI 572 Lab 4"},{"metadata":{"colab":{},"colab_type":"code","id":"AoVG1Z3FO2pQ","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nfrom sklearn.model_selection import train_test_split\nfrom scipy.signal import convolve2d\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom skimage.color import rgb2gray\nfrom skimage.transform import resize\n\nimport tensorflow as tf\n\nfrom tensorflow.keras.datasets import mnist\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Activation, Conv2D, MaxPooling2D, GlobalAveragePooling2D\nfrom tensorflow.keras.models import Sequential, Model, load_model\nfrom tensorflow.keras.optimizers import Adam, SGD\nfrom tensorflow.keras.preprocessing.image import img_to_array, load_img\nfrom tensorflow.keras import utils\nfrom tensorflow.keras.applications.inception_v3 import InceptionV3\n\n# se imports\nimport time\nfrom sklearn.dummy import DummyClassifier","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"G-ePax-DO2pS","trusted":true},"cell_type":"code","source":"plt.rcParams['font.size'] = 16","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"BYYZaUzyO2qN"},"cell_type":"markdown","source":"## Exercise 2. Convolutional networks for MNIST\n\nSorry to continue with MNIST so long. It's just _THE_ classic data set for this stuff.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# define a simple CNN model\ndef build_mnist_CNN():\n    mnist_model = Sequential()\n    mnist_model.add(Conv2D(32, (5, 5), input_shape=(28, 28, 1), activation='relu'))\n    mnist_model.add(MaxPooling2D(pool_size=(2, 2)))\n    mnist_model.add(Dropout(0.2))\n    mnist_model.add(Flatten())\n    mnist_model.add(Dense(128, activation='relu'))\n    mnist_model.add(Dense(num_classes, activation='softmax'))\n\n    # Compile model\n    mnist_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    return mnist_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Run with 10,000 training samples"},{"metadata":{"colab":{},"colab_type":"code","id":"vc8-vt_WO2qQ","trusted":true},"cell_type":"code","source":"# load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\n\n# reshape to be [samples][channels][width][height]\nX_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype('float32')\nX_test = X_test.reshape(X_test.shape[0], 28, 28, 1).astype('float32')\n\n# normalize inputs from 0-255 to 0-1\nX_train = X_train / 255\nX_test = X_test / 255\n\n# one hot encode outputs\ny_train = utils.to_categorical(y_train)\ny_test = utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]\n\n# take a subset of the data for speed\nsubset_size = 10000\nX_train = X_train[:subset_size]\ny_train = y_train[:subset_size]\n\nmnist_model = build_mnist_CNN()\n\n# Fit the model\nstart_time = time.time()\nmnist_model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=256)\n# Final evaluation of the model\nscores = mnist_model.evaluate(X_test, y_test, verbose=0)\nprint(\"CNN Error: %.2f%%\" % (100-scores[1]*100))\nelapsed_time = time.time()-start_time\nprint(\"---Running Time: %s seconds ---\" % elapsed_time)\n\nmnist_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Run with full 60,000 samples:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\n\n# reshape to be [samples][channels][width][height]\nX_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype('float32')\nX_test = X_test.reshape(X_test.shape[0], 28, 28, 1).astype('float32')\n\n# normalize inputs from 0-255 to 0-1\nX_train = X_train / 255\nX_test = X_test / 255\n\n# one hot encode outputs\ny_train = utils.to_categorical(y_train)\ny_test = utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]\n\n# take a subset of the data for speed\n# subset_size = 10000\n# X_train = X_train[:subset_size]\n# y_train = y_train[:subset_size]\n\nmnist_model = build_mnist_CNN()\n\n# Fit the model\nstart_time = time.time()\nmnist_model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=256)\n# Final evaluation of the model\nscores = mnist_model.evaluate(X_test, y_test, verbose=0)\nprint(\"CNN Error: %.2f%%\" % (100-scores[1]*100))\nelapsed_time = time.time()-start_time\nprint(\"---Running Time: %s seconds ---\" % elapsed_time)\n\nmnist_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"markdown","source":"# Exercise 3: Transfer learning"},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.read_csv('../input/dog-breed-identification/labels.csv')\ndata = data[:2000]\ndata['image_path'] = data.apply( lambda row: (os.path.join(\"../input/dog-breed-identification/train\", row[\"id\"] + \".jpg\") ), axis=1)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target_labels = data['breed']\ntotal_classes = len(set(target_labels))\nprint(\"number of dog breeds:\", total_classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# read images from the image directory. \nimages = np.array([img_to_array(\n                    load_img(img, target_size=(256,256))\n                    ) for img in data['image_path'].values.tolist()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = images.astype('float32')/255.0 ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(images[0]);\nplt.grid(True);\nplt.xticks([]);\nplt.yticks([]);\nplt.title(\"Breed = \" + target_labels[0]);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid = train_test_split(images, target_labels, \n                                                    stratify=np.array(target_labels), \n                                                    random_state=42)\n\nprint(X_train.shape)\nprint(X_valid.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 3(a)\nrubric={reasoning:10}\n\nBefore we start, do some EDA to assess whether there is serious class imbalance in the training data. What training accuracy would you get with `DummyClassifier`? Briefly discuss your results."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.value_counts().hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> There is some class inbalance. The histogram shows the distribution of class size. The distribution is very roughly Gaussian. For some breeds we have many observations (20), and for some breeds we only have a few.\n>\n> Using the dummy classifier with the `most_frequent` strategy, we would expect a very high error rate. The code below demonstrates an error rate of ~ 98.6%. This makes sense since the model is just predicting the most common occurence everytime.\n>\n> It looks like `bernese_mountain_dog` is the most common class with 20 occurences. Based on this, we would expect the model to only have an accuracy of 20/1500."},{"metadata":{"trusted":true},"cell_type":"code","source":"1 - 20/1500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dummy = DummyClassifier(strategy=\"most_frequent\").fit(X_train, y_train)\nprint(f\"Train error: {1 - dummy.score(X_train, y_train)}\")\nprint(f\"Valid error: {1 - dummy.score(X_valid, y_valid)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"#### 3(b)\nrubric={reasoning:5}\n\nHow many training examples do we have per breed of dog, roughly? In the context of other classification tasks we've done in MDS, do you consider this to be a lot or a little?"},{"metadata":{},"cell_type":"markdown","source":"> The histogram above demonstrates how many training samples we have roughly per breed of dog. Most breeds only have 8 to 14 training samples. This does not seem like very many, and is much less than what we are used to for most MDS problems we have tackled to date."},{"metadata":{"trusted":true},"cell_type":"code","source":"# OHE\n\nY_train = pd.get_dummies(y_train.reset_index(drop=True)).values\nY_valid = pd.get_dummies(y_valid.reset_index(drop=True)).values\n\nprint(Y_train.shape)\nprint(Y_valid.shape)\n\n# Note: it would be better to use keras.utils.to_categorical, or something else like that,\n# just in case one of the classes is absent in one of the two sets.\n# But this works for now.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Approach 1\n\nNow, we try Approach 1, which is training an end-to-end CNN on the dog breed classification task."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(32, (3, 3), input_shape=(256, 256, 3)))\nmodel.add(Activation('relu')) # this is just different syntax for specifying the activation function\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(32, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(64, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten())\nmodel.add(Dense(64))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(total_classes))\nmodel.add(Activation('softmax'))\n\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\nmodel.summary()\n\nhistory = model.fit(X_train, Y_train, epochs=10, validation_data=(X_valid, Y_valid))\n\n# FYI: it's often a good idea to save your weights after training or during training.\n# But you don't have to here.\n# model.save_weights('my_conv_net.h5')  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate(X_valid, Y_valid, verbose=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 3(c)\nrubric={reasoning:1}\n\nWhat do you think of the results? Are you impressed? "},{"metadata":{},"cell_type":"markdown","source":"> I am not very impressed with the results. The validation accuracy of ~ 0.02 is not very good. My assumption is that our small number of training examples for each class is hurting our results"},{"metadata":{},"cell_type":"markdown","source":"### Approach 2\n\nHere we load a pre-trained model and add some layers on top. The syntax is not what you're used to - that's OK, don't worry about it."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get the InceptionV3 model trained on the ImageNet data set\nbase_inception = InceptionV3(weights='imagenet', include_top=False, input_shape=(256, 256, 3))\n\ntop_block = base_inception.output\ntop_block = GlobalAveragePooling2D()(top_block) # pool over height/width to reduce number of parameters\ntop_block = Dense(256, activation='relu')(top_block) # add a Dense layer\npredictions = Dense(total_classes, activation='softmax')(top_block) # add another Dense layer\n\nmodel_transfer = Model(inputs=base_inception.input, outputs=predictions)\n\nfor layer in base_inception.layers:\n    layer.trainable = False\n    \nmodel_transfer.compile(Adam(lr=.001), loss='categorical_crossentropy', metrics=['accuracy']) \nmodel_transfer.summary() # run me if you dare\nhistory = model_transfer.fit(X_train, Y_train, validation_data=(X_valid, Y_valid), epochs=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_transfer.evaluate(X_valid, Y_valid, verbose=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 3(d)\nrubric={reasoning:1}\n\nHow does this result compare to the \"from scratch\" CNN?"},{"metadata":{},"cell_type":"markdown","source":"> The above code made some major improvements compared to the \"from scratch\" CNN. The validation accuracy has increased to ~ 75%."},{"metadata":{},"cell_type":"markdown","source":"### Approach 3\n\nBelow, we un-freeze the last \"15\" layers, which is really only the last one or two layers, since the list of Keras layer objects doesn't really correspond to our idea of a layer (see `model.summary()`)."},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, layer in enumerate(reversed(model_transfer.layers)):\n    layer.trainable = True\n#     print(layer)\n    if i > 15:\n        break\n\n# compile the model with a SGD/momentum optimizer and a very slow learning rate.\nmodel_transfer.compile(loss='categorical_crossentropy',\n              optimizer=SGD(lr=1e-4, momentum=0.9),\n              metrics=['accuracy'])\n\n# fine-tune the unfrozen layers\nhistory = model_transfer.fit(X_train, Y_train, validation_data=(X_valid, Y_valid), epochs=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_transfer.evaluate(X_valid, Y_valid, verbose=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### (optional) 3(e)\nrubric={reasoning:1}\n\nUn-freezing some of the layers seems to have a small effect here. Was it actually useful at all, or could we have achieved the same results by just training our top layers for more epochs?"},{"metadata":{},"cell_type":"markdown","source":"> Unfreezing additional layers does not seem useful in this case. The bump in validation accuracy was very small. Assuming that the original layers were well trained on the additional training class it is probably best to not re-train, and just do more epochs on the top layers."},{"metadata":{},"cell_type":"markdown","source":"#### 3(f)\nrubric={reasoning:5}\n\nIn Lab 3 we noticed that unlike scikit-learn's `fit`, Keras's `fit` doesn't re-initialize the weights, but rather continues on from where you were. In the above code, we benefitted from this. Briefly describe how/why this behaviour was useful to us."},{"metadata":{},"cell_type":"markdown","source":"> This is useful because it means that hopefully we are starting with weights that have already been optimized. Best case, the weights were optimized by someone who knows what they are doing, and had access to a really powerful computer. This saves us a lot of compute time and power if we are starting with meaningful weights instead of randomly initializing them. It should take less epochs and iterations to converge to a minimum."},{"metadata":{},"cell_type":"markdown","source":"#### 3(g)\nrubric={reasoning:10}\n\nBrainstorm 3 other applications of this type of transfer learning, where you use a pre-trained network plus some modifications. In each case, what is the original task and what is the new task? (It's OK if you don't actually have access to a pre-trained network to do the original task; we're just brainstorming here.)"},{"metadata":{},"cell_type":"markdown","source":"> **(1) Self Driving cars and existing computer vision**\n> - There are many existing models for image classification\n> - To train self driving cars could be a timley endeavor, because you would need to rack up many millions of miles of real people driving to get good training data\n> - Self driving car models could \"shortcut\" some of this by using transfer learning from existing image classifications. These existing image classifications may already be able to identify useful things (such as stop signs), and also may be able to extract useful features that are relevant to this task (where a road ends, where water is, etc.)\n>\n> **(2) Sentiment Classification**\n> - Imagine in 20 years from now, everyday langauge will have probably changed a little bit (new sayings, new slang terms, etc.).\n> - Old sentiment classifications may not work as good as they used to.\n> - Instead of building completely new models, you could start with as a base an existing model. Then continue to train it on new data.\n> - This will likely reduce the number of new training samples you need because the model will already be good at identifying features that lead to positive or negative sentiment.\n>\n> **(3) Writing code in a new langauge**\n> - There are many existing AI tools that currently exist that can right code, or provide reccomendations for code.\n> - Imagine a new language is created.\n> - It may require a lot of training data (that may not exist yet) to train an AI tool that can write code or provide code completion reccomendations.\n> - Intead of starting from scratch you could use an existing model, say for Python. It will likely be good at identifying sctructres that are common accross many langauges such as for loops."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"accelerator":"GPU","colab":{"name":"lab4.ipynb","provenance":[],"version":"0.3.2"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":1}