{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n"},{"metadata":{},"cell_type":"markdown","source":"# WashU Math Undergrad Seminar Dec 9 2020\n\n## Thank you for attending and Adeli Hutton for hosting."},{"metadata":{},"cell_type":"markdown","source":"# How computers learn to recognize cats and dogs: an introduction to deep learning and the optimization methods behind the curtain\n\nWelcome to an undergrad seminar like you have never seen before! Today we will learn how computers learn to tell cats from dogs using machine learning! and some Python language.\n\n\n
\n
\nIn this notebook we will use a software package called tensorflow!\n![\"drawing\"](\"https://upload.wikimedia.org/wikipedia/commons/thumb/1/11/TensorFlowLogo.svg/1200px-TensorFlowLogo.svg.png\")
\n
\n
\n
\nWe will leave PyTorch for next semester's [Math 450: Optimization Methods in Machine Learning](https://scaomath.github.io/teaching/sp2021-math450).\n![\"drawing\"](\"https://upload.wikimedia.org/wikipedia/commons/9/96/Pytorch_logo.png\")
\n
\n
\nIf you have already registered on Kaggle, now please click the **COPY and EDIT** button on the upper right corner.\n"},{"metadata":{},"cell_type":"markdown","source":"# Notebook style Python\n\nThis is called a \"**Notebook**\".\n\nThis a markdown cell. We can write words in this cell. \n\nWelcome to our math seminar.\n\n## Command vs. Edit Modes\n\nThere two different keyboard input modes:\n\n1. **Command mode** - binds the keyboard to notebook level actions. Indicated by a grey cell border with a blue left margin.\n2. **Edit mode** - when you're typing in a cell. Indicated by a green cell border\n\nExperiment with switching between command and edit modes in this cell. \n\nHint: If you're in command mode, type `enter` or double-click to enter edit mode. If you're in edit mode, type `esc` or `cmd`+`m` (`ctrl`+`m` in Windows/Linux) to enter command mode."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"print(\"This is a code cell.\")\nprint(\"Hello world\")\nprint(f\"3+2 is {3+2}\")\n# a more advanced f-string example","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Simple variable assignment\n# This is a comment\nx = 5.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(type(x), '\\n', dir(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# simple calculations\n# ** means exponential\nprint(2**3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# simple lambda functions\nf = lambda x: x**3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Logic\n\nComputer are really good at computing, when being given **EXACT and CLEAR** instructions. For example, what is $f(5)$ if $f(x) = x^2 - x + 2$, or if something is true, do another thing. However, computer is not so good at many things (used to). For examples, recognizing cats and dogs from photos. \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# == vs =\na = 2 # we let a be 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a == 3 # check if a is 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# simple if-then condition\n# flow control","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a = 1\nif a == 2:\n print(f'a is {a}')\nelse:\n print('Nothing')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# browse and introduce the Cats vs dog competition","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Computer can learn from examples!\n\nJust like us! Imagine we are preparing an exam, teachers will give us some practice exams (with answers available), we will train ourselves by doing these practice problems, honing our skills, then in the actual exam, we will be able to tackle exam problems without knowing the answer beforehand (hopefully). The examples are call data.\n\nImaging how computers learn calculus just by looking at the problems and solutions (no theorems)...\n\nFirst let us load some packages into our system."},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport random","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dir(pd)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir(\"../input/dogs-vs-cats/\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us unzip the compressed images in the `train.zip` and `test1.zip` (this may take a while)."},{"metadata":{"trusted":true},"cell_type":"code","source":"!unzip -q '../input/dogs-vs-cats/train.zip'\n!unzip -q '../input/dogs-vs-cats/test1.zip'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"filenames = os.listdir(\"./train\")\nprint(filenames[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In a computer system, we need to represent \"cat\" or \"dog\" these abstract words into 0s and 1s so that computer can understand! We store our data in a Dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"categories = []\nfor filename in filenames:\n category = filename.split('.')[0]\n if category == 'dog':\n categories.append(1)\n else: # cat\n categories.append(0)\n\ndf = pd.DataFrame({\n 'filename': filenames,\n 'category': categories\n})\nprint(df.head(20))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us view a sample image (randomly chosen)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import load_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = df.sample(1)\nimage = load_img(\"./train/\"+sample['filename'].values[0])\nfig = plt.figure()\nfig.set_size_inches(6,6)\nplt.imshow(image)\nprint(sample)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Deep learning model"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import layers, applications, optimizers, callbacks\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense, Activation,GlobalMaxPooling2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications import VGG16\nfrom keras.models import Model, load_model\nfrom keras.utils import plot_model, to_categorical\n\nimage_size = 224\ninput_shape = (image_size, image_size, 3)\n\nepochs = 6\nbatch_size = 16\n\npre_trained_model = VGG16(input_shape=input_shape, include_top=False, weights=\"imagenet\")\n \nfor layer in pre_trained_model.layers[:15]:\n layer.trainable = False\n\nfor layer in pre_trained_model.layers[15:]:\n layer.trainable = True\n \nlast_layer = pre_trained_model.get_layer('block5_pool')\nlast_output = last_layer.output\n \n# Flatten the output layer to 1 dimension\nx = GlobalMaxPooling2D()(last_output)\n# Add a fully connected layer with 512 hidden units and ReLU activation\nx = Dense(512, activation='relu')(x)\n# Add a dropout rate of 0.3\nx = Dropout(0.3)(x)\n# Add a final sigmoid layer for classification\nx = layers.Dense(1, activation='sigmoid')(x)\n\nmodel = Model(pre_trained_model.input, x)\n\nmodel.compile(loss='binary_crossentropy',\n optimizer=optimizers.Adam(lr=1e-4),\n metrics=['accuracy'])\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_model(model, to_file='/model_vgg16.png', show_shapes=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing the data for the model\n\n- `train_df`: data for training the model.\n- `validate_df`: data for validating the trained model (the model has not seen these data before)."},{"metadata":{"trusted":true},"cell_type":"code","source":"df['category'] = df['category'].astype('str')\ntrain_df, validate_df = train_test_split(df, test_size=0.1)\ntrain_df = train_df.reset_index()\nvalidate_df = validate_df.reset_index()\n\n# validate_df = validate_df.sample(n=100).reset_index() # use for fast testing code purpose\n# train_df = train_df.sample(n=1800).reset_index() # use for fast testing code purpose\n\ntotal_train = train_df.shape[0]\ntotal_validate = validate_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n rotation_range=16,\n rescale=1./255,\n shear_range=0.2,\n zoom_range=0.2,\n horizontal_flip=True,\n fill_mode='nearest',\n width_shift_range=0.1,\n height_shift_range=0.1\n)\n\ntrain_generator = train_datagen.flow_from_dataframe(\n train_df, \n \"./train/\", \n x_col='filename',\n y_col='category',\n class_mode='binary',\n target_size=(image_size, image_size),\n batch_size=batch_size\n)\n\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\nvalidation_generator = validation_datagen.flow_from_dataframe(\n validate_df, \n \"./train/\", \n x_col='filename',\n y_col='category',\n class_mode='binary',\n target_size=(image_size, image_size),\n batch_size=batch_size\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augment the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_df = train_df.sample(n=1).reset_index(drop=True)\nexample_generator = train_datagen.flow_from_dataframe(\n example_df, \n \"./train/\", \n x_col='filename',\n y_col='category',\n# class_mode='binary'\n)\nplt.figure(figsize=(12, 12))\nfor i in range(0, 9):\n plt.subplot(3, 3, i+1)\n for X_batch, Y_batch in example_generator:\n image = X_batch[0]\n plt.imshow(image)\n break\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test if our model can recognize this image!"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_filenames = os.listdir(\"./test1/\")\ntest_df = pd.DataFrame({\n 'filename': test_filenames[:64]\n})\n\nnb_samples = test_df.shape[0]\ntest_gen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_gen.flow_from_dataframe(\n test_df, \n \"./test1/\", \n x_col='filename',\n y_col=None,\n class_mode=None,\n batch_size=batch_size,\n target_size=(image_size, image_size),\n shuffle=False\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# this may take a while\npredict = model.predict_generator(test_generator, steps=np.ceil(nb_samples/batch_size))\nthreshold = 0.5\ntest_df['category'] = np.where(predict > threshold, 1,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## check prediction results\n\nWithout training, the model just assigns every images it sees as a \"dog\" (\"1\" label), also the `dog` or `cat` strings are removed from the testing image filenames to avoid \"cheating\"."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_test = test_df.sample(n=9).reset_index()\nsample_test.head()\nplt.figure(figsize=(12, 12))\nfor index, row in sample_test.iterrows():\n filename = row['filename']\n category = row['category']\n category = 'cat' if category == 0 else 'dog'\n img = load_img(\"./test1/\"+filename, target_size=(256, 256))\n plt.subplot(3, 3, index+1)\n plt.imshow(img)\n plt.xlabel(f'{filename} : {category} ')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Now let's load a trained model\n\nThe prediction is like a random guess. Imaging we have spent three days doing practice exam for the actual exam!"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = load_model('../input/vgg16catsvsdogs/model_0_vgg16.h5')\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let this model see the images again and check the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"# this may take a while\npredict = model.predict_generator(test_generator, steps=np.ceil(nb_samples/batch_size))\nthreshold = 0.5\ntest_df['category'] = np.where(predict > threshold, 0, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_test = test_df.sample(n=9).reset_index()\nsample_test.head()\nplt.figure(figsize=(12, 12))\nfor index, row in sample_test.iterrows():\n filename = row['filename']\n category = row['category']\n category = 'cat' if category == 0 else 'dog'\n img = load_img(\"./test1/\"+filename, target_size=(256, 256))\n plt.subplot(3, 3, index+1)\n plt.imshow(img)\n plt.xlabel(f'{filename} : {category} ')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pretty accurate! isn't it"},{"metadata":{},"cell_type":"markdown","source":"# How this computer algorithm achieves that?!\nlong story...first we have to learn how computer represent images."},{"metadata":{},"cell_type":"markdown","source":"\n\n### Computer stores image as a \"matrix\""},{"metadata":{"trusted":true},"cell_type":"code","source":"# scala, vector\na = 1\nv = [1,2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# matrix\nm = [[1,2], [3,4]]\nprint(np.array(m))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# tensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## This is a tensor\n\n\n\nReference: Introduction to Tensors at TensorFlow guide. https://www.tensorflow.org/guide/tensor"},{"metadata":{"trusted":true},"cell_type":"code","source":"# example of imshow\na = np.array([[0,4], [2,10]])\nplt.imshow(a);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let us load an image from Pokemon dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"pokemon_filename = os.listdir(\"../input/pokemon-images-dataset/pokemon_jpg/pokemon_jpg/\")\nrandom_pokemon = random.choice(pokemon_filename)\nG = plt.imread(\"../input/pokemon-images-dataset/pokemon_jpg/pokemon_jpg/\"+random_pokemon)\nplt.imshow(G)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"random_pokemon","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## But what is G????"},{"metadata":{"trusted":true},"cell_type":"code","source":"# check G\ntype(G)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"G.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# G is a tensor!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# show only 1 color channel\nG1 = G[:,:,0]\nplt.imshow(G1, cmap='Reds');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## computer stores these images as tensors!\n"},{"metadata":{},"cell_type":"markdown","source":"# How computer learns?\n\nWe tranform the problem into an optimization problem! First the neural network is a nonlinear function: \n$$\n\\hat{y} = h(x; w), \\quad \\text{vs } y\n$$\nwhere $x$ is the datum (sample, matrix or tensor), $\\hat{y}$ is the output of the model, $y$ is called the ground truth, and $w$ is the parameter of our model.\n\nIf $y$ are just a real scalar value (for example, stock prices) then we can tweak our model's $w$ by solving the following minimization problem: $i$ stands for indices for $i$-th samples:\n$$\n\\min_{w\\in \\Omega} L(w):= \\min_{w\\in \\Omega} \\|h(x; w) - y\\|^2,\n$$\nwhere $L(w)$ is called the loss function (it is a function in $w$!!!).\n\nYou might have seen this picture:\n\n\nEach layer can be written as the following:\n$$\na^{(l+1)} = \\sigma(W a^{(l)} + b)\n$$\n
\n
\n
\nThe VGG16 is a deep convolutional neural network, and this is a miniature of our model: imagine those little blocks are like magnifiers + translators.\n"},{"metadata":{},"cell_type":"markdown","source":"## Cross-entropy\n\nWhat if the ground truth we are interested in is a probability distribution, for example, if $x$ stands for an image of a dog:\n$$\nP(y = 1| x) = 1 \\text{ and } P(y= 0 | x) = 0.\n$$\nSo we are really interested in approximating $P(y|\\mathbf{x})$:\n\n$$\nh(\\mathbf{x}) := h(\\mathbf{x};\\mathbf{w}) = \\frac{1}{1 + \\exp(-\\mathbf{w}^\\top \\mathbf{x})}\n=: \\sigma(\\mathbf{w}^\\top \\mathbf{x}) \\in (0,1)\n$$\n\nwhere $\\sigma(z)$ is the Sigmoid function $1/(1+e^{z})$\nor more compactly.\n\nNow $h(\\mathbf{x})$ is our estimate of $ P(y=1|\\mathbf{x})$ (conditional probability of giving sample $\\mathbf{x}$, it is in class 1), and $1 - h(\\mathbf{x})$ is our estimate of $P(y=0|\\mathbf{x}) = 1 - P(y=1|\\mathbf{x})$, moreover, because $y = 0$ or $1$, \n\n$$\nP(y|\\mathbf{x}) \\text{ is estimated by } h(\\mathbf{x})^y \\big(1 - h(\\mathbf{x}) \\big)^{1-y}.\n$$\n\nWhen the true $y$ is 1, we want $h(\\mathbf{x})$ closer to 1, and vice versa.\n\nThe cross entropy loss for two probability distribution is defined as, $K=2$ is the no. of classes, $\\hat {y}$ is the prediction from the model (try to estimate $y$)\n\n$$\nH(p,q)\\ =\\ -\\sum^{K}_{k=1}p_{k}\\log q_{k}\\ =\\ -y\\log {\\hat {y}}-(1-y)\\log(1-{\\hat {y}})\n$$\n\nSince we estimate $y$ using $h(\\mathbf{x})$,\n\n$$\nL (\\mathbf{w}; X, \\mathbf{y}) = - \\frac{1}{N}\\sum_{i=1}^N \n\\Bigl\\{y^{(i)} \\ln\\big( h(\\mathbf{x}^{(i)}; \\mathbf{w}) \\big) \n+ (1 - y^{(i)}) \\ln\\big( 1 - h(\\mathbf{x}^{(i)};\\mathbf{w}) \\big) \\Bigr\\}.\n\\tag{$\\star$}\n$$\n\nand the minimization problem we are solving is:\n\n$$\n\\min_{\\mathbf{w}} L (\\mathbf{w}; X, \\mathbf{y})\n$$"},{"metadata":{},"cell_type":"markdown","source":"# Stochastic Gradient descent\n\nLoss\n\n$$L(\\mathbf{w}) := L(\\mathbf{w}; X,\\mathbf{y}) = \\frac{1}{N}\\sum_{i=1}^N f_i(\\mathbf{w}; \\mathbf{x}^{(i)},y^{(i)})$$ \n\n# Gradient descent for this loss:\n> Choose an initial guess $\\mathbf{w}_0$, step size (learning rate) $\\eta$, number of iterations $M$
\n> For $k=0,1,2, \\cdots, M$
\n> $\\displaystyle\\mathbf{w}_{k+1} = \\mathbf{w}_k - \\eta\\nabla_{\\mathbf{w}} L(\\mathbf{w}_k) = \\mathbf{w}_k - \\eta\\frac{1}{N}\\sum_{i=1}^N \\nabla_{\\mathbf{w}} f_i(\\mathbf{w}; \\mathbf{x}^{(i)},y^{(i)})$\n\n\n### SGD\n* > Choose an initial guess $\\mathbf{w}_0$, step size (learning rate) $\\eta$, number of iterations $M$
\n> For $k=0,1,2, \\cdots, M$
\n> $\\displaystyle\\mathbf{w}_{k+1} = \\mathbf{w}_k - \\eta\\nabla_{\\mathbf{w}} L(\\mathbf{w}_k) = \\mathbf{w}_k - \\eta\\frac{1}{n_{\\text{batch}}}\\sum_{i=1}^{n_{\\text{batch}}} \\nabla_{\\mathbf{w}} f_i(\\mathbf{w}; \\mathbf{x}^{(i)},y^{(i)})$"},{"metadata":{},"cell_type":"markdown","source":"# Summary:\n\n1. Computer represents data as vector, matrix, or tensor.\n2. A deep learning model learns how to classify images through optimization.\n3. Our Neural network model (VGG16) uses mathematical operations to extract features from images.\n4. The model is \"trained\" through solving an optimization problem.\n5. We will learn how to code these in Math 450 in Spring 2021.\n\nEmail: s.cao@wustl.edu"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}