{"cells":[{"metadata":{},"cell_type":"markdown","source":"![Washu Math](https://sites.wustl.edu/scao/files/2020/10/Screen-Shot-2020-10-25-at-1.03.49-PM.png)\n"},{"metadata":{},"cell_type":"markdown","source":"# Math Circle Fall 2020 Introduction\n\nWelcome to a math circle session like you have never seen before! Today we will learn how computers learn to tell cats from dogs using machine learning! and some Python language.\n\n![](https://www.python.org/static/community_logos/python-logo-master-v3-TM.png)\n\nIf you have already registered on Kaggle, now please click the **COPY and EDIT** button on the upper right corner.\n![](https://sites.wustl.edu/scao/files/2020/10/Screen-Shot-2020-10-25-at-1.09.19-PM.png)"},{"metadata":{},"cell_type":"markdown","source":"This a markdown cell. We can write words in this cell. \n\n## Headers\nHello\n\n### smaller header"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"print(\"This is a code cell.\")\nprint(\"Hello world\")\nprint(f\"Hello world, time is {3} o'clock\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Simple variable assignment\n# This is a comment\n# x variable like 3x = 5\nx = 5/3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y = x*3\nprint(y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# simple calculations\n# ** means exponential\nprint(2**5)\nprint(2**3*3) # computer 2**3 first, 8 is multiplied with 3\nprint(1/2+1) # 1/2 first, +1 computed secondly","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(2***5) # unfortunately tetration is not built-in Python","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Logic\nComputer are really good at computing (recognizing numbers), following **EXACT and CLEAR** instructions. However, computer is not so good at many things (used to). For examples, recognizing cats and dogs from photos. \n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/3362/media/woof_meow.jpg)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# browse the Cats vs dog competition","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# == vs =\na = 2 # we let a be 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(a)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"dir(a)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# == is checking whether a variable is equal to whatever on the right side of ==\na == 3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"s = 'David'\nprint(type(s))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"False is not True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a = 5.\ntype(a)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a = 7\ntype(a)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"s1 = [a, s]\nprint(s1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type({2,3})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dict1 = {'Shuhao': 'instructor'}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# simple logical checking == or using \"is\"\ndict1['Shuhao'] == 'student'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"1 is 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# simple if-then condition\n# flow control\n\nif 1 is 1:\n    print(\"1 is 1\")\n\nif 1 is 0:\n    print(\"1 is 0\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# range(a,b) the integer greater than or equal to a but less than b\nfor i in range(0,5):\n    if i > 2: # press Tab to indent\n        print(i)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Computer can learn from examples!\n\nJust like us! Imagine we are preparing an exam, teachers will give us some practice exams (with answers available), we will train ourselves by doing these practice problems, honing our skills, then in the actual exam, we will be able to tackle exam problems without knowing the answer beforehand (hopefully). The examples are call data.\n\nFirst let us load some packages into our system."},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport random","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dir(pandas)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"dir(pd)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import site\nsite.getsitepackages()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"help(pd)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir(\"../input/dogs-vs-cats/\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us unzip the compressed images in the `train.zip` and `test1.zip` (this may take a while)."},{"metadata":{"trusted":true},"cell_type":"code","source":"!unzip -q '../input/dogs-vs-cats/train.zip'\n!unzip -q '../input/dogs-vs-cats/test1.zip'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"filenames = os.listdir(\"./train\")\nprint(filenames[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In a computer system, we need to represent \"cat\" or \"dog\" these abstract words into 0s and 1s so that computer can understand! We store our data in a Dataframe (like the inventory list in Minecraft)."},{"metadata":{"trusted":true},"cell_type":"code","source":"categories = []\nfor filename in filenames:\n    category = filename.split('.')[0]\n    if category == 'dog':\n        categories.append(0)\n    else:\n        categories.append(1)\n\ndf = pd.DataFrame({\n    'filename': filenames,\n    'category': categories\n})\nprint(df.head(20))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us view a sample image (randomly chosen)."},{"metadata":{"trusted":true},"cell_type":"code","source":"len(filenames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import load_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = random.choice(filenames)\nimage = load_img(\"./train/\"+sample)\nfig = plt.figure()\nfig.set_size_inches(6,6)\nplt.imshow(image);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Deep learning model"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import layers, applications, optimizers, callbacks\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense, Activation,GlobalMaxPooling2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications import VGG16\nfrom keras.models import Model, load_model\nfrom keras.utils import plot_model, to_categorical\n\nimage_size = 224\ninput_shape = (image_size, image_size, 3)\n\nepochs = 6\nbatch_size = 16\n\npre_trained_model = VGG16(input_shape=input_shape, include_top=False, weights=\"imagenet\")\n    \nfor layer in pre_trained_model.layers[:15]:\n    layer.trainable = False\n\nfor layer in pre_trained_model.layers[15:]:\n    layer.trainable = True\n    \nlast_layer = pre_trained_model.get_layer('block5_pool')\nlast_output = last_layer.output\n    \n# Flatten the output layer to 1 dimension\nx = GlobalMaxPooling2D()(last_output)\n# Add a fully connected layer with 512 hidden units and ReLU activation\nx = Dense(512, activation='relu')(x)\n# Add a dropout rate of 0.3\nx = Dropout(0.3)(x)\n# Add a final sigmoid layer for classification\nx = layers.Dense(1, activation='sigmoid')(x)\n\nmodel = Model(pre_trained_model.input, x)\n\nmodel.compile(loss='binary_crossentropy',\n              optimizer=optimizers.Adam(lr=1e-4),\n              metrics=['accuracy'])\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"plot_model(model, to_file='/model_vgg16.png', show_shapes=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing the data for the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"df['category'] = df['category'].astype('str')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df, validate_df = train_test_split(df, test_size=0.1)\ntrain_df = train_df.reset_index()\nvalidate_df = validate_df.reset_index()\n\n# validate_df = validate_df.sample(n=100).reset_index() # use for fast testing code purpose\n# train_df = train_df.sample(n=1800).reset_index() # use for fast testing code purpose\n\ntotal_train = train_df.shape[0]\ntotal_validate = validate_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n    rotation_range=16,\n    rescale=1./255,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest',\n    width_shift_range=0.1,\n    height_shift_range=0.1\n)\n\ntrain_generator = train_datagen.flow_from_dataframe(\n    train_df, \n    \"./train/\", \n    x_col='filename',\n    y_col='category',\n    class_mode='binary',\n    target_size=(image_size, image_size),\n    batch_size=batch_size\n)\n\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    validate_df, \n    \"./train/\",  \n    x_col='filename',\n    y_col='category',\n    class_mode='binary',\n    target_size=(image_size, image_size),\n    batch_size=batch_size\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augment the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_df = train_df.sample(n=1).reset_index(drop=True)\nexample_generator = train_datagen.flow_from_dataframe(\n    example_df, \n    \"./train/\", \n    x_col='filename',\n    y_col='category',\n#     class_mode='binary'\n)\nplt.figure(figsize=(12, 12))\nfor i in range(0, 9):\n    plt.subplot(3, 3, i+1)\n    for X_batch, Y_batch in example_generator:\n        image = X_batch[0]\n        plt.imshow(image)\n        break\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test if our model can recognize this image!"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_filenames = os.listdir(\"./test1/\")\ntest_df = pd.DataFrame({\n    'filename': test_filenames[:128]\n})\n\nnb_samples = test_df.shape[0]\ntest_gen = ImageDataGenerator(rescale=1./255)\ntest_generator = test_gen.flow_from_dataframe(\n    test_df, \n    \"./test1/\", \n    x_col='filename',\n    y_col=None,\n    class_mode=None,\n    batch_size=batch_size,\n    target_size=(image_size, image_size),\n    shuffle=False\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# this may take a while\npredict = model.predict_generator(test_generator, steps=np.ceil(nb_samples/batch_size))\nthreshold = 0.5\ntest_df['category'] = np.where(predict > threshold, 1,0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## check prediction results\n\nWithout training, the model just assigns every images it sees as a \"dog\" (\"0\" label)."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_test = test_df.sample(n=9).reset_index()\nsample_test.head()\nplt.figure(figsize=(12, 12))\nfor index, row in sample_test.iterrows():\n    filename = row['filename']\n    category = row['category']\n    img = load_img(\"./test1/\"+filename, target_size=(256, 256))\n    plt.subplot(3, 3, index+1)\n    plt.imshow(img)\n    plt.xlabel(filename + '(' + \"{}\".format(category) + ')')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Now let's load a trained model\n\nImaging we have spent three days doing practice exam for the actual exam!"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = load_model('../input/vgg16catsvsdogs/model_0_vgg16.h5')\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let this model see the images again and check the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"# this may take a while\npredict = model.predict_generator(test_generator, steps=np.ceil(nb_samples/batch_size))\nthreshold = 0.5\ntest_df['category'] = np.where(predict > threshold, 1,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_test = test_df.sample(n=9).reset_index()\nsample_test.head()\nplt.figure(figsize=(12, 12))\nfor index, row in sample_test.iterrows():\n    filename = row['filename']\n    category = row['category']\n    img = load_img(\"./test1/\"+filename, target_size=(256, 256))\n    plt.subplot(3, 3, index+1)\n    plt.imshow(img)\n    plt.xlabel(filename + '(' + \"{}\".format(category) + ')')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pretty accurate! isn't it"},{"metadata":{},"cell_type":"markdown","source":"# How this computer algorithm achieves that?!\nlong story...first we have to learn how computer represent images."},{"metadata":{},"cell_type":"markdown","source":"![](https://sites.wustl.edu/scao/files/2020/10/linear_dogs.jpg)\n\n### Computer stores image as a \"matrix\""},{"metadata":{"trusted":true},"cell_type":"code","source":"# example of scalar, vector, matrix\ns = 1.34992","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"v = [1, 3, 5.2]\nprint(v)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m = [[1,3], [8,9]]\nprint(m)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m = np.array(m)\nprint(type(m))\nprint(m)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# example of imshow\na = np.array([[0,4], [2,10]])\nplt.imshow(a);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let us load an image from Pokemon dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"pokemon_filename = os.listdir(\"../input/pokemon-images-dataset/pokemon_jpg/pokemon_jpg/\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"random_pokemon = random.choice(pokemon_filename)\nG = plt.imread(\"../input/pokemon-images-dataset/pokemon_jpg/pokemon_jpg/\"+random_pokemon)\nplt.imshow(G)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## But what is G????"},{"metadata":{"trusted":true},"cell_type":"code","source":"G","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"G.shape # 3 represents the Red, Green, Blue color ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# indexing\nG1 = G[:,:,0] # take the red channel","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"G1.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# G is a tensor!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# show only 1 color channel\nplt.imshow(G1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## computer stores these images as tensors!\n\nThe name of the software package we are using is called tensorflow!\n![](https://upload.wikimedia.org/wikipedia/commons/thumb/1/11/TensorFlowLogo.svg/1200px-TensorFlowLogo.svg.png)"},{"metadata":{},"cell_type":"markdown","source":"# How computer learns?\n\nLet us load some pokemon data."},{"metadata":{"trusted":true},"cell_type":"code","source":"pkmn = pd.read_csv('../input/pokemon/Pokemon.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By just looking at a Pokemon's attributes (features), we will know which pokemon it is without knowing its name."},{"metadata":{"trusted":true},"cell_type":"code","source":"pkmn.sample(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(pkmn['Generation']) # sns means seaborn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Generation and Legendary variables were added to the dataset.\nWe are going to add a step here to drop the variables so that the presentation is simpler."},{"metadata":{"trusted":true},"cell_type":"code","source":"pkmn = pkmn.drop(['Generation', 'Legendary'],1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Features visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.jointplot(x=\"HP\", y=\"Speed\", data=pkmn);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.jointplot(x=\"Attack\", y=\"Defense\", data=pkmn, kind='hex');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Now let us go back to the VGG16 model!"},{"metadata":{},"cell_type":"markdown","source":"# Summary:\n\n1. Computer represents data as vector, matrix, or tensor.\n2. Computer learns how to classify images using its features.\n3. Our Neural network model (VGG16) uses mathematical operations to extract features from images.\n\nEmail: s.cao@wustl.edu"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}