{"cells":[{"metadata":{"_uuid":"2e7f38f0b46ae39d3ce510e1ff12d1800acedf2c","_cell_guid":"594af098-2562-4b87-9c4f-f3ae674906ad"},"cell_type":"markdown","source":"**Where are we?**\n\n[Machine Learning Chapter](https://docs.google.com/document/d/1xlIywWb56hxKhdoZS7PLH4OxChewoaPKEpv6O57aJ0M/edit#)\n\n**Recap of last session**\n* Supervised Learning and Algorithms\n* Python Notebooks\n* What else?\n\n**Goals for this session**\n1. Clear understanding of what a Neural Network is and how does it come up with a decision.\n1. Train and run, pretty much by hand, a neural network that predicts a simple dataset.\n1. Train and run, a neural network that predicts the number within an image.\n"},{"metadata":{"_uuid":"0a9cdc0c370373ddf6f824082d798800af074732","_cell_guid":"dbe92ab7-141f-469e-a7dc-d44c4b6d5a7c"},"cell_type":"markdown","source":"**Simple Introduction to (Artificial) Neural Networks**\n\nSlides 31 to 45\n\nhttps://docs.google.com/presentation/d/1vWrIE81q6-DKt4WKolxJU51b1Z-PwQ0ePkFUsmS9z6w/edit#slide=id.g13cb2ed31c_0_125"},{"metadata":{"_uuid":"88595c304af11f36b7cfb5039c211a502b99eac3","_cell_guid":"62638333-0384-4989-a43f-294a5fecf5f0"},"cell_type":"markdown","source":"**So, what's a Neuron?**\n\nA Neuron is composed of:\n1. Inputs - These inputs have numeric values\n2. Weights - Each input has an associated weight, also a numeric value.\n3. An 'activation' function. We will go through activations function in a second."},{"metadata":{"_uuid":"9e376be4f0726b145cabe2b48b7c0318d4eac115","_cell_guid":"b4294494-bd78-4a49-a133-41c892026fa5"},"cell_type":"markdown","source":"**Quiz**\n\nGiven a neuron,\n\n3 inputs with values: 1, 0 and -3. \n\nThe weights for this neuron are 0.5, 1.5 and 2.5. \n\nWhat's the output?\n\n![](https://imgur.com/u0o25AM.png)"},{"metadata":{"_uuid":"091d384b2fef5d49f542e2affe46ad69a5efdf37","_cell_guid":"f410474c-abe6-4380-be9d-c9064a857a2f","trusted":true},"cell_type":"code","source":"# Write code here that prints the output of the above neuron\nprint (1*2 + 3*4 + 4*5)","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"f4acbdc636a1bff33b45c40ba78488edd92d73e4","_cell_guid":"90f9c2ce-2243-47df-9f94-3877ba8dc062"},"cell_type":"markdown","source":"**Let's learn an XOR function**\n\nA | B | A ⊕ B\n------------ | ------------- | -------------\n0 | 0 | 0\n0 | 1 | 1\n1 | 0 | 1\n1 | 1 | 0\n\nLet's create a Neural Network that models this function. \n\nWhy such a simple model? The objective is to make the model so simple that you can follow at each step of the way what's going on. Later on we will take a more complex example.\n\n![](https://imgur.com/8WXgiOt.png)\n\nThis Neural Network will have two inputs and one output. Let's have one single hidden layer with 3 neurons.\n\nEach input node is connected to each node in the hidden layer. Each node in the hidden layer is connected to the output. This is the most common configuration in a Neural Network.\n\n*What do you think are the \"1\" nodes or \"Bias\" ?*\n\n*If each line represents a \"weight\", how many weights do you think we need to define in order to model the XOR function ?*"},{"metadata":{"_uuid":"ec3ac738fd4d2ed0c6866529b423f8651e4e2774","_cell_guid":"061a3a76-d2f9-4ea5-8f46-96ebc2cbb594","trusted":true},"cell_type":"code","source":"# Import some libraries\n\nimport numpy as np\nimport pandas as pd\nimport numpy as np\nfrom keras.utils import np_utils\nfrom keras.models import Sequential\nfrom keras.layers.core import Dense\nimport matplotlib.image as mpimg\nimport matplotlib.pyplot as plt\nimport matplotlib\n%matplotlib inline","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"48a0cfdea91bde607bddc8f94aae68763427f5f9","collapsed":true,"_cell_guid":"5ce996e1-ea42-4de9-9096-778f699d3bd4","trusted":true},"cell_type":"code","source":"dataset = np.array([[0, 0, 0],\n                    [0, 1, 1],\n                    [1, 0, 1],\n                    [1, 1, 0]])\n\nX = dataset[:, :2]\ny = dataset[:, -1]","execution_count":4,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ae092c4393ebeeeb720d54170ef662e56b74394"},"cell_type":"code","source":"# Trivial exercise, print the values of X, print the values of y. \n# Confirm that you understand how are we slicing the dataset into X's and y's.\n# Confirm that the X's and y's are the ones from the table above by doing the following exercise, assign into the variable 'xyz' all the elements from the second row.\n# Print the \"shape\" of X and y, what does shape mean?\n\n# print the values of X\nprint (X)\n\n# print the values of y\n\n# assign into the variable 'xyz' all the elements from the second row\nxyz = None\n\n# print the shape of X and y, what does shape mean?\nprint (X.shape)\n\n","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"19eaab4134bc0c58b0a09fc16e31e8b968e58379","_cell_guid":"e294ba84-0406-4038-95e4-537ddcf588ed"},"cell_type":"markdown","source":"In order to create our Neural Network, we will use a Python Package called \"**Keras**\".\n\n**What's Keras?**\n\nFrom its [website](https://keras.io/):\n\n*Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.*\n\nThis is a really popular package and I have found it to be really easy and friendly to use."},{"metadata":{"_uuid":"f5a0ebf2124301be0216d69a0f4cdcb9b998d78f","collapsed":true,"_cell_guid":"a19561cb-95cb-423f-ace0-fe371c1d27a0","trusted":true},"cell_type":"code","source":"# A Neural Network is a sequence of layers. \n# In Keras, you define one by instantiating \"Sequential\"\n# https://keras.io/models/sequential/\n\nmodel = Sequential()\nmodel.add(Dense(3, input_dim=2, activation='relu')) # This is our hidden layer\nmodel.add(Dense(1, activation='sigmoid')) # This is our output layer\n\n# What the above is saying is that we are adding two layers (three actually...).\n# The first layer is our hidden layer with 3 nodes. We also specify that it has an \"input_dim\" \n# or input dimension of 2 (our two inputs).\n\n# The next layer is our output layer with just one node.\n# We will worry later about the activation functions.","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"a2f41e1d552430d4ae8752b14b03e5f570e3e961","collapsed":true,"_cell_guid":"a79394b6-756c-41e2-968a-2c05a4271677","trusted":true},"cell_type":"code","source":"# We then proceed to compile our model. \n# Two required parameters are the 'loss' and an 'optimizer'.\n\n# 'loss' is the objective function. Suffice to say for now that when the network\n# has just one output and it's expected to have two values (like true/false or 0/1)\n# we can use 'binary_crossentropy'\n# Later on we will use another objective function when we have more than one output.\n\n# We will not dive deeper into the 'optimizer' and stick to use 'adam'.\nmodel.compile(loss='binary_crossentropy', optimizer='adam')","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"a72cb6b66484b75b9950a2784615c522955ec6cf","_cell_guid":"b4a196d2-1209-49b8-bc95-f2ca501d1a66","trusted":true},"cell_type":"code","source":"# We can now do the training, Keras calls it 'fit', as in, fit our model to the X -> y mapping.\n\n# batch_size is how many samples to do within a training epoch.\n# We don't need to dive deeper into it.\n# We only have 4 elements in our dataset, so let's use a batch_size of 4.\n\n# 'epochs' is the number of training iterations to run. This is an important parameter.\n# The more iterations we do, the better our model will map the X to the y's that we give.\n# However, notice that that means that we are mapping the training set, having a really high\n# number of epochs can cause 'overfit' !.\n# In this particular case, we don't have a test set, so we don't care.\nmodel.fit(X, y, batch_size=4, epochs=5000)","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"15dbc10063dce150708093445d0de0c25ccde07b","_cell_guid":"471b7510-08b5-4b4e-9693-01f197ac6828"},"cell_type":"markdown","source":"Here's a sample graph showing the effects of overfitting.\n![](https://lh6.googleusercontent.com/-6P-HsoKSxUY/VBjWSvxPN-I/AAAAAAAAukg/HhmPfLu7K9I/s400/05_caffe_loss.png)\n\nLet's confirm that this model follows the XOR function using \"predict_classes\"."},{"metadata":{"_uuid":"da726580d4136da48b06d30785bff76f040409af","_cell_guid":"cec90bba-9a1d-4fa8-a490-352d7ddd4b7c","trusted":true},"cell_type":"code","source":"print (model.predict_classes(X))","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"191f999e69f42ceef144e8d9fd53763bb95ab7f1","_cell_guid":"6d3afc75-49d6-4106-b160-e4f75cd2ee0f"},"cell_type":"markdown","source":"Now let's get the actual predictions, this is what folks typically call the \"probabilities\" of the prediction. We will see in a second where are those numbers coming from."},{"metadata":{"_uuid":"533737dca662abdbcd38278d9b536a3c154050a9","_cell_guid":"972da8cc-c3c8-4ac5-9145-9f540fb42725","trusted":true},"cell_type":"code","source":"print (model.predict(X))","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"e4b030f4819eaeb8b7dad2458fb780fe14971d85","_cell_guid":"94c86c35-6897-4291-893f-3d2e63816c17"},"cell_type":"markdown","source":"Earlier on, we asked the question, how many weights or parameters would we need to learn for this particular neural network? We can answer that by taking a look at the summary of our model."},{"metadata":{"_uuid":"e1a60aa1c0bc27a04be151e214faf9be925cd6b8","_cell_guid":"8a429179-7287-46c2-8e2f-4001facbeb89","trusted":true},"cell_type":"code","source":"print (model.summary())","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"089103c9f1f45e81851073548d382e19ff53292f","_cell_guid":"6dd853e4-2a43-428d-a4e4-2611b37fc4a8"},"cell_type":"markdown","source":"From the summary, we can see that the first layer has 9 parameters. This number comes from 3 hidden nodes, each node is connected to the two inputs and the \"bias\" input. Therefore 3*3 = 9.\n\n*Why is the second layer 4 parameters?*\n\n9+4 = 13, therefore we have a total of 13 parameters or weights.\n\nLet's go ahead and print those weights."},{"metadata":{"_uuid":"85123526ef673e8ca644c2d688f62e2d8e6e588c","_cell_guid":"bd84a914-284b-4048-aa4c-a00f7a4ff060","trusted":true},"cell_type":"code","source":"print (model.get_weights())","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"7cecc783e7d5c99e9f0513a241b31cec318ef392","_cell_guid":"6ef1a0d6-c566-4752-ad99-e29fc3bc7eef"},"cell_type":"markdown","source":"Let's take the above weights and plug them into a spreadsheet to really confirm the numbers that we are seeing and where they are coming from:\n\n[XOR Neural Network in Google Spreadsheets](https://docs.google.com/spreadsheets/d/1sWKqqNFhQUaFOFD8LSonowFp2ou-gy2QNnQSGhTP1-o/edit)\n\nWhat are the activation functions?\n* [Rectifier 'relu'](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)\n* [Sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function)"},{"metadata":{"_uuid":"af125cebd9743f746b93d1bfa780cad0952fe23f"},"cell_type":"markdown","source":"**Comprehension validation questions**\n1. What is the Neural Networking 'predicting' ?\n1. What are the code steps that we did to define and train a neural network above?\n1. What does it mean to say that we \"trained\" the neural network?\n"},{"metadata":{"_uuid":"141a79d9e3b371d3cc1e6aa92698fa51beaef8d1","_cell_guid":"d94982d3-751c-45b3-980e-b405e44bfbd4"},"cell_type":"markdown","source":"**Can we use a Neural Network to classify something more complex?**\n\nLet's try to classify images using a famous dataset called 'MNIST'.\n\n'MNIST' is a dataset of images of digits from 0 to 9. Each image is 28 by 28 pixels. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\nThe objective of this exercise will be to learn to classify those images using a Neural Network.\n\n* [MNIST in Kaggle](https://www.kaggle.com/c/digit-recognizer)\n* [Google Images](https://www.google.com/search?tbm=isch&source=hp&biw=1920&bih=1099&ei=LjhYWv-LPImi_QbjuYXYBw&q=mnist&oq=mnist&gs_l=img.3..0l10.954.1671.0.1807.6.6.0.0.0.0.52.221.5.5.0....0...1ac.1.64.img..1.5.221.0...0.rYn5A1NpvXk#imgrc=_)"},{"metadata":{"_uuid":"f929ce4ffce4ea83ace174b4442ed44a363f5ce1","collapsed":true,"_cell_guid":"fb168fdb-28b1-41ca-afe6-cd8f55cec8c3","trusted":true},"cell_type":"code","source":"# Let's read our training and test set from the attached dataset\ntrain = pd.read_csv(\"../input/train.csv\").values\n\n# The first column of the dataset tell us the actual value\nx_train = train[:, 1:]\ny_train = train[:, 0]\n\n# So far we have loaded the 2 variables that we need\n# 1. x_train -> has the images to train on\n# 2. y_train -> has the value for those images","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"b34bbcb692f3ab8e625bac440556b750e0710a16","_cell_guid":"c8e5c89a-ae6d-4def-89bc-b587646920c3","trusted":true},"cell_type":"code","source":"# How many images do we have?\nprint (x_train.shape)","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"cc715deae0874213ac5b6d4355461e92e955e392","_cell_guid":"6e26645f-2233-4f62-a689-e5f9eb1cbbe0"},"cell_type":"markdown","source":"It's a collection of 42,000 images of digits from 0 to 9. Each image is 28x28 (784) pixels, each pixel is a value from 0 to 255. 0 is black and 255 is white."},{"metadata":{"_uuid":"eaa66d5c84349fb95530d6b557ab99c23c6bd66d","scrolled":false,"_cell_guid":"58159033-1e91-42fe-8f09-774488e5bfe7","trusted":true},"cell_type":"code","source":"print (x_train[0])","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"8904cc087e65fda5e09a210b639f504876d12e36","_cell_guid":"8450d7da-306a-457d-9741-76c4cba3f910","trusted":true},"cell_type":"code","source":"# Let's visualize the first ten images.\n\nfig1, ax1 = plt.subplots(1,12, figsize=(12,10))\nfor i in range(12):\n    ax1[i].imshow(x_train[i].reshape((28,28)), cmap='gray')\n    ax1[i].axis('off')\n    ax1[i].set_title(y_train[i])","execution_count":16,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"7bcd1f96cfae512a282c252fa38f30e184a380b2"},"cell_type":"code","source":"# Exercise, visualize the last figure in the set, what number does it contain?\n\n# TODO","execution_count":17,"outputs":[]},{"metadata":{"_uuid":"b0f75bfa5de92bc36e9f828b713ea0e9fb06749f","_cell_guid":"9cdd51a9-3e0d-4db0-b0d3-b29efcbf4af2"},"cell_type":"markdown","source":"Our job is to construct a neural network capable of learning these digits.\n\nSame as with the XOR function, let's build a very similar neural network. Instead of 2 inputs, we will now have 784, one per each pixel.\nLet's also have one hidden layer. A good heuristic is for the hidden layer to be 2x, x or x/2 size of the first layer."},{"metadata":{"_uuid":"33f73b763e6cd7971b60859624240fe71f92fc85","_cell_guid":"7259581d-465d-4c20-b767-4498d2edcb9b","trusted":true},"cell_type":"code","source":"num_pixels = x_train.shape[1]\n\nprint (num_pixels)","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"314c31c529486dda95f9f4705b12ea02f3890cc2","_cell_guid":"86a58ba2-5615-49f6-8734-c385c25c20c0","trusted":true},"cell_type":"code","source":"# Let's normalize from 0 to 255 to 0 to 1. It's a good practice when dealing with \n# neural networks that the inputs have short range of values.\nx_train = x_train.astype('float32')\nx_train /= 255\n\nprint (x_train[0])","execution_count":19,"outputs":[]},{"metadata":{"_uuid":"3a4b9abef531276bf86e9ab164345a5f972b03e3","_cell_guid":"51f1f897-5905-437e-9c08-2c1dd3f5c6d0","trusted":true},"cell_type":"code","source":"# Now, the \"y_train\" is an array of values from 0 to 9.\n# Let's print the first 10.\nprint (y_train[:10])","execution_count":20,"outputs":[]},{"metadata":{"_uuid":"acbc62961c4aa20eb30828298d5aebe74b1d1128","_cell_guid":"2be6e87d-3a21-452c-9f67-e2b3d8dfa88f"},"cell_type":"markdown","source":"The type of problem that we are dealing with is \"multiclass classification\". We have multiple \"classes\", each digit is a class. And therefore, for each image, we want to assign it a \"class\".\n\nTo solve a multiclass classification problem using a Neural Network, is that in your output layer, you will have one node per class and you want the node to output a 1 whenever the input belongs to that class.\n\n![](http://neuralnetworksanddeeplearning.com/images/tikz12.png)"},{"metadata":{"_uuid":"50e3c6d8ccaf8261105ea0d54856648de1961623","_cell_guid":"c367a03d-9146-4d2f-a132-4d155c70c8bc","trusted":true},"cell_type":"code","source":"# What this means for us is that we need to transform our y_train from 0-9 to a 'one-hot encoding'\nprint ('Before')\nprint (y_train[:10])\n\ny_train = np_utils.to_categorical(y_train)\nnum_classes = y_train.shape[1]\n\nprint ('After one hot encoding')\nprint (y_train[:10])","execution_count":21,"outputs":[]},{"metadata":{"_uuid":"de13bbcd6e1390ee163e0934a3b5b2bfd4e9f7ca","_cell_guid":"281adaa9-dc81-4285-bfe5-302ff1cbfac3","trusted":true},"cell_type":"code","source":"# We're finally ready to construct our model.\n# It's practically identical as the one for the XOR.\n\n# The few differences are:\n# 1. We are using a 'softmax' activation for the output layer.\n# 2. We are using a 'categorical_crossentropy' loss function.\nmodel = Sequential()\n\nmodel.add(Dense(num_pixels, input_dim=num_pixels, activation='relu'))\nmodel.add(Dense(num_classes, activation='softmax'))\n\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\nprint (model.summary())","execution_count":22,"outputs":[]},{"metadata":{"_uuid":"6f670ae3cbff93b0a1a7899dfd51dd9b63f83246","_cell_guid":"ff3b719d-7662-436c-8c97-46bef723a0c9"},"cell_type":"markdown","source":"We are going to train 623,290 parameters"},{"metadata":{"_uuid":"7cd4fd0f858d518f787d50427ad1648f3b85da0d","_cell_guid":"0eab9359-d8c9-424c-bbda-36163c748fd7","trusted":true},"cell_type":"code","source":"model.fit(x_train, y_train, epochs=10, batch_size=200, verbose=2, validation_split=0.2)","execution_count":23,"outputs":[]},{"metadata":{"_uuid":"33a951bf09d288668a132666ffb4c6a107484714","_cell_guid":"2a6ee2cf-01d6-4803-b794-1d68339a62e2"},"cell_type":"markdown","source":"**Notice that this an accuracy of ~98% !**\n\nNext session we will cover 'deep learning' with more advanced Neural Networks.\n\n**Comprehension validation questions**\n1. What is the Neural Networking 'predicting' ?\n1. What are the inputs to the neural network?\n1. What node does it need to be activated so that we predict that the image is a zero?\n1. Can this Neural network predict an 'A' ? If so, why? otherwise, what needs to be done so it can predict it?\n1. Why are we splitting the dataset into training and validation set?\n1. Why we didn't split the dataset for XOR?\n1. Why did we massaged the data to normalize it from 0-255 to 0-1 ? Could we have done the training without it?\n1. What activation functions did we use?\n1. What's an epoch?\n1. How many epochs did we use to learn the XOR function?\n1. How many epochs did we use to learn the MNIST dataset?\n1. What could go wrong (if any) if we keep increasing the number of epochs for either the XOR function or the MNIST dataset?\n\n\n**Recap, what did we learn today?**\n1. X\n\n*Exercises: *\n* Design a Neural network for another boolean function (AND?) or a more complex boolean function with 3 or more parameters.\n* Learn what's the \"backpropagation\" algorithm, or what does it mean to train a neural network? (https://docs.google.com/spreadsheets/d/1pJrj9utxo69RLamTOxkF95vEm9eR9WJg6LPDtwEzMws/edit#gid=1658224758)\n* Use a Neural Network to model the \"voice gender\" dataset that we studied last section (https://www.kaggle.com/primaryobjects/voicegender) or (https://www.kaggle.com/carlosaguayo/gender-recognition-by-voice-introduction-to-ml)"}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}