{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Practical Machine Learning ML with PyTorch (TBU)\n\nThis kernel is empowering you to use machine learning to get valuable insights from data.\n- 🔥 Implement basic ML algorithms and deep neural networks with PyTorch.\n- 📦 Learn object-oriented ML to code for products, not just tutorials.\n\n> #### **Credits**: Thanks to **Practical AI - Goku Mohandas** and other contributers for such wonderful work!\n\n### Here are some of *my kernel notebooks* for **Machine Learning and Data Science** as follows, ***Upvote*** them if you *like* them\n\n> * [Awesome Deep Learning Basics and Resources](https://www.kaggle.com/arunkumarramanan/awesome-deep-learning-resources)\n> * [Data Science with R - Awesome Tutorials](https://www.kaggle.com/arunkumarramanan/data-science-with-r-awesome-tutorials)\n> * [Data Science and Machine Learning Cheetcheets](https://www.kaggle.com/arunkumarramanan/data-science-and-machine-learning-cheatsheets)\n> * [Awesome ML Frameworks and MNIST Classification](https://www.kaggle.com/arunkumarramanan/awesome-machine-learning-ml-frameworks)\n> * [Awesome Data Science for Beginners with Titanic Exploration](https://kaggle.com/arunkumarramanan/awesome-data-science-for-beginners)\n> * [Tensorflow Tutorial and House Price Prediction](https://www.kaggle.com/arunkumarramanan/tensorflow-tutorial-and-examples)\n> * [Data Scientist's Toolkits - Awesome Data Science Resources](https://www.kaggle.com/arunkumarramanan/data-scientist-s-toolkits-awesome-ds-resources)\n> * [Awesome Computer Vision Resources (TBU)](https://www.kaggle.com/arunkumarramanan/awesome-computer-vision-resources-to-be-updated)\n> * [Machine Learning and Deep Learning - Awesome Tutorials](https://www.kaggle.com/arunkumarramanan/awesome-deep-learning-ml-tutorials)\n> * [Data Science with Python - Awesome Tutorials](https://www.kaggle.com/arunkumarramanan/data-science-with-python-awesome-tutorials)\n> * [Awesome TensorFlow and PyTorch Resources](https://www.kaggle.com/arunkumarramanan/awesome-tensorflow-and-pytorch-resources)\n> * [Awesome Data Science IPython Notebooks](https://www.kaggle.com/arunkumarramanan/awesome-data-science-ipython-notebooks)\n> * [Machine Learning Engineer's Toolkit with Roadmap](https://www.kaggle.com/arunkumarramanan/machine-learning-engineer-s-toolkit-with-roadmap) \n> * [Hands-on ML with scikit-learn and TensorFlow](https://www.kaggle.com/arunkumarramanan/hands-on-ml-with-scikit-learn-and-tensorflow)\n> * [Practical Machine Learning with PyTorch](https://www.kaggle.com/arunkumarramanan/practical-machine-learning-with-pytorch)\n\n> ***Practical Machine Learning ML with TensorFlow***\n\n> The above highlighted work will soon be released and also be based on the below coursework;\n\n* [Google Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/) - Machine Learning ML Crash Course with TensorFlow APIs (Google's fast-paced, practical introduction to machine learning, A self-study guide for aspiring machine learning practitioners: Machine Learning Crash Course features a series of lessons with video lectures, real-world case studies, and hands-on practice exercises.) is highly recommended by Google as it's developed by googlers along with Notebooks for exercises.\n\n
\n \n## Kernel Notebook Content\n \n## [Basics](#1) \n- 🐍 [Python](#1) \n- 🔢 [NumPy](#1) \n- 🐼[Pandas](#1) \n- 📈 [Linear Regression](#1) \n- 📊 [Logistic Regression](#1)\n- 🌳 [Random Forests](#1)\n- 💥 KMeans Clustering\n \n## [Deep Learning](#2)\n- 🔥 [PyTorch](#2)\n- 🎛️ [Multilayer Perceptrons](#2)\n- 🔎 [Data & Models](#2)\n- 📦 [Object-Oriented ML](#2)\n- 🖼️ [Convolutional Neural Networks](#2)\n- 📝 [Embeddings](#2)\n- 📗 [Recurrent Neural Networks](#2)\n\n## [Advanced](#3)\n- 📚 [Advanced RNNs](#3)\n- 🏎️ Highway and Residual Networks\n- 🔮 Autoencoders\n- 🎭 Generative Adversarial Networks\n- 🐝 Spatial Transformer Networks\n\n## [Topics](#4)\n- 📸 [Computer Vision](#4)\n- ⏰ Time Series Analysis\n- 🏘️ Topic Modeling\n- 🛒 Recommendation Systems\n- 🗣️ Pretrained Language Modeling\n- 🤷 Multitask Learning\n- 🎯 Low Shot Learning|\n- 🍒 Reinforcement Learning|\n\n"},{"metadata":{"_uuid":"1b749587d2b8e714b26e66be37685528b0226643"},"cell_type":"markdown","source":"
\n## 1. Basics \n\n
\n## 1.1 Introduction to Python\nIn this lesson we will learn the basics of the Python programming language (version 3). We won't learn everything about Python but enough to do some basic machine learning.\n\n"},{"metadata":{"_uuid":"87095c52a5451bb48c8a673a875caabac089a09d"},"cell_type":"markdown","source":"### Variables\nVariables are objects in python that can hold anything with numbers or text. Let's look at how to make some variables.\n"},{"metadata":{"trusted":true,"_uuid":"d5544a43ebcea37fb183d9f3f177fb0e1bb1252a"},"cell_type":"code","source":"# Numerical example\nx = 5\nprint (x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d528a64d9d5f75733ae4b4e1d5d53ff2182ffa0"},"cell_type":"code","source":"# Text example\nx = \"hello\"\nprint (x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6edb0a0c488afa6650d1a201f923a4a3b9e4abaf"},"cell_type":"code","source":"# int variable\nx = 5\nprint (x)\nprint (type(x))\n\n# float variable\nx = 5.0\nprint (x)\nprint (type(x))\n\n# text variable\nx = \"5\" \nprint (x)\nprint (type(x))\n\n# boolean variable\nx = True\nprint (x)\nprint (type(x))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e5968fd96b80c3a99322a0a8baadf21d1c110d8"},"cell_type":"markdown","source":"It's good practice to know what types your variables are. When you want to use numerical operations on then, they need to be compatible. "},{"metadata":{"trusted":true,"_uuid":"42049443dde91565117a5d1bfc03ce95d9cc8609"},"cell_type":"code","source":"# int variables\na = 5\nb = 3\nprint (a + b)\n\n# string variables\na = \"5\"\nb = \"3\"\nprint (a + b)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9581d2f609646494e986a67b6a217b289aff31f4"},"cell_type":"markdown","source":"### Lists\nLists are objects in python that can hold a ordered sequence of numbers **and** text."},{"metadata":{"trusted":true,"_uuid":"f431aad36b52abcb9c35e45e86d4c85fa6af4f2a"},"cell_type":"code","source":"# Making a list\nlist_x = [3, \"hello\", 1]\nprint (list_x)\n\n# Adding to a list\nlist_x.append(7)\nprint (list_x)\n\n# Accessing items at specific location in a list\nprint (\"list_x[0]: \", list_x[0])\nprint (\"list_x[1]: \", list_x[1])\nprint (\"list_x[2]: \", list_x[2])\nprint (\"list_x[-1]: \", list_x[-1]) # the last item\nprint (\"list_x[-2]: \", list_x[-2]) # the second to last item\n\n# Slicing\nprint (\"list_x[:]: \", list_x[:])\nprint (\"list_x[2:]: \", list_x[2:])\nprint (\"list_x[1:3]: \", list_x[1:3])\nprint (\"list_x[:-1]: \", list_x[:-1])\n\n# Length of a list\nlen(list_x)\n\n# Replacing items in a list\nlist_x[1] = \"hi\"\nprint (list_x)\n\n# Combining lists\nlist_y = [2.4, \"world\"]\nlist_z = list_x + list_y\nprint (list_z)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8ffa6a28990e63112886f4d1f014bfce4c7e774"},"cell_type":"markdown","source":"### Tuples\nTuples are also objects in python that can hold data but you cannot replace values (for this reason, tuples are called immutable, whereas lists are known as mutable)."},{"metadata":{"trusted":true,"_uuid":"525c03dc23b2c5d11c54a0262c105fdf0849835f"},"cell_type":"code","source":"# Creating a tuple\ntuple_x = (3.0, \"hello\")\nprint (tuple_x)\n\n# Adding values to a tuple\ntuple_x = tuple_x + (5.6,)\nprint (tuple_x)\n\n# Trying to change a tuples value (you can't)\ntuple_x[1] = \"world\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e11c939df2558d599042363d538de6a7e5ca865"},"cell_type":"markdown","source":"### Dictionaries\nDictionaries are python objects that hold key-value pairs. In the example dictionary below, the keys are the \"name\" and \"eye_color\" variables. They each have a value associated with them. A dictionary cannot have two of the same keys. "},{"metadata":{"trusted":true,"_uuid":"0323766d32f39df54855c350814b798468c5e833"},"cell_type":"code","source":"# Creating a dictionary\narun = {\"name\": \"Arun\",\n \"eye_color\": \"brown\"}\nprint (arun)\nprint (arun[\"name\"])\nprint (arun[\"eye_color\"])\n\n# Changing the value for a key\narun[\"eye_color\"] = \"black\"\nprint (arun)\n\n# Adding new key-value pairs\narun[\"age\"] = 24\nprint (arun)\n\n# Length of a dictionary\nprint (len(arun))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e52fbfd7bedad38a53e097211e470c5c6d78c0a"},"cell_type":"markdown","source":"### If statements\nYou can use if statements to conditionally do something."},{"metadata":{"trusted":true,"_uuid":"ae5386cdd9ca623b7aa1e67c5baf55b127a6144f"},"cell_type":"code","source":"# If statement\nx = 4\nif x < 1:\n score = \"low\"\nelif x <= 4:\n score = \"medium\"\nelse:\n score = \"high\"\nprint (score)\n\n# If statment with a boolean\nx = True\nif x:\n print (\"it worked\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a66d7b3d8df594906da2de1b355032a1de83769"},"cell_type":"markdown","source":"### Loops\nYou can use for or while loops in python to do something repeatedly until a condition is met."},{"metadata":{"trusted":true,"_uuid":"966ce1cf4c72f321f67df25e846c49143c808746"},"cell_type":"code","source":"# For loop\nx = 1\nfor i in range(3): # goes from i=0 to i=2\n x += 1 # same as x = x + 1\n print (\"i={0}, x={1}\".format(i, x)) # printing with multiple variables","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4717504e37b5cc2f4c17ac40116a2a0264214a43"},"cell_type":"code","source":"# While loop\nx = 3\nwhile x > 0:\n x -= 1 # same as x = x - 1\n print (x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e32ef45562520a4cf43bfb0b90bcd0e19d8990b"},"cell_type":"markdown","source":"### Functions\nFunctions are a way to modularize reusable pieces of code. "},{"metadata":{"trusted":true,"_uuid":"62daebe39d26019ade16824007adc4deea388989"},"cell_type":"code","source":"# Create a function\ndef add_two(x):\n x += 2\n return x\n\n# Use the function\nscore = 0\nscore = add_two(x=score)\nprint (score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6fd44e94b94791782016efdf533d486c2b1ef613"},"cell_type":"code","source":"# Function with multiple inputs\ndef join_name(first_name, last_name):\n joined_name = first_name + \" \" + last_name\n return joined_name\n\n# Use the function\nfirst_name = \"Arunkumar\"\nlast_name = \"Venkataramanan\"\njoined_name = join_name(first_name=first_name, last_name=last_name)\nprint (joined_name)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4696aedb528f468d1434446375667c9dfa8b56da"},"cell_type":"markdown","source":"### Classes\nClasses are a fundamental piece of object oriented Python programming."},{"metadata":{"trusted":true,"_uuid":"e7ddd3260f71f295fe3d72807b955fceabff382e"},"cell_type":"code","source":"# Create the function\nclass Pets(object):\n \n # Initialize the class\n def __init__(self, species, color, name):\n self.species = species\n self.color = color\n self.name = name\n\n # For printing \n def __str__(self):\n return \"{0} {1} named {2}.\".format(self.color, self.species, self.name)\n\n # Example function\n def change_name(self, new_name):\n self.name = new_name","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d882a339bc23ba5211a9fb09ebf819811de0d65"},"cell_type":"code","source":"# Making an instance of a class\nmy_dog = Pets(species=\"dog\", color=\"orange\", name=\"Guiness\",)\nprint (my_dog)\nprint (my_dog.name)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"052bd705025c1ac1a0b796aed21a88628fb32830"},"cell_type":"code","source":"# Using a class's function\nmy_dog.change_name(new_name=\"Johny Kutty\")\nprint (my_dog)\nprint (my_dog.name)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6b9d3b0335194de011eb821e51d819fd7c3e8825"},"cell_type":"markdown","source":"### Additional resources\nThis was a very quick look at python and we'll be learning more in future lessons. If you want to learn more right now before diving into machine learning, check out this free course: [Free Python Course](https://www.codecademy.com/learn/learn-python) and [Kaggle Learn](https://www.kaggle.com/learn/python)\n\n###### [Go to top](#top)"},{"metadata":{"_uuid":"764100c75a2b848625ec70752317b92a63339567"},"cell_type":"markdown","source":"
\n## 1.2 NumPy\n\nIn this lesson we will learn the basics of numerical analysis using the NumPy package.\n\n\n\n"},{"metadata":{"_uuid":"960229bdaf361f77f0209d048dcf635ac448f2d3"},"cell_type":"markdown","source":"### NumPy basics"},{"metadata":{"trusted":true,"_uuid":"711cbbf9bdf62c98acc93e76ad7228845145996a"},"cell_type":"code","source":"import numpy as np","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33450d265e1ddb7a203dd19febb4e4fcf37ec602"},"cell_type":"code","source":"# Set seed for reproducability\nnp.random.seed(seed=1234)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a1b9e773e9d7d680dc02833fdef12960f88db914"},"cell_type":"code","source":"# Scalars\nx = np.array(6) # scalar\nprint (\"x: \", x)\nprint(\"x ndim: \", x.ndim)\nprint(\"x shape:\", x.shape)\nprint(\"x size: \", x.size)\nprint (\"x dtype: \", x.dtype)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b72b71984025d728d6a332c2c106ab2da6347139"},"cell_type":"code","source":"# 1-D Array\nx = np.array([1.3 , 2.2 , 1.7])\nprint (\"x: \", x)\nprint(\"x ndim: \", x.ndim)\nprint(\"x shape:\", x.shape)\nprint(\"x size: \", x.size)\nprint (\"x dtype: \", x.dtype) # notice the float datatype","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e70b7b2e5cd004f67fc6bdabed61c55cd4de599"},"cell_type":"code","source":"# 3-D array (matrix)\nx = np.array([[1,2,3], [4,5,6], [7,8,9]])\nprint (\"x:\\n\", x)\nprint(\"x ndim: \", x.ndim)\nprint(\"x shape:\", x.shape)\nprint(\"x size: \", x.size)\nprint (\"x dtype: \", x.dtype)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f00b454c421832f511a49c2f1da5a707559d5dd"},"cell_type":"code","source":"# Functions\nprint (\"np.zeros((2,2)):\\n\", np.zeros((2,2)))\nprint (\"np.ones((2,2)):\\n\", np.ones((2,2)))\nprint (\"np.eye((2)):\\n\", np.eye((2)))\nprint (\"np.random.random((2,2)):\\n\", np.random.random((2,2)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"20613e4bca9fa9c90e771eb0e9c7c2983e9e12e3"},"cell_type":"markdown","source":"### Indexing"},{"metadata":{"trusted":true,"_uuid":"75fe13478f87a3d40ab2e9c138c4d2d8d3e7e194"},"cell_type":"code","source":"# Indexing\nx = np.array([1, 2, 3])\nprint (\"x[0]: \", x[0])\nx[0] = 0\nprint (\"x: \", x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa63063e81bf01c79d880f1f2607c3250382bd6a"},"cell_type":"code","source":"# Slicing\nx = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\nprint (x)\nprint (\"x column 1: \", x[:, 1]) \nprint (\"x row 0: \", x[0, :]) \nprint (\"x rows 0,1,2 & cols 1,2: \\n\", x[:3, 1:3]) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"632dca3dba68ba681e7ae77d1aa30a19e78ba373"},"cell_type":"code","source":"# Integer array indexing\nprint (x)\nrows_to_get = np.arange(len(x))\nprint (\"rows_to_get: \", rows_to_get)\ncols_to_get = np.array([0, 2, 1])\nprint (\"cols_to_get: \", cols_to_get)\nprint (\"indexed values: \", x[rows_to_get, cols_to_get])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d498b265582b8af4c12af8c057400ef20db74c5"},"cell_type":"code","source":"# Boolean array indexing\nx = np.array([[1,2], [3, 4], [5, 6]])\nprint (\"x:\\n\", x)\nprint (\"x > 2:\\n\", x > 2)\nprint (\"x[x > 2]:\\n\", x[x > 2])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e9f890e728f269f498054bf237a68cb96a02fcf4"},"cell_type":"markdown","source":"### Array math"},{"metadata":{"trusted":true,"_uuid":"375d060ee3b69f064f1ca547f857789fc7c63920"},"cell_type":"code","source":"# Basic math\nx = np.array([[1,2], [3,4]], dtype=np.float64)\ny = np.array([[1,2], [3,4]], dtype=np.float64)\nprint (\"x + y:\\n\", np.add(x, y)) # or x + y\nprint (\"x - y:\\n\", np.subtract(x, y)) # or x - y\nprint (\"x * y:\\n\", np.multiply(x, y)) # or x * y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"682a1d0f478beb00cc54de61cf9ba22a61df27e6"},"cell_type":"markdown","source":"\n"},{"metadata":{"trusted":true,"_uuid":"0e40c88c67be967dc2fe2ee8bc38483ece7c3d59"},"cell_type":"code","source":"# Dot product\na = np.array([[1,2,3], [4,5,6]], dtype=np.float64) # we can specify dtype\nb = np.array([[7,8], [9,10], [11, 12]], dtype=np.float64)\nprint (a.dot(b))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bab3441df4dc5dcfb5fc5e1b864160c6429be6ad"},"cell_type":"code","source":"# Sum across a dimension\nx = np.array([[1,2],[3,4]])\nprint (x)\nprint (\"sum all: \", np.sum(x)) # adds all elements\nprint (\"sum by col: \", np.sum(x, axis=0)) # add numbers in each column\nprint (\"sum by row: \", np.sum(x, axis=1)) # add numbers in each row","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ed6c5c3d3e48982e90c46c45373f741449ca67f"},"cell_type":"code","source":"# Transposing\nprint (\"x:\\n\", x)\nprint (\"x.T:\\n\", x.T)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa9e906d2df76601f8300c83db6efaeb423db582"},"cell_type":"markdown","source":"### Advanced"},{"metadata":{"trusted":true,"_uuid":"ccde7b95cd9c341686c6c3ddee7d521e04be3aba"},"cell_type":"code","source":"# Tile\nx = np.array([[1,2], [3,4]])\ny = np.array([5, 6])\naddent = np.tile(y, (len(x), 1))\nprint (\"addent: \\n\", addent)\nz = x + addent\nprint (\"z:\\n\", z)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a8550625e21aa662328575ea7c93b96a65c2044"},"cell_type":"code","source":"# Broadcasting\nx = np.array([[1,2], [3,4]])\ny = np.array([5, 6])\nz = x + y\nprint (\"z:\\n\", z)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c0130c0ef050d6fa56b58e460e31a51f5cc48bb"},"cell_type":"code","source":"# Reshaping\nx = np.array([[1,2], [3,4], [5,6]])\nprint (x)\nprint (\"x.shape: \", x.shape)\ny = np.reshape(x, (2, 3))\nprint (\"y.shape: \", y.shape)\nprint (\"y: \\n\", y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64e5cc2601c1ed16974a6c5a8fee4d63da3a4f3a"},"cell_type":"code","source":"# Removing dimensions\nx = np.array([[[1,2,1]],[[2,2,3]]])\nprint (\"x.shape: \", x.shape)\ny = np.squeeze(x, 1) # squeeze dim 1\nprint (\"y.shape: \", y.shape) \nprint (\"y: \\n\", y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"30904692b20825932661bba16185d1ccd2e79a5a"},"cell_type":"code","source":"# Adding dimensions\nx = np.array([[1,2,1],[2,2,3]])\nprint (\"x.shape: \", x.shape)\ny = np.expand_dims(x, 1) # expand dim 1\nprint (\"y.shape: \", y.shape) \nprint (\"y: \\n\", y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2dbf08a26cc17522627029c04732e6c15ffc10a"},"cell_type":"markdown","source":"### Additional resources\n\nYou don't to memorize anything here and we will be taking a closer look at NumPy in the later lessons. If you are curious about more checkout the [NumPy reference manual](https://docs.scipy.org/doc/numpy-1.15.1/reference/).\n\n###### [Go to top](#top)"},{"metadata":{"_uuid":"5af6113318872cbdcb243c078885b0d9cf55f2d0"},"cell_type":"markdown","source":"
\n## 1.3 Pandas"},{"metadata":{"_uuid":"faa8d2425043e34f04340f3d167353f2d95699e1"},"cell_type":"markdown","source":"In this notebook, we'll learn the basics of data analysis with the Python Pandas library.\n\n\n"},{"metadata":{"_uuid":"5930f07ab617e0256869ea850108c3359d6812f5"},"cell_type":"markdown","source":"### Uploading the data\n\nWe're first going to get some data to play with. We're going to load the titanic dataset from the public link below."},{"metadata":{"_uuid":"edd4d22270b4f5bab79091dad72c36bdf0152726"},"cell_type":"markdown","source":"### Loading the data\n\nNow that we have some data to play with, let's load into a Pandas dataframe. Pandas is a great python library for data analysis."},{"metadata":{"trusted":true,"_uuid":"246051dc8a3d32afc09afecae83cfe5cedc06d3e"},"cell_type":"code","source":"import pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2a508dd75e33bf7c88ae8851435b13e8f5bf3d3"},"cell_type":"code","source":"# Read from CSV to Pandas DataFrame\ndf = pd.read_csv(\"../input/train.csv\", header=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76562f85e9fa8e1a25d26db45ccdde434c411af4"},"cell_type":"code","source":"# First five items\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe312dc4282d7aabd280caf0fc970053cbee1e45"},"cell_type":"markdown","source":"These are the diferent features: \n* pclass: class of travel\n* name: full name of the passenger\n* sex: gender\n* age: numerical age\n* sibsp: # of siblings/spouse aboard\n* parch: number of parents/child aboard\n* ticket: ticket number\n* fare: cost of the ticket\n* cabin: location of room\n* emarked: port that the passenger embarked at (C - Cherbourg, S - Southampton, Q = Queenstown)\n* survived: survial metric (0 - died, 1 - survived)"},{"metadata":{"_uuid":"66d4545b5d79ccf92f21f6e71b077999005adcf0"},"cell_type":"markdown","source":"### Exploratory Dats Analysis EDA\n\nWe're going to explore the Pandas library and see how we can explore and process our data."},{"metadata":{"trusted":true,"_uuid":"d84ceeb5f1216f1ba40196b21494316f5b9b86e3"},"cell_type":"code","source":"# Describe features\ndf.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eec517892313feabf8ccbbbf64da420facc868df"},"cell_type":"code","source":"# Histograms\ndf[\"Age\"].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a3581d2f0f45f342975e051b68c73ed6fbb5b8c5"},"cell_type":"code","source":"# Unique values\ndf[\"Embarked\"].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"81d5d5cf1b29ba6fb547275f46913dc0083743e3"},"cell_type":"code","source":"# Selecting data by feature\ndf[\"Name\"].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cd968a00e5015c3a8d5ee274c7a2bcd4ec432392"},"cell_type":"code","source":"# Filtering\ndf[df[\"Sex\"]==\"female\"].head() # only the female data appear","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47ddc8132bcd0f43ea9f1963601c36e0cef266c9"},"cell_type":"code","source":"# Sorting\ndf.sort_values(\"Age\", ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"72e69569b4031c821ea7b0e57e7d5147841653f6"},"cell_type":"code","source":"# Grouping\nsex_group = df.groupby(\"Survived\")\nsex_group.mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"912f9f5f46a0c89b40c7dd5947847168260aebdd"},"cell_type":"code","source":"# Selecting row\ndf.iloc[0, :] # iloc gets rows (or columns) at particular positions in the index (so it only takes integers)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c1b1e04dc23976019fd6ca10912c85142feb3ad"},"cell_type":"code","source":"# Selecting specific value\ndf.iloc[0, 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eac58e4174fb4a85e3f01eaca0f5abe4d94d0b81"},"cell_type":"code","source":"# Selecting by index\ndf.loc[0] # loc gets rows (or columns) with particular labels from the index","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be1ec6bc45e8684f841169c88a81d49a6d6081eb"},"cell_type":"markdown","source":"### Data Preprocessing"},{"metadata":{"trusted":true,"_uuid":"79a04db00df5fb0f5cac46789c71632d0946ebe5"},"cell_type":"code","source":"# Rows with at least one NaN value\ndf[pd.isnull(df).any(axis=1)].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"146ac8eeba5c2d6ed6ef7cfcae38e273d3ec0374"},"cell_type":"code","source":"# Drop rows with Nan values\ndf = df.dropna() # removes rows with any NaN values\ndf = df.reset_index() # reset's row indexes in case any rows were dropped\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4f58d4d72ebfa8bacd9acc22eed1ddcdf2ebe50"},"cell_type":"code","source":"# Dropping multiple rows\ndf = df.drop([\"Name\", \"Cabin\", \"Ticket\"], axis=1) # we won't use text features for our initial basic models\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"06d03c3ff550fc1be20980075fe4bebd9eaa3663"},"cell_type":"code","source":"# Map feature values\ndf['Sex'] = df['Sex'].map( {'female': 0, 'male': 1} ).astype(int)\ndf[\"Embarked\"] = df['Embarked'].dropna().map( {'S':0, 'C':1, 'Q':2} ).astype(int)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6feac353b9f121820565c70b7588dd97a1dd3c88"},"cell_type":"markdown","source":"### Feature Engineering"},{"metadata":{"trusted":true,"_uuid":"cba71f8d029ab2c22e17dc24997f9d1a4758b006"},"cell_type":"code","source":"# Lambda expressions to create new features\ndef get_family_size(sibsp, parch):\n family_size = sibsp + parch\n return family_size\n\ndf[\"Family_Size\"] = df[[\"SibSp\", \"Parch\"]].apply(lambda x: get_family_size(x[\"SibSp\"], x[\"Parch\"]), axis=1)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d7a87c0a0c4443faf51d3836a050b33d9b89b95"},"cell_type":"code","source":"# Reorganize headers\ndf = df[['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Family_Size', 'Fare', 'Embarked', 'Survived']]\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3684a910938d24ad041c2f3fbc1914ddacb3f652"},"cell_type":"markdown","source":"### Saving data"},{"metadata":{"trusted":true,"_uuid":"d486db0a2e95f63f4822230a9a6f4485f9101d85"},"cell_type":"code","source":"# Saving dataframe to CSV\ndf.to_csv(\"processed_titanic.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c867a6c86d9944c7826d58723d0cdbbd6b99bc8"},"cell_type":"code","source":"# See your saved file\n!ls -l","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d38f9c750165f5a7c633519b4944e62bd8dc6ffd"},"cell_type":"markdown","source":"###### [Go to top](#top)"},{"metadata":{"_uuid":"71e4fdafa8cc44119db847d0880622138e119211"},"cell_type":"markdown","source":"
\n## 1.4 Linear Regression\n\nIn this lesson we will learn about linear regression. We will first understand the basic math behind it and then implement it in Python. We will also look at ways of interpreting the linear model."},{"metadata":{"_uuid":"7b533575f347d8fce703a4643d13088f11d2f7ed"},"cell_type":"markdown","source":"### Overview\n\n\n\n$\\hat{y} = XW$\n\n*where*:\n* $\\hat{y}$ = prediction | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ "},{"metadata":{"_uuid":"e38e8c84c3df9ceac30a6a1973f5a784406bb4b0"},"cell_type":"markdown","source":"* **Objective:** Use inputs $X$ to predict the output $\\hat{y}$ using a linear model. The model will be a line of best fit that minimizes the distance between the predicted and target outcomes. Training data $(X, y)$ is used to train the model and learn the weights $W$ using stochastic gradient descent (SGD).\n* **Advantages:**\n * Computationally simple.\n * Highly interpretable.\n * Can account for continuous and categorical features.\n* **Disadvantages:**\n * The model will perform well only when the data is linearly separable (for classification).\n * Usually not used for classification and only for regression.\n* **Miscellaneous:** You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuos regression tasks only.\n"},{"metadata":{"_uuid":"c2aab2e031206d6c50e1c393a7164ecc00e84ddc"},"cell_type":"markdown","source":"### Training\n\n*Steps*: \n1. Randomly initialize the model's weights $W$.\n2. Feed inputs $X$ into the model to receive the predictions $\\hat{y}$.\n3. Compare the predictions $\\hat{y}$ with the actual target values $y$ with the objective (cost) function to determine loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it. (the $\\frac{1}{2}$ is just for convenicing the derivative operation).\n * $MSE = J(\\theta) = \\frac{1}{2}\\sum_{i}(\\hat{y}_i - y_i)^2$\n4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights.\n * $J(\\theta) = \\frac{1}{2}\\sum_{i}(\\hat{y}_i - y_i)^2 = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 $\n * $\\frac{\\partial{J}}{\\partial{W}} = X(\\hat{y} - y)$\n4. Apply backpropagation to update the weights $W$ using a learning rate $\\alpha$ and an optimization technique (ie. stochastic gradient descent). The simplified intuition is that the gradient tells you the direction for how to increase something so subtracting it will help you go the other way since we want to decrease loss $J(\\theta)$.\n * $W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n5. Repeat steps 2 - 4 until model performs well."},{"metadata":{"_uuid":"2cc0cc3eaf2cda8417409530bd1e888e5c158440"},"cell_type":"markdown","source":"### Data\n\nWe're going to create some simple dummy data to apply linear regression on."},{"metadata":{"trusted":true,"_uuid":"9391dc4f0d493d98079efd70c9ac4bb4de160c6e"},"cell_type":"code","source":"from argparse import Namespace\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26b3b64416157b292f552caba0ce980ba3dc02da"},"cell_type":"code","source":"# Arguments\nargs = Namespace(\n seed=1234,\n data_file=\"sample_data.csv\",\n num_samples=100,\n train_size=0.75,\n test_size=0.25,\n num_epochs=100,\n)\n\n# Set seed for reproducability\nnp.random.seed(args.seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"492ed691531684dcb4746bc2fc58c7d7576f97ef"},"cell_type":"code","source":"# Generate synthetic data\ndef generate_data(num_samples):\n X = np.array(range(num_samples))\n y = 3.65*X + 10\n return X, y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2dda300e1a1889c5ed6cd176ea3fc364190f0fab"},"cell_type":"code","source":"# Generate random (linear) data\nX, y = generate_data(args.num_samples)\ndata = np.vstack([X, y]).T\ndf = pd.DataFrame(data, columns=['X', 'y'])\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"769ac7ca18d21f84142c97d565faf72236847a97"},"cell_type":"code","source":"# Scatter plot\nplt.title(\"Generated data\")\nplt.scatter(x=df[\"X\"], y=df[\"y\"])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24f7d1745b0db34f367f744f9b7581e2be57d98d"},"cell_type":"markdown","source":"### Scikit-learn Implementation\n\n**Note**: The `LinearRegression` class in Scikit-learn uses the normal equation to solve the fit. However, we are going to use Scikit-learn's `SGDRegressor` class which uses stochastic gradient descent. We want to use this optimization approach because we will be using this for the models in subsequent lessons."},{"metadata":{"trusted":true,"_uuid":"d9871d90bd05fee1f669d8212d0ee0198e3e5162"},"cell_type":"code","source":"# Import packages\nfrom sklearn.linear_model.stochastic_gradient import SGDRegressor\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90b8cca836e9281e87504652d106b40583b2cd5b"},"cell_type":"code","source":"# Create data splits\nX_train, X_test, y_train, y_test = train_test_split(\n df[\"X\"].values.reshape(-1, 1), df[\"y\"], test_size=args.test_size, \n random_state=args.seed)\nprint (\"X_train:\", X_train.shape)\nprint (\"y_train:\", y_train.shape)\nprint (\"X_test:\", X_test.shape)\nprint (\"y_test:\", y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72afc98b081abee3a19a2b082e6a1bcaa0873f8f"},"cell_type":"markdown","source":"We need to standardize our data (zero mean and unit variance) in order to properly use SGD and optimize quickly."},{"metadata":{"trusted":true,"_uuid":"649deeac37867b5b20139d54510e71ff77a83661"},"cell_type":"code","source":"# Standardize the data (mean=0, std=1) using training data\nX_scaler = StandardScaler().fit(X_train)\ny_scaler = StandardScaler().fit(y_train.values.reshape(-1,1))\n\n# Apply scaler on training and test data\nstandardized_X_train = X_scaler.transform(X_train)\nstandardized_y_train = y_scaler.transform(y_train.values.reshape(-1,1)).ravel()\nstandardized_X_test = X_scaler.transform(X_test)\nstandardized_y_test = y_scaler.transform(y_test.values.reshape(-1,1)).ravel()\n\n\n# Check\nprint (\"mean:\", np.mean(standardized_X_train, axis=0), \n np.mean(standardized_y_train, axis=0)) # mean should be ~0\nprint (\"std:\", np.std(standardized_X_train, axis=0), \n np.std(standardized_y_train, axis=0)) # std should be 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99bca189430dc9ef5236d2564a2f62aed4cf615b"},"cell_type":"code","source":"# Initialize the model\nlm = SGDRegressor(loss=\"squared_loss\", penalty=\"none\", max_iter=args.num_epochs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a6ab84903a85ed43aab9d78ea058df05647a9b14"},"cell_type":"code","source":"# Train\nlm.fit(X=standardized_X_train, y=standardized_y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0f2fad5c7d39c71f31e64378507480df71e2e322"},"cell_type":"code","source":"# Predictions (unstandardize them)\npred_train = (lm.predict(standardized_X_train) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\npred_test = (lm.predict(standardized_X_test) * np.sqrt(y_scaler.var_)) + y_scaler.mean_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4113921082df0eb011a6c25f679d023da1a263fd"},"cell_type":"markdown","source":"### Evaluation\n\nThere are several evaluation techniques to see how well our model performed."},{"metadata":{"trusted":true,"_uuid":"80b8f9d5f6ea03ef285ba8bb428ad18059c78b38"},"cell_type":"code","source":"import matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47fa4877946a76e5fa0464014661380ee3472007"},"cell_type":"code","source":"# Train and test MSE\ntrain_mse = np.mean((y_train - pred_train) ** 2)\ntest_mse = np.mean((y_test - pred_test) ** 2)\nprint (\"train_MSE: {0:.2f}, test_MSE: {1:.2f}\".format(train_mse, test_mse))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d3668039d6392edfebeac97448251b68daee9bc"},"cell_type":"markdown","source":"Besides MSE, when we only have one feature, we can visually inspect the model."},{"metadata":{"trusted":true,"_uuid":"3fb8d118527a8c295ed60f63628c547388952812"},"cell_type":"code","source":"# Figure size\nplt.figure(figsize=(15,5))\n\n# Plot train data\nplt.subplot(1, 2, 1)\nplt.title(\"Train\")\nplt.scatter(X_train, y_train, label=\"y_train\")\nplt.plot(X_train, pred_train, color=\"red\", linewidth=1, linestyle=\"-\", label=\"lm\")\nplt.legend(loc='lower right')\n\n# Plot test data\nplt.subplot(1, 2, 2)\nplt.title(\"Test\")\nplt.scatter(X_test, y_test, label=\"y_test\")\nplt.plot(X_test, pred_test, color=\"red\", linewidth=1, linestyle=\"-\", label=\"lm\")\nplt.legend(loc='lower right')\n\n# Show plots\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"888ebefbbc0430e64f93f1d8ae4462ceb135e296"},"cell_type":"markdown","source":"### Inference"},{"metadata":{"trusted":true,"_uuid":"fdcebba46c73362ecd3d084911957cfb1769f6ff"},"cell_type":"code","source":"# Feed in your own inputs\nX_infer = np.array((0, 1, 2), dtype=np.float32)\nstandardized_X_infer = X_scaler.transform(X_infer.reshape(-1, 1))\npred_infer = (lm.predict(standardized_X_infer) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\nprint (pred_infer)\ndf.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ddb09ca0df1572a7ce2119077dedb4455fc51ee"},"cell_type":"markdown","source":"### Interpretability\n\nLinear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies it's importance/impact on the output variable y. We can interpret our coefficient as follows: By increasing X by 1 unit, we increase y by $W$ (~3.65) units. \n\n**Note**: Since we standardized our inputs and outputs for gradient descent, we need to apply an operation to our coefficients and intercept to interpret them. See proof below."},{"metadata":{"trusted":true,"_uuid":"b0a55407d607ad63c1ad90c08570bc599b4430ec"},"cell_type":"code","source":"# Unstandardize coefficients \ncoef = lm.coef_ * (y_scaler.scale_/X_scaler.scale_)\nintercept = lm.intercept_ * y_scaler.scale_ + y_scaler.mean_ - np.sum(coef*X_scaler.mean_)\nprint (coef) # ~3.65\nprint (intercept) # ~10","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed069f4162451dd62a580f847fc12327d6587ea9"},"cell_type":"markdown","source":"### Proof for unstandardizing coefficients:\n\n\nNote that both X and y were standardized.\n\n$\\frac{\\mathbb{E}[y] - \\hat{y}}{\\sigma_y} = W_0 + \\sum_{j=1}^{k}W_jz_j$\n\n$z_j = \\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n\n$ \\hat{y}_{scaled} = \\frac{\\hat{y}_{unscaled} - \\bar{y}}{\\sigma_y} = \\hat{W_0} + \\sum_{j=1}^{k} \\hat{W}_j (\\frac{x_j - \\bar{x}_j}{\\sigma_j}) $\n\n$\\hat{y}_{unscaled} = \\hat{W}_0\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} \\hat{W}_j(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j + \\sum_{j=1}^{k}(\\frac{\\sigma_y}{\\sigma_j})x_j $\n"},{"metadata":{"_uuid":"ff6076ced433e1193d609644cebd22faa28068e5"},"cell_type":"markdown","source":"### Regularization\n\nRegularization helps decrease over fitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing the weights with large magnitudes by decaying them. Having certain weights with high magnitudes will lead to preferential bias with the inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature cofficients are zeroed out, or elastic which combines L1 and L2 penalties. \n\n**Note**: Regularization is not just for linear regression. You can use it to regualr any model's weights including the ones we will look at in future lessons."},{"metadata":{"_uuid":"f6862a8b61e4b240e454a5530ac57bf974988653"},"cell_type":"markdown","source":"* $ J(\\theta) = = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 + \\frac{\\lambda}{2}\\sum\\sum W^2$\n* $ \\frac{\\partial{J}}{\\partial{W}} = X (\\hat{y} - y) + \\lambda W $\n* $W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\nwhere:\n * $\\lambda$ is the regularzation coefficient"},{"metadata":{"trusted":true,"_uuid":"97119da9b555347ddda37d984be6f7f5bf77d240"},"cell_type":"code","source":"# Initialize the model with L2 regularization\nlm = SGDRegressor(loss=\"squared_loss\", penalty='l2', alpha=1e-2, \n max_iter=args.num_epochs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2a94b1c40fc075facb1b5b41c49a84528064e992"},"cell_type":"code","source":"# Train\nlm.fit(X=standardized_X_train, y=standardized_y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd317ec5cd6c35798b65561babc0e7eaaaaa97c7"},"cell_type":"code","source":"# Predictions (unstandardize them)\npred_train = (lm.predict(standardized_X_train) * np.sqrt(y_scaler.var_)) + y_scaler.mean_\npred_test = (lm.predict(standardized_X_test) * np.sqrt(y_scaler.var_)) + y_scaler.mean_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6fcbeb98a1af7605ed175573cfab8966a90ba2f4"},"cell_type":"code","source":"# Train and test MSE\ntrain_mse = np.mean((y_train - pred_train) ** 2)\ntest_mse = np.mean((y_test - pred_test) ** 2)\nprint (\"train_MSE: {0:.2f}, test_MSE: {1:.2f}\".format(\n train_mse, test_mse))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96415731104e638d1f186b440659808d30a2e70f"},"cell_type":"markdown","source":"Regularization didn't help much with this specific example because our data is generation from a perfect linear equation but for realistic data, regularization can help our model generalize well."},{"metadata":{"trusted":true,"_uuid":"6176ac8e13e7c8850081c08d7a4913ccab9b0035"},"cell_type":"code","source":"# Unstandardize coefficients \ncoef = lm.coef_ * (y_scaler.scale_/X_scaler.scale_)\nintercept = lm.intercept_ * y_scaler.scale_ + y_scaler.mean_ - (coef*X_scaler.mean_)\nprint (coef) # ~3.65\nprint (intercept) # ~10","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c1c5a821a01e87864206f2bfb575a3193a66d848"},"cell_type":"markdown","source":"### Categorical variables\n\nIn our example, the feature was a continuous variable but what if we also have features that are categorical? One option is to treat the categorical variables as one-hot encoded variables. This is very easy to do with Pandas and once you create the dummy variables, you can use the same steps as above to train your linear model."},{"metadata":{"trusted":true,"_uuid":"5f7c6aeb9b9be038a7d730576a60b95016b6805c"},"cell_type":"code","source":"# Create data with categorical features\ncat_data = pd.DataFrame(['a', 'b', 'c', 'a'], columns=['favorite_letter'])\ncat_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9099b0413c416d384e9013f47dab702afdb71beb"},"cell_type":"code","source":"dummy_cat_data = pd.get_dummies(cat_data)\ndummy_cat_data.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4725ce85b87ddc4fb1c370219a76499e37f00090"},"cell_type":"markdown","source":"Now you can concat this with your continuous features and train the linear model.\n\n### TODO\n\n- polynomial regression\n- simple example with normal equation method (sklearn.linear_model.LinearRegression) with pros and cons vs. SGD linear regression\n\n###### [Go to top](#top)"},{"metadata":{"_uuid":"f33b5e3196388b80d421c1008c5d6ac056dc96ad"},"cell_type":"markdown","source":"
\n## 1.5 Logistic Regression\n\nIn the previous lesson, we saw how linear regression works really well for predicting continuous outputs that can easily fit to a line/plane. But linear regression doesn't fare well for classification asks where we want to probabilititcally determine the outcome for a given set on inputs."},{"metadata":{"_uuid":"7edf8b2cb3b904cf41959d904bf7af5d1a6d1dad"},"cell_type":"markdown","source":"\n\n$ \\hat{y} = \\frac{1}{1 + e^{-XW}} $ \n\n*where*:\n* $\\hat{y}$ = prediction | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n\nThis is the binomial logistic regression. The main idea is to take the outputs from the linear equation ($z=XW$) and use the sigmoid (logistic) function ($\\frac{1}{1+e^{-z}}$) to restrict the value between (0, 1). "},{"metadata":{"_uuid":"b01e52b990f8a8c88c5cf7598736cbc3974e4bb3"},"cell_type":"markdown","source":"When we have more than two classes, we need to use multinomial logistic regression (softmax classifier). The softmax classifier will use the linear equation ($z=XW$) and normalize it to product the probabiltiy for class y given the inputs.\n\n$ \\hat{y} = \\frac{e^{XW_y}}{\\sum e^{XW}} $ \n\n*where*:\n* $\\hat{y}$ = prediction | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n* $W$ = weights | $\\in \\mathbb{R}^{DXC}$ ($C$ is the number of classes)\n"},{"metadata":{"_uuid":"1f485bbf5698bebce757f6e76ab8598d4b347c14"},"cell_type":"markdown","source":"* **Objective:** Predict the probability of class $y$ given the inputs $X$. The softmax classifier normalizes the linear outputs to determine class probabilities. \n* **Advantages:**\n * Can predict class probabilities given a set on inputs.\n* **Disadvantages:**\n * Sensitive to outliers since objective is minimize cross entropy loss. (Support vector machines ([SVMs](https://towardsdatascience.com/support-vector-machine-vs-logistic-regression-94cc2975433f)) are a good alternative to counter outliers).\n* **Miscellaneous:** Softmax classifier is going to used widely in neural network architectures as the last layer since it produces class probabilities."},{"metadata":{"_uuid":"2699700c3a6c06ff1504c946ec55bec898586567"},"cell_type":"markdown","source":"# Training\n\n*Steps*:\n\n1. Randomly initialize the model's weights $W$.\n2. Feed inputs $X$ into the model to receive the logits ($z=XW$). Apply the softmax operation on the logits to get the class probabilies $\\hat{y}$ in one-hot encoded form. For example, if there are three classes, the predicted class probabilities could look like [0.3, 0.3, 0.4]. \n3. Compare the predictions $\\hat{y}$ (ex. [0.3, 0.3, 0.4]]) with the actual target values $y$ (ex. class 2 would look like [0, 0, 1]) with the objective (cost) function to determine loss $J$. A common objective function for logistics regression is cross-entropy loss. \n * $J(\\theta) = - \\sum_i y_i ln (\\hat{y_i}) = - \\sum_i y_i ln (\\frac{e^{X_iW_y}}{\\sum e^{X_iW}}) $\n * $y$ = [0, 0, 1]\n * $\\hat{y}$ = [0.3, 0.3, 0.4]]\n * $J(\\theta) = - \\sum_i y_i ln (\\hat{y_i}) = - \\sum_i y_i ln (\\frac{e^{X_iW_y}}{\\sum e^{X_iW}}) = - \\sum_i [0 * ln(0.3) + 0 * ln(0.3) + 1 * ln(0.4)] = -ln(0.4) $\n * This simplifies our cross entropy objective to the following: $J(\\theta) = - ln(\\hat{y_i})$ (negative log likelihood).\n * $J(\\theta) = - ln(\\hat{y_i}) = - ln (\\frac{e^{X_iW_y}}{\\sum_i e^{X_iW}}) $\n4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights. Let's assume that our classes are mutually exclusive (a set of inputs could only belong to one class).\n * $\\frac{\\partial{J}}{\\partial{W_j}} = \\frac{\\partial{J}}{\\partial{y}}\\frac{\\partial{y}}{\\partial{W_j}} = - \\frac{1}{y}\\frac{\\partial{y}}{\\partial{W_j}} = - \\frac{1}{\\frac{e^{W_yX}}{\\sum e^{XW}}}\\frac{\\sum e^{XW}e^{W_yX}0 - e^{W_yX}e^{W_jX}X}{(\\sum e^{XW})^2} = \\frac{Xe^{W_jX}}{\\sum e^{XW}} = XP$\n * $\\frac{\\partial{J}}{\\partial{W_y}} = \\frac{\\partial{J}}{\\partial{y}}\\frac{\\partial{y}}{\\partial{W_y}} = - \\frac{1}{y}\\frac{\\partial{y}}{\\partial{W_y}} = - \\frac{1}{\\frac{e^{W_yX}}{\\sum e^{XW}}}\\frac{\\sum e^{XW}e^{W_yX}X - e^{W_yX}e^{W_yX}X}{(\\sum e^{XW})^2} = \\frac{1}{P}(XP - XP^2) = X(P-1)$\n5. Apply backpropagation to update the weights $W$ using gradient descent. The updates will penalize the probabiltiy for the incorrect classes (j) and encourage a higher probability for the correct class (y).\n * $W_i = W_i - \\alpha\\frac{\\partial{J}}{\\partial{W_i}}$\n6. Repeat steps 2 - 4 until model performs well."},{"metadata":{"_uuid":"8cdbfb1a227da5b940d59156d59265b2573bd290"},"cell_type":"markdown","source":"# Data\n\nWe're going to the load the titanic dataset we looked at in lesson 03_Pandas."},{"metadata":{"trusted":true,"_uuid":"ff2e95b231eb4c6d3988b0536231c443d6648f24"},"cell_type":"code","source":"from argparse import Namespace\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport urllib","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d2fc07ea4bb6156205254b5350b3ad6c33fc8de1"},"cell_type":"code","source":"# Arguments\nargs = Namespace(\n seed=1234,\n data_file=\"titanic.csv\",\n train_size=0.75,\n test_size=0.25,\n num_epochs=100,\n)\n\n# Set seed for reproducability\nnp.random.seed(args.seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18866041b8859feebfab7b71f979feacc9312ffb"},"cell_type":"code","source":"# Upload data from GitHub to notebook's local drive\nurl = \"https://raw.githubusercontent.com/ArunkumarRamanan/practicalAI/master/data/titanic.csv\"\nresponse = urllib.request.urlopen(url)\nhtml = response.read()\nwith open(args.data_file, 'wb') as f:\n f.write(html)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4abc8ad4f5ccc0cfe1fbee9155a19a3841506d08"},"cell_type":"code","source":"# Read from CSV to Pandas DataFrame\ndf = pd.read_csv(args.data_file, header=0)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5969b7f723ed23a133f9533c7abeaac48368b30e"},"cell_type":"markdown","source":"# Scikit-learn implementation\n\n**Note**: The `LogisticRegression` class in Scikit-learn uses coordinate descent to solve the fit. However, we are going to use Scikit-learn's `SGDClassifier` class which uses stochastic gradient descent. We want to use this optimization approach because we will be using this for the models in subsequent lessons."},{"metadata":{"trusted":true,"_uuid":"b44faa037f6b1180a9abafd5e4665c72ba6facb1"},"cell_type":"code","source":"# Import packages\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8bd9559592191e5ee512f7ebe187c17302abe4af"},"cell_type":"code","source":"# Preprocessing\ndef preprocess(df):\n \n # Drop rows with NaN values\n df = df.dropna()\n\n # Drop text based features (we'll learn how to use them in later lessons)\n features_to_drop = [\"name\", \"cabin\", \"ticket\"]\n df = df.drop(features_to_drop, axis=1)\n\n # pclass, sex, and embarked are categorical features\n categorical_features = [\"pclass\",\"embarked\",\"sex\"]\n df = pd.get_dummies(df, columns=categorical_features)\n\n return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3ebe964fdc508f8d7873285ee438c1cdc05dc5fc"},"cell_type":"code","source":"# Preprocess the dataset\ndf = preprocess(df)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a7088ddbf9c2e7b3dcf3895b4b9b19074b137bcc"},"cell_type":"code","source":"# Split the data\nmask = np.random.rand(len(df)) < args.train_size\ntrain_df = df[mask]\ntest_df = df[~mask]\nprint (\"Train size: {0}, test size: {1}\".format(len(train_df), len(test_df)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dbfc38f7cddaed0a693163d31ad11954d40ce620"},"cell_type":"markdown","source":"**Note**: If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before spplying those operations. This is because we cannot apply any knowledge gained from the test set accidentally during preprocessing/training."},{"metadata":{"trusted":true,"_uuid":"017398d945754bcf74858f7c5e11bfbd3ab8c948"},"cell_type":"code","source":"# Separate X and y\nX_train = train_df.drop([\"survived\"], axis=1)\ny_train = train_df[\"survived\"]\nX_test = test_df.drop([\"survived\"], axis=1)\ny_test = test_df[\"survived\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61f00cbfd7d67eb2e4f639281c6b6ae6fff711be"},"cell_type":"code","source":"# Standardize the data (mean=0, std=1) using training data\nX_scaler = StandardScaler().fit(X_train)\n\n# Apply scaler on training and test data (don't standardize outputs for classification)\nstandardized_X_train = X_scaler.transform(X_train)\nstandardized_X_test = X_scaler.transform(X_test)\n\n# Check\nprint (\"mean:\", np.mean(standardized_X_train, axis=0)) # mean should be ~0\nprint (\"std:\", np.std(standardized_X_train, axis=0)) # std should be 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"efeb5023e484eead48481496b04725b9ebfb3c73"},"cell_type":"code","source":"# Initialize the model\nlog_reg = SGDClassifier(loss=\"log\", penalty=\"none\", max_iter=args.num_epochs, \n random_state=args.seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e94da629053684f5fe97fe8f02e17776c45fac1b"},"cell_type":"code","source":"# Train\nlog_reg.fit(X=standardized_X_train, y=y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"728aa520a2967544f4f0319536c3aadcd59cae90"},"cell_type":"code","source":"# Probabilities\npred_test = log_reg.predict_proba(standardized_X_test)\nprint (pred_test[:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7ea24ad6510bab4c7c31fa0fd986112eab0870e"},"cell_type":"code","source":"# Predictions (unstandardize them)\npred_train = log_reg.predict(standardized_X_train) \npred_test = log_reg.predict(standardized_X_test)\nprint (pred_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60ff27e91252efd8f47a74da153c007517e1c909"},"cell_type":"markdown","source":"# To Be Updated Soon\n###### [Go to top](#top)"},{"metadata":{"_uuid":"edbdc600a631c345dd59b9a7a370b311383dbba6"},"cell_type":"markdown","source":"## Credits (Reference)\n\n> - [practicalAI - Goku Mohandas](https://github.com/GokuMohandas/practicalAI/)\n> - [GitHub Awesome Lists Topic](https://github.com/topics/awesome)\n> - [GitHub Machine Learning Topic](https://github.com/topics/machine-learning)\n> - [GitHub Deep Learning Topic](https://github.com/topics/deep-learning)\n> - [GitHub Awesome Lists Topic](https://github.com/topics/awesome)\n\n## License\n\n[![MIT](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://raw.githubusercontent.com/ArunkumarRamanan/practicalAI/master/LICENSE)\n\n### Please ***UPVOTE*** my kernel if you like it or wanna fork it.\n\n##### Feedback: If you have any ideas or you want any other content to be added to this curated list, please feel free to make any comments to make it better.\n#### I am open to have your *feedback* for improving this ***kernel***\n###### Hope you enjoyed this kernel!\n\n### Thanks for visiting my *Kernel* and please *UPVOTE* to stay connected and follow up the *further updates!*"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}