{"cells":[{"metadata":{"id":"eirndQVvL9_Q","colab_type":"text","_uuid":"3d13eb477e91cb3af5f81d998f9004ee296f4da5"},"cell_type":"markdown","source":"#  Intro to Machine Learning\nThis is a free course offered by [fast.ai](http://www.fast.ai/) (currently [unlisted](http://forums.fast.ai/t/another-treat-early-access-to-intro-to-machine-learning-videos/6826)). There's a github [repository](https://github.com/fastai/fastai/tree/master/courses/ml1).\n\n## About this course\nSome machine learning courses can leave you confused by the enormous range of techniques shown and can make it difficult to have a practical understanding of how to apply them.\n\nThe good news is that modern machine learning can be distilled down to a couple of key techniques that are of very wide applicability. Recent studies have shown that the vast majority of datasets can be best modeled with just two methods:\n\n- **Ensembles of decision trees** (i.e. Random Forests and Gradient Boosting Machines), mainly for **structured data** (such as you might find in a database table at most companies)\n- **Multi-layered neural networks learnt with Stochastic Gradient Descent** (SGD) (i.e. shallow and/or deep learning), mainly for **unstructured data** (such as audio, vision, and natural language)\n\n### The lessons\nIn this course we'll be learning about:\n- **Random Forests** \n- **Stochastic Gradient Descent**.\n- **Gradient Boosting** \n- **Deep Learning**\n\n### The dataset\nWe will be teaching the course using the [Blue Book for Bulldozers Kaggle Competition](https://www.kaggle.com/c/bluebook-for-bulldozers): \n- \"The goal of the contest is to predict the sale price of a particular piece of heavy equiment at auction based on it's usage, equipment type, and configuration. The data is sourced from auction result postings and includes information on usage and equipment configurations.\"\n\n### Note:\nThese are personal notes. For the original code, check the github repository of the course. Also, I will be importing things as I need them.\n"},{"metadata":{"_uuid":"f76f5fe1f9b96b115ec894d4c318751dff3a3cfc"},"cell_type":"markdown","source":"# Lecture 1\nIt is recommended to [watch the video-lecture first](https://youtu.be/CzdWqFTmn0Y), then follow the notebook."},{"metadata":{"id":"Rc6l_PkHeCot","colab_type":"text","_uuid":"161169c7b8fcfa19f4516da9d0c35904b636e1d9"},"cell_type":"markdown","source":"# Introduction to Random Forests\n## The data\nFor [this competition](https://www.kaggle.com/c/bluebook-for-bulldozers/data), you are **predicting the sale price of bulldozers sold at auctions**.\n\nThe data for this competition is split into three parts:\n\n- **Train.csv** is the training set, which contains data through the end of 2011.\n- **Valid.csv** is the validation set, which contains data from January 1, 2012 - April 30, 2012 You make predictions on this set throughout the majority of the competition. Your score on this set is used to create the public leaderboard.\n- **Test.csv** is the test set, which won't be released until the last week of the competition. It contains data from May 1, 2012 - November 2012. Your score on the test set determines your final rank for the competition.\n\nThe key fields are in train.csv are:\n\n- **SalesID**: the uniue identifier of the sale\n- **MachineID**: the unique identifier of a machine. A machine can be sold multiple times\n- **saleprice**: what the machine sold for at auction (only provided in train.csv)\n- **saledate**: the date of the sale\n\n### Exploring the data"},{"metadata":{"id":"78yfUkWxL-Cl","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"8daeb3bb57d38da1fc83035a5b7382d882b97e78","collapsed":true},"cell_type":"code","source":"%%time\n# times the whole cell\n\n# import pandas\nimport pandas as pd\n\n# set the path to read Train.csv\npath = '../input/Train.csv'\n\n# read the data into a pandas DataFrame\ndf_raw = pd.read_csv(path,\n                     low_memory=False, # use as much memory as necessary to figure out dtypes\n                     parse_dates=[\"saledate\"]) # read saledate column as datetime dtype","execution_count":null,"outputs":[]},{"metadata":{"id":"dDjFACw-L-C3","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"4e163c2ec4e14e18ac822277e4f01cc717560d12","collapsed":true},"cell_type":"code","source":"# dimensions (rows, columns)\ndf_raw.shape","execution_count":null,"outputs":[]},{"metadata":{"id":"0Z2qo1fGLPRZ","colab_type":"text","_uuid":"c60c0e0492758f3e85434acb164e2a77372127db"},"cell_type":"markdown","source":"It's a relatively **large dataset**."},{"metadata":{"id":"R3fHc3GTL-DG","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"95b12a7a37c41e021a51c237f6020e3ce2b999ea","collapsed":true},"cell_type":"code","source":"# print 5 first rows\ndf_raw.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"YNXU7EECgLPG","colab_type":"text","_uuid":"9d6fb5f5c2e95109b008e525e17fed3e8a8972b2"},"cell_type":"markdown","source":"Pandas switchs to **truncate view** when there are too many rows and columns, but we can change the display options if we want to see all of them."},{"metadata":{"id":"GbZAJkq3L-DU","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"21b233508616a39acb2cb9b6524464f4c6d15338"},"cell_type":"code","source":"# create a function that displays up to a thousand rows and columns of a dataframe\n# pd.option_context(*args) is a context manager to temporarily set options in the `with` statement context\ndef display_all(df):\n    with pd.option_context(\"display.max_rows\", 1000, \"display.max_columns\", 1000): \n        display(df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24d83bf31fab8e2a394413a75a120c9dafe9faf0"},"cell_type":"markdown","source":"Now we can display all the rows and columns whenever we need it.\n- For readability, we'll use the **.T attribute**, equal to the .transpose() method, which cause the rows and columns to exchange places.\n- Like so, we have to **scroll down** instead of sideways to see all the columns.\n"},{"metadata":{"id":"7dDQrDV9L-ED","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"b5255ba0088693e5f72ee24e0cdf62ad3de637fd","collapsed":true},"cell_type":"code","source":"# show the tail (last 5 rows)\n# view transposed\ndisplay_all(df_raw.tail().T)","execution_count":null,"outputs":[]},{"metadata":{"id":"CxYnT1eJL-EW","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"984b3031a2efaa98231a19c9183b488a172cf48a","collapsed":true},"cell_type":"code","source":"# generate descriptive statistics of all columns (including object columns)\ndisplay_all(df_raw.describe(include='all'))","execution_count":null,"outputs":[]},{"metadata":{"id":"8tUsrZmHmW8q","colab_type":"text","_uuid":"2acde466cc28fa2bbfa97baea97c9c814507a3bb"},"cell_type":"markdown","source":"### Initial processing\n#### SalePrice\n- SalePrice is the **dependent variable**, the target.\n- Kaggle tells us that the evaluation metric for this competition is the **RMSLE** (root mean squared log error) between the actual and predicted auction prices.\n- Therefore we take the **log of the prices**, so that RMSE will give us what we need."},{"metadata":{"id":"0pcScd3GL-Ey","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"1a51c4f3439c3bfd17374fce31e436de5c949cba","collapsed":true},"cell_type":"code","source":"#import numpy\nimport numpy as np\n\n# replacing SalePrice column values with the calculated natural logarithm of SalePrice values.\ndf_raw['SalePrice'] = np.log(df_raw.SalePrice)\n\n# show head of column\ndf_raw.SalePrice.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"xrkzKRDUOl73","colab_type":"text","_uuid":"e8b6deb7a1f26df662e4f42fcfd4535b46971481"},"cell_type":"markdown","source":"We need to **split** the **dependent variable** from the dataset.\n- Later on, we'll use a method from the **fastai library** , **`proc_df`**,  to do just that and more.\n- We show some alternative code below to illustrate the process."},{"metadata":{"id":"guxW0qdFOmhG","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"8c85ea00615863e38333dd0dbc5b2894ed68e688"},"cell_type":"code","source":"# DataFrame with a column removed\nindependent = df_raw.drop('SalePrice', axis=1)\n\n# select a Series from the DataFrame\ndependent = df_raw.SalePrice","execution_count":null,"outputs":[]},{"metadata":{"id":"WX11fFC4n4S1","colab_type":"text","_uuid":"c332e858edc5990b4dad2f90fd21bd2932668110"},"cell_type":"markdown","source":"#### SaleDate\nWe already know that one of the features is a date and we've explicitly told pandas to read it as  **datetime dtype**."},{"metadata":{"id":"6ZmNXGMKnu_Y","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"db800371fb036f8a1783cc679e57ae3736ce307b","collapsed":true},"cell_type":"code","source":"# prints first 5 rows of the column\ndf_raw.saledate.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"E4SeBD0hm3EX","colab_type":"text","_uuid":"3cbfbf0de37705bf043dda015f6361ace22ff3ee"},"cell_type":"markdown","source":" **What's in a date is one of the more important pieces of feaute engineering you can do** (i.e. Was it a holiday? Was it raining?).\n\n- We'll use a method from the **fastai library**, **`add_datepart`**, that extracts particular date fields from a complete datetime for the purpose of constructing categoricals. \n- You should always consider this **feature extraction** step when working with date-time dtypes. Without expanding your date-time into these additional fields, you can't capture any trend/cyclical behavior as a function of time at any of these granularities."},{"metadata":{"id":"0Zfk_RvsL-Ga","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"6fb9b833f14e89552ee18cad8fbd28e612d37dc4","collapsed":true},"cell_type":"code","source":"# import add_datepart\nfrom fastai.structured import add_datepart\n\nadd_datepart(df_raw, 'saledate')","execution_count":null,"outputs":[]},{"metadata":{"id":"l5nBAir9o2Pm","colab_type":"text","_uuid":"c01240c428c311a300c67049a634eb0fb8def9c9"},"cell_type":"markdown","source":"We can check the **source code** of a function  by typing its name and **??** (before or after the name)."},{"metadata":{"id":"-_xnk_oPphEc","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"bd27aaf0180e270399e8f26d40682a28d3e845be"},"cell_type":"code","source":"add_datepart??","execution_count":null,"outputs":[]},{"metadata":{"id":"p3EcJkwrpz16","colab_type":"text","_uuid":"4b61fd6d01af087681c5a02cb35f1733ff6dd4a3"},"cell_type":"markdown","source":"So `add_datepart` converts a column of a DataFramefrom a **datetime64 to many columns containing the information from the date** (changes occur inplace)."},{"metadata":{"id":"tbVdJcflqwnB","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"8501fa06bc456a48ed4dbcd527e07c1064a491d4","collapsed":true},"cell_type":"code","source":"# column labels of the DataFrame.\ndf_raw.columns","execution_count":null,"outputs":[]},{"metadata":{"id":"P41PpXZFrTaR","colab_type":"text","_uuid":"03e0b19b452a53fd9d8fd395d373d8450e4f545b"},"cell_type":"markdown","source":"There are a bunch of new columns. Let's take a look."},{"metadata":{"id":"Gf6jDjjnrtvY","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"40ad4eea0d040d5773d7da634953cd331aad1e69","collapsed":true},"cell_type":"code","source":"# acces the selected [rows, columns]. View transposed.\ndf_raw.loc[:,['saleYear', 'saleMonth',\n       'saleWeek', 'saleDay', 'saleDayofweek', 'saleDayofyear',\n       'saleIs_month_end', 'saleIs_month_start', 'saleIs_quarter_end',\n       'saleIs_quarter_start', 'saleIs_year_end', 'saleIs_year_start',\n       'saleElapsed']].T","execution_count":null,"outputs":[]},{"metadata":{"id":"czl3fxYemQEc","colab_type":"text","_uuid":"a085de8b952014cb557d96806b1b33c53191f85f"},"cell_type":"markdown","source":"#### Categorical Data\nAs in most cases when dealing with datasets, this dataset contains a mix of **continuous** and **categorical** variables.\n- We need all of them to be **numeric** so they can be used by the model.\n- We must make the necessary changes (**feature engineering**)."},{"metadata":{"id":"F9Jhp4e6mhmo","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"610fb2fd648d578a7693682494149a2ecd3232f1","collapsed":true},"cell_type":"code","source":"# count dtypes in the dataframe\ndf_raw.dtypes.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"id":"rYR8qDCw3rZ9","colab_type":"text","_uuid":"a7b58c39979c8d17924e2c71c641950c54b258e7"},"cell_type":"markdown","source":"If we look at the numeric columns, we can se that **most of them** are not cotinuous but **categorical**.\n- Even though it's not ideal that they stay that way, **random forest** works fine with it so there is **no problem**."},{"metadata":{"id":"fv-PoSwe3sOS","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"0572fbc514168ffb1507cd02714ea646943239b4","collapsed":true},"cell_type":"code","source":"# select numerical columns\ndf_raw.select_dtypes(['int64','float64']).head()","execution_count":null,"outputs":[]},{"metadata":{"id":"fVQTlOuFL-Gq","colab_type":"text","_uuid":"82bdc41baab70ceec425f99d23f8cda0f9776a7b"},"cell_type":"markdown","source":"Most of the **categorical variables are currently stored as strings** (objects).\n- Apart from being inefficient, it doesn't provide the **numeric coding** required for our model.\n\n- We'll use a method from the **fastai library** , **`train_cats`**,  to convert strings to pandas categories."},{"metadata":{"id":"hVGHnM34L-Gs","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"28a8a73b9c2d62a470b33e8803cdb2414da1f5b4"},"cell_type":"code","source":"# import train_cats\nfrom fastai.structured import train_cats\n\ntrain_cats(df_raw)","execution_count":null,"outputs":[]},{"metadata":{"id":"ihgaiGHBvh29","colab_type":"text","_uuid":"0f9d61bb871a9c5165ed3e90a511f81f5e678b09"},"cell_type":"markdown","source":"We can again check the **source code** of the function for more information."},{"metadata":{"id":"YNKi5Y9RvpX3","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"1f67b5bf043e164fecff2a582f5c00eaebb3f06c"},"cell_type":"code","source":"??train_cats","execution_count":null,"outputs":[]},{"metadata":{"id":"ne3UOKcvwfv3","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"52556b8cff000a1909af0f7f430b228614c22019","collapsed":true},"cell_type":"code","source":"# examine all the data types\ndisplay_all(df_raw.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"id":"luxkAugFwoms","colab_type":"text","_uuid":"355a26262f02baade8a6cabe038356c98c7ec3f1"},"cell_type":"markdown","source":"Indeed it has changed inplace all columns that contained **strings to category dtypes**.\n- It doesn't make the DataFrame look different but behind the scenes all string are encoded with **integers mapped to the strings**.\n- Let's look at UsageBand, our first category column."},{"metadata":{"id":"KL4WKoMWL-G0","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"3ddd050941bb687dcfa782428bc1ec6a1971bb79","collapsed":true},"cell_type":"code","source":"# categories of this categorical\ndf_raw.UsageBand.cat.categories","execution_count":null,"outputs":[]},{"metadata":{"id":"Px8-2ryz4sbl","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"07c6f24699bbb39cc3f0a802dc5926771c6cdf0a","collapsed":true},"cell_type":"code","source":"# encoded variables\ndf_raw.UsageBand.cat.codes.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"-pHHmBjRL-Gy","colab_type":"text","_uuid":"0ccb5a5caac379058b171f01407edc4a7e8b6470"},"cell_type":"markdown","source":"We can **specify the order** to use for categorical variables if we wish:"},{"metadata":{"id":"ofI4CXPNL-HD","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"d1100434259f72575c3b48ade1a46fc0a9d9f68c"},"cell_type":"code","source":"# sets the categories to the specified new_categories, ordered in place.\ndf_raw.UsageBand.cat.set_categories(['High', 'Medium', 'Low'], ordered=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"5Eb9YWOr6lM6","colab_type":"text","_uuid":"f2628086f1b884b1aff30965d379dbc524591527"},"cell_type":"markdown","source":"Finally, to fit this data to a random forest, we would need to **replace the text categories with their numeric codes**.\n- We'll use a method from the **fastai library** , **`proc_df`**,  to do just that and more.\n- We show some alternative code below to illustrate the process."},{"metadata":{"id":"TtwgUCDj6lqA","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"acc6d7181223c9398ce1a9667185b63854033a76","collapsed":true},"cell_type":"code","source":"# make a copy for demonstration purposes\ndf_codes = df_raw.copy()\n\n# iterate through column name and column values and if column values are categories, replace them with their numeric code.\nfor name, column in df_codes.items():\n  if column.dtype.name == 'category':\n    df_codes[name] = column.cat.codes\n    \n# count dtypes in the dataframe\ndf_codes.dtypes.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"id":"J9aL9HJG9A6e","colab_type":"text","_uuid":"896866ff74417b31a2ffcafdcdff2d1320ab49dc"},"cell_type":"markdown","source":"This is a **two step process** for being able to **fit categorical** variables to our model: first we convert **strings to categories**, then **categories to codes**/numbers.\n- The result is that all variables are recoded in **only one column** using different integers for different categories, which is **useful for** columns with **lots of possible values**. \n- An **alternative** to this would be to **get dummy variables** for the categorical variables, but it would resolve it by recoding all the variables in **different columns** with **zeroes** and **ones**.\n\n#### Missing Data\nHandling missing data is important as **many** machine learning **algorithms do not support** data with **missing values** ([source](https://machinelearningmastery.com/handle-missing-data-python/)).\n\nTypically, random forest methods/packages encourage **two ways** of handling missing values ([source](https://medium.com/airbnb-engineering/overcoming-missing-values-in-a-random-forest-classifier-7b1fc1fc03ba)): \n- **Drop data points** with missing values (not recommended)\n- **Fill** in missing values with the **median** (for numerical values)."},{"metadata":{"id":"JRGJMDO3L-HY","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"0bc3edd84b6e5560339359279fe915ea3891be8f","collapsed":true},"cell_type":"code","source":"# display all columns with the percentage of missing values (NaN)\ndisplay_all(df_raw.isnull().sum().sort_index()/len(df_raw))","execution_count":null,"outputs":[]},{"metadata":{"id":"hcuIZImOu-Dm","colab_type":"text","_uuid":"0bed3a1297cedf078457c44536c674f820c04bb2"},"cell_type":"markdown","source":"Pandas has handled all **missing values** in the **categories** automatically by encoding  them with a **minus one**."},{"metadata":{"id":"dzJjalRYIkim","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"3d8c538ea43990fb6cb80da6d91c28ebfcf8ab27","collapsed":true},"cell_type":"code","source":"# categories of this categorical\nprint(df_raw.Hydraulics.cat.categories)\n\n# encoded variables\ndf_raw.Hydraulics.cat.codes.head(6)","execution_count":null,"outputs":[]},{"metadata":{"id":"vP-jL8i-K0UH","colab_type":"text","_uuid":"687c402bb74bc1991d5d4fe95c07ed258a20c8c4"},"cell_type":"markdown","source":"For the **numerical data**, we need to **replace** the **missing values** with the **median** (also called median imputation).\n- We'll use a method from the **fastai library** , **`proc_df`**,  to do just that and more.\n- We show some alternative code below to illustrate the process."},{"metadata":{"id":"hPYcXGp4wAe9","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"de6dbebde62474d0895d126eb419c719527ed2e0","collapsed":true},"cell_type":"code","source":"# select numerical columns\nnum_col = df_raw.select_dtypes(include='number')\n\n# display numerical columns with the percentage of missing values (NaN)\nnum_col.isnull().sum().sort_index()/len(df_raw)","execution_count":null,"outputs":[]},{"metadata":{"id":"z0ltzl5gvG2B","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"2fca6c661302f59f82a4d74672166076e02e432b","collapsed":true},"cell_type":"code","source":"# make a copy for demonstration purposes\ndf_imputed = df_raw.copy()\n\n# iterate through column name and column values and if column values are numerical, replace NA values with the column median.\nfor name, column in df_imputed.items():\n  if column.dtype == np.number:\n    df_imputed[name] = df_imputed[name].fillna(column.median())\n\n# select numerical columns\nnum_col = df_imputed.select_dtypes(include='number')\n\n# display numerical columns with the percentage of missing values (NaN)\nnum_col.isnull().sum().sort_index()/len(df_raw)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"424e27c7b2239332eab8604556c86f3f1d9016ba"},"cell_type":"markdown","source":"## Saving the progress\nWe will **save the dataset** in its current form so we can **save ourselves** the trouble of **repeating the process** of getting the data ready to be to be passed to a random forest in **future lessons**.\n- In kaggle, when we **commit** the kernel, everything that is writed to the current directory is saved as **output**."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"00c9088629ad76d02a31e24025f3657445f80904"},"cell_type":"code","source":"# Save to feather file\ndf_raw.to_feather('df_raw')","execution_count":null,"outputs":[]},{"metadata":{"id":"IN2rX7vYL-ID","colab_type":"text","_uuid":"f9571ae001b1a91008416b643cc2167e4a22b6d8"},"cell_type":"markdown","source":"## Pre-processing\nWe'll use a method from the **fastai library** , **`proc_df`**,  to get the dataset ready for the random forest.\n- We'll **replace categories** with their numeric codes.\n- **Handle missing** continuous **values**.\n- **Split the dependent variable** into a separate variable."},{"metadata":{"id":"XUgqLRWVlxiA","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"d615da2dbde158bc31d3527889ed08bdf28864f8"},"cell_type":"code","source":"# import proc_df and the functions it depends on\nfrom fastai.structured import numericalize, fix_missing, proc_df\n\nX, y, nas_dict = proc_df(df_raw, 'SalePrice')","execution_count":null,"outputs":[]},{"metadata":{"id":"zQdiXfhJ7lDE","colab_type":"text","_uuid":"dc9beb0c3c5406e1e8943c677b92c7b22a61ce83"},"cell_type":"markdown","source":"For more information, we can again check the **source code**:"},{"metadata":{"id":"1dKCRo9m77BC","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"ce8c6ea5eb7c90444662a71e3623bedaa77eaa86"},"cell_type":"code","source":"??proc_df","execution_count":null,"outputs":[]},{"metadata":{"id":"rgM2diC1JJN_","colab_type":"text","_uuid":"0758b39d3e0ab204853df8fba1d97699b54d0d67"},"cell_type":"markdown","source":"The method **`fix_missing`** has returned `nas_dict`, which is a dictionary with the name of the columns that had missing data.\n- From the function docstring: Fill missing data in a column of df with the median, and add a {name}_na column which specifies if the data was missing (boolean).\n- Our model will **not be using it** but we can check it out."},{"metadata":{"id":"Fo_un7fhL-IF","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"eca51125c05ee6ac6b9f3f8bbc57ad6d2d7f7c43","collapsed":true},"cell_type":"code","source":"nas_dict","execution_count":null,"outputs":[]},{"metadata":{"id":"WD2wM04588Li","colab_type":"text","_uuid":"63bca0e3b58ba236cb514a864b4328b1854e3281"},"cell_type":"markdown","source":"So we know the columns MachineHoursCurrentMeter' and 'auctioneerID' had missing values and according to the docstring, `fix_missing` has added **new columns** which specifies if the data was missing. Let's see.'"},{"metadata":{"id":"lgb74IyU-NEW","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"14a44dd416e96fcb9193a08dc0633c594975e114","collapsed":true},"cell_type":"code","source":"# acces columns with names containing some strings\nX.loc[:, X.columns.str.contains('MachineHoursCurrentMeter|auctioneerID')].head()","execution_count":null,"outputs":[]},{"metadata":{"id":"cX-C20IsL-IL","colab_type":"text","_uuid":"43300c9de83865577fefd2845b7b554347774bb0"},"cell_type":"markdown","source":"Anyhow, now the **dataset is ready** to be to be passed to a **random forest**."},{"metadata":{"_uuid":"8e12541d3fe9003d12fbd08e1c3ab3a4969d4406"},"cell_type":"markdown","source":"# Lecture 2\nIt is recommended to [watch the video-lecture first](https://youtu.be/blyXCk4sgEg), then follow the notebook."},{"metadata":{"id":"RAqlKgo8IR5x","colab_type":"text","_uuid":"d164a104d481017abb5b5fb5a379bab54d36cfb3"},"cell_type":"markdown","source":"# Random Forests\n## Base model\n### Validation set\nTo **avoid  overfitting** , we'll split the dataset to have two **separate training and validation sets**.\n- Since we are trying to predict new prices, we should pick the **latest samples** from our train set.\n- We'll use the **same size as** the **Kaggle** validation set, so when we evaluate our model it gives us an estimation of how will it perform in the **public leaderboard**."},{"metadata":{"id":"Lpt-W9jE-_uy","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"763747677f25041880292d5dbbf9d660cd9f3bf9","collapsed":true},"cell_type":"code","source":"# set the path to read the Valid.csv\npath = '../input/Test.csv'\n\n# dimensions of the dataframe (rows, columns)\npd.read_csv(path).shape","execution_count":null,"outputs":[]},{"metadata":{"id":"srvGkHmRCy1Z","colab_type":"text","_uuid":"ed2676fc2c3b92b1e0273db9fc7e8c59812c9413"},"cell_type":"markdown","source":"The set Kaggle used for creating the public leaderboard has roughtly **12,000 samples**, so that is the size we'll use for our validation set."},{"metadata":{"id":"DhBgirfLL-Ih","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"a8ad6873d7811b08e919dcbf62a348397ebcd907","collapsed":true},"cell_type":"code","source":"# create a function for splitting X and y into train and test sets of customizable sizes\ndef split_vals(a,n): return a[:n].copy(), a[n:].copy()\n\n# validation set size: 12000.\nvalidation = 12000\n\n# split point: length of dataset minus validation set size.\nsplit_point = len(X)-validation\n\n# split X\nX_train, X_valid = split_vals(X, split_point)\n\n# split y\ny_train, y_valid = split_vals(y, split_point)\n\n# dimensions (row, columns) of X, y and X_valid\nX_train.shape, y_train.shape, X_valid.shape","execution_count":null,"outputs":[]},{"metadata":{"id":"kxrhqIS_L-Id","colab_type":"text","_uuid":"4d799e7d7806123c7e5e9104ffdf44576367035f"},"cell_type":"markdown","source":"### Model evaluation\n#### RMSE\nRemember the evaluation metric that Kaggle is going to use for this competition is the Root Mean Squared Log Error (**RMSLE**) between the actual and predicted auction prices.\n- Because we **already took the log** of the prices, we can use the Root Mean Squared Error (RMSE) instead.\n\nThe **RMSE** is an evaluation metric that expresses the **average error of the model predictions** by comparing the predicted values with the actual known values.\n- The score can range from **0** (best score possible, never achieved in practice) to **∞**, so the **lower the score the better**.\n\n#### R Squared\nThe coefficient of determination R^2 (**R squared**) is an evaluation metric for regression problems that essentially tells us **how good is our model** compared with **a model that just predicts the average** of the target (saleprice) all the time .\n- The **best** possible score is **1**.\n- The score of the **average** model is **0**.\n- A **negative** score means the model is even **worse** than simply predicting the average of the target."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"025b9759396bc182c1ae2bd6959c7cf25811a09c"},"cell_type":"code","source":"# create a function that takes the RMSE\ndef rmse(pred,known): return np.sqrt(((pred-known)**2).mean())\n\n# create a function that rounds to 5 decimal places (like kaggle leaderboard)\ndef rounded(value): return np.round(value, 5)\n\n# create a function that prints a list of 4 scores, rounded:\n# [RMSE of X_train, RMSE of X_valid, R Squared of X_train, R Squared of X_valid]\ndef print_scores(model):\n    RMSE_train = rmse(model.predict(X_train), y_train)\n    RMSE_valid = rmse(model.predict(X_valid), y_valid)\n    R2_train = model.score(X_train, y_train)\n    R2_valid = model.score(X_valid, y_valid)\n    scores = [rounded(RMSE_train), rounded(RMSE_valid), rounded(R2_train), rounded(R2_valid)]\n    print(scores)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68514c3ac6eacd0859078bcc9f89d6b473bf4e13"},"cell_type":"markdown","source":"We'll use the [random_state](https://www.kaggle.com/questions-and-answers/49890) parameter for reproducibility."},{"metadata":{"id":"twcxynECL-I1","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"604d16174b8eb1bc2fb2fa347ff5a594f2d54478","collapsed":true},"cell_type":"code","source":"# import the class\nfrom sklearn.ensemble import RandomForestRegressor\n\n# instantiate the model \nm = RandomForestRegressor(n_jobs=-1, # n_jobs is a performance parameter. -1 is to paralelize the computations across the number of CPU cores.\n                          random_state=17190)\n# fit the model with data and calculate the running time\n%time m.fit(X_train, y_train)\n\n# print a list of 4 scores:\n# [RMSE of X_train, RMSE of X_valid, R Squared of X_train, R Squared of X_valid]\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"58zIVvfkphM3","colab_type":"text","_uuid":"59f1fbdd63c05d091beff195a806d7f8dcc2a8a9"},"cell_type":"markdown","source":"These four scores allows us to evaluate the accuracy of our model.\n- An **R Squared** of the validation set in the **high-80's** tells us that our model is significantly **better** than simply **predicting the average** price.\n- The large difference of the **RMSLE** of the training (**0.09**) and the validation sets (**0.25**) tells us that we're **over-fitting badly**, because the average error of the model predictions are much greater when dealing with data it hasn't been trained with.\n\n## Speeding things up\nIn order to make the model development process more **interactive**, we need to make sure that the model runs in a reasonable time to be able to **make some changes and see the results fast**.\n- We''ll use the paramater **`subset`** from **`proc_df`** that takes a random subset of the selected size from the dataframe.\n- We'll randomly sample **32,000** from the original 401,125 rows.\n- We'll use the **same validation set** as before.\n- To make sure our training set doesn't overlap with the dates of the validation set we'll **split again the subset** we've taken and **discard the last 12,000** rows."},{"metadata":{"id":"CiZnao-jL-JL","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"6348e3a9185a4ca337d851625eb5fb7429da957f"},"cell_type":"code","source":"# randomly sample 32000 thousand rows\nX_subset, y_subset, nas_dict = proc_df(df_raw, 'SalePrice', subset=32000)\n\n# split the train subset and discard the last 12000 rows: X_train [:20000], _ [20000:].\nX_train, _ = split_vals(X_subset, 20000)\n\n# split the target subset and discard the last 12000 rows: X_train [:20000], _ [20000:].\ny_train, _ = split_vals(y_subset, 20000)","execution_count":null,"outputs":[]},{"metadata":{"id":"4oOB-dhCEMID","colab_type":"text","_uuid":"a41dcb9ed7c2422f0afee864580a50d01f1a85c4"},"cell_type":"markdown","source":"Now we can **train our model with 20,000 randomly chosen samples** from the total 401,125 rows and we can be sure that we're not cheating because the validation set is a different set entirely."},{"metadata":{"id":"_Vo1Fd45L-JV","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"02faa258525adc4d8a662651960e26f0959e0947","collapsed":true},"cell_type":"code","source":"m = RandomForestRegressor(n_jobs=-1)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"MSU2l2OtHK1K","colab_type":"text","_uuid":"c3be02a20683b84f1c14eed73f9d0126a0860885"},"cell_type":"markdown","source":"Well, that was fast.\n## Single tree\nTo **understand** why we were overfitting, we'll create a **random forest** so simple that we can actually **take a look inside**.\n- With the parameter `n_estimators`, we'll create a forest with **only one tree**.\n- With the parameter `max_depth`, we'll create a forest which trees **split only three times**.\n- With the parameter `bootstrap`, we'll **avoid using randomly generated training sets**."},{"metadata":{"id":"3UK_-aTAL-Jc","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"c22f2ff0bca6e04e28dbd9f3498b61799bb02a12","collapsed":true},"cell_type":"code","source":"m = RandomForestRegressor(n_estimators=1, max_depth=3, bootstrap=False, n_jobs=-1, random_state=17190)\nm.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"id":"4pr0AExeL-fN","colab_type":"text","_uuid":"61d3077ea2bf5168fc0cdb660581f3edecd25b9c"},"cell_type":"markdown","source":"Now we'll **take a look inside** this random forest simplified to a **single small deterministic tree**.\n- We'll use a method fromt the **fastai library**, **`draw_tree`**, to draw a representation of the random forest in IPython.\n- This method uses the function **`export_graphviz`**, which generates a GraphViz representation of the decision tree."},{"metadata":{"id":"ruJOs9BuL-Jm","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"8b18a1aaab3bb17fd43a675cb2e156c0b059e4a8","collapsed":true},"cell_type":"code","source":"# import the export_graphviz exporter \nfrom sklearn.tree import export_graphviz\n\n# import draw_tree method\nfrom fastai.structured import draw_tree\n\n# draw a random forest\ndraw_tree(m.estimators_[0], X_train, precision=3)","execution_count":null,"outputs":[]},{"metadata":{"id":"S28ub8s4L-Ju","colab_type":"text","_uuid":"17c04890cb1b8c17a24158484f36b8ad1332039c"},"cell_type":"markdown","source":"A tree consist of a **sequence of binary decisions** or splits.\n- Each step **partitions the data** into two subsets.\n- Each split point is a decision to split the data on a **particular variable** (feature) and in a **particular location** (value).\n- **All possible split points are evaluated** (all features and values).\n- The very **best split point is chosen** each time trying to **minimize the error** (for regression).\n- The error is calculated by **comparing the predicted values with the known values** of the subsets of a particular split point.\n\nEach box in the graphic that represents a split point consists of:\n- A **variable** and a **value** to split on.\n- **MSE** quantifies the error, the lower the better.\n- **Samples** is the size (rows) of the subset.\n- The **average** of the **target** value (log of price) of the subset.\n\nSources: [1](https://machinelearningmastery.com/classification-and-regression-trees-for-machine-learning/), [2](https://infocenter.informationbuilders.com/wf80/index.jsp?topic=%2Fpubdocs%2FRStat16%2Fsource%2Ftopic47.htm).\n\nLooking at our little tree, we can see that we **start with 20,000 rows** (the root node),  **split only three times** (trying to minimze the MSE) and end up with **many subsets of different sizes** (leaf nodes), which are the result of the last split.\n\nIf we create the largest (**deepest**) tree possible, it'll **keep splitting until each leaf node has only one sample**. Such tree would learn all there is to learn about the training data but would be a model that would **not be able to generalize a pattern** from it. And that is exactly what overfitting is: to **learn the noise instead of the signal**. Let's see it in practice."},{"metadata":{"id":"6J1Z9XcqL-Jw","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"ac67c96601d2149df69ef3849ea90b2bbdc94f0e","collapsed":true},"cell_type":"code","source":"# remove the max_depth parameter (by default to None) so the nodes expand until they can't do it anymore. \nm = RandomForestRegressor(n_estimators=1, bootstrap=False, n_jobs=-1, random_state=17190)\n\nm.fit(X_train, y_train)\n\n# [RMSE of X_train, RMSE of X_valid, R Squared of X_train, R Squared of X_valid]\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"smV3Kj2RL-KL","colab_type":"text","_uuid":"90d9b7bd8337d932b12c77385d207bb7443a058b"},"cell_type":"markdown","source":"The scores are **great in the training set**: the RMSE is 0 and the R Squared is 1 (the best possible scores). This is because we can in fact **predict everything** from the training data.\n\nOur interest, of course, is to do **better in the validation set** to get more generalizable results and for that we need to use **multiple trees** and some kind of **model averaging approach** .\n## Bagging\n### Intro to bagging\nBootstrap aggregating (**Bagging**) is a way of doing repeated statistical analyses on the same data and combining them to form a single result.\n- The predictions that are combined are all different thanks to the use of **resampling** (bootstrap samples).\n- **Bootstraping** is a way of generating random non-repeated samples from a dataset.\n- The samples have the **same size** of the original dataset **but** they are all **different** to each other because each time there are rows that are left out and rows that appear more than once ([samples with replacement](https://www.emathzone.com/tutorials/basic-statistics/sampling-with-replacement.html)).\n- These random differences make each sample unique so each model build on that sample can be different.\n\nSo bagging is a way of taking **many** training sets **from the single** original one and build a different prediction model using each training set in order to **average** the resulting **predictions** ([source](http://www2.stat.duke.edu/~rcs46/lectures_2017/08-trees/08-tree-advanced.pdf)).\n\nA **random forest** is simply a way of **bagging trees**.\n- It combines the **unique predictions** of different trees.\n- Each tree is trained on a **randomly generated training set**.\n- It grows **deep trees** that overfit their individual non-repeated subsets.\n- The prediction **error** of the trees are **random** so the **average of the predictions doesn't overfit**.\n\nSources: [1](https://onlinecourses.science.psu.edu/stat857/node/181/), [2](https://machinelearningmastery.com/bagging-and-random-forest-ensemble-algorithms-for-machine-learning/).\n\nTo learn about bagging in random forests, let's look at our **base model** again."},{"metadata":{"id":"r3ufPXDpL-KO","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"4716224267eb35fcc393a776b0d25c4e761947e4","collapsed":true},"cell_type":"code","source":"# 10 trees: it's the default number of trees in the random forest\nm = RandomForestRegressor(n_jobs=-1, random_state=171900)\nm.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"y5OjqWMSL-KW","colab_type":"text","_uuid":"d20138969e9fd989b1aa624740d842ff0141c5ea"},"cell_type":"markdown","source":"We'll grab the **predictions for each individual tree**, and look at one example."},{"metadata":{"id":"3jC3BG75L-Ki","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"08abf6d1c8602ec62bb5455c7a37c7865aba8f60","collapsed":true},"cell_type":"code","source":"# use a list comprehension to loop through the random forest and concatenates the predictions of each individual tree on a new axis\npreds = np.stack([t.predict(X_valid) for t in m.estimators_])\n\n# dimensions of the predictions (rows, columns)\npreds.shape","execution_count":null,"outputs":[]},{"metadata":{"id":"-HkSwtUpVMmH","colab_type":"text","_uuid":"fd9b76d6423df2d844e298308970c0d6db707cd0"},"cell_type":"markdown","source":"There are 10 sets of predictions (trees) with 12,000 values (predictions), which corresponds to the size of the validation set.\n- Let's see what the **first prediction of each tree** does look like and how they **compare** with the actual value **when averaged out**."},{"metadata":{"id":"4UwzEQwvL-Kz","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"63ce9b845343fc6af9841866e4e260d1d7ef4496","collapsed":true},"cell_type":"code","source":"# print the first prediction for each of the ten trees [all tree rows, first prediction column]\nprint(preds[:,0])\n\n# print the mean of the first ten predictions\nprint(np.mean(preds[:,0]))\n\n# the first value of the validation set\ny_valid[0]","execution_count":null,"outputs":[]},{"metadata":{"id":"fFztVWhcL-L_","colab_type":"text","_uuid":"4230dcc11a7bf83eb7171ba30ee7e9603d9a2e2f"},"cell_type":"markdown","source":"Some of the individual tree's predictions are better than the others, but **the mean of them is actually good enough**.\n### Out-of-bag (OOB) score\nOut-of-bag (**OOB**) **error**  is another method of measuring the **prediction performance** of models utilizing **bagging**.\n- Out-of-bag samples are **samples not used** during training for **any given tree**.\n- For **every tree** in the forest there is **unseen data** available to make **new predictions** on.\n- For each **training sample**, calculates the **mean prediction error** using only the trees that were not trained on that sample.\n- **Evaluates** the model on the training set **without** needing a separate **validation set**.\n\nSources: [1](https://en.wikipedia.org/wiki/Out-of-bag_error), [2](http://scikit-learn.org/stable/auto_examples/ensemble/plot_ensemble_oob.html), [3](https://stackoverflow.com/questions/18541923/what-is-out-of-bag-error-in-random-forests).\n\nThis also has the benefit of allowing us to **assess** whether **our model generalizes** even if we **only have** a **small amount of data** so want to avoid separating some out to create a validation set.\n\nWe can add one more parameter to our model, **`oob_score`**, to use out-of-bag samples to estimate the **R^2** on unseen data.\n- There is **minimum number of trees** to be used for any given data that are **necessary** to compute a reliable OOB error (or **sklearn warns** us).\n- We'll also update the `print_scores` function to **print** the  **OOB error** last."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"83228c9021d37849a878b28f1a81ed9a24710c0c"},"cell_type":"code","source":"# appends the oob score to the list of scores if the model has the parameter\ndef print_scores(model):\n    RMSE_train = rmse(model.predict(X_train), y_train)\n    RMSE_valid = rmse(model.predict(X_valid), y_valid)\n    R2_train = model.score(X_train, y_train)\n    R2_valid = model.score(X_valid, y_valid)\n    scores = [rounded(RMSE_train), rounded(RMSE_valid), rounded(R2_train), rounded(R2_valid)]\n    if hasattr(m, 'oob_score_'): scores.append(m.oob_score_) # appends OOB score (if any) to the list \n    print(scores)","execution_count":null,"outputs":[]},{"metadata":{"id":"aJcqwdJlL-MF","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"2e043072ab6c7b4584a80a66ad851fe32a94d0b5","collapsed":true},"cell_type":"code","source":"# 10 trees\nm = RandomForestRegressor(n_jobs=-1, oob_score=True)\nm.fit(X_train, y_train)\n\nprint('10 trees:')\n# [RMSE of X_train, RMSE of X_valid, R Squared of X_train, R Squared of X_valid, OOB score]\nprint_scores(m)\n\n# 30 trees\nm = RandomForestRegressor(n_estimators=30, n_jobs=-1, oob_score=True)\nm.fit(X_train, y_train)\n\nprint('\\n30 trees:')\n# [RMSE of X_train, RMSE of X_valid, R Squared of X_train, R Squared of X_valid, OOB score]\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"dYaY-ukKL-Mm","colab_type":"text","_uuid":"9f1fff36b09995b34fd8d4ceb2b6fb4b5226ce9b"},"cell_type":"markdown","source":"The **OOB** is **better** than the **R Squared** of the validation set because the validation set is a **different time period** entirely, while the OOB is calculated on random samples of the same time period as the training set.\n- In this case the **validation set is much harder to predict** and the R Squared and will consistently be lower than the OOB.\n\n## Hyperparameter tuning\n### Subsampling\nEarlier we used **subsampling** to speed up the analysis, which basically consist in **limiting** the total **amount of data** that our **model can access** so it **trains faster**. There is a **better way** to do it that is actually **also** one of the easiest ways to **avoid overfitting**.\n- Rather than use **one random subset** of the data for our **entire model** (and for all its trees), we'll use a **different random subset per tree**.\n- **Given enough trees**, and enough random subsets, the model will **eventually** be able to **train with** much of **all the available data**.\n- Each **individual tree** it'll be **just as fast** as if we were **using one subset** for all the trees, **like** we did **before**.\n\nWe'll use a method from the **fastai library** , **`set_rf_samples`**, to do just that.\n- From the documentation: \"**changes Scikit learn's random forests** to give each tree a random sample of n random rows\".\n- It is **not compatible** with **OOB score** right now, so will simply do without it.\n- To turn off `set_rf_samples` to use random forest in the normal way, it's necessary to call **`reset_rf_samples`**.\n\nFirst we must **return** to using our **full dataset** so the **subsampling** is done on the **entire data**."},{"metadata":{"id":"sxq1qXRTL-Mo","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"e1bf7b5e24e984a6adc50963a975796c77fe9764"},"cell_type":"code","source":"X, y, nas_dict = proc_df(df_raw, 'SalePrice')\nX_train, X_valid = split_vals(X, split_point)\ny_train, y_valid = split_vals(y, split_point)","execution_count":null,"outputs":[]},{"metadata":{"id":"VCu2sX1WSNdP","colab_type":"text","_uuid":"425036e9be2c0a0421af689bca49bd462e249032"},"cell_type":"markdown","source":"We'll use the same subset size as before: **20,000** random samples, but this time will be a different subset for each tree."},{"metadata":{"id":"HiO9NQDlL-M3","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"260e1308c0727ee98eccdc23ae7ab7dbe4c3c23f"},"cell_type":"code","source":"# import set_rf_samples\nfrom fastai.structured import set_rf_samples\n\nset_rf_samples(20000)","execution_count":null,"outputs":[]},{"metadata":{"id":"10N2mWqsL-M8","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"ec9de5c28a26998d5b13169d19a2793d99e02086","collapsed":true},"cell_type":"code","source":"# 10 trees\nm = RandomForestRegressor(n_jobs=-1, oob_score=True, random_state=17190)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"CigYuwBFL-NB","colab_type":"text","_uuid":"3a6e7edfbefe9548cf18d7b277650033167f6ace"},"cell_type":"markdown","source":"Since each additional tree allows the model to see more data, this approach can make **additional trees more useful without needing additional fine-tuning**."},{"metadata":{"id":"CEmUupvaL-NC","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"b1426f4110f8097369a25cd9e248567bc766b113","collapsed":true},"cell_type":"code","source":"# 40 trees\nm = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True, random_state=17190)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"qqbRcPkjW9sw","colab_type":"text","_uuid":"a48657981c1f66eef918a4e52f221d99ae916aa6"},"cell_type":"markdown","source":"### Number of trees\nWe'll create a graphic to see what happens to the **R Squared** when using **1 tree up to 100 trees**.\n- We'll **store** the **predictions** of each tree in a **numpy array** like we did before for our base model.\n- We'll **loop** through the **predictions** of each tree to **plot the change** in **R Squared** as we add more trees."},{"metadata":{"id":"2pOPwCqUU56n","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"81ed891aca56d6840524a44e3cecaff102a88a48","collapsed":true},"cell_type":"code","source":"# 100 trees\nm = RandomForestRegressor(n_estimators=100, n_jobs=-1)\n%time m.fit(X_train, y_train)\nprint_scores(m)\n\n# use a list comprehension to loop through the random forest and concatenates the predictions of each individual tree on a new axis\npreds = np.stack([t.predict(X_valid) for t in m.estimators_])\n\n# dimensions of the predictions (rows, columns)\npreds.shape","execution_count":null,"outputs":[]},{"metadata":{"id":"t9hAA3WoL-LB","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"9531f7a4d32106d3f7427c3dc04e4b4638ccca48","collapsed":true},"cell_type":"code","source":"# import matplotlib\nimport matplotlib.pyplot as plt\n\n# import metrics module\nfrom sklearn import metrics\n\n# plot the calculated r^2 of the true values and the predicted values up to [:i] trees (looping through a range from 1 to 100). \nplt.plot([metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(100)]);","execution_count":null,"outputs":[]},{"metadata":{"id":"BD0bNVTLL-LO","colab_type":"text","_uuid":"f8a4fa459a448f793c6662b0bb5faf381765efd0"},"cell_type":"markdown","source":"The shape of this curve clearly shows that the **model improves** the **more trees** we use, but it also suggests that adding more trees is beneficial **up to a point**, because it **flattens out** the closer it gets to 100 trees.\n- There is **no lineal relationship** between number of trees and our evaluation metric score.\n- The **number of trees** we use can **vary wether** we are:\n - **developing the model**, we'll propably use less trees but do lots of iterations, or\n - **finishing the model**, in that case, we'll probably use as much trees as our processor can handle to fit the model with all the available data and using the best found parameters.\n- **Without** our **subsampling** approach, building **lots of trees** on **lots of data** can be computationally expensive and very **time comsuming**.\n\n### Depth of the trees\nAnother way to reduce over-fitting is to grow our trees **less deeply**.\n- We can do this with **`min_samples_leaf`** parameter, which requires some minimum number of rows in every leaf node (by default 1).\n- For each tree there will be less levels and **less decisions** being made so it will result in simpler models.\n- The predictions are made by **averaging more samples** in the leaf node.\n- This can make the random forest to **generalize better**.\n\nPossible good values to try:\n- 1, 3, 5, 10, 25, 100"},{"metadata":{"id":"kgSX7D3zL-NP","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"1293fce2312a6473794539465a527dce0fd17284","collapsed":true},"cell_type":"code","source":"# baseline for comparison\n\n# 40 trees: 1 minimum samples per leaf (default)\nm = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True, random_state=17190)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"heAnqe4lL-Ng","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"f99ec277afcf969105ed91abdc91ca0b5cf5afdb","collapsed":true},"cell_type":"code","source":"# 40 trees: 2 minimum samples per leaf\nm = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, n_jobs=-1, oob_score=True, random_state=17190)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"4jgkqBBhL-Nl","colab_type":"text","_uuid":"01317bc5c5a741fa244efba912615dc1f6110ec0"},"cell_type":"markdown","source":"### Variance\nAnother way to reduce over-fitting is to increase the amount of **variation amongst the trees**.\n- We can do this by specifying **`max_features`**, which is the **number of features** to consider at **each split**.\n- This way not only randomly selects a **sample of rows** for each tree, but **also** randomly selects a **sample of columns** for each split.\n- This can be critical for creating variance when there are features that are so much predictive than others and all the trees are very similar to each other because they are doing similar splits.\n\nPossible good values to try:\n- None: use all columns.\n- 0.5: use half of the columns\n- 'sqrt' (default): use the square root of total number of columns."},{"metadata":{"id":"YTBazX2SL-Nq","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"6a8f14033efe93a74f5da84a6d9d63dacdcc418f","collapsed":true},"cell_type":"code","source":"# 40 trees: consider all columns per split\nm = RandomForestRegressor(n_estimators=40, max_features=None, n_jobs=-1, oob_score=True, random_state=17190)\nm.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"Q4Plp4DRTDJh","colab_type":"text","_uuid":"5368b28bd4aa44ff1f600648ef07b8e13583b6ac"},"cell_type":"markdown","source":"### Experimenting\nWe'll revert to using a full bootstrap sample in order to experiment with these hyperparameters further and to see their full impact.\n- The models will **train much slower**.\n- For more information, you can check this post on stackoverflow: [Practical questions on tuning Random Forests](https://stats.stackexchange.com/questions/53240/practical-questions-on-tuning-random-forests)."},{"metadata":{"id":"71FFzJh6T32A","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"collapsed":true,"_uuid":"8781b17fb7b47efe4f6da63f8e79373dde2e6781"},"cell_type":"code","source":"# import reset_rf_samples\nfrom fastai.structured import reset_rf_samples\n\n# use full bootstrap sample\nreset_rf_samples()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3bf6a9e03415ce5d75425b6fe7ab54951afee350","collapsed":true},"cell_type":"code","source":"# baseline for comparison\n\nm = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True, random_state=17190)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"lLWMpvleTpo8","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"9151f96d6a9e3a9b6b618defc5d9290f7d2c614e","collapsed":true},"cell_type":"code","source":"# 40 trees: 3 minimum samples per leaf\nm = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, n_jobs=-1, oob_score=True, random_state=17190)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"id":"SQMP10PqTuIh","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"trusted":true,"_uuid":"2bdad918dcf28ae5e187b20525372fb0aef160b2","collapsed":true},"cell_type":"code","source":"# 40 trees: half columns per split\nm = RandomForestRegressor(n_estimators=40, max_features=0.5, n_jobs=-1, oob_score=True, random_state=17190)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c79042fe73ff5e538e24e86a59245fbcb60f6726","collapsed":true},"cell_type":"code","source":"# 40 trees: 3 minimum samples per leaf and half columns per split\nm = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True, random_state=1719)\n%time m.fit(X_train, y_train)\nprint_scores(m)","execution_count":null,"outputs":[]}],"metadata":{"colab":{"name":"lesson1-rf.ipynb","version":"0.3.2","views":{},"default_view":{},"provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"accelerator":"GPU","language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}