{"cells":[{"metadata":{"_uuid":"83c373b34ff4ff2acfefb808ede5a5e2545518cd"},"cell_type":"markdown","source":"## Chapter 5. Categorical Variables: Counting Eggs in the Age of Robotic Chickens\n\nThis chapter is one of the chapters of the book, Feature Engineering for Machine Learning. Since I do not have enough resource on my local machine, I have ended up creating this kernel to practice the source code of the chapter 5 while reading the book. You can also examine all the main chapters' code over the original GitHub repository of the book: https://github.com/alicezheng/feature-engineering-book\n"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"8d3ace32d2351db730f1e9504df82f4cba3f4b01"},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import sklearn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31a4cabc8c8e73912ea1255d208bc10934163711"},"cell_type":"code","source":"sklearn.__version__","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b617a10e7dff95280659547d28c649125b55d82"},"cell_type":"markdown","source":"## Encoding Categorical Variables"},{"metadata":{"trusted":true,"_uuid":"092e7836349d8bafba9a467c2d8338e0d64f0900"},"cell_type":"code","source":"# Linear regression on a categorical variable using one-hot and dummy codes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"744026b1ad05fcc57511cf7876f4dce2cdede1af"},"cell_type":"code","source":"from sklearn import linear_model\n\n# Define a toy dataset of apartment rental prices in \n# New York, San Francisco, and Seattle\ndf = pd.DataFrame({\n     'City': ['SF', 'SF', 'SF', 'NYC', 'NYC', 'NYC', \n              'Seattle', 'Seattle', 'Seattle'],\n     'Rent': [3999, 4000, 4001, 3499, 3500, 3501, 2499, 2500, 2501]\n})\ndf['Rent'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2a0e31d52beceecd0f9e5288658b5ad5253e915"},"cell_type":"code","source":"# Convert the categorical variables in the DataFrame to one-hot encoding\n# and fit a linear regression model\none_hot_df = pd.get_dummies(df, prefix=['city'])\none_hot_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0b7cd5b04a6e2645547fe6934a924deac8ca667"},"cell_type":"code","source":"model = linear_model.LinearRegression()\nmodel.fit(one_hot_df[['city_NYC', 'city_SF', 'city_Seattle']],\n           one_hot_df['Rent'])\nmodel.coef_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"971d632d1935c60f793e270275a1702a7a70cd22"},"cell_type":"code","source":"model.intercept_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"81d5dd7ef8b23206986ab3117d72ab9c0aeb6b7d"},"cell_type":"code","source":"dummy_df = pd.get_dummies(df, prefix=['city'], drop_first=True)\ndummy_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"244f9bf645b6cf72128eb3f1f1dfe7a8f343f8ad"},"cell_type":"code","source":"model.fit(dummy_df[['city_SF', 'city_Seattle']], dummy_df['Rent'])\nmodel.coef_\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db3027965ea62dadbf8d31b3f717dc34c80a3381"},"cell_type":"code","source":"model.intercept_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d70bb3b5ac7ce593e4cb8aa7f8ef0949306237d"},"cell_type":"markdown","source":"Linear regression learned coefficients\n\n|                  \t| x1     \t|   x2   \t|      x3 \t| b       \t|\n|------------------\t|--------\t|:------:\t|--------:\t|---------\t|\n| One-hot encoding \t| 166.67 \t| 666.67 \t| –833.33 \t| 3333.33 \t|\n| Dummy coding     \t| 0      \t|   500  \t|   -1000 \t| 3500    \t|"},{"metadata":{"trusted":true,"_uuid":"321e0661499d52fa8f56d04ae94c78480645fb5a"},"cell_type":"code","source":"# bias coefficient\ndf[df['City'] == 'NYC'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8baab04b9ec54dc758355d4bdb5249615c7f7219"},"cell_type":"code","source":"# The coefficient for the x1 feature\ndf[df['City'] == 'NYC'].mean() - 3500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"cfeb17946c0be6965fc7c11e4d1bee5f6287ff91"},"cell_type":"code","source":"# The coefficient for the x2 feature\ndf[df['City'] == 'SF'].mean() - 3500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"a9b0f93f3bdf4415ab0e48c8912b059c4e67a02b"},"cell_type":"code","source":"# The coefficient for the x3 feature\ndf[df['City'] == 'Seattle'].mean() - 3500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b02f235d203c1333b1661e13893191cfc9f38055"},"cell_type":"markdown","source":"## Effect Coding"},{"metadata":{"trusted":true,"_uuid":"6615b9adb35b95349794f3af4c9803e962769291"},"cell_type":"code","source":"effect_df = dummy_df.copy()\neffect_df.loc[3:5, ['city_SF', 'city_Seattle']] = -1.0\neffect_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8b7f8925f1ce56c34f64aee7b31195c7e8b07ff4"},"cell_type":"code","source":"model.fit(effect_df[['city_SF', 'city_Seattle']], effect_df['Rent'])\nmodel.coef_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7d9e82d6684f7c91d8019860a4ea564d547e9cfc"},"cell_type":"code","source":"model.intercept_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f61aa8d94cc7519d3cbc39d825d4a704487c4c3e"},"cell_type":"markdown","source":"## Feature Hashing"},{"metadata":{"trusted":true,"_uuid":"0b34cf10d8458781cb9f7442c1e05bdb24df414a"},"cell_type":"code","source":"import json","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a9d6e04cf03e99b2e10f03e7508f8d2fdc74fcf7"},"cell_type":"code","source":"# Load Yelp reviews data\nwith open('../input/yelp-dataset/yelp_academic_dataset_review.json') as review_file:\n    review_df = pd.DataFrame([json.loads(next(review_file)) for x in range(10000)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea420d35859da50c57fb9dd84e4019265d20c13c"},"cell_type":"code","source":"# Define m as equal to the unique number of business_ids\nm = len(review_df.business_id.unique())\nm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d69a548361325ddd77fe79e919c0a2e3a33b47e6"},"cell_type":"code","source":"from sklearn.feature_extraction import FeatureHasher\n\nh = FeatureHasher(n_features=m, input_type='string')\nf = h.transform(review_df['business_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f7af9c7afcaebe17941bc9aefe0d1fe81bf2c6d"},"cell_type":"code","source":"# How does this affect feature interpretability?\nreview_df['business_id'].unique().tolist()[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c840d8bf8e2a5b03c161b8922ff4866c4d3a6fd"},"cell_type":"code","source":"f.toarray()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4d25c0c92ed170300712e80a773b955a0ab39b3"},"cell_type":"code","source":"# Not great. BUT, let's see the storage size of our features.\nfrom sys import getsizeof\nprint('Our pandas Series, in bytes: ', getsizeof(review_df['business_id']))\nprint('Our hashed numpy array, in bytes: ', getsizeof(f))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a3e4855506ddc75a463024e97e5fa6959f082436"},"cell_type":"markdown","source":"## Bin Counting"},{"metadata":{"trusted":true,"_uuid":"99c1ab4735b626f0a74746ac99d433eb2fa17cf0"},"cell_type":"code","source":"# train_subset data is first 10K rows of 6+GB set\ndf = pd.read_csv('../input/avazu-ctr-prediction/train/train.csv', nrows=10000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"adf2f483fa13141da1f6977b6229a08a50271336"},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e379eb7baea78d2c2be34b286774a80d6e492b55"},"cell_type":"code","source":"# How many unique features should we have after?\nlen(df['device_id'].unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"604543956757d287fb8a019ceeeec66fece59ab0"},"cell_type":"code","source":"def click_counting(x, bin_column):\n    clicks = pd.Series(x[x['click'] > 0][bin_column].value_counts(), name='clicks')\n    no_clicks = pd.Series(x[x['click'] < 1][bin_column].value_counts(), name='no_clicks')\n    \n    counts = pd.DataFrame([clicks,no_clicks]).T.fillna('0')\n    counts['total_clicks'] = counts['clicks'].astype('int64') + counts['no_clicks'].astype('int64')\n    return counts\n\ndef bin_counting(counts):\n    counts['N+'] = counts['clicks']\\\n                    .astype('int64')\\\n                    .divide(counts['total_clicks'].astype('int64'))\n    counts['N-'] = counts['no_clicks']\\\n                    .astype('int64')\\\n                    .divide(counts['total_clicks'].astype('int64'))\n    counts['log_N+'] = counts['N+'].divide(counts['N-'])\n    # If we wanted to only return bin-counting properties, \n    # we would filter here\n    bin_counts = counts.filter(items= ['N+', 'N-', 'log_N+'])\n    return counts, bin_counts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c17ca5edc7f35eaa0002d52ba4a5fa260d45f70"},"cell_type":"code","source":"bin_column = 'device_id'\ndevice_clicks = click_counting(df.filter(items=[bin_column, 'click']), bin_column)\ndevice_all, device_bin_counts = bin_counting(device_clicks.copy())\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"79927dc8f52f5f29f8f5ae752181ed762ddaab14"},"cell_type":"code","source":"device_clicks.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e2dc43314039de6b021ac7d3d120c06f99e2983"},"cell_type":"code","source":"device_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2fc648e30d17f0c34c823eeae6448b48a71f4d21"},"cell_type":"code","source":"device_bin_counts.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4920ee969e995568559ce01945aef3064a3ffe7b"},"cell_type":"code","source":"len(device_bin_counts)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e09b8c7217114760dd38cf82bd4c9043dffaf97b"},"cell_type":"code","source":"device_all.sort_values(by = 'total_clicks', ascending=False).head(4)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"References:\n1. https://scikit-learn.org/stable/modules/preprocessing.html#encoding-categorical-features"},{"metadata":{"trusted":true,"_uuid":"68e73407a1220cf0b09a766c7c2a2e360d5453f5"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}