{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"In this notebook, I will explore the GA Customer Revenue dataset, do some quick data preprocessing, \nand train a simple model that will serve as a benchmark.\n\nLet's get started!"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport missingno as msno","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"488216f0a3fb5589988eb50b05f1c1f827c8081a"},"cell_type":"code","source":"# Some constants\n# Otherwise, pandas will try to interpret this column as an integer \n# (which is wrong according to the competition's guidelines).\nVISITOR_ID_COL = \"fullVisitorId\"\nDTYPES = {VISITOR_ID_COL: 'str'}\nTARGET_COL = \"transactionRevenue\"\nTRAIN_DATA_PATH = \"../input/train.csv\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"9970701f91544d4931aea8037cd10be0f5f28b0f"},"cell_type":"code","source":"train_df = pd.read_csv(TRAIN_DATA_PATH, dtype=DTYPES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"455ffc3db2c069e3599fc607f729cb3bfc8d94cd"},"cell_type":"code","source":"train_df.sample(2).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"80810256c1e227ea608433bca142bfa0f4cef7bb","scrolled":true},"cell_type":"code","source":"msno.matrix(train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"440aad2fef0380907a5e96b02e4c7976de40e027"},"cell_type":"markdown","source":"No missing data, awesome! Or maybe one shouldn't be that enthusisat since there are a lot of \nnested columns (more about this later) ;)"},{"metadata":{"_uuid":"614fe4292007f6e4aaeeaaf32c90ea28c1a30044"},"cell_type":"markdown","source":"Alright, after loading the data and having a look at some samples, the first\nthing one needs to do is extract the target for this problem and (basic for now) \nfeatures. Let's do that!"},{"metadata":{"_uuid":"1c3ebdb439f8c96edadbf7a8aa0b3a2576d4ae52"},"cell_type":"markdown","source":"# Unnesting the target"},{"metadata":{"_uuid":"3f9838c82b7525b52a4d71c183efa869b8450d5f"},"cell_type":"markdown","source":"So, where is the target? As mentioned in the competition's directions, it is inside\nthe `totals` column. Let's have a look, shall we?"},{"metadata":{"trusted":true,"_uuid":"eb54eb6c1c536ba308cc3b80d761822da0227f51"},"cell_type":"code","source":"RAW_TARGET_COL = \"totals\"\nraw_target_s = train_df[RAW_TARGET_COL]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"45c741524a0c7dec9321cf6de661141aea27abcf"},"cell_type":"code","source":"for index, raw_target_row in raw_target_s.sample(30).iteritems():\n    print(eval(raw_target_row))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5490934a61b36e662ab75986d6ba82aa5c637754"},"cell_type":"markdown","source":"As you can see, this is a nested column (it is a dict). Moreover, the \ntarget of interest `transactionRevenue` isn't always available. \nLet's unnest this column and explore the missing values."},{"metadata":{"trusted":true,"_uuid":"ad336833460f7193755e948238491e60eade0476"},"cell_type":"code","source":"records = []\nfor index, raw_target_row in raw_target_s.iteritems():\n    parsed_target_row = eval(raw_target_row)\n    records.append(parsed_target_row)\nparsed_target_df = pd.DataFrame(records)\n# Don't forget the visitor id!\nparsed_target_df[VISITOR_ID_COL] = train_df[VISITOR_ID_COL]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"735ade5161cca1029c22ffb0c6715ee79d8f32dc"},"cell_type":"code","source":"parsed_target_df.sample(3).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac281a44ce69eb1fb8af86908376645481d105c7"},"cell_type":"code","source":"msno.matrix(parsed_target_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3fd4d72bb57aeca94010d7e57022e9bd7e13caf0"},"cell_type":"markdown","source":"Waw, it seems that the target to predict is missing a lot of times. How many times?"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"4762d97571932d0201650ea5f9777266abf68191"},"cell_type":"code","source":"def percentage_of_missing(df, col):\n    return 100 * df[col].isnull().sum() / df.shape[0]\n\nmissing_target_percent = percentage_of_missing(parsed_target_df, \n                                              TARGET_COL)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"60b5d2702ec46e453de67d79607266b9e8239b7a"},"cell_type":"code","source":"\"The target column contains {}% missing data!\".format(missing_target_percent.round(2))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"485e88cbcbd15a682e7c8db8ec237f56d99a9b41"},"cell_type":"markdown","source":"In what follows, I will assume that a missing value for `transactionRevenue`\nmeans that the transaction value is 0 (even though it could be a \"real\" missing \nvalue). Let's fill the missing values with this information."},{"metadata":{"_uuid":"33145b808aefbcc879936b32b7b4c68d857694d5"},"cell_type":"markdown","source":"Let's check the distribution of transactionRevenue."},{"metadata":{"trusted":true,"_uuid":"54724280815afb16fd37da2390622a2d68e2c873"},"cell_type":"code","source":"target_df = (parsed_target_df.loc[:, [TARGET_COL, VISITOR_ID_COL]]\n                            .assign(**{TARGET_COL: lambda df: df[TARGET_COL].fillna(0.0)\n                                                                            .astype(int)}))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"22c6c7eddffdb189b503c3c4d486cb29c4b1d5bf"},"cell_type":"code","source":"target_df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"44290d3b5a3fd0b0c8f442acde8cfb5ecab1773d"},"cell_type":"code","source":"import seaborn as sns\nimport numpy as np\nimport matplotlib.pylab as plt\n\n\nfig, ax = plt.subplots(1, 1, figsize=(12, 8))\n# Since most of the transactions are 0$, I will remove these when plotting\n# the distribution.\nsns.distplot(np.log(target_df.loc[lambda df: df[TARGET_COL] >0, \n                                  TARGET_COL]), ax=ax)\nax.set_xlabel(\"Log of transaction revenue ($)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"88d5e8bc26561e209f18defa606aa52f547471c0"},"cell_type":"code","source":"# The same thing as above but this time aggregated using the \n# visitor unique id.\nfig, ax = plt.subplots(1, 1, figsize=(12, 8))\n# Since most of the transactions are 0$, I will remove these when plotting\n# the distribution.\n\ndef _log_sum_agg(g):\n    \"\"\" Take the natural logarithm of the aggregated sum\n    (+1 to avoid -inf for a 0 sum).\n    \"\"\"\n    return np.log(g.sum() + 1)\n\ngrouped_target_a = (target_df.groupby(VISITOR_ID_COL)\n                             .agg({TARGET_COL: _log_sum_agg})\n                             .values)\nsns.distplot(grouped_target_a[grouped_target_a > np.log(1)], ax=ax)\nax.set_xlabel(\"Log of sum of transaction revenue ($)\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c342871f239235d21d7007660c19725c66fadc0b"},"cell_type":"markdown","source":"# Basic features extraction"},{"metadata":{"_uuid":"e56cde149dfbe41d97d20b41420316309d3cac0f"},"cell_type":"markdown","source":"In order to build the benchmark model, one needs some features. Let's use the following ones: \n\n*  `date`: this is the date of the transaction. I assume that it is in UTC.\n* `geoNetwork`: this is a nested column that contains information about location of the transaction. \n\nIn what follows, I will extract these features and engineer some basic ones (day of week, month, year, and so on...)"},{"metadata":{"trusted":true,"_uuid":"88ba3ffee4825500b290ff2a131b17736a31aab0"},"cell_type":"code","source":"DATE_COL = \"date\"\nTMS_GMT_COL = \"tms_gmt\"\n# Here, I parse the DATE_COL to extract year, month, and day information \n# (using there positions). Then, I build the TMS_GMT column (using pandas' \n# to_datetime function) and extract additional calendar features: \n# day of week, week of year, and day of year. \n# Notice that I drop DATE_COL and TMS_GMT columns since these\n# aren't numerical columns.\ndate_df = (train_df[[DATE_COL]].assign(year=lambda df: df[DATE_COL].astype(str)\n                                                                   .str[0:4]\n                                                                   .astype(int),\n                                       month=lambda df: df[DATE_COL].astype(str)\n                                                                    .str[4:6]\n                                                                    .astype(int),\n                                       day=lambda df: df[DATE_COL].astype(str)\n                                                                  .str[6:8]\n                                                                  .astype(int))\n                               .drop(DATE_COL, axis=1)\n                               .assign(tms_gmt=lambda df: pd.to_datetime(df))\n                               .assign(dow=lambda df: df[TMS_GMT_COL].dt.dayofweek,\n                                       woy=lambda df: df[TMS_GMT_COL].dt.week,\n                                       doy=lambda df: df[TMS_GMT_COL].dt.day)\n                               .drop(TMS_GMT_COL, axis=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c206bd96b92e9337c38d081cb4a6ea7f65bfb497"},"cell_type":"code","source":"date_df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"07311a47585035054102c032f65d50e5fc8ca70b"},"cell_type":"code","source":"records = []\nGEO_COL = \"geoNetwork\"\nfor index, row in train_df[GEO_COL].iteritems():\n    parsed_row = eval(row)\n    records.append(parsed_row)\n\ngeo_df = pd.DataFrame(records)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"640b295cd3f8574afb91194241d4e12e9bf9b75b"},"cell_type":"code","source":"geo_df.sample(2).T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0032981fbcd3c2423429f0e77bc126b57da30015"},"cell_type":"markdown","source":"To make things simpler, I will only keep the `country` and `continent` features \nfrom the `geoNetwork` parsed column. I will also dummify these features. Finally, I will combine\nthe various engineered features. Let's do that!"},{"metadata":{"_uuid":"1905b6a4a4c4044e7d957d802300ed8f00a033ca"},"cell_type":"markdown","source":" "},{"metadata":{"trusted":true,"_uuid":"79e65f33f974954676283c755e4a64b527c7eb82"},"cell_type":"code","source":"GEO_COLS_TO_KEEP = [\"country\", \"continent\"]\nengineered_train_df = (geo_df.loc[:, GEO_COLS_TO_KEEP]\n                             .pipe(pd.get_dummies)\n                             .pipe(pd.merge, date_df, \n                                   left_index=True,\n                                   right_index=True)\n                             .pipe(pd.merge, \n                                   train_df[[VISITOR_ID_COL]],\n                                   left_index=True,\n                                   right_index=True))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"bb642ac00e44d77ce94ac3ac9feb05786712c399"},"cell_type":"code","source":"engineered_train_df.sample(2).T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d7b850a52144a8aa20b74e52c0b1c867aa7a6257"},"cell_type":"markdown","source":"Awesome! Time to do some (basic) modeling."},{"metadata":{"_uuid":"04ba67a0c8edc054d338ae7db82a123a2c0da76d"},"cell_type":"markdown","source":"# LASSO as a benchmark"},{"metadata":{"_uuid":"ec98424a2ed4636c2e3992afa455f920254fd3bf"},"cell_type":"markdown","source":"Now that I have prepared some features, I will train a LASSO model (i.e. a linear regression model that\ndoes features selection automatically) and compute its CV score. \nNotice that I can't use the cross_val_score from sklearn since I need to aggregate the out-of-fold \npredictions before computing the score. "},{"metadata":{"trusted":true,"_uuid":"883f48df1ccd58111b1c8dc0a42ef178fd29f772"},"cell_type":"code","source":"from sklearn.linear_model import Lasso\nfrom sklearn.model_selection import KFold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"824c90f075f9d3b362729cbb81b5dc887d45536b","scrolled":true},"cell_type":"code","source":"# For reproducibility\nSEED = 314\nCV = 5\n# Resources are limited! \nN_SAMPLES = 10000\nkf = KFold(CV, random_state=SEED)\n\n\nbenchmark = Lasso(random_state=SEED)\n\ndf = engineered_train_df.sample(N_SAMPLES).drop(VISITOR_ID_COL, axis=1)\n\n# TODO: Do some cleaning and refactoring of the CV computation. \n# Also check the grouping step...\n# LASSO warnings are annoying. :)\nimport warnings\nwarnings.simplefilter(\"ignore\")\n\n\ncv_rmse = []\nfor train_index, test_index in kf.split(df):\n    train_features_df = df.iloc[train_index, :]\n    test_features_df = df.iloc[test_index, :]\n    train_target_s = target_df.loc[train_index, TARGET_COL]\n    test_target_df = target_df.iloc[test_index, :].reset_index(drop=True)\n    benchmark.fit(train_features_df, train_target_s)\n    test_target_df.loc[:, \"predictions\"] = benchmark.predict(test_features_df)\n    grouped_df  = (test_target_df.groupby(VISITOR_ID_COL)\n                                 .agg({\"predictions\": _log_sum_agg, \n                                       TARGET_COL: _log_sum_agg})\n                                 .reset_index())\n    rmse = ((grouped_df[\"predictions\"] - grouped_df[TARGET_COL]) ** 2).mean() ** 0.5\n    cv_rmse.append(rmse)\n\ncv_rmse = np.array(cv_rmse)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7b36c4d8a80116ee4e748d39834c6b79085b5221"},"cell_type":"code","source":"cv_rmse","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b1a2fa8a62d18afb7ebfa4e87cfa0e063ec4c550","scrolled":true},"cell_type":"code","source":"\"The mean CV RMSE for the benchmark is: {}\".format(cv_rmse.mean())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"840780d45eb6082982147498cfc19cd2422c0cce"},"cell_type":"markdown","source":"That's it for now, I hope you have enjoyed this introductory notebook. \nStay tuned for more!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}