{"cells":[{"metadata":{"_uuid":"f0f383056618671a00c1e6369d784015d498584f"},"cell_type":"markdown","source":"**Kernel Approach**\n\nThis kernel is a quick ML solution built based on XGBoost, and a Feature Importance Analisys using [Permutation Importance](https://www.kaggle.com/dansbecker/permutation-importance?utm_medium=email&utm_source=mailchimp&utm_campaign=ml4insights). Further feature analisys were done in my previous kernel [GA Challenge - DAta Analisys](https://www.kaggle.com/wesleyjr01/google-analytics-challenge-data-analisys)."},{"metadata":{"_uuid":"a1697e03f622bebf38585893ec0558df1281a595"},"cell_type":"markdown","source":"**Reshaping the given dataset**\n\nAs mentioned in the challenge description, some fields of the datasets train.csv and test.csv are in json format, and for better data manipulation with DataFrames, we should make some type conversions first. For this task, we already have a [pretty nice kernel](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook) built by [Julián Peller](https://www.kaggle.com/julian3833)."},{"metadata":{"trusted":true,"_uuid":"238c7ef5b302fb5659a082ed8b56e270172f3aa9"},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\n\ndef load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\ndf_train = load_df()\ndf_test = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41b6cdc5f1e8b800b3c8f2c8a6fdef10073376c5"},"cell_type":"code","source":"#Lets have a look at the data\ndf_train.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d45dd7371c2f0c4b5a85ee343f4669f0f131b331"},"cell_type":"markdown","source":"**Datasets Analisys**"},{"metadata":{"trusted":true,"_uuid":"6d16f6770547d1ac108adbc5f2327e1db3c3a904"},"cell_type":"code","source":"df_train.columns\nprint('Is there more than one transaction by VisitorId in train dataset?',\n      len(df_train['fullVisitorId'])!=df_train['fullVisitorId'].nunique())\nprint('Is there more than one transaction by VisitorId in test dataset?',\n      len(df_test['fullVisitorId'])!=df_test['fullVisitorId'].nunique())\n#Confirming that we have more rows than unique Visitors Id's in both train and test datasets.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd24b5f53948a8a97b6fb2d3db53e4b2f9ca6f1c"},"cell_type":"markdown","source":"**Dropping columns with constant values**\n\nSome of the features imported in the json-csv process have only one unique value, which doesn't give us any information, and it would be a problem for any ML model. So we should just drop them."},{"metadata":{"trusted":true,"_uuid":"21882880e9ca278ec825a8b662b14ec08fa7e95c"},"cell_type":"code","source":"dropcols = [c for c in df_train.columns if df_train[c].nunique()==1]\ndropcols_test = [c for c in df_test.columns if df_test[c].nunique()==1]\ndf_train = df_train.drop(dropcols,axis=1)\ndf_test = df_test.drop(dropcols_test,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2326390868c012ffd7e52bde22626b32bc83329a"},"cell_type":"markdown","source":"## Building Date Features"},{"metadata":{"trusted":true,"_uuid":"8fd4ed938dd94de5953830a4f6086be8392f156b"},"cell_type":"code","source":"#Date Information: Lets split the information in Years/Months/Days and split the analisys.\ndf_train['year']= df_train['date'].astype(str).str[:4]\ndf_test['year']= df_test['date'].astype(str).str[:4]\ndf_train['month']= df_train['date'].astype(str).str[4:6]\ndf_test['month']= df_test['date'].astype(str).str[4:6]\ndf_train['day']= df_train['date'].astype(str).str[6:8]\ndf_test['day']= df_test['date'].astype(str).str[6:8]\ndf_train.drop('date',axis=1,inplace=True)\ndf_test.drop('date',axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e3723c7fc5bd044587c54166e59e02cdd329aebc"},"cell_type":"code","source":"# As stated in previous kernel analisys, this information does very little for us, lets drop it.\ndf_train.drop('year',axis=1,inplace=True)\ndf_test.drop('year',axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ad8634226b0ea53c00819fdb297f1170d3378de"},"cell_type":"markdown","source":"## Verify if features in both train and test datasets are equal"},{"metadata":{"trusted":true,"_uuid":"69672217f105a414c0d58f246e54392c826d8bef"},"cell_type":"code","source":"#Missing Data on Train Dataset\nprint('Any different features between Train and Test datasets?',\n      False in df_train.drop('totals.transactionRevenue',axis=1).columns == df_test.columns)\n#So, apart from the target values, there are no features difference between these datasets.\n#The 'sessionId' and 'visitId' features are not going to be usefull for us, so we can drop it.\ndf_train.drop(['sessionId','visitId'],axis=1,inplace=True)\ndf_test.drop(['sessionId','visitId'],axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9c920d82edfb458f1ad93fcbb2d4e40afc6e3f7"},"cell_type":"markdown","source":"## Store and Drops Id's from datasets, and Target from Train Dataset"},{"metadata":{"trusted":true,"_uuid":"a6f12f2135188fcf16b24915b8b1a952d3a7b10d"},"cell_type":"code","source":"df_train[\"totals.transactionRevenue\"].fillna(0, inplace=True)# Impute 0 for missing target values\ny_train = df_train[\"totals.transactionRevenue\"]\n#df_train.drop(\"totals.transactionRevenue\",axis=1,inplace=True)\ntrain_id = df_train[\"fullVisitorId\"]\npred_id = df_test[\"fullVisitorId\"]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2d5e1e5d83acbf1b5af3c30e036e08103906299"},"cell_type":"markdown","source":"## Missing Data per features type"},{"metadata":{"trusted":true,"_uuid":"f40dc45fad7257061ad046e321604e5831fec552"},"cell_type":"code","source":"df_merge = pd.concat([df_train.drop(\"totals.transactionRevenue\",axis=1),df_test],axis=0)\n\ndf_merge['day'] = df_merge['day'].astype(float)\ndf_merge['month'] = df_merge['month'].astype(float)\n\n\ndf_merge['trafficSource.adwordsClickInfo.page'] = df_merge['trafficSource.adwordsClickInfo.page'].astype(str)\ndf_merge['trafficSource.adwordsClickInfo.page'].fillna('None',inplace=True)\n\ndf_merge['totals.pageviews'] = df_merge['totals.pageviews'].astype(str)\ndf_merge['totals.pageviews'].fillna('None',inplace=True)\n\nmergeId = df_merge['fullVisitorId']\ndf_merge.drop('fullVisitorId',axis=1,inplace=True)\n\nqualitative_features = [f for f in df_merge.dropna().columns \n                        if df_merge.dropna().dtypes[f] == 'object' or 'bool'] #Lista de Features Qualitativas.\nquantitative_features = [f for f in df_merge.dropna().columns \n                         if df_merge.dropna().dtypes[f] != 'object' or 'bool'] #Lista de Features Qualitativas.\n\n\ndef missingData(df,features):\n    total = df[features].isnull().sum().sort_values(ascending=False)\n    percent = (df[features].isnull().sum()/df[features].isnull().count()).sort_values(ascending=False)\n    missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\n    return missing_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d71053838b5623b102d88bb3cd84a5d8ecf826eb"},"cell_type":"code","source":"#Missing data qualitative feature\nmissing_data_quali = missingData(df_merge,qualitative_features)\nmissing_data_quali.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6466605ca33cd8f46b3a93ffdc44729fd46bb9a9"},"cell_type":"code","source":"#Missing data quantitative feature\nmissing_data_quanti = missingData(df_merge,quantitative_features)\nmissing_data_quanti.head(20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1772330db1093e1335a43fbbd2ff96493c3dc1bd"},"cell_type":"markdown","source":"## Imputation with 'None' and LabelEncoder"},{"metadata":{"trusted":true,"_uuid":"e937f81822c8947db0ed6aef37750aa614ae5696"},"cell_type":"code","source":"for i in qualitative_features:\n    if df_merge[i].isnull().any():\n        df_merge[i].fillna('None',inplace=True)\nprint('\\nIs there any NaN value  in the dataset after Imputing?:',df_merge.isnull().sum().any())\n\n# Encoding the variable\nfrom collections import defaultdict\nfrom sklearn.preprocessing import LabelEncoder\nd = defaultdict(LabelEncoder)\n# Encoding the variable\nfit = df_merge[qualitative_features].apply(lambda x: d[x.name].fit_transform(x))\ndf_merge[qualitative_features] = fit\n\n#Restore datraframes df_train and df_test\ndf_train = df_merge[:len(df_train)]\ndf_train[\"totals.transactionRevenue\"] = y_train.tolist()\ndf_test = df_merge[len(df_train):]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5365edae21bed50a55abf6cb44ad28ee4701c2a7"},"cell_type":"markdown","source":"\n## Division between X_train,X_val,y_train,y_val "},{"metadata":{"trusted":true,"_uuid":"d30f4b55c8d2a9ff3c8e4a1fc1205742a944ee81"},"cell_type":"code","source":"from sklearn.utils import shuffle\nfrom sklearn.model_selection import train_test_split\nsize_test = 0.3\ndf_train = shuffle(df_train) #shuffle data before division\ntrain_target = np.log1p(df_train[\"totals.transactionRevenue\"].astype(float).tolist()) # Just for code readibility\npredictors = df_train.drop(\"totals.transactionRevenue\", axis=1)\nX_train, X_val, y_train, y_val = train_test_split(predictors, \n                                                    train_target,\n                                                    train_size=1-size_test, \n                                                    test_size=size_test, \n                                                    random_state=0)\nX_pred = df_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3ae8b1e9c21fe1b0ea9afa5e5c9a072dd51bd24f"},"cell_type":"code","source":"train_target","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dbf26e546dcf86124782fbbbfdaa027032d27566"},"cell_type":"markdown","source":"## XGBoost Model\n\nIf you want to start with XGBoost, [this kernel](https://www.kaggle.com/dansbecker/xgboost) written by [DanB](https://www.kaggle.com/dansbecker) might help you."},{"metadata":{"trusted":true,"_uuid":"01b70f90e1aa7ed84c154d28430ba0af93ca997b"},"cell_type":"code","source":"from xgboost import XGBRegressor\nxgb_model = XGBRegressor(n_estimators=500, learning_rate=0.01)\n\nxgb_model.fit(X_train.values, y_train, early_stopping_rounds=5, \n             eval_set=[(X_val.values, y_val)], verbose=False)\n\n\n# Training the model #\npred_test = xgb_model.predict(X_pred.values)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b63427b746dd72687fd1f8ebb2a988a2501eef12"},"cell_type":"markdown","source":"## Permutation Importante\n\nThere is a[ great kernel](https://www.kaggle.com/dansbecker/permutation-importance?utm_medium=email&utm_source=mailchimp&utm_campaign=ml4insights) written by[ DanB](https://www.kaggle.com/dansbecker) to helps us get insighsts about the feature importance in our model, and we will use it here."},{"metadata":{"trusted":true,"_uuid":"9affca21c455909ecc66061e7088dee5cb93c1dc"},"cell_type":"code","source":"import eli5\nfrom eli5.sklearn import PermutationImportance\n\nperm = PermutationImportance(xgb_model, random_state=1).fit(X_val, y_val)\neli5.show_weights(perm, feature_names = X_val.columns.tolist())\n\n#Looks like totals.pageviews is the most important feature, followed by totals.hits","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01a9f995d9d1d3d280ccb5f570decef71539e42c"},"cell_type":"markdown","source":"## Build the solution DataFrame"},{"metadata":{"trusted":true,"_uuid":"4e33fc4a1489594515b323ccc935eacb6442eab0"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":pred_id})\n#Round to Zero the negative predictions\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\n#sub_df[\"PredictedLogRevenue\"] = pred_test\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"xgb_predictions.csv\", index=False)\nsub_df.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}