{"cells":[{"metadata":{"_uuid":"503b27f4751cef3db7bed4a5cf78d660df1ae146"},"cell_type":"markdown","source":"## LIGHT-GBM starter\n*aknowledgment: a quick hello at [Olivier](https://www.kaggle.com/ogrellier) to whom I borrowed many lines of code*"},{"metadata":{"_uuid":"5943b283627d826313f860a3af3fabb37f7266de"},"cell_type":"markdown","source":"## Permutation Importance\n1. [What is Permutation Importance ? ](#1.What-is-Permuatation-Importance?)\n2. [How it's works.](#2.-How-it's-works.)\n3.  [Permutation Importance by Kaggle](https://www.kaggle.com/dansbecker/permutation-importance)"},{"metadata":{"_uuid":"dfd39af7f5aadc5a6c8cedd61b894094b7bda8df"},"cell_type":"markdown","source":"## 1.What is Permuatation Importance?\n#### Paper Reference : [Model Class Reliance: Variable Importance Measures for any Machine Learning Model Class, from the “Rashomon” Perspective](https://arxiv.org/pdf/1801.01489.pdf)\n\n* Everytime when Training Every model we check feature importance but what about it's bias?\n* Feature Importance calculating the increase model prediction error that permuting the feature.\n* feature is **`Importnant`**, If permuting its values increase the model error because the model relied on the feature for the prediction.\n* feature is **`un-Importnant`**, If permuting its values keeps the model error unchanged, because the model ignored the feature for the prediction.\n\n### Algorithm:\n![](https://www.kaggle.com/ashishpatel26/mlimage/downloads/algo.JPG)"},{"metadata":{"_uuid":"24a3df8dd169726de0b7de2f851fb0fecc70eeb0"},"cell_type":"markdown","source":"### Advantages\n* Nice interpretation: Feature importance is the increase of model error when the feature’s information is destroyed.\n* Feature importance provides a highly compressed, global insight into the model’s behavior.\n* A positive aspect of using the error ratio instead of the error difference is that the feature importance measurements are comparable across different problems.\n\n### Disadvantages\n* The feature importance measure is tied to the error of the model. This is not inherently bad, but in some cases not what you need. In some cases you would prefer to know how much the model’s output varies for one feature, ignoring what it means for the performance. For example: You want to find out how robust your model’s output is, given someone manipulates the features. In this case, you wouldn’t be interested in how much the model performance drops given the permutation of a feature, but rather how much of the model’s output variance is explained by each feature. Model variance (explained by the features) and feature importance correlate strongly when the model generalizes well (i.e. it doesn’t overfit).\n* You need access to the actual outcome target. If someone only gives you the model and unlabeled data - but not the actual target - you can’t compute the permutation feature importance.\n\n"},{"metadata":{"_uuid":"25cff80197cc69b3e0d26785db2a705843aa7612"},"cell_type":"markdown","source":"## 2. Examples\nReference : [Link](https://christophm.github.io/interpretable-ml-book/feature-importance.html)\n\n### Cervical cancer (Classification)\n\n> * *We fit a random forest model to predict cervical cancer. We measure the error increase by:  1 − AUC (one minus the area under the ROC curve). Features that are associated model error increase by a factor of 1 (= no change) were not important for predicting cervical cancer.*\n\n![](https://christophm.github.io/interpretable-ml-book/images/importance-cervical-1.png)\n\n> * **The importance for each of the features in predicting cervical cancer with a random forest. The importance is the factor by which the error is increased compared to the original model error.**\n\n> The feature with the highest importance was associated with an error increase of 7.8 after permutation.\n\nThis Example From above Reference and you can get another example from there."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dd1714f80d3aba317196e64400d62f6d85ff873d"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport time\nimport lightgbm as lgb\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import KFold,StratifiedKFold\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e250f1cde4b580075fd1e1152ff13e211f4635a"},"cell_type":"markdown","source":"First, we load the dataset and thanks to [Juliàn Peller](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields), this is made quite simple:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# https://www.kaggle.com/youhanlee/stratified-sampling-for-regression-lb-1-6595/notebook\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport time\nfrom datetime import datetime\nimport gc\nimport psutil\nfrom sklearn.preprocessing import LabelEncoder\n\nPATH=\"../input/\"\nNUM_ROUNDS = 20000\nVERBOSE_EVAL = 500\nSTOP_ROUNDS = 100\nN_SPLITS = 10\n\n #the columns that will be parsed to extract the fields from the jsons\ncols_to_parse = ['device', 'geoNetwork', 'totals', 'trafficSource']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae71114ba1bfc9a440c9113af30103bdead2341b"},"cell_type":"code","source":"def read_parse_dataframe(file_name):\n    #full path for the data file\n    path = PATH + file_name\n    #read the data file, convert the columns in the list of columns to parse using json loader,\n    #convert the `fullVisitorId` field as a string\n    data_df = pd.read_csv(path, \n        converters={column: json.loads for column in cols_to_parse}, \n        dtype={'fullVisitorId': 'str'})\n    #parse the json-type columns\n    for col in cols_to_parse:\n        #each column became a dataset, with the columns the fields of the Json type object\n        json_col_df = json_normalize(data_df[col])\n        json_col_df.columns = [f\"{col}_{sub_col}\" for sub_col in json_col_df.columns]\n        #we drop the object column processed and we add the columns created from the json fields\n        data_df = data_df.drop(col, axis=1).merge(json_col_df, right_index=True, left_index=True)\n    return data_df\n    \ndef process_date_time(data_df):\n    print(\"process date time ...\")\n    data_df['date'] = data_df['date'].astype(str)\n    data_df[\"date\"] = data_df[\"date\"].apply(lambda x : x[:4] + \"-\" + x[4:6] + \"-\" + x[6:])\n    data_df[\"date\"] = pd.to_datetime(data_df[\"date\"])   \n    data_df[\"year\"] = data_df['date'].dt.year\n    data_df[\"month\"] = data_df['date'].dt.month\n    data_df[\"day\"] = data_df['date'].dt.day\n    data_df[\"weekday\"] = data_df['date'].dt.weekday\n    data_df['weekofyear'] = data_df['date'].dt.weekofyear\n    data_df['month_unique_user_count'] = data_df.groupby('month')['fullVisitorId'].transform('nunique')\n    data_df['day_unique_user_count'] = data_df.groupby('day')['fullVisitorId'].transform('nunique')\n    data_df['weekday_unique_user_count'] = data_df.groupby('weekday')['fullVisitorId'].transform('nunique')\n\n    return data_df\n\ndef process_format(data_df):\n    print(\"process format ...\")\n    for col in ['visitNumber', 'totals_hits', 'totals_pageviews']:\n        data_df[col] = data_df[col].astype(float)\n    data_df['trafficSource_adwordsClickInfo.isVideoAd'].fillna(True, inplace=True)\n    data_df['trafficSource_isTrueDirect'].fillna(False, inplace=True)\n    return data_df\n    \ndef process_device(data_df):\n    print(\"process device ...\")\n    data_df['browser_category'] = data_df['device_browser'] + '_' + data_df['device_deviceCategory']\n    data_df['browser_os'] = data_df['device_browser'] + '_' + data_df['device_operatingSystem']\n    return data_df\n\ndef process_totals(data_df):\n    print(\"process totals ...\")\n    data_df['visitNumber'] = np.log1p(data_df['visitNumber'])\n    data_df['totals_hits'] = np.log1p(data_df['totals_hits'])\n    data_df['totals_pageviews'] = np.log1p(data_df['totals_pageviews'].fillna(0))\n    data_df['mean_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('mean')\n    data_df['sum_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('sum')\n    data_df['max_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('max')\n    data_df['min_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('min')\n    data_df['var_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('var')\n    data_df['mean_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('mean')\n    data_df['sum_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('sum')\n    data_df['max_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('max')\n    data_df['min_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('min')    \n    return data_df\n\ndef process_geo_network(data_df):\n    print(\"process geo network ...\")\n    data_df['sum_pageviews_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_pageviews'].transform('sum')\n    data_df['count_pageviews_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_pageviews'].transform('count')\n    data_df['mean_pageviews_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_pageviews'].transform('mean')\n    data_df['sum_hits_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_hits'].transform('sum')\n    data_df['count_hits_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_hits'].transform('count')\n    data_df['mean_hits_per_network_domain'] = data_df.groupby('geoNetwork_networkDomain')['totals_hits'].transform('mean')\n    return data_df\n\ndef process_traffic_source(data_df):\n    print(\"process traffic source ...\")\n    data_df['source_country'] = data_df['trafficSource_source'] + '_' + data_df['geoNetwork_country']\n    data_df['campaign_medium'] = data_df['trafficSource_campaign'] + '_' + data_df['trafficSource_medium']\n    data_df['medium_hits_mean'] = data_df.groupby(['trafficSource_medium'])['totals_hits'].transform('mean')\n    data_df['medium_hits_max'] = data_df.groupby(['trafficSource_medium'])['totals_hits'].transform('max')\n    data_df['medium_hits_min'] = data_df.groupby(['trafficSource_medium'])['totals_hits'].transform('min')\n    data_df['medium_hits_sum'] = data_df.groupby(['trafficSource_medium'])['totals_hits'].transform('sum')\n    return data_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d2462d9b47b800d55aa97d82bd93fa23db5243f"},"cell_type":"code","source":"#Feature processing\n## Load data\nprint('reading train')\ntrain_df = read_parse_dataframe('train.csv')\ntrn_len = train_df.shape[0]\ntrain_df = process_date_time(train_df)\nprint('reading test')\ntest_df = read_parse_dataframe('test.csv')\ntest_df = process_date_time(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"852ac853e68ea042afa35787c78cdd01de82cefd"},"cell_type":"code","source":"## Drop columns\ncols_to_drop = [col for col in train_df.columns if train_df[col].nunique(dropna=False) == 1]\ntrain_df.drop(cols_to_drop, axis=1, inplace=True)\ntest_df.drop([col for col in cols_to_drop if col in test_df.columns], axis=1, inplace=True)\n\n###only one not null value\ntrain_df.drop(['trafficSource_campaignCode'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4212a2ae637c86777579de007dec13e4e39b5783"},"cell_type":"code","source":"###converting columns format\ntrain_df['totals_transactionRevenue'] = train_df['totals_transactionRevenue'].astype(float)\ntrain_df['totals_transactionRevenue'] = train_df['totals_transactionRevenue'].fillna(0)\ntrain_df['totals_transactionRevenue'] = np.log1p(train_df['totals_transactionRevenue'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da30d151f70114502a7d6a00d034432e1586323f"},"cell_type":"code","source":"## Features engineering\ntrain_df = process_format(train_df)\ntrain_df = process_device(train_df)\ntrain_df = process_totals(train_df)\ntrain_df = process_geo_network(train_df)\ntrain_df = process_traffic_source(train_df)\n\ntest_df = process_format(test_df)\ntest_df = process_device(test_df)\ntest_df = process_totals(test_df)\ntest_df = process_geo_network(test_df)\ntest_df = process_traffic_source(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70685f78a9318a7f3b9441cd76579f1d63017144"},"cell_type":"code","source":"## Categorical columns\nprint(\"process categorical columns ...\")\nnum_cols = ['month_unique_user_count', 'day_unique_user_count', 'weekday_unique_user_count',\n            'visitNumber', 'totals_hits', 'totals_pageviews', \n            'mean_hits_per_day', 'sum_hits_per_day', 'min_hits_per_day', 'max_hits_per_day', 'var_hits_per_day',\n            'mean_pageviews_per_day', 'sum_pageviews_per_day', 'min_pageviews_per_day', 'max_pageviews_per_day',\n            'sum_pageviews_per_network_domain', 'count_pageviews_per_network_domain', 'mean_pageviews_per_network_domain',\n            'sum_hits_per_network_domain', 'count_hits_per_network_domain', 'mean_hits_per_network_domain',\n            'medium_hits_mean','medium_hits_min','medium_hits_max','medium_hits_sum']\n            \nnot_used_cols = [\"visitNumber\", \"date\", \"fullVisitorId\", \"sessionId\", \n        \"visitId\", \"visitStartTime\", 'totals_transactionRevenue', 'trafficSource_referralPath']\ncat_cols = [col for col in train_df.columns if col not in num_cols and col not in not_used_cols]\n\nmerged_df = pd.concat([train_df, test_df])\nprint('Cat columns : ', len(cat_cols))\nohe_cols = []\nfor i in cat_cols:\n    if len(set(merged_df[i].values)) < 100:\n        ohe_cols.append(i)\n\nprint('ohe_cols : ', ohe_cols)\nprint(len(ohe_cols))\nmerged_df = pd.get_dummies(merged_df, columns = ohe_cols)\ntrain_df = merged_df[:trn_len]\ntest_df = merged_df[trn_len:]\ndel merged_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b6b66238b6d8c7163c21fc048a635dbdcecf7f4"},"cell_type":"code","source":"for col in cat_cols:\n    if col in ohe_cols:\n        continue\n    #print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))\n\nprint('FINAL train shape : ', train_df.shape, ' test shape : ', test_df.shape)\n#print(train_df.columns)\ntrain_df = train_df.sort_values('date')\nX = train_df.drop(not_used_cols, axis=1)\ny = train_df['totals_transactionRevenue']\nX_test = test_df.drop([col for col in not_used_cols if col in test_df.columns], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cdbfb33cba21ae0a6d532969750c9d0cd6b61218"},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import KFold, StratifiedKFold\nfrom sklearn import model_selection, preprocessing, metrics\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cfce11ecce4b12a63ef4bf273ca68321de862cbe"},"cell_type":"code","source":"def categorize_target(x):\n    if x < 2:\n        return 0\n    elif x < 4:\n        return 1\n    elif x < 6:\n        return 2\n    elif x < 8:\n        return 3\n    elif x < 10:\n        return 4\n    elif x < 12:\n        return 5\n    elif x < 14:\n        return 6\n    elif x < 16:\n        return 7\n    elif x < 18:\n        return 8\n    elif x < 20:\n        return 9\n    elif x < 22:\n        return 10\n    else:\n        return 11","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d77e160f444680e831038ef3844cecf10c65dd51"},"cell_type":"code","source":"y_categorized = y.apply(categorize_target)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c52a9d042f9a074d204a479e0c2b49ed39511c6"},"cell_type":"markdown","source":"## Light GBM\nWe adopt some ad-hoc hyperparameters, set the objective function to regression and use a random forest as learning method:"},{"metadata":{"trusted":true,"_uuid":"db371e433ce22a690eac7f91f13f598fcdbac8b1"},"cell_type":"code","source":"lgb_params1 = {\"objective\" : \"regression\", \"metric\" : \"rmse\", \n               \"max_depth\": 8, \"min_child_samples\": 21, \n               \"reg_alpha\": 1, \"reg_lambda\": 1,\n               \"num_leaves\" : 257, \"learning_rate\" : 0.01,\n               \"subsample\" : 0.82, \"colsample_bytree\" : 0.84, \n               \"verbosity\": -1}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b9ee707d98cf0b78164c685ac6384c2d792fb8c"},"cell_type":"markdown","source":"The train set is split with a Kfold method and the prediction on the test set are averaged:"},{"metadata":{"trusted":true,"_uuid":"c8f56b2a86dc20cd1063391d7e324b278866c1f1"},"cell_type":"code","source":"def rmse(predictions, targets):\n    return np.sqrt(((predictions - targets) ** 2).mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35ee1800e472a1ba87b6d3d2b7bb35cdfbd50812","scrolled":false},"cell_type":"code","source":"\nFOLDs = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)\n\noof_lgb = np.zeros(len(train_df))\npredictions_lgb = np.zeros(len(test_df))\n\nfeatures_lgb = list(X.columns)\nfeature_importance_df_lgb = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(X, y_categorized)):\n    trn_data = lgb.Dataset(X.iloc[trn_idx], label=y.iloc[trn_idx])\n    val_data = lgb.Dataset(X.iloc[val_idx], label=y.iloc[val_idx])\n    X_val = X.iloc[val_idx]\n    y_val = y.iloc[val_idx]\n    print(\"LGB \" + str(fold_) + \"-\" * 50)\n    num_round = 20000\n    clf = lgb.train(lgb_params1, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=1000, early_stopping_rounds = 100)\n    oof_lgb[val_idx] = clf.predict(X.iloc[val_idx], num_iteration=clf.best_iteration)\n\n    fold_importance_df_lgb = pd.DataFrame()\n    fold_importance_df_lgb[\"feature\"] = features_lgb\n    fold_importance_df_lgb[\"importance\"] = clf.feature_importance()\n    fold_importance_df_lgb[\"fold\"] = fold_ + 1\n    feature_importance_df_lgb = pd.concat([feature_importance_df_lgb, fold_importance_df_lgb], axis=0)\n    predictions_lgb += clf.predict(X_test, num_iteration=clf.best_iteration) / FOLDs.n_splits\n#     perm = PermutationImportance(clf, random_state=1).fit(val_data)\n#     eli5.show_weights(perm, feature_names = X.iloc[val_idx].columns.tolist())\n\n#lgb.plot_importance(clf, max_num_features=30)    \ncols = feature_importance_df_lgb[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(by=\"importance\", ascending=False)[:50].index\nbest_features_lgb = feature_importance_df_lgb.loc[feature_importance_df_lgb.feature.isin(cols)]\nplt.figure(figsize=(14,10))\nsns.barplot(x=\"importance\", y=\"feature\", data=best_features_lgb.sort_values(by=\"importance\", ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')\nx = []\nfor i in oof_lgb:\n    if i < 0:\n        x.append(0.0)\n    else:\n        x.append(i)\ncv_lgb = mean_squared_error(x, y)**0.5\ncv_lgb = str(cv_lgb)\ncv_lgb = cv_lgb[:10]\n\npd.DataFrame({'preds': x}).to_csv('lgb_oof_' + cv_lgb + '.csv', index = False)\n\nprint(\"CV_LGB : \", cv_lgb)\n# return (cv_lgb, predictions_lgb, clf, X_val, y_val, best_features_lgb)\n\n# cv_lgb, lgb_ans, clf, X_val, y_val= kfold_lgb_xgb()\nx = []\nfor i in predictions_lgb:\n    if i < 0:\n        x.append(0.0)\n    else:\n        x.append(i)\nnp.save('lgb_ans.npy', x)\nsubmission = test_df[['fullVisitorId']].copy()\nsubmission.loc[:, 'PredictedLogRevenue'] = x\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].apply(lambda x : 0.0 if x < 0 else x)\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].fillna(0.0)\ngrouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\ngrouped_test.to_csv('lgb_' + cv_lgb + '.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}