{"cells":[{"metadata":{"_uuid":"a200ed0ecbdc649f78653ddb25c99514fcb8caf1"},"cell_type":"markdown","source":"# This kernel is to predict the future of customer will come back purchase or not\n* Fot train_v2 data, we have 2016/08/01 ~ 2018/04/30 period data\n* For test_v2 data, we have 2018/05/1 ~ 2018/10/15 period data\n* The Public LB  score is base on timeframe 2018/05/1~ 2018/10/15\n* The Private LB score is base on timeframe of 2018/12/1 ~ 2019/01/31 with same visitor ID that in test_v2\n* So this competition become the future prediction question ....."},{"metadata":{"_uuid":"b4481f5fd211db7c42245fd88fbd0d432f8c131c"},"cell_type":"markdown","source":"## Discussion topic about this idea from AmirH\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71427\n* I use LGBM to predict the user will come back purchase or not (Classification)\n"},{"metadata":{"_uuid":"8179a534108d35599df7310c0620d5e1e55724aa"},"cell_type":"markdown","source":"## Training Set\n* Training period set 1==> 2016/08/01 ~ 2017/1/15 (5.5 month)\n* Target period set 1  ==> 2017/03/1 ~ 2017/04/30 (2 month)\n* Training period set 2==> 2017/06/01 ~ 2017/11/15 (5.5 month)\n* Target period set 2  ==> 2018/1/1 ~ 2018/02/30 (2 month)\n* Concate set 1 and set 2 to be training data\n* Feature engineering on training period feature\n* Target set that those come back purchased user in target period"},{"metadata":{"_uuid":"472a9b331122cba560deb5f043c27c81578d8089"},"cell_type":"markdown","source":"## Valid Set (1 year ago of our test set and target )\n* Valid period set ==> 2017/5/1 ~ 2017/10/15\n* Valid target period set ==> 2017/12/1 ~ 2018/1/31"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport gc\nimport os\nfrom sklearn.metrics import confusion_matrix,accuracy_score, roc_curve, auc\nimport matplotlib.pyplot as plt\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9941ff3bdefd781bf5543fad4876bdf6c8dea7ea"},"cell_type":"code","source":"import seaborn as sns\nimport json\nimport pandas.io.json as pdjson\nimport ast\n\nfrom pandas.io.json import json_normalize\ndef load_df(csv_path='../input/train_v2.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    for column in JSON_COLUMNS:\n        column_as_df = pdjson.json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d526af7211ee5b6e5fd05c2032e2b94523cc5b61"},"cell_type":"markdown","source":"## Load Data\n* use Aguiar's dataset (Many thanks): https://www.kaggle.com/jsaguiar/parse-json-v2-without-hits-column"},{"metadata":{"trusted":true,"_uuid":"1661752d11eea3c3ea34746ed17a26493ca4533e"},"cell_type":"code","source":"%%time\npath = \"../input/parse-json-v2-without-hits-column/\"\ntrain_df = pd.read_pickle(path + 'train_v2_clean.pkl')\ntest_df = pd.read_pickle(path + 'test_v2_clean.pkl')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81d150e09e39e0bfdc02e15ad833f8998fde8551"},"cell_type":"markdown","source":"## Add time feature"},{"metadata":{"trusted":true,"_uuid":"16dce7cf1ac1db473bb1838b4cc3b1ce61544e12"},"cell_type":"code","source":"for df in [train_df,test_df]:\n    df['date'] = pd.to_datetime(df['visitStartTime'], unit='s')\n    df[\"day\"] = df['date'].dt.day\n    df['month'] = df['date'].dt.month\n    df['weekday'] = df['date'].dt.weekday\n    df['weekofyear'] = df['date'].dt.weekofyear","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"94f2aad8992eba09d70158cc2f6b4b6b5166d622"},"cell_type":"code","source":"train_df.shape, test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81188dedc2ecc5bd434692c99f9e71f33211e3ce"},"cell_type":"markdown","source":"## Feature engineering \n* mean, max, min for \"totals_pagevies\" and \"totals_hits \"\n* Change to lable encoding for categorical feature\n* Drop 'trafficSource_referralPath','trafficSource_source'"},{"metadata":{"trusted":true,"_uuid":"76fda8305f4dde63c10e8007bb0a0dcfc2569b77"},"cell_type":"code","source":"train_df['totals_pageviews']=train_df['totals_pageviews'].astype('float')\ntrain_df['totals_hits']=train_df['totals_hits'].astype('float')\ntest_df['totals_pageviews']=test_df['totals_pageviews'].astype('float')\ntest_df['totals_hits']=test_df['totals_hits'].astype('float')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d617364b846a2b9850a19c72c8d548b9978adbd9"},"cell_type":"code","source":"train_df['totals_pageviews_mean']=train_df.groupby(['fullVisitorId'])['totals_pageviews'].transform('mean')\ntrain_df['totals_pageviews_max']=train_df.groupby(['fullVisitorId'])['totals_pageviews'].transform('max')\ntrain_df['totals_pageviews_min']=train_df.groupby(['fullVisitorId'])['totals_pageviews'].transform('min')\ntrain_df['totals_hits_mean']=train_df.groupby(['fullVisitorId'])['totals_hits'].transform('mean')\ntrain_df['totals_hits_max']=train_df.groupby(['fullVisitorId'])['totals_hits'].transform('max')\ntrain_df['totals_hits_min']=train_df.groupby(['fullVisitorId'])['totals_hits'].transform('min')\ntest_df['totals_pageviews_mean']=test_df.groupby(['fullVisitorId'])['totals_pageviews'].transform('mean')\ntest_df['totals_pageviews_max']=test_df.groupby(['fullVisitorId'])['totals_pageviews'].transform('max')\ntest_df['totals_pageviews_min']=test_df.groupby(['fullVisitorId'])['totals_pageviews'].transform('min')\ntest_df['totals_hits_mean']=test_df.groupby(['fullVisitorId'])['totals_hits'].transform('mean')\ntest_df['totals_hits_max']=test_df.groupby(['fullVisitorId'])['totals_hits'].transform('max')\ntest_df['totals_hits_min']=test_df.groupby(['fullVisitorId'])['totals_hits'].transform('min')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"badb9e56a17e87312a1500c0ef203b69fe939a40"},"cell_type":"code","source":"\"\"\"\ndef process_totals(data_df):\n    print(\"process totals ...\")\n    #data_df['visitNumber'] = np.log1p(data_df['visitNumber'])\n    #data_df['totals_hits'] = np.log1p(data_df['totals_hits'])\n    #data_df['totals_pageviews'] = np.log1p(data_df['totals_pageviews'].fillna(0))\n    data_df['mean_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('mean')\n    data_df['sum_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('sum')\n    data_df['max_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('max')\n    data_df['min_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('min')\n    data_df['var_hits_per_day'] = data_df.groupby(['day'])['totals_hits'].transform('var')\n    data_df['mean_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('mean')\n    data_df['sum_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('sum')\n    data_df['max_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('max')\n    data_df['min_pageviews_per_day'] = data_df.groupby(['day'])['totals_pageviews'].transform('min')    \n    return data_df\ntrain_df = process_totals(train_df)\ntest_df = process_totals(test_df)\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c56d3e36cb418494a5e0e880437770ed9da79941"},"cell_type":"code","source":"train_df.drop(['trafficSource_referralPath', 'trafficSource_source'], axis=1, inplace=True)\ntest_df.drop(['trafficSource_referralPath', 'trafficSource_source'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d1fb0ea9828e886877a102db92b39263be37f6fb"},"cell_type":"code","source":"excluded_features = [\n    'date','fullVisitorId', 'sessionId','classfication_target','totals_totalTransactionRevenue','totals_transactionRevenue',\n    'visitId', 'visitStartTime', 'vis_date', 'nb_sessions', 'max_visits','next_session_1','next_session_2'\n]\ncategorical_features = [\n    _f for _f in train_df.columns\n    if (_f not in excluded_features) & (train_df[_f].dtype == 'object')\n]\none_hot_features = ['day','month','weekday']\n#'totals.totalTransactionRevenue','totals.TransactionRevenue','classfication_target'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c16234133115b16b36267859ec8a74fed5edd4c2"},"cell_type":"markdown","source":"## Process one hot encoding on time "},{"metadata":{"trusted":true,"_uuid":"f9ff2f02137dd0bbe62b8e3bbf462087d8c8eff2"},"cell_type":"code","source":"\nfor i in one_hot_features:\n    print(\"Process feature =====>\"+str(i))\n    train_df[\"one_hot_feature\"] = train_df[i]\n    train_df[\"one_hot_feature\"] =  str(i) + \".\" + train_df[\"one_hot_feature\"].astype('str')\n    one_hot_combine = pd.get_dummies(train_df[\"one_hot_feature\"])\n    print(one_hot_combine.shape)\n    train_df = train_df.join(one_hot_combine)\n    del train_df[\"one_hot_feature\"]\n    del train_df[i]\n    del one_hot_combine\n    print(train_df.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f89bbbec852c61d5dc9bb0ee0bae06fa6d0d225d"},"cell_type":"markdown","source":"### Factoriza  categorical featuers"},{"metadata":{"trusted":true,"_uuid":"8b5feb002f24aad5fab269297ac79f0d93b77184"},"cell_type":"code","source":"\nfor f in categorical_features:\n    train_df[f], indexer = pd.factorize(train_df[f])\n    test_df[f] = indexer.get_indexer(test_df[f])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b56c10500413c36b737d80bc8cac5499bd4da194"},"cell_type":"code","source":"train_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c92c35f5fb7e6f216f03ed11ca915a0944dcb8f2"},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"26df9a4b5da8840d8e4974101b5a9464433a3ff4"},"cell_type":"markdown","source":"## Split Validate and Train data by timeframe"},{"metadata":{"_uuid":"76effd5e7a823f59a1fc32b4445cec8a3c3d3dd7"},"cell_type":"markdown","source":"## Training Set\n* Training period set 1==> 2016/08/01 ~ 2017/1/15 (5.5 month)\n* Target period set 1  ==> 2017/03/1 ~ 2017/04/30 (2 month)\n* Training period set 2==> 2017/06/01 ~ 2017/11/15 (5.5 month)\n* Target period set 2  ==> 2018/1/1 ~ 2018/02/30 (2 month)"},{"metadata":{"trusted":true,"_uuid":"0146fdd7b9c451261a220114da26822b2c074099"},"cell_type":"code","source":"train_df['date'].max(),train_df['date'].min()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20bbe3024034eb867b76cd8fe7eeb92632535742"},"cell_type":"code","source":"test_df['date'].max(),test_df['date'].min()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6c12f2df0e6d140e62304e5d0400aca0613658b"},"cell_type":"markdown","source":"## Training period"},{"metadata":{"trusted":true,"_uuid":"65caa2a9f01a39e502e85efdd94ba1c8a221ad45"},"cell_type":"code","source":"train_period_1 = train_df[(train_df['date']<=pd.datetime(2017,1,15)) & (train_df['date']>=pd.datetime(2016,8,1))]\ntrain_predict_preiod_1 = train_df[(train_df['date']<=pd.datetime(2017,4,30)) & (train_df['date']>=pd.datetime(2017,3,1))]\ntrain_period_2 = train_df[(train_df['date']<=pd.datetime(2017,11,15)) & (train_df['date']>=pd.datetime(2017,6,1))]\ntrain_predict_preiod_2 = train_df[(train_df['date']<=pd.datetime(2018,2,28)) & (train_df['date']>=pd.datetime(2018,1,1))]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07d88ff57a35b5bb4e2e8b3c583921d8dff6a324"},"cell_type":"markdown","source":"## Valid period"},{"metadata":{"trusted":true,"_uuid":"7fbfa75b35371f1a0ad12eb10982a9e4a397931f"},"cell_type":"code","source":"valid_period = train_df[(train_df['date']<=pd.datetime(2017,10,15)) & (train_df['date']>=pd.datetime(2017,5,1))]\nvalid_predict_preiod = train_df[(train_df['date']<=pd.datetime(2018,1,31)) & (train_df['date']>=pd.datetime(2017,12,1))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02b1930e73eaa7482ef7300398f6c6a8723a3215"},"cell_type":"code","source":"print('train_period1_shape',train_period_1.shape) \nprint('train_target1_period_shape',train_predict_preiod_1.shape)\nprint('train_period2_shape',train_period_2.shape) \nprint('train_target2_period_shape',train_predict_preiod_2.shape)\nprint('valid_period_shape',valid_period.shape) \nprint('valid_target_period_shape',valid_predict_preiod.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1170d6fdfb1d003fe9214cb12db17c3b2b306c0f"},"cell_type":"markdown","source":"## Add the target on training data and validation data"},{"metadata":{"trusted":true,"_uuid":"fb3d87921620617cf4638499fd903b2094c70f64"},"cell_type":"code","source":"def add_target(train_period,target_period):\n    \n    train_period['totals_totalTransactionRevenue'] = train_period['totals_totalTransactionRevenue'].fillna(0).astype('float64')\n    target_period['totals_totalTransactionRevenue'] =target_period['totals_totalTransactionRevenue'].fillna(0).astype('float64')\n    train_period['totals_transactionRevenue'] = train_period['totals_transactionRevenue'].fillna(0).astype('float64')\n    target_period['totals_transactionRevenue'] = target_period['totals_transactionRevenue'].fillna(0).astype('float64')\n    #train_period['totals_transactions'] = train_period['totals_transactions'].fillna(0).astype('float64')\n    #target_period['totals_transactions'] = target_period['totals_transactions'].fillna(0).astype('float64')\n    \n    #train_pd=train_period\n    train_pd = train_period.groupby('fullVisitorId').mean().reset_index()\n    target_pd = target_period.groupby('fullVisitorId').mean().reset_index()\n    #target_pd=target_period\n    #Find the visitors those back puchased in future period\n    train_visitors = train_pd.fullVisitorId.unique()\n    train_predict_visitors = target_pd.fullVisitorId.unique()\n    same_visitors = np.intersect1d(train_visitors, train_predict_visitors)\n    \n    #Process data type\n    \n    \n    #Process back user df\n    back_user = target_pd[(target_pd.fullVisitorId.isin(same_visitors)) & (target_pd['totals_transactionRevenue'] > 0)]\n    back_user = back_user[['fullVisitorId','totals_transactionRevenue']]\n    print('we have',len(back_user['fullVisitorId'].value_counts()),'visitors back to purchase at target periods')\n    \n    #Add target\n    train_pd['classfication_target'] = train_pd['fullVisitorId'].map(lambda x: 1 if x in list(back_user['fullVisitorId']) else 0)\n    train_pd['totals_totalTransactionRevenue'] = np.log1p(train_pd['totals_totalTransactionRevenue'])\n    train_pd['totals_transactionRevenue'] = np.log1p(train_pd['totals_transactionRevenue'])\n    print (train_pd.shape)\n    return train_pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9132e2a1760c5f813cb51a54b5fc299b8acb743f"},"cell_type":"code","source":"train_pd_1=add_target(train_period_1,train_predict_preiod_1)\ntrain_pd_2=add_target(train_period_2,train_predict_preiod_2)\nvalid_pd = add_target(valid_period,valid_predict_preiod)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3376fc31a94609517ab41801f78597649b16bd95"},"cell_type":"code","source":"train_set = pd.concat([train_pd_1,train_pd_2], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ef89b47b74864171e58f1f453bcb1d13f9d6707"},"cell_type":"code","source":"train_set.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10df27027c512b309eb39a93b8b37006e2e7da53"},"cell_type":"code","source":"excluded_features = [\n    'date','fullVisitorId', 'sessionId','classfication_target',\n    'visitId', 'visitStartTime', 'vis_date', 'nb_sessions', 'max_visits','next_session_1','next_session_2'\n]\ntrain_features = [_f for _f in train_set.columns if _f not in excluded_features ]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"53462ca1b47527cfcf3773b9ef67376924d29f83"},"cell_type":"markdown","source":"## Set K fold"},{"metadata":{"trusted":true,"_uuid":"90de0c70f900db5c1872211a589f7e59b349c4a3"},"cell_type":"code","source":"from sklearn.model_selection import GroupKFold\ndef get_folds(df=None, n_splits=5):\n    \"\"\"Returns dataframe indices corresponding to Visitors Group KFold\"\"\"\n    # Get sorted unique visitors\n    unique_vis = np.array(sorted(df['fullVisitorId'].unique()))\n\n    # Get folds\n    folds = GroupKFold(n_splits=n_splits)\n    fold_ids = []\n    ids = np.arange(df.shape[0])\n    for trn_vis, val_vis in folds.split(X=unique_vis, y=unique_vis, groups=unique_vis):\n        fold_ids.append(\n            [\n                ids[df['fullVisitorId'].isin(unique_vis[trn_vis])],\n                ids[df['fullVisitorId'].isin(unique_vis[val_vis])]\n            ]\n        )\n\n    return fold_ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"440183e8cf6c21e8abd230aa6ca48c75fa7b026a"},"cell_type":"code","source":"y_target = train_set['classfication_target']\nvalid_target = valid_pd['classfication_target']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcd651d50e8f0300dea71cddc1899ec6d656c4ef"},"cell_type":"markdown","source":"## Start training (5 fold LightGBM)"},{"metadata":{"trusted":true,"_uuid":"10423f549785753603a7ea619f58dc9289ff51b4"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import TimeSeriesSplit, KFold\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import roc_auc_score\nimport lightgbm as lgb\nfrom sklearn.model_selection import GroupKFold\nparams = {\n    \"max_bin\": 512,\n    \"learning_rate\": 0.02,\n    \"boosting_type\": \"gbdt\",\n    \"objective\": \"binary\",\n    \"metric\": \"binary_logloss\",\n    \"num_leaves\": 10,\n    \"min_data\": 100,\n    \"boost_from_average\": True\n}\nn_fold = 5\n#print(train_features)\nfolds = get_folds(df=train_set, n_splits=5)\n\nmodel = lgb.LGBMClassifier(**params, n_estimators = 20000, nthread = 4, n_jobs = -1)\n\noof_reg_preds = np.zeros(train_set.shape[0])\nprediction = np.zeros(valid_pd.shape[0])\n\nfor fold_n, (trn_, val_) in enumerate(folds):\n    print('Fold:', fold_n)\n    #print(f'Train samples: {len(train_index)}. Valid samples: {len(test_index)}')\n    trn_x, trn_y = train_set[train_features].iloc[trn_], y_target.iloc[trn_]\n    val_x, val_y = train_set[train_features].iloc[val_], y_target.iloc[val_]\n    \n\n    model.fit(trn_x, trn_y, \n            eval_set=[(trn_x, trn_y), (val_x, val_y)], eval_metric='AUC',\n            verbose=500, early_stopping_rounds=100)\n    \n    oof_reg_preds[val_] = model.predict(val_x, num_iteration=model.best_iteration_)\n    \n    pred = model.predict(valid_pd[train_features], num_iteration=model.best_iteration_)\n    prediction += pred\n    \nprediction /= n_fold\n#print(accuracy_score(y_target,np.float64(oof_reg_preds>=0.5)))\n#print(accuracy_score(valid_target,np.float64(prediction>=0.5)))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"785cc8f481f3068dbca19471eb68be8814cecd28"},"cell_type":"markdown","source":"## Plot feature important"},{"metadata":{"trusted":true,"_uuid":"1be3a4728644392695b85951dcf0f3bd4e006712","scrolled":false},"cell_type":"code","source":"lgb.plot_importance(model, figsize=(15, 10))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6986ced66043054b04b70712e731e539448091b2"},"cell_type":"code","source":"prediction_ans = np.where(prediction >= 0.2, 1, 0)\n#valid_ans = np.where(prediction>=0.5,1,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d1b377064f8e575655ffbce61b5e7f279572e80"},"cell_type":"code","source":"plt.figure(figsize=(16,6))\nfalse_positive_rate, recall, thresholds = roc_curve(y_target, oof_reg_preds)\nroc_auc = auc(false_positive_rate, recall)\nplt.subplot(121)\nplt.title('Receiver Operating Characteristic (ROC)_train')\nplt.plot(false_positive_rate, recall, 'b', label = 'AUC = %0.3f' %roc_auc)\nplt.legend(loc='lower right')\nplt.plot([0,1], [0,1], 'r--')\nplt.xlim([0.0,1.0])\nplt.ylim([0.0,1.0])\nplt.ylabel('Recall')\nplt.xlabel('Fall-out (1-Specificity)')\n\nfalse_positive_rate, recall, thresholds = roc_curve(valid_target, prediction_ans)\nroc_auc = auc(false_positive_rate, recall)\nplt.subplot(122)\nplt.title('Receiver Operating Characteristic (ROC)_Valid')\nplt.plot(false_positive_rate, recall, 'b', label = 'AUC = %0.3f' %roc_auc)\nplt.legend(loc='lower right')\nplt.plot([0,1], [0,1], 'r--')\nplt.xlim([0.0,1.0])\nplt.ylim([0.0,1.0])\nplt.ylabel('Recall')\nplt.xlabel('Fall-out (1-Specificity)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b789ee63f29ea7b0242b46c5755d555684076676"},"cell_type":"markdown","source":"## Plot confusion matrix of prediction"},{"metadata":{"trusted":true,"_uuid":"3ba81fbe12abf55536d9de60402ddcb2c8650431"},"cell_type":"code","source":"import seaborn as sns\n#Print Confusion Matrix\nplt.figure(figsize=(16,6))\ncm1 = confusion_matrix(y_target, oof_reg_preds)\nlabels = ['0', '1']\nplt.subplot(121)\nsns.heatmap(cm1, xticklabels = labels, yticklabels = labels, annot = True, fmt='d', cmap=\"Blues\", vmin = 0.2);\nplt.title('Confusion Matrix_train')\nplt.ylabel('True Class')\nplt.xlabel('Predicted Class')\n\ncm2 = confusion_matrix(valid_target, prediction_ans)\nlabels = ['0', '1']\nplt.subplot(122)\nsns.heatmap(cm2, xticklabels = labels, yticklabels = labels, annot = True, fmt='d', cmap=\"Blues\", vmin = 0.2);\nplt.title('Confusion Matrix_valid')\nplt.ylabel('True Class')\nplt.xlabel('Predicted Class')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cfe4e70ac8b7429a3707e90e9192f10c8828e48f"},"cell_type":"markdown","source":"## Conclusion \n* We only can see there are only 5 true positives labels....   \n* Try find the key feature, and do another feature enginnering for the future predict \n* Did anyone have better idea and improve AUC for classification?\n\n## Next Step\n* Doing regression for future revenue prediction...\n"},{"metadata":{"trusted":true,"_uuid":"c86232c46e06cf0429a7f4837298c383f5689ec3"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}