{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"<a id=\"0\"></a> <br>\n## Kernel Headlines\n1. [Features at a Glance](#1)\n    1. [Train.csv](#2)\n    \n        1. [first_active_month](#3)\n        2. [card_id](#4)\n        3. [features_#](#5)\n        4. [target](#6)\n        \n\t2. [merchants.csv](#7)\n\t\n\t    1. [merchant_group_id & merchant_category_id & subsector_id](#8)\n\t\t2. [numerical_1 & numerical_2](#9)\n\t\t3. [category_1 & category_2 & category_4 ](#10)\n\t\t4. [most_recent_sales_range & most_recent_purchases_range](#11)\n        5. [city_id & state_id](#12)\n        6. [avg_sales_lag3 & avg_sales_lag6 & avg_sales_lag12](#13)\n        7. [avg_purchases_lag3 & avg_purchases_lag6 & avg_purchases_lag12](#14)\n\t\n    3. [Historical_transactions_df.csv](#15)\n\t\n\t\t1. [authorized_flag & catgory_1 & category_2 & category_3](#16)\n\t\t2. [purchase_amount](#17)\n\t\t3. [installments](#18)\n\t\t4. [churn rate using merchant_id](#19)\n\t\t5. [churn rate using card_id](#20)\n\t\t6. [nested_pie_chart historical state_id & city_id](#21)\n\t\n\t4. [Answering Basic Question about Datasets](#22)\n\t\t1. [Which features could be helpful ?! ](#23)\n\t\t2. [Status of Test Set](#24)\n\t\t3. [Who are disloyal merchants and who are affordable merchants](#25)\n\t\t4. [Who are disloyal cards and who are affordable cards](#26)\n       \n2. [Training](#27)\n      1. [Preprocessing ](#28)\n      2. [SimpleTraining ](#29)\n      3. [Feature Importances](#30)\n      4. [Improving accuracy using boosting methods](#31)"},{"metadata":{"_uuid":"ae640a18b9965f635fc47dbbeef81b5485c0dcd8"},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n#  1-FEATURES AT A GLANCE\n\nIn the first paragraph we will take a glance to our feature spaces.  We will try to data feature by feature.\n\nLets import necessary packages ;-)."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport warnings\nfrom datetime import datetime,timedelta\nimport seaborn as sns\n%matplotlib inline\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f56307116b9d9e296586e25c9920a20a4bde8b16"},"cell_type":"markdown","source":"Considering the memory limitations, we need to reduce the size of historical_transactions. It is the biggest dataset existed in this kernel. \nInspiring form [Fabien Kernel](https://www.kaggle.com/fabiendaniel/elo-world), we decrease the size of our data to prevent probable mermory overflow."},{"metadata":{"trusted":true,"_uuid":"a7b38b09ac0e19ee324df4e51725d02ff3b6ec52","_kg_hide-input":true},"cell_type":"code","source":"\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6eb89eba353499cc32c2760dc22ab82e234b4271","_kg_hide-input":true},"cell_type":"code","source":"train_df = reduce_mem_usage(pd.read_csv(\"../input/train.csv\",parse_dates=[\"first_active_month\"]))\ntest_df = reduce_mem_usage(pd.read_csv(\"../input/test.csv\",parse_dates=[\"first_active_month\"]))\nhistorical_df = reduce_mem_usage(pd.read_csv(\"../input/historical_transactions.csv\",parse_dates=[\"purchase_date\"]))\nnew_merchants_df = reduce_mem_usage(pd.read_csv(\"../input/new_merchant_transactions.csv\",parse_dates=[\"purchase_date\"]))\nmerchants_df = reduce_mem_usage(pd.read_csv(\"../input/merchants.csv\"))\n# tmp_train_df = pd.merge(train_df,pd.concat([new_merchants_df,historical_df]),on=\"card_id\")\n# tmp_test_df =  pd.merge(test_df,pd.concat([new_merchants_df,historical_df]),on=\"card_id\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a165249f5258120eb4fb80e71570cf199781b3a"},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n**A. Train.csv**"},{"metadata":{"trusted":true,"_uuid":"7a2c396268cd375269b4d3dfabb953ba5cd8283a"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"649d078e342b100cc2a8246869339d0c4812f52a"},"cell_type":"code","source":"train_df.first_active_month.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b447199d880aee0a77cd5905beac922aa8590f7"},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n1. FIRST_ACTIVE_MONTH"},{"metadata":{"trusted":true,"_uuid":"252be21c019d2e338a3ef7cced860184edbec8e6"},"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(15,10))\naxes.set_title(\"First Active Month\")\naxes.set_ylabel(\"#\")\naxes.set_xlabel(\"years\")\ntrain_df.first_active_month.value_counts().plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2d7ede26693c214a0588f3c738f667af0b56ef4"},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n2. CARD_ID"},{"metadata":{"trusted":true,"_uuid":"fd552e7896717a26011f3d51ee21a1f5f442d268"},"cell_type":"code","source":"train_df.card_id.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f10239385fe65e7a18e3381cd5fb0b4804783da"},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n3. FEATURES_1 & FEATURES_2 & FEATURES_3"},{"metadata":{"trusted":true,"_uuid":"2940a1904f4c928a4974593f75165c38dc753515"},"cell_type":"code","source":"fig,axes = plt.subplots(1,3,figsize=(15,8),sharey=True)\naxes[0].set_ylabel(\"#\")\naxes[1].set_ylabel(\"#\")\naxes[2].set_ylabel(\"#\")\n# axes.set_xlabel(\"feature\")\ntrain_df[\"feature_1\"].value_counts().plot(kind=\"bar\",ax = axes[0],title=\"feature_1\",rot=0,color=\"tan\")\ntrain_df[\"feature_2\"].value_counts().plot(kind=\"bar\",ax=axes[1],title=\"feature_2\",rot=0,color=\"teal\")\ntrain_df[\"feature_3\"].value_counts().plot(kind=\"bar\",ax=axes[2],title=\"feature_3\",rot=0,color=\"gray\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"897404cba0a6f3d06ef3202fe485e79c5b56e2c2"},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n4. TARGET"},{"metadata":{"trusted":true,"_uuid":"a035bdfe8ed0b714b556c1f6694bdd8f640863a1"},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8,10))\naxes.set_title(\"target boxplot\")\naxes.set_ylabel(\"target\")\naxes.boxplot(list(train_df[\"target\"].values),showmeans=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c7a50cf8a4d4c2f5e4590d32579f2927fde53595"},"cell_type":"markdown","source":"<a id=\"7\"></a> <br>\n**B. Merchants.csv**"},{"metadata":{"trusted":true,"_uuid":"a1ad64f63a4c2daf7770e831f5c24ca415d843c0"},"cell_type":"code","source":"merchants_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82c224e92cfe741de7ca7494993a5c1406915926"},"cell_type":"code","source":"merchants_df.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"522173bab01480961ea0109c1250431b25b747f3"},"cell_type":"markdown","source":"<a id=\"8\"></a> <br>\n1. MERCHANT_GROUP_ID & MERCHANT_CATEGORY_ID & SUBSCTOR_ID\n\n*merchant_group_id: Merchant group (anonymized )*\n\n*merchant_category_id: Unique identifier for merchant category (anonymized )*\n\n*subsector_id: Merchant category group (anonymized )*"},{"metadata":{"trusted":true,"_uuid":"35abd4f4f1f8a957b117b7f5ccfb131f5a12b9c2"},"cell_type":"code","source":"fig,axes = plt.subplots(1,3,figsize=(15,8))\nnp.log(merchants_df[\"merchant_group_id\"].value_counts()[:10]).plot(kind=\"bar\",ax = axes[0],title=\"Log(#(merchant_group_id))\",rot=0,color=\"tan\")\nmerchants_df[\"merchant_category_id\"].value_counts()[:10].plot(kind=\"bar\",ax=axes[1],title=\"#(merchant_category_id)\",rot=0,color=\"teal\")\nmerchants_df[\"subsector_id\"].value_counts()[:10].plot(kind=\"bar\",ax=axes[2],title=\"#(subsector_id)\",rot=0,color=\"blue\")\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d873001e318f36560392ce41384e60146a2d798"},"cell_type":"markdown","source":"<a id=\"9\"></a> <br>\n2. NUMERICAL_1 & NUMERICAL_2\n\n*numerical_1: anonymized measure*\n\n*numerical_2: anonymized measure*"},{"metadata":{"trusted":true,"_uuid":"a32d01efd43ce781971e6add613164fec8b60282"},"cell_type":"code","source":"pd.concat([merchants_df[\"numerical_1\"],merchants_df[\"numerical_2\"]],axis=1).describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70960232c6bd2fb037f54411cb62257873d1a74b"},"cell_type":"markdown","source":"<a id=\"10\"></a> <br>\n3. CATEGORY_1 & CATEGORY_2 & CATEGORY_4\n\n*category_1: anonymized category*\n\n*category_2: anonymized category*\n\n*category_4: anonymized category*"},{"metadata":{"trusted":true,"_uuid":"e0b6d4a721650c72f98fc4612795b6928f288b2c"},"cell_type":"code","source":"fig,axes = plt.subplots(1,3,figsize=(21,7))\nmerchants_df[\"category_1\"].value_counts().plot(kind=\"pie\",explode=(0,0.1),autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[0])\nmerchants_df[\"category_2\"].value_counts().plot(kind=\"pie\",explode=(0,0.01,0.02, 0.03, 0.04),autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[1])\nmerchants_df[\"category_4\"].value_counts().plot(kind=\"pie\",explode=(0,0.1),autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[2])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69729daea062b38681a94b8341d9b1cba3ca435d"},"cell_type":"markdown","source":"<a id=\"11\"></a> <br>\n4. MOST_RECENT_SALES_RANGE & MOST_RECENT_PURCHASE_RANGE\n\n*most_recent_sales_range: Range of revenue (monetary units) in last active month --> A > B > C > D > E *\n\n*most_recent_purchases_range: Range of quantity of transactions in last active month --> A > B > C > D > E*"},{"metadata":{"trusted":true,"_uuid":"30af2ce1ebf963d4ac2461fd4348aef1878066b5"},"cell_type":"code","source":"fig,axes = plt.subplots(1,2,figsize=(16,8))\nmerchants_df[\"most_recent_purchases_range\"].value_counts().plot(kind=\"pie\",autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[0])\nmerchants_df[\"most_recent_sales_range\"].value_counts().plot(kind=\"pie\",autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[1])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c5cc86ad3b0665de28482dd646214f3854ee140"},"cell_type":"markdown","source":"<a id=\"12\"></a> <br>\n5. CITY_ID & STATE_ID\n\n*city_id: City identifier (anonymized )*\n\n*state_id: State identifier (anonymized)*"},{"metadata":{"trusted":true,"_uuid":"42e3d6142b60316ad882c6e35de223ad58790497"},"cell_type":"code","source":"#city_id && #state_id\nfig,axes = plt.subplots(2,1,figsize=(16,8))\nnp.log(merchants_df[\"city_id\"].value_counts())[:20].plot(kind=\"bar\",title=\"Log(#(city_id))\",rot=0,ax=axes[0],color=\"b\")\nnp.log(merchants_df[\"state_id\"].value_counts()).plot(kind=\"bar\",title=\"Log(#(state_id))\",rot=0,ax=axes[1],color=\"b\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd5bc770f4f8a06bba5e41b2643d21482cc56774"},"cell_type":"markdown","source":"<a id=\"13\"></a> <br>\n6. AVG_SALES_LAG3 & AVG_SALES_LAG6 & AVG_SALES_LAG12\n\n*avg_sales_lag3: Monthly average of revenue in last 3 months divided by revenue in last active month*\n\n*avg_sales_lag6: Monthly average of revenue in last 6 months divided by revenue in last active month*\n\n*avg_sales_lag12: Monthly average of revenue in last 12 months divided by revenue in last active month*"},{"metadata":{"trusted":true,"_uuid":"8eeba0ce1a69ec3d52b68f2db212b4cae83a7cff"},"cell_type":"code","source":"merchants_df[[\"avg_sales_lag3\",\"avg_sales_lag6\",\"avg_sales_lag12\"]].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d2a0ecd589d2df9a53868fd305638f4bbdbe8d6"},"cell_type":"markdown","source":"<a id=\"14\"></a> <br>\n7. AVG_PURCHASES_LAG3 & AVG_PURCHASES_LAG6 & AVG_PURCHASES_LAG12\n\n*avg_purchases_lag3: Monthly average of transactions in last 3 months divided by transactions in last active month*\n\n*avg_purchases_lag6: Monthly average of transactions in last 6 months divided by transactions in last active month*\n\n*avg_purchases_lag12: Monthly average of transactions in last 12 months divided by transactions in last active month*"},{"metadata":{"trusted":true,"_uuid":"1eea89595470c1678f9e36cecc2d7bb7dbdc0131"},"cell_type":"code","source":"merchants_df[[\"avg_purchases_lag3\",\"avg_purchases_lag6\",\"avg_purchases_lag12\"]].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e22765691c40e4332e39ebb274742fef8c91ff2"},"cell_type":"markdown","source":"<a id=\"15\"></a> <br>\n**C. History_transactions.csv**"},{"metadata":{"trusted":true,"_uuid":"b77480c29f4eb41f94efcdefe3879acbac6a7d16"},"cell_type":"code","source":"historical_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5545c75a7a128e98e0a229637c95a8afa7a0942"},"cell_type":"code","source":"historical_df.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d9ffc9081c9fabbc522a950eb294e21f26ac42f"},"cell_type":"markdown","source":"<a id=\"16\"></a> <br>\n1. AUTHORIZED_FLAG & CATEGORY_1 & CATEGORY_2 & CATEGORY_3\n"},{"metadata":{"trusted":true,"_uuid":"ca1b96cfa70c16cbb8e8a399a85edc6665f9aec2"},"cell_type":"code","source":"fig,axes = plt.subplots(2,2,figsize=(8,8))\nhistorical_df[\"authorized_flag\"].value_counts().plot(kind=\"pie\",explode=(0,0.1),autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[0][0])\nhistorical_df[\"category_1\"].value_counts().plot(kind=\"pie\",explode=(0,0.1),autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[0][1])\nhistorical_df[\"category_3\"].value_counts().plot(kind=\"pie\",explode=(0,0.1,0.1),autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[1][0])\nhistorical_df[\"category_2\"].value_counts().plot(kind=\"pie\",explode=(0,0.1,0.1,0.1,0.1),autopct='%1.1f%%',shadow=False, startangle=90,ax=axes[1][1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"783ac3a7cb5ab8a5896e8de0061ee5ce559438fe"},"cell_type":"markdown","source":"<a id=\"17\"></a> <br>\n2.  PURCHASE_AMOUNT\n"},{"metadata":{"trusted":true,"_uuid":"cdd308b5ebaa0b378676ca42931bd6fd6b2b64e1"},"cell_type":"code","source":"historical_df[\"purchase_amount\"].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ae2be55e3b604588c0a6521fd6cca705af26574"},"cell_type":"markdown","source":"There is a huge difference between 75% percentile and maximum. You can conclude that there is a peak which may not be related to normal activeities. Lets get 99% percentile to check this. "},{"metadata":{"trusted":true,"_uuid":"2fe5d58d50e23b4e1fcaf7caf25ba5ec6850d9a0"},"cell_type":"code","source":"np.percentile(historical_df[\"purchase_amount\"].values,q=99)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e62328f60dc3d91f73db393bc4f33a3e3258ec1"},"cell_type":"markdown","source":"99 percent of purchase amounts are less than 1.22. Lets assume the remaining 1 percent is outlier and move to visualize it."},{"metadata":{"trusted":true,"_uuid":"02e3f2f3b5bfcdf4ef90955402a28f542cadcebd"},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 8))\naxes.set_title(\"purchase amount violin plot (removing top 1%)\")\naxes.set_ylabel(\"purchase_amount\")\nax = sns.violinplot(y=list(historical_df[historical_df[\"purchase_amount\"] < np.percentile(historical_df[\"purchase_amount\"],99)][\"purchase_amount\"]),showmeans=True,showmedians=True, palette=\"muted\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f5aa7d9fc24fbf20b436a70353e26358544ee56"},"cell_type":"markdown","source":"<a id=\"18\"></a> <br>\n3.  INSTALLMENTS\n"},{"metadata":{"trusted":true,"_uuid":"a17f2b0fe30c1efbb01e61ffd7058ae20cba99cb"},"cell_type":"code","source":"fig,axes = plt.subplots(1,1,figsize=(8,8))\naxes.set_title(\"log(#(installments))\")\naxes.set_ylabel(\"#\")\naxes.set_xlabel(\"records\")\nnp.log(historical_df.installments.value_counts()).plot(kind=\"bar\",color=\"b\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"152c8dbcbb0440b683c670d1a8e79c56008621bc"},"cell_type":"markdown","source":"<a id=\"19\"></a> <br>\n4.  CHURN_RATE WITH MERCHANT_ID\n"},{"metadata":{"trusted":true,"_uuid":"849b0ad639b685051a2c8dd1e1d74681fa3035d0"},"cell_type":"code","source":"historical_df[\"purchase_date\"].head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"829d745f09f13d3fbe588ae668d07e29c3624dd6"},"cell_type":"code","source":"\"first_day:'{}' and last_day:'{}'\".format(np.min(historical_df[\"purchase_date\"]), np.max(historical_df[\"purchase_date\"]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b01897b99c7c0599844ba08e2573fcf63d660661"},"cell_type":"code","source":"start_date = datetime(2017, 1, 1)\nend_date = datetime(2018, 2, 1)\ndef daterange(start_date, end_date):\n    for n in range(int((end_date - start_date).days)):\n        yield start_date + timedelta(n)\ndates_month = []\nfor single_date in daterange(start_date, end_date):\n    dates_month.append(single_date.strftime(\"%Y-%m\"))\ndates_month = list(set(dates_month))\n\ntmp_churn_df = pd.DataFrame()\ntmp_churn_df[\"date\"] = historical_df[\"purchase_date\"]\ntmp_churn_df[\"yaer\"] = pd.DatetimeIndex(tmp_churn_df[\"date\"]).year\ntmp_churn_df[\"month\"] =pd.DatetimeIndex(tmp_churn_df[\"date\"]).month\ntmp_churn_df[\"merchant_id\"] = historical_df[\"merchant_id\"]\ntmp_churn_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"435eb3b5c5513d711875c2bb2eb301abd28a3f39"},"cell_type":"code","source":"\"distinct merchants who have bought on the website on 2017-01 are:'{}'merchants\".format(len(set(historical_df[(tmp_churn_df.yaer == 2017) & (tmp_churn_df.month == 1) ][\"merchant_id\"])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"889fa5e4f20deafa082131b2aa70451fc335c027"},"cell_type":"code","source":"target_intervals_list = [(2017,1),(2017,2),(2017,3),(2017,4),(2017,5),(2017,6),(2017,7),(2017,8),(2017,9),(2017,10),(2017,11),(2017,12),(2018,1),(2018,2)]\nintervals_visitors = []\nfor tmp_tuple in target_intervals_list:\n    intervals_visitors.append(tmp_churn_df[(tmp_churn_df.yaer == tmp_tuple[0]) & (tmp_churn_df.month == tmp_tuple[1]) ][\"merchant_id\"])\n\"Size of intervals_visitors:{} \".format(len(intervals_visitors))\n\ntmp_matrix = np.zeros((14,14))\n\nfor i in range(0,14):\n    k = False\n    tmp_set = []\n    for j in range(i,14): \n        if k:\n            tmp_set = tmp_set & set(intervals_visitors[j])\n        else:\n            tmp_set = set(intervals_visitors[i]) & set(intervals_visitors[j])\n        tmp_matrix[i][j] = len(list(tmp_set))\n        k = True\nxticklabels = [\"interval 1\",\"interval 2\",\"interval 3\",\"interval 4\",\"interval 5\",\"interval 6\",\"interval 7\",\"interval 8\",\n              \"interval 9\",\"interval 10\",\"interval 11\", \"interval 12\",\"interval 13\",\"interval 14\"]\nyticklabels = [(2017,1),(2017,2),(2017,3),(2017,4),(2017,5),(2017,6),(2017,7),(2017,8),(2017,9),(2017,10),(2017,11),(2017,12),(2018,1),(2018,2)]\nfig, ax = plt.subplots(figsize=(14,14))\nax = sns.heatmap(np.array(tmp_matrix,dtype=int), annot=True, cmap=\"RdBu_r\",xticklabels=xticklabels,fmt=\"d\",yticklabels=yticklabels)\nax.set_title(\"Churn-rate using merchant_id\")\nax.set_xlabel(\"intervals\")\nax.set_ylabel(\"months\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0f3b2b3bea7c05fce22448cf68c505aefa9e121e"},"cell_type":"code","source":"import gc;gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41574df079609f02db1fb60a3ac5eac1a1473d56"},"cell_type":"markdown","source":"<a id=\"20\"></a> <br>\n5.  CHURN_RATE WITH CARD_ID\n"},{"metadata":{"trusted":true,"_uuid":"5fcad0c95fb9031bfadaf8269ce7c1edbd429d16"},"cell_type":"code","source":"start_date = datetime(2017, 1, 1)\nend_date = datetime(2018, 2, 1)\ndef daterange(start_date, end_date):\n    for n in range(int((end_date - start_date).days)):\n        yield start_date + timedelta(n)\ndates_month = []\nfor single_date in daterange(start_date, end_date):\n    dates_month.append(single_date.strftime(\"%Y-%m\"))\ndates_month = list(set(dates_month))\n\ntmp_churn_df = pd.DataFrame()\ntmp_churn_df[\"date\"] = historical_df[\"purchase_date\"]\ntmp_churn_df[\"yaer\"] = pd.DatetimeIndex(tmp_churn_df[\"date\"]).year\ntmp_churn_df[\"month\"] =pd.DatetimeIndex(tmp_churn_df[\"date\"]).month\ntmp_churn_df[\"card_id\"] = historical_df[\"card_id\"]\ntmp_churn_df.head()\n\n\"distinct cards who have bounght on the website on 2017-01 are:'{}'merchants\".format(len(set(historical_df[(tmp_churn_df.yaer == 2017) & (tmp_churn_df.month == 1) ][\"merchant_id\"])))\ntarget_intervals_list = [(2017,1),(2017,2),(2017,3),(2017,4),(2017,5),(2017,6),(2017,7),(2017,8),(2017,9),(2017,10),(2017,11),(2017,12),(2018,1),(2018,2)]\nintervals_visitors = []\nfor tmp_tuple in target_intervals_list:\n    intervals_visitors.append(tmp_churn_df[(tmp_churn_df.yaer == tmp_tuple[0]) & (tmp_churn_df.month == tmp_tuple[1]) ][\"card_id\"])\n\"Size of intervals_visitors:{} \".format(len(intervals_visitors))\n\ntmp_matrix = np.zeros((14,14))\n\nfor i in range(0,14):\n    k = False\n    tmp_set = []\n    for j in range(i,14): \n        if k:\n            tmp_set = tmp_set & set(intervals_visitors[j])\n        else:\n            tmp_set = set(intervals_visitors[i]) & set(intervals_visitors[j])\n        tmp_matrix[i][j] = len(list(tmp_set))\n        k = True\nxticklabels = [\"interval 1\",\"interval 2\",\"interval 3\",\"interval 4\",\"interval 5\",\"interval 6\",\"interval 7\",\"interval 8\",\n              \"interval 9\",\"interval 10\",\"interval 11\", \"interval 12\",\"interval 13\",\"interval 14\"]\nyticklabels = [(2017,1),(2017,2),(2017,3),(2017,4),(2017,5),(2017,6),(2017,7),(2017,8),(2017,9),(2017,10),(2017,11),(2017,12),(2018,1),(2018,2)]\nfig, ax = plt.subplots(figsize=(14,14))\nax = sns.heatmap(np.array(tmp_matrix,dtype=int), annot=True, cmap=\"RdBu_r\",xticklabels=xticklabels,fmt=\"d\",yticklabels=yticklabels)\nax.set_title(\"Churn-rate using card_ids\")\nax.set_xlabel(\"intervals\")\nax.set_ylabel(\"months\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c972ecaa61fe9c4a8d1d33362623836ce0d43f98"},"cell_type":"markdown","source":"Comparing two diagram represents that in overall number of buys have been increased and the system have satisfied churn rate. We will go deeper to churn rates and conversion rates in next sections."},{"metadata":{"trusted":true,"_uuid":"d65b2f933e0ce6222417c5987ec9f4c3f3aedcee"},"cell_type":"code","source":"del tmp_churn_df\nimport gc;gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8a18a94edb9875900fd32b8fec881df4c2fb391"},"cell_type":"markdown","source":"<a id=\"21\"></a> <br>\n5.  NESTED PIE CHART FOR STATE_ID AND CITY_ID\n"},{"metadata":{"trusted":true,"_uuid":"03a995c62a2d3ab281edc9b16cb49bd24fa6f6fd"},"cell_type":"code","source":"city_state_df = historical_df[[\"city_id\" , \"state_id\"]]\n\ntmp_df = city_state_df.groupby(by=\"city_id\").count().reset_index()\ntmp_df = tmp_df.rename(index=str,columns={\"state_id\" : \"count\"})\ntmp_df.sort_values(ascending=False,by=\"count\",inplace=True)\n# tmp_df = tmp_df.head(100)\nlables=list(tmp_df.city_id)\nsizes=list(tmp_df[\"count\"])\n\ntmp_df=city_state_df.groupby(by=\"state_id\").count().reset_index()\ntmp_df = tmp_df.rename(index=str,columns={\"city_id\" : \"count\"})\ntmp_df.sort_values(ascending=False,by=\"count\",inplace=True)\nlabels_gender = list(tmp_df.state_id)\nsizes_gender = list(tmp_df[\"count\"])\n\nfig, ax = plt.subplots(figsize=(15,15))\nplt.pie(sizes, labels=lables, startangle=90,frame=True)\nplt.pie(sizes_gender,radius=0.75,startangle=90)\ncentre_circle = plt.Circle((0,0),0.5,color='black', fc='white',linewidth=0)\nfig = plt.gcf()\nfig.gca().add_artist(centre_circle)\nplt.show()\nprint()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"05cce1da8b247f5e384cda36682bc647887c3d2e"},"cell_type":"code","source":"del city_state_df\ndel tmp_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ff957a349ee0b3c43de160fbba430f402169b72"},"cell_type":"markdown","source":"states -1, 1 and 2 have more than half of buy records. "},{"metadata":{"_uuid":"1940c9d110c6029584344a92c7f739ad5bde9980"},"cell_type":"markdown","source":"<a id=\"22\"></a> <br>\n**D. ANSWERING BASIC QUESTIONS ABOUT DATASETS. **\n\n<a id=\"23\"></a> <br>\n1.  WHICH FEATURES COULD BE HELPFUL\n"},{"metadata":{"trusted":true,"_uuid":"5812192b70f7562b0eda2d515ee15ce945838cef"},"cell_type":"code","source":"historical_df[\"year\"] = historical_df[\"purchase_date\"].dt.year\nhistorical_df[\"month\"] = historical_df[\"purchase_date\"].dt.month\nhistorical_df[\"day\"] = historical_df[\"purchase_date\"].dt.day\nhistorical_df[\"hour\"] = historical_df[\"purchase_date\"].dt.hour\n#  you can also use single line commands\n# historical_df = pd.concat(\n#     [\n#         historical_df,\n#         historical_df[\"purchase_date\"].apply(lambda x:\n#                                              pd.Series({\n#                                                  'year':x.year,\n#                                                  'month':x.month,\n#                                                  'day':x.day\n#                                              }))\n#     ]\n#     ,axis=1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f6cccd1a832de9e3aa5b4500d91bc7e8aa43d73f"},"cell_type":"code","source":"fig,axes = plt.subplots(4,2,figsize=(10,30))\naxes[0][0].set_title(\"yearly histogram\")\naxes[0][1].set_title(\"monthly histogram\")\naxes[1][0].set_title(\"daily histogram\")\naxes[1][1].set_title(\"hourly histogram\")\naxes[2][0].set_title(\"state_id histogram\")\naxes[2][1].set_title(\"city_id histogram\")\naxes[3][0].set_title(\"subsector_id histogram\")\naxes[3][1].remove()\n\nhistorical_df.year.hist(ax=axes[0][0],normed=True,bins=2)\nhistorical_df.month.hist(ax=axes[0][1],normed=True,bins=12)\nhistorical_df.day.hist(ax=axes[1][0],normed=True,bins=30)\nhistorical_df.hour.hist(ax=axes[1][1],normed=True,bins=24)\nhistorical_df[\"state_id\"].hist( bins=30,normed=True,ax=axes[2][0])\nhistorical_df[\"city_id\"].hist( bins=30,normed=True,ax=axes[2][1])\nhistorical_df[\"subsector_id\"].hist( bins=30,normed=True,ax=axes[3][0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5dfa9ee058bc0406497f5af540511782055b3cd5"},"cell_type":"markdown","source":"Although daily histogram reveals that there is a uniform distribution for historical data but hourly histogram represents that it could be usefull data for our competition goal. city id, state id and also subsector id have acceptable entropy and they may have considerable potential for enhacing the accuracy of model."},{"metadata":{"trusted":true,"_uuid":"e29f0c2f44550849bf58ae8b0ae20eb64f951cc8"},"cell_type":"code","source":"import gc; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e54fa9a4a518396312650892e3002e4dce2fbadb"},"cell_type":"markdown","source":"<a id=\"24\"></a> <br>\n2.  STATUS OF TEST SET ?!"},{"metadata":{"trusted":true,"_uuid":"ac8b4b0d4591f7a24284e4c0fcfe49e103f93e14"},"cell_type":"code","source":"test_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"84bd4cd2641d2c43a8ef790be2e9984c18ff2dc2"},"cell_type":"code","source":"test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54da963e67551edf3167681c3f5068d4c3fa1b28"},"cell_type":"code","source":"# test_df = test_df.dropna()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25b040ad6e5c504e1400b6d19bc3d45059e6d360"},"cell_type":"markdown","source":"test_df contains a null in first_active_month in record number #11578"},{"metadata":{"trusted":true,"_uuid":"834c45f27d14e6ff958935a8969c6ebcd3f6becd"},"cell_type":"code","source":"datetimes_df = pd.concat([train_df[[\"first_active_month\"]].groupby(by=[\"first_active_month\"],axis=0).size(),test_df[[\"first_active_month\"]].groupby(by=[\"first_active_month\"],axis=0).size()],axis=1)\n\nfig, ax1 = plt.subplots(figsize=(20,10))\nt = datetimes_df.index\ns1 = datetimes_df.iloc[:,0]\nax1.plot(t, s1, 'b-')\nax1.set_xlabel('date')\n# Make the y-axis label, ticks and tick labels match the line color.\nax1.set_ylabel('train', color='b')\nax1.tick_params('y', colors='b')\n\nax2 = ax1.twinx()\ns2 = datetimes_df.iloc[:,1]\nax2.plot(t, s2, 'r--')\nax2.set_ylabel('test', color='r')\nax2.tick_params('y', colors='r')\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0186af0f2e0e5000293e09e88f54ca0689d7bdd5"},"cell_type":"markdown","source":"Test sets are completeltely in the same distribution the train data are."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"14d7f5598639305ea5c3f4120decad9c1af1f890"},"cell_type":"code","source":"fig,axes = plt.subplots(1,2,figsize=(10,5))\naxes[0].set_title(\"train card_id distru\")\naxes[1].set_title(\"test card_id distru\")\naxes[0].hist(train_df[[\"card_id\"]].groupby(by=[\"card_id\"]).size(),bins=3)\naxes[1].hist(test_df[[\"card_id\"]].groupby(by=[\"card_id\"]).size(),bins=3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1de7d69dc5d09665c0da7549bf6c85e6f29e012a"},"cell_type":"markdown","source":"   There is only one card_id in both of the train and test datasets."},{"metadata":{"_uuid":"ccc9e3c341ac8f5e9455386188cc56a7a6c63a9b"},"cell_type":"markdown","source":"<a id=\"25\"></a> <br>\n3.  WHO ARE DISLOYAL MERCHANTS & WHO ARE AFFORDABLE MERCHANTS  ?!"},{"metadata":{"trusted":true,"_uuid":"0460814999b54e5e822e927495094f139c7079be"},"cell_type":"code","source":"# calling gargage collector\ndel datetimes_df\nimport gc; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5103e1f602853ede573907a9e0707021a7fc1f8f"},"cell_type":"code","source":"card_merchant_df = historical_df[[\"card_id\",\"merchant_id\",\"purchase_amount\"]]\ncard_merchant_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a0eb21a1ca2287dab38bb12478d63b22451dfc28"},"cell_type":"code","source":"merchants_loyality_and_revenue = card_merchant_df.groupby(by=[\"merchant_id\"]).agg({\"card_id\":\"count\",\"purchase_amount\":\"sum\"})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b51be6b94d902f6b3d4d57a5bfcc0fe5b121114"},"cell_type":"code","source":"merchants_loyality_and_revenue = merchants_loyality_and_revenue.reset_index().rename(index=str , columns={\"purchase_amount\" : \"amount\" , \"card_id\" : \"count\"})\nmerchants_loyality_and_revenue[\"avg\"] = merchants_loyality_and_revenue[\"amount\"] / merchants_loyality_and_revenue[\"count\"]\nmerchants_loyality_and_revenue.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"89897d10359961426f0832274cfc111971d5d59c"},"cell_type":"code","source":"# who are disloyal merchants\nmerchants_loyality_and_revenue.sort_values(by=\"count\",ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d57ab1486de66f7d14269e6f7079fbc27e44e2ca"},"cell_type":"markdown","source":"As you can see merchant_id 'M_ID_00a6ca8a8a' has the most interactions with card_ids."},{"metadata":{"trusted":true,"_uuid":"551e88eb97f175530ac7719cab6254687418e127"},"cell_type":"code","source":"#who are the most affordable merchants\nmerchants_loyality_and_revenue.sort_values(by=\"avg\",ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1559a87dbc88c117f7b7099c75cc4ab870d06f8d"},"cell_type":"markdown","source":"<a id=\"26\"></a> <br>\n4.  WHO ARE DISLOYAL CARDS & WHO ARE AFFORDABLE CARDS  ?!"},{"metadata":{"trusted":true,"_uuid":"15c3baf69f9ea74ad6b84165e0c571daa50ac359"},"cell_type":"code","source":"del merchants_loyality_and_revenue","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9df2665401b5b6b761d24e499c303f9c5ff6d145"},"cell_type":"code","source":"cards_loyality_and_revenue = card_merchant_df.groupby(by=[\"card_id\"]).agg({\"merchant_id\":\"count\",\"purchase_amount\":\"sum\"})\ncards_loyality_and_revenue = cards_loyality_and_revenue.reset_index().rename(index=str , columns={\"purchase_amount\" : \"amount\" , \"merchant_id\" : \"count\"})\ncards_loyality_and_revenue[\"avg\"] = cards_loyality_and_revenue[\"amount\"] / cards_loyality_and_revenue[\"count\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02b0f943c011a7e72cfe20d0c7be4118db8d46e7"},"cell_type":"code","source":"cards_loyality_and_revenue.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9213ebea391c41d1df2e751545f12d89a4368cc2"},"cell_type":"code","source":"# who are disloyal cards\ncards_loyality_and_revenue.sort_values(by=\"count\",ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e94c6200e805ae6d41cbe1d5d524bb64916e941"},"cell_type":"code","source":"#who are the most affordable cards\ncards_loyality_and_revenue.sort_values(by=\"avg\",ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e89e0dd7c0869a9180f240960c3881f83a32cbc6"},"cell_type":"code","source":"import gc; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ee9130aef105c3936cc009411eeac5910691eb3"},"cell_type":"markdown","source":"<a id=\"27\"></a> <br>\n#  2-TRAINING\n\nNow, lets move on training steps. In the first poit we will try to do simple regression on our data."},{"metadata":{"trusted":true,"_uuid":"ffba87c2a2c3d44b9367d7b39ce7191fda3c65c8"},"cell_type":"code","source":"historical_df.head(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0536667fcf09cec66322cf0c55d6a2d52921ed40"},"cell_type":"markdown","source":"<a id=\"28\"></a> <br>\n**A. PREPROCESSING**"},{"metadata":{"trusted":true,"_uuid":"95c024321c48f4b251db43e24c079d14f917a404"},"cell_type":"code","source":"#extracting information from historical_df\nimport gc;gc.collect()\nhistorical_df['authorized_flag'] = historical_df['authorized_flag'].map({'Y':1, 'N':0})\ndef aggregate_historical_transactions(history):\n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'state_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max'],\n        'year': ['nunique'],\n        'month': ['nunique'],\n        'day': ['nunique'],\n        'hour': ['nunique'],\n        'merchant_category_id': ['nunique'],\n        }\n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['hist_' + '_'.join(col).strip() \n                           for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='hist_transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history\n\nnew_history = aggregate_historical_transactions(historical_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f93b5329eaf0c74152638f237d5c5c7f5a4003dd"},"cell_type":"code","source":"new_history.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"50374eee189427c54f3b6fd99af76c434cd034f7"},"cell_type":"code","source":"new_merchants_df.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67896e06c1ce49794c3ace0258b805243a1c6b5f"},"cell_type":"code","source":"new_merchants_df['authorized_flag'] = new_merchants_df['authorized_flag'].map({'Y':1, 'N':0})\ndef aggregate_new_transactions(new_trans):    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'month_lag': ['min', 'max'],\n        'subsector_id':['nunique'],\n        'state_id':['nunique']\n        }\n    agg_new_trans = new_trans.groupby(['card_id']).agg(agg_func)\n    agg_new_trans.columns = ['new_' + '_'.join(col).strip() \n                           for col in agg_new_trans.columns.values]\n    agg_new_trans.reset_index(inplace=True)\n    \n    df = (new_trans.groupby('card_id')\n          .size()\n          .reset_index(name='new_transactions_count'))\n    \n    agg_new_trans = pd.merge(df, agg_new_trans, on='card_id', how='left')\n    \n    return agg_new_trans\n\nnew_merchants = aggregate_new_transactions(new_merchants_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e518e8c84e5ece5c025a3382886fb927c594d83c","trusted":true},"cell_type":"code","source":"new_merchants.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19b8703733e6a9768eb2bb1b2b96255f79963e2b"},"cell_type":"code","source":"train_df['elapsed_time'] = (datetime(2018, 2, 1) - train_df['first_active_month']).dt.days\ntest_df['elapsed_time'] = (datetime(2018, 2, 1) - test_df['first_active_month']).dt.days","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d8cb4b6638bf561cd099b1f58d412c43ff0a5801"},"cell_type":"code","source":"train_df = pd.merge(train_df, new_history, on='card_id', how='left')\ntest_df = pd.merge(test_df, new_history, on='card_id', how='left')\n\ntrain_df = pd.merge(train_df, new_merchants, on='card_id', how='left')\ntest_df = pd.merge(test_df, new_merchants, on='card_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"84e145eb349d03256f26e951ebfbc671acbad6cd"},"cell_type":"code","source":"train_df.shape, test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9e824cd539c27e589e2793df6fddfd01c2868fc"},"cell_type":"code","source":"set(train_df.columns)-set(test_df.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c238fa47386b5477e1bdf866bcd49a0bf8b69451"},"cell_type":"code","source":"# categorizing feature_1 & feature_2 & feature_3\nfrom sklearn.preprocessing import LabelEncoder\nfor col in [\"feature_1\", \"feature_2\",\"feature_3\"]:\n#     print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd9d24360179bc7cdd149811f0a31e36bd5f4103"},"cell_type":"code","source":"exclude_features = ['card_id', 'first_active_month']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"a5cdb971a332073d527556215d5808473360cbf5"},"cell_type":"code","source":"train_df = train_df.loc[:,train_df.columns[~train_df.columns.isin(exclude_features)]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f06d08b6d27ce161cc2687c5d1fd69e00c3c810"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n_train, _eval = train_test_split(train_df, test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c736263b8756ff7e125ae3f0eec824ad7670093f"},"cell_type":"code","source":"_train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2000b4edb7ee1d83d67f66b96ac1d02e6a9163b9"},"cell_type":"markdown","source":"<a id=\"29\"></a> <br>\n**B. SIMPLE TRAINING**"},{"metadata":{"trusted":true,"_uuid":"2bad6e52ccffa516e70daddda7c4e34c60433428"},"cell_type":"code","source":"import lightgbm as lgb\nparams = {\n    \"objective\": \"regression\",\n    \"metric\": \"rmse\",\n    \"num_leaves\": 30,\n    \"min_child_samples\": 100,\n    \"learning_rate\": 0.1,\n    \"bagging_fraction\": 0.7,\n    \"feature_fraction\": 0.5,\n    \"bagging_frequency\": 5,\n    \"bagging_seed\": 2018,\n    \"verbosity\": -1\n}\nlgb_train = lgb.Dataset(_train.loc[:,_train.columns[~_train.columns.isin([\"target\"])]], _train[\"target\"])\nlgb_eval = lgb.Dataset(_eval.loc[:,_eval.columns[~_eval.columns.isin([\"target\"])]], _eval[\"target\"], reference=lgb_train)\ngbm = lgb.train(params, lgb_train, num_boost_round=2000, valid_sets=[lgb_eval], early_stopping_rounds=100,verbose_eval=100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"44f26cf101a9338568b91368741d7cb192b5c05f"},"cell_type":"code","source":"exclude_features = [\"first_active_month\",\"card_id\"]\ntest_df = test_df.loc[:,test_df.columns[~test_df.columns.isin(exclude_features)]]\n\npredicted_target = gbm.predict(test_df, num_iteration=gbm.best_iteration)\npredicted_target[predicted_target < 0] = 0 \n\nsub_df = pd.read_csv('../input/sample_submission.csv')\nsub_df[\"target\"] = predicted_target\n\nsub_df.to_csv(\"submission_raw.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30b75b1a29cc51e47fc83ba9188eee26617fbd5d"},"cell_type":"markdown","source":"You will get 3.92 score if use the output for submission.\n\nIn next steps we will use some methods for improving the performance of the regressioner. "},{"metadata":{"trusted":true,"_uuid":"4a07bbd00370f52681825e8a4d642afa94a9c135"},"cell_type":"code","source":"test_df.shape, sub_df.shape","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"8f2b86fcbd912b4aabf56a002f87efd2bd2a2785"},"cell_type":"markdown","source":"<a id=\"30\"></a> <br>\n**C. FEATURE IMPORTANCE**"},{"metadata":{"trusted":true,"_uuid":"2bd6772f1dbe9f6d4fb596655538d24694cdb9fd"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nlgb.plot_importance(gbm, max_num_features=50, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ad7bc6ef265b7c39e4c0b68a72ac5d86681d406"},"cell_type":"markdown","source":"<a id=\"31\"></a> <br>\n**D. IMPROVING PERFORMANCE USING BOOSTING METHODS**\n\n\nNow, lets improve our accuracy by using boosting approaches. We have used [This kernel](https://www.kaggle.com/youhanlee/hello-elo-ensemble-will-help-you) because of easy and straight forward road map it has. Thanks to @youhanlee."},{"metadata":{"trusted":true,"_uuid":"d6665536fe0ee652ef52fd57846ed375812293fc"},"cell_type":"code","source":"target = train_df['target']\ndel train_df['target']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d0c705bad346512ef0488c580c0a92f6595e5366"},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\n\n\nlgb_params = {\"objective\" : \"regression\", \"metric\" : \"rmse\", \n               \"max_depth\": 7, \"min_child_samples\": 20, \n               \"reg_alpha\": 1, \"reg_lambda\": 1,\n               \"num_leaves\" : 64, \"learning_rate\" : 0.001, \n               \"subsample\" : 0.8, \"colsample_bytree\" : 0.8, \n               \"verbosity\": -1}\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=1989)\n\noof_lgb = np.zeros(len(train_df))\npredictions_lgb = np.zeros(len(test_df))\n\nfeatures_lgb = list(train_df.columns)\nfeature_importance_df_lgb = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train_df)):\n    trn_data = lgb.Dataset(train_df.iloc[trn_idx], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train_df.iloc[val_idx], label=target.iloc[val_idx])\n\n    print(\"LGB \" + str(fold_) + \"-\" * 50)\n    num_round = 2000\n    clf = lgb.train(lgb_params, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=500, early_stopping_rounds = 2000)\n    oof_lgb[val_idx] = clf.predict(train_df.iloc[val_idx], num_iteration=clf.best_iteration)\n\n    fold_importance_df_lgb = pd.DataFrame()\n    fold_importance_df_lgb[\"feature\"] = features_lgb\n    fold_importance_df_lgb[\"importance\"] = clf.feature_importance()\n    fold_importance_df_lgb[\"fold\"] = fold_ + 1\n    feature_importance_df_lgb = pd.concat([feature_importance_df_lgb, fold_importance_df_lgb], axis=0)\n    predictions_lgb += clf.predict(test_df, num_iteration=clf.best_iteration) / FOLDs.n_splits\n    \n\nprint(np.sqrt(mean_squared_error(oof_lgb, target)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e34c9dce03b451fe821cea3862ffdf064099221","scrolled":false},"cell_type":"code","source":"import xgboost as xgb\n\nxgb_params = {'eta': 0.001, 'max_depth': 7, 'subsample': 0.8, 'colsample_bytree': 0.8, \n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True}\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=1989)\n\noof_xgb = np.zeros(len(train_df))\npredictions_xgb = np.zeros(len(test_df))\n\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train_df)):\n    trn_data = xgb.DMatrix(data=train_df.iloc[trn_idx], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train_df.iloc[val_idx], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 2000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=100, verbose_eval=500)\n    oof_xgb[val_idx] = xgb_model.predict(xgb.DMatrix(train_df.iloc[val_idx]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb += xgb_model.predict(xgb.DMatrix(test_df), ntree_limit=xgb_model.best_ntree_limit+50) / FOLDs.n_splits\n\nnp.sqrt(mean_squared_error(oof_xgb, target))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9e02dfe21f8bd0ff3d11d772a1b88f2ca3ea7159"},"cell_type":"code","source":"print('lgb', np.sqrt(mean_squared_error(oof_lgb, target)))\nprint('xgb', np.sqrt(mean_squared_error(oof_xgb, target)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"096a3597cfb312875a9c2c2431fad5e0f616f53c"},"cell_type":"code","source":"total_sum = 0.5 * oof_lgb + 0.5 * oof_xgb\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(total_sum, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42821918b162d3f7ac6a1220950b351789cd8059"},"cell_type":"code","source":"cols = (feature_importance_df_lgb[[\"feature\", \"importance\"]]\n        .groupby(\"feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df_lgb.loc[feature_importance_df_lgb.feature.isin(cols)]\n\nplt.figure(figsize=(14,14))\nsns.barplot(x=\"importance\",\n            y=\"feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"603d1fa72b433079c58f6fa041fe6f67e2a486c3"},"cell_type":"code","source":"sub_df = pd.read_csv('../input/sample_submission.csv')\nsub_df[\"target\"] = 0.5 * predictions_lgb + 0.5 * predictions_xgb\nsub_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4a80d0e420b69b4fa8c8ad19a0c169a35c051a4"},"cell_type":"markdown","source":"Now, the accuracy have been increased thanks to boosting methods ;-)."},{"metadata":{"_uuid":"29dbe43f27a833a064ca034009c80b0944c1c396"},"cell_type":"markdown","source":"\n**In progress ...**\n\n**Be in touch to get last commits ...**\n\n**I'll try to complete it as soon as possible**\n\n**Your upvote will be motivation for me for continuing the kernel ;-)**"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}