{"cells":[{"metadata":{"_uuid":"cb7281970e7c44017ff87d82a98828d023f03f45"},"cell_type":"markdown","source":"## Beginner Guide of Elo Merchant Category Recommendation\n---\n### Help understand customer loyalty\n"},{"metadata":{"_uuid":"98e6945de0ae1e7a7227a9f878a6e29919cd4e0d"},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/10445/logos/header.png)\n\n---\n## *About Competition*\n---\n\n<div class=\"competition-overview__content\"><div><div class=\"markdown-converter__text--rendered\"><p><img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/10445/logos/thumb76_76.png?t=2018-10-24-17-14-05\" alt=\"TGS\" width=\"180\" style=\"float: right;\"></p>\n\n<p style = \"text-align:justify;\">Imagine being hungry in an unfamiliar part of town and getting restaurant recommendations served up, based on your personal preferences, at just the right moment. The recommendation comes with an attached discount from your credit card provider for a local place around the corner!</p>\n\n<p style = \"text-align:justify;\">Right now, <a href=\"https://www.cartaoelo.com.br/\" rel=\"nofollow\">Elo</a>, one of the largest payment brands in Brazil, has built partnerships with merchants in order to offer promotions or discounts to cardholders. But do these promotions work for either the consumer or the merchant? Do customers enjoy their experience? Do merchants see repeat business? Personalization is key.\n</p>\n\n<p style = \"text-align:justify;\">Elo has built machine learning models to understand the most important aspects and preferences in their customers’ lifecycle, from food to shopping. But so far none of them is specifically tailored for an individual or profile. This is where you come in.</p>\n\n<p style = \"text-align:justify;\">In this competition, Kagglers will develop algorithms to identify and serve the most relevant opportunities to individuals, by uncovering signal in customer loyalty. Your input will improve customers’ lives and help Elo reduce unwanted campaigns, to create the right experience for customers.</p>\n\n</div></div></div>\n\n---\n## *Objective*\n---\n### Identify and serve the most relevant opportunity to all individual customer by covering their loyalty\n---\n## *Evaluation Metrics*\n---\n\n## Root Mean Squared Error (RMSE)\n\nSubmissions are scored on the root mean squared error. RMSE is defined as:\n![](http://patentimages.storage.googleapis.com/EP2270530A1/imgb0001.png)\nwhere y^ is the predicted loyalty score for each card_id, and y is the actual loyalty score assigned to a `card_id`.\n\n---\n## Data Description\n---\n\n<div class=\"markdown-content-box__converter\"><div class=\"markdown-converter__text--rendered competition-data__content\"><p>Note: All data is simulated and fictitious, and is not real customer data</p>\n\n<h2>What files do I need?</h2>\n\n<p>You will need, at a minimum, the <strong>train.csv</strong> and <strong>test.csv</strong> files.  These contain the <code>card_id</code>s that we'll be using for training and prediction.</p>\n\n<p>The <strong>historical_transactions.csv</strong> and <strong>new_merchant_transactions.csv</strong> files contain information about each card's transactions.  <strong>historical_transactions.csv</strong> contains up to 3 months' worth of transactions for every card at any of the provided <code>merchant_id</code>s.  <strong>new_merchant_transactions.csv</strong> contains the transactions at <em>new</em> merchants (<code>merchant_id</code>s that this particular <code>card_id</code> has not yet visited) over a period of two months.</p>\n\n<p><strong>merchants.csv</strong> contains aggregate information for each <code>merchant_id</code> represented in the data set.</p>\n\n<h2>What should I expect the data format to be?</h2>\n\n<p>The data is formatted as follows:</p>\n\n<p><strong>train.csv</strong> and <strong>test.csv</strong> contain <code>card_id</code>s and information about the card itself - the first month the card was active, etc.  <strong>train.csv</strong> also contains the <code>target</code>.</p>\n\n<p><strong>historical_transactions.csv</strong> and <strong>new_merchant_transactions.csv</strong> are designed to be joined with <strong>train.csv</strong>, <strong>test.csv</strong>, and <strong>merchants.csv</strong>.  They contain information about transactions for each card, as described above.</p>\n\n<p><strong>merchants</strong> can be joined with the transaction sets to provide additional merchant-level information.</p>\n\n<h2>What am I predicting?</h2>\n\n<p>You are predicting a <em>loyalty score</em> for each <code>card_id</code> represented in <strong>test.csv</strong> and <strong>sample_submission.csv</strong>.</p>\n\n<h2>File descriptions</h2>\n\n<ul>\n<li><strong>train.csv</strong> - the training set</li>\n<li><strong>test.csv</strong> - the test set</li>\n<li><strong>sample_submission.csv</strong> - a sample submission file in the correct format - contains all <code>card_id</code>s you are expected to predict for.</li>\n<li><strong>historical_transactions.csv</strong> - up to 3 months' worth of historical transactions for each <code>card_id</code></li>\n<li><strong>merchants.csv</strong> - additional information about all merchants / <code>merchant_id</code>s in the dataset.</li>\n<li><strong>new_merchant_transactions.csv</strong> - two months' worth of data for each <code>card_id</code> containing ALL purchases that <code>card_id</code> made at <code>merchant_id</code>s that were <em>not visited in the historical data</em>.</li>\n</ul>\n\n<h2>Data fields</h2>\n\n<p>Data field descriptions are provided in <strong>Data Dictionary.xlsx</strong>.</p></div></div>\n\n---\n### *Outline of the note book*\n---\n* [**1.Import packages**](#1.Import-packages)\n* [**2.Helping Function**](#2.Helping-Function)\n* [**3.Data Types Count By Dataframe**](#3.Data-Types-Count-By-Dataframe)\n* [**4.Missing Value Count By Dataframe**](#4.Missing-Value-Count-By-Dataframe)\n* [**5. Impute the Missing Value**](#5.-Impute-the-Missing-Value)\n* [**6.Check Distribution of Target variable**](#6.Check-Distribution-of-Target-variable)\n* [**7.Descriptive statistics of Feature Variable**](#7.Descriptive-statistics-of-Feature-Variable)\n* [**8.Basic Feature Engineering**](#8.Basic-Feature-Engineering)\n    * [**8.1 Historical Transaction Feature**](#8.1-Historical-Transaction-Feature)\n    * [**8.2 New Merchant Feature**](8.2-New-Merchant-Feature)\n* [**9.Model Training with kfold**](#9.Model-Training-with-kfold)\n* [**10.Feature Importance**](#10.Feature-Importance)\n* [**11.Ensembling**](#11.Ensembling)\n* [**12.Final Submission**](#12.Final-Submission)\n\n---"},{"metadata":{"_uuid":"f14c8d1fcdc264a818506cd207f4d870796e1121"},"cell_type":"markdown","source":"## 1.Import-packages\n---"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pylab as plt\n# plt.style.use(\"fivethirtyeight\")\nplt.style.use('ggplot')\nimport seaborn as sns\nimport gc\n\nsns.set(style=\"ticks\", color_codes=True)\nimport matplotlib.pyplot as plt\nfrom tqdm._tqdm_notebook import tqdm_notebook as tqdm\ntqdm.pandas()\nimport datetime\n\nimport plotly.offline as ply\nply.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Read in the dataframes\ndef load_data():\n    train = pd.read_csv('../input/train.csv',parse_dates=[\"first_active_month\"])\n    test = pd.read_csv('../input/test.csv',parse_dates=[\"first_active_month\"])\n    merchant = pd.read_csv('../input/merchants.csv')\n    hist_trans = pd.read_csv('../input/historical_transactions.csv')\n    print('train shape', train.shape)\n    print('test shape', train.shape)\n    print('merchants shape', merchant.shape)\n    print('historical_transactions', hist_trans.shape)\n    return (train,test,merchant,hist_trans)\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c36ebc410c1d6b68d62356c84d15ccb89af6abf8"},"cell_type":"markdown","source":"## 2.Helping Function\n---"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f0fb9df73bf3fe6a9e9256ba806b94cddef2c040"},"cell_type":"code","source":"######### Function##################\ndef mis_value_graph(data, name = \"\"):\n    data = [\n    go.Bar(\n        x = data.columns,\n        y = data.isnull().sum(),\n        name = name,\n        textfont=dict(size=20),\n        marker=dict(\n        color= generate_color(),\n        line=dict(\n            color='#000000',\n            width=1,\n        ), opacity = 0.85\n    )\n    ),\n    ]\n    layout= go.Layout(\n        title= 'Total Missing Value of'+ str(name),\n        xaxis= dict(title='Columns', ticklen=5, zeroline=False, gridwidth=2),\n        yaxis=dict(title='Value Count', ticklen=5, gridwidth=2),\n        showlegend=True\n    )\n    fig= go.Figure(data=data, layout=layout)\n    ply.iplot(fig, filename='skin')\n    \ndef datatypes_pie(data, title = \"\"):\n    # Create a trace\n    colors = ['#FEBFB3', '#E1396C', '#96D38C', '#D0F9B1']\n    trace1 = go.Pie(\n        labels = ['float64','Int64'],\n        values = data.dtypes.value_counts(),\n        textfont=dict(size=20),\n        marker=dict(colors=colors,line=dict(color='#000000', width=2)), hole = 0.45)\n    layout = dict(title = \"Data Types Count Percentage of \"+ str(title))\n    data = [trace1]\n    ply.iplot(dict(data=data, layout=layout), filename='basic-line')\n    \n\ndef mis_impute(data):\n    for i in data.columns:\n        if data[i].dtype == \"object\":\n            data[i] = data[i].fillna(\"other\")\n        elif (data[i].dtype == \"int64\" or data[i].dtype == \"float64\"):\n            data[i] = data[i].fillna(data[i].mean())\n        else:\n            pass\n    return data\n\n\nimport random\n\ndef generate_color():\n    color = '#{:02x}{:02x}{:02x}'.format(*map(lambda x: random.randint(0, 255), range(3)))\n    return color","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1341842455acc4eb06f7790d520e35d59b5c6625"},"cell_type":"code","source":"%%time\ntrain,test,merchant, hist_trans = load_data()\ntrain.name,test.name,merchant.name, hist_trans.name = \"train\",\"test\",\"merchant\", \"hist_trans\"\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d22054396bd0b19ceb141c3bf51d4a4193c56d6"},"cell_type":"markdown","source":"## 3.Data Types Count By Dataframe\n---"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"10d8bfd8fb14b128fa8699e85da852504d48bba0"},"cell_type":"code","source":"%%time\nfor i in [train,test, merchant, hist_trans]:\n    print(\"Data Types Cont of \",i.name)\n    display(i.dtypes.value_counts())\n    datatypes_pie(i, title = i.name)\n    \ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36376dd76db38eeaa2525539f6b3a03e1bf76cd1"},"cell_type":"markdown","source":"## 4.Missing Value Count By Dataframe\n---"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"599db695267e0473c5b5302707910ce5c2d34d1a"},"cell_type":"code","source":"%%time\nfor i in [train,test,merchant, hist_trans]:\n    print(\"Missing Value Count of \",i.name)\n    mis_value_graph(i, name = i.name)\n    \ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68ba656517ac637404ad6f29063c8f56dcf12d22"},"cell_type":"markdown","source":"## 5. Impute the Missing Value\n---"},{"metadata":{"trusted":true,"_uuid":"c5bea8df5a69830745f934a18bed541c2336aa7c","_kg_hide-input":true},"cell_type":"code","source":"# %%time\n# for i in [train,test,merchant, hist_trans]:\n#     print(\"Impute the Missing value of \", i.name)\n#     mis_impute(i)\n#     print(\"Done Imputation on\", i.name)\n\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af2da81879364aa41d802f0535650d2d92dc0198"},"cell_type":"markdown","source":"## 6.Check Distribution of Target variable\n---"},{"metadata":{"trusted":true,"_uuid":"fe6c2ec45f93a6c6d782f527b35e3584585d48f7"},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ddf6f29534f7e6c8d9b95b1922a133b1e367ca2a"},"cell_type":"code","source":"# train = train[train['target'] > -33]\n# train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42ddfc7962031a8070c2a16cc2fab4cd91fc0d03","scrolled":false,"_kg_hide-input":true},"cell_type":"code","source":"%%time\nx = train.target\ndata = [go.Histogram(x=x,\n                     histnorm='probability')]\nlayout = go.Layout(\n    title='Target Distribution',\n    xaxis=dict(title='Value'),yaxis=dict(title='Count'),\n    bargap=0.2,\n    bargroupgap=0.1\n)\nfig = go.Figure(data=data, layout=layout)\nply.iplot(fig, filename='normalized histogram')\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89a7376c8d91e45d1a4ac9241cb589f66f77f900"},"cell_type":"markdown","source":"## 7.Datacount By Date in Probability\n---"},{"metadata":{"trusted":true,"_uuid":"a00a807e6b0f1deeae462806307fced1e9317edd","_kg_hide-input":true},"cell_type":"code","source":"x = train['first_active_month'].dt.date.value_counts()\nx = x.sort_index()\ndata0 = [go.Histogram(x=x.index,y = x.values,histnorm='probability', marker=dict(color=generate_color()))]\nlayout = go.Layout(\n    title='First active month count Train Data',\n    xaxis=dict(title='First active month',ticklen=5, zeroline=False, gridwidth=2),\n    yaxis=dict(title='Number of cards',ticklen=5, gridwidth=2),\n    bargap=0.1,\n    bargroupgap=0.2\n)\nfig = go.Figure(data=data0, layout=layout)\nply.iplot(fig, filename='normalized histogram')\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56b35bb4e90244b39ee1d97432b459997626498e"},"cell_type":"markdown","source":"## 8.Basic Feature Engineering\n---"},{"metadata":{"trusted":true,"_uuid":"1bba53f6f332d43261357982aedf3144713f3141"},"cell_type":"code","source":"%%time\n##---------------Time based Feature\ntrain['day']= train['first_active_month'].dt.day \ntrain['dayofweek']= train['first_active_month'].dt.dayofweek\ntrain['dayofyear']= train['first_active_month'].dt.dayofyear\ntrain['days_in_month']= train['first_active_month'].dt.days_in_month\ntrain['daysinmonth']= train['first_active_month'].dt.daysinmonth \ntrain['month']= train['first_active_month'].dt.month\ntrain['week']= train['first_active_month'].dt.week \ntrain['weekday']= train['first_active_month'].dt.weekday\ntrain['weekofyear']= train['first_active_month'].dt.weekofyear\ntrain['year']= train['first_active_month'].dt.year\ntrain['elapsed_time'] = (datetime.date(2018, 2, 1) - train['first_active_month'].dt.date).dt.days\n\n      \n##---------------Time based Test Feature      \ntest['day']= test['first_active_month'].dt.day \ntest['dayofweek']= test['first_active_month'].dt.dayofweek\ntest['dayofyear']= test['first_active_month'].dt.dayofyear\ntest['days_in_month']= test['first_active_month'].dt.days_in_month\ntest['daysinmonth']= test['first_active_month'].dt.daysinmonth \ntest['month']= test['first_active_month'].dt.month\ntest['week']= test['first_active_month'].dt.week \ntest['weekday']= test['first_active_month'].dt.weekday\ntest['weekofyear']= test['first_active_month'].dt.weekofyear\ntest['year']= test['first_active_month'].dt.year\ntest['elapsed_time'] = (datetime.date(2018, 2, 1) - test['first_active_month'].dt.date).dt.days\n\n\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ngccollect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f22c324790b6028d994f75c5c3a983f69299e5ec"},"cell_type":"code","source":"%%time\n#-----------------One-hot encode features\nfeat1 = pd.get_dummies(train['feature_1'], prefix='f1_')\nfeat2 = pd.get_dummies(train['feature_2'], prefix='f2_')\nfeat3 = pd.get_dummies(train['feature_3'], prefix='f3_')\nfeat4 = pd.get_dummies(test['feature_1'], prefix='f1_')\nfeat5 = pd.get_dummies(test['feature_2'], prefix='f2_')\nfeat6 = pd.get_dummies(test['feature_3'], prefix='f3_')\n\n##---------------Numerical representation of the first active month\ntrain = pd.concat([train,feat1, feat2, feat3], axis=1, sort=False)\ntest = pd.concat([test,feat4, feat5, feat6], axis=1, sort=False)\n\n#shape of data\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8153c1b7b1a14ac5434a353c4e255707b1a108db"},"cell_type":"markdown","source":"#### 8.1 Historical Transaction Feature"},{"metadata":{"trusted":true,"_uuid":"3ac8d5122dd2532bd6526efac2e68e0362982b41"},"cell_type":"code","source":"%%time\nhist_trans = pd.get_dummies(hist_trans, columns=['category_2', 'category_3'])\nhist_trans['authorized_flag'] = hist_trans['authorized_flag'].map({'Y': 1, 'N': 0})\nhist_trans['category_1'] = hist_trans['category_1'].map({'Y': 1, 'N': 0})\nhist_trans.head()\n#shape of data\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cfe7305eca4b57f643e6d4474112f6ca708570da"},"cell_type":"code","source":"%%time\ndef aggregate_transactions(trans, prefix):  \n    trans.loc[:, 'purchase_date'] = pd.DatetimeIndex(trans['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'category_1': ['mean'],\n        'category_2_1.0': ['mean'],\n        'category_2_2.0': ['mean'],\n        'category_2_3.0': ['mean'],\n        'category_2_4.0': ['mean'],\n        'category_2_5.0': ['mean'],\n        'category_3_A': ['mean'],\n        'category_3_B': ['mean'],\n        'category_3_C': ['mean'],\n        'merchant_id': ['nunique'],\n        'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n        'installments': ['sum', 'mean', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n    }\n    agg_trans = trans.groupby(['card_id']).agg(agg_func)\n    agg_trans.columns = [prefix + '_'.join(col).strip() \n                           for col in agg_trans.columns.values]\n    agg_trans.reset_index(inplace=True)\n    \n    df = (trans.groupby('card_id')\n          .size()\n          .reset_index(name='{}transactions_count'.format(prefix)))\n    \n    agg_trans = pd.merge(df, agg_trans, on='card_id', how='left')\n    \n    return agg_trans\n#shape of data\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"954a6321108e59e3b90660f39f45f7af43ce9dc2"},"cell_type":"code","source":"%%time\nmerch_hist = aggregate_transactions(hist_trans, prefix='hist_')\ntrain = pd.merge(train, merch_hist, on='card_id',how='left')\ntest = pd.merge(test, merch_hist, on='card_id',how='left')\n#shape of data\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f1263232c515b158ae3df783d4bfff4245414c8","_kg_hide-input":true},"cell_type":"code","source":"train.head()\ntrace0 = go.Box(y=train.feature_1,name=\"feature_1\", marker=dict(color=generate_color()))\ntrace1 = go.Box(y=train.feature_2,name=\"feature_2\", marker=dict(color=generate_color()))\ntrace2 = go.Box(y=train.feature_3,name=\"feature_3\", marker=dict(color=generate_color()))\ndata = [trace0, trace1, trace2]\nlayout = go.Layout(\n    title='Feature Boxplot Train',\n    xaxis=dict(title='Value'),yaxis=dict(title='Count'),\n    bargap=0.2,\n    bargroupgap=0.1\n)\nfig = go.Figure(data=data, layout=layout)\nply.iplot(fig)\n\ntest.head()\ntrace0 = go.Box(y=test.feature_1,name=\"feature_1\", marker=dict(color=generate_color()))\ntrace1 = go.Box(y=test.feature_2,name=\"feature_2\", marker=dict(color=generate_color()))\ntrace2 = go.Box(y=test.feature_3,name=\"feature_3\", marker=dict(color=generate_color()))\ndata = [trace0, trace1, trace2]\nlayout = go.Layout(\n    title='Feature Boxplot Test',\n    xaxis=dict(title='Value'),yaxis=dict(title='Count'),\n    bargap=0.2,\n    bargroupgap=0.1\n)\nfig = go.Figure(data=data, layout=layout)\nply.iplot(fig)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f28ae032ada30711e7a011d7cdf831ff845d25b3"},"cell_type":"markdown","source":"#### 8.2 New Merchant Feature"},{"metadata":{"trusted":true,"_uuid":"fec287b78b3ee17e4d8afd99c8f6f56962c1ea07","_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"%%time\nnew_trans_df = pd.read_csv(\"../input/new_merchant_transactions.csv\")\ndisplay(new_trans_df.head())\nnew_trans_df.hist(figsize = (17,12))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"daac03a4392ef5e39939abd2bc879f975c34301b"},"cell_type":"code","source":"%%time\nnew_trans_df = pd.get_dummies(new_trans_df, columns=['category_2', 'category_3'])\nnew_trans_df['authorized_flag'] = new_trans_df['authorized_flag'].map({'Y': 1, 'N': 0})\nnew_trans_df['category_1'] = new_trans_df['category_1'].map({'Y': 1, 'N': 0})\nnew_trans_df.head()\n#shape of data\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c71be90321f18e4c9521ec0a0f0cd9631a61f1fb"},"cell_type":"code","source":"%%time\nmerch_hist = aggregate_transactions(hist_trans, prefix='hist_')\ntrain = pd.merge(train, merch_hist, on='card_id',how='left')\ntest = pd.merge(test, merch_hist, on='card_id',how='left')\n#shape of data\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b57604e49e8951be281440bffbc0bbe62233eca1"},"cell_type":"code","source":"target = train['target']\ndrops = ['card_id', 'first_active_month', 'target', 'date']\nuse_cols = [c for c in train.columns if c not in drops]\nfeatures = list(train[use_cols].columns)\ntrain[features].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d749b857aa93d443345f973ebf13622ef1cbcf1f"},"cell_type":"code","source":"%%time\nprint('train shape', train.shape)\nprint('test shape', test.shape)\ntrain_df = train.copy()\ntest_df = test.copy()\n\nprint('train shape', train_df.shape)\nprint('test shape', test_df.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d3bfd97203c6d4cda0cedcb85dc9f85507b7df05","_kg_hide-input":true},"cell_type":"code","source":"# train.dtypes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"86870a25eb66e1544fb3d18829243992cb4eef30"},"cell_type":"markdown","source":"## 9.Model Training with kfold\n---"},{"metadata":{"trusted":true,"_uuid":"abaea8d16136affc70c09186696a192c6307909e"},"cell_type":"code","source":"correlation = train_df.corr()\nplt.figure(figsize=(20,15))\n# mask = np.zeros_like(correlation)\n# mask[np.triu_indices_from(mask)] = True\nsns.heatmap(correlation, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b727718c9b3365b76467d3fd01582c90763468c","_kg_hide-output":true},"cell_type":"code","source":"%%time\n# train_df.isnull().sum()\ntrain_X = train_df[features]\ntest_X = test_df[features]\ntrain_y = target\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c237ab45cc8e88f8ddf41f07b1684bbe2945d79"},"cell_type":"code","source":"%%time\nprint(\"X_train : \",train_X.shape)\nprint(\"X_test : \",test_X.shape)\nprint(\"Y_train : \",train_y.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"712d24038f49e04e603fb85073f3cdf31fad2218"},"cell_type":"code","source":"from sklearn.datasets import load_boston\nfrom sklearn.model_selection import (cross_val_score, train_test_split, \n                                     GridSearchCV, RandomizedSearchCV)\nfrom sklearn.metrics import r2_score\n \nfrom lightgbm.sklearn import LGBMRegressor\n\nhyper_space = {'n_estimators': [1000, 1500, 2000, 2500],\n               'max_depth':  [4, 5, 8, -1],\n               'num_leaves': [15, 31, 63, 127],\n               'subsample': [0.6, 0.7, 0.8, 1.0],\n               'colsample_bytree': [0.6, 0.7, 0.8, 1.0],\n               'learning_rate' : [0.01,0.02,0.03]\n              }\n\nest = lgb.LGBMRegressor(n_jobs=-1, random_state=2018)\ngs = GridSearchCV(est, hyper_space, scoring='r2', cv=4, verbose=1)\ngs_results = gs.fit(train_X, train_y)\nprint(\"BEST PARAMETERS: \" + str(gs_results.best_params_))\nprint(\"BEST CV SCORE: \" + str(gs_results.best_score_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e5e27dd725e87a726d9b2356a733a99d559df42"},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\n\n\nlgb_params = {\"objective\" : \"regression\", \"metric\" : \"rmse\", \n               \"max_depth\": 7, \"min_child_samples\": 20, \n               \"reg_alpha\": 1, \"reg_lambda\": 1,\n               \"num_leaves\" : 64, \"learning_rate\" : 0.01, \n               \"subsample\" : 0.8, \"colsample_bytree\" : 0.8, \n               \"verbosity\": -1}\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=42)\n\noof_lgb = np.zeros(len(train_X))\npredictions_lgb = np.zeros(len(test_X))\n\nfeatures_lgb = list(train_X.columns)\nfeature_importance_df_lgb = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train_X)):\n    trn_data = lgb.Dataset(train_X.iloc[trn_idx], label=train_y.iloc[trn_idx])\n    val_data = lgb.Dataset(train_X.iloc[val_idx], label=train_y.iloc[val_idx])\n\n    print(\"-\" * 20 +\"LGB Fold:\"+str(fold_)+ \"-\" * 20)\n    num_round = 10000\n    clf = lgb.train(lgb_params, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=1000, early_stopping_rounds = 50)\n    oof_lgb[val_idx] = clf.predict(train_X.iloc[val_idx], num_iteration=clf.best_iteration)\n\n    fold_importance_df_lgb = pd.DataFrame()\n    fold_importance_df_lgb[\"feature\"] = features_lgb\n    fold_importance_df_lgb[\"importance\"] = clf.feature_importance()\n    fold_importance_df_lgb[\"fold\"] = fold_ + 1\n    feature_importance_df_lgb = pd.concat([feature_importance_df_lgb, fold_importance_df_lgb], axis=0)\n    predictions_lgb += clf.predict(test_X, num_iteration=clf.best_iteration) / FOLDs.n_splits\n    \n\nprint(\"Best RMSE: \",np.sqrt(mean_squared_error(oof_lgb, train_y)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a8ce6af961aae9d4d19a13a944ace257c330a8b5"},"cell_type":"code","source":"from sklearn import model_selection, preprocessing, metrics\nimport lightgbm as lgb\n\ndef run_lgb(train_X, train_y, val_X, val_y, test_X):\n#     params = {\n#         \"objective\" : \"regression\",\n#         \"metric\" : \"rmse\",\n#         \"num_leaves\" : 128,\n#         'max_depth' : 7,\n#         \"min_child_weight\" : 20,\n#         \"learning_rate\" : 0.001,\n#         \"reg_alpha\": 1, \"reg_lambda\": 1,\n#         \"learning_rate\" : 0.01,\n#         \"subsample\" : 0.8, \"colsample_bytree\" : 0.8, \n#         \"verbose\": 1\n#     }\n    params={'learning_rate': 0.01,\n            'objective':'regression',\n            'metric':'rmse',\n            'num_leaves': 31,\n            'verbose': 1,\n            'bagging_fraction': 0.9,\n            'feature_fraction': 0.9,\n            \"random_state\":1,\n#             'max_depth': 5,\n#             \"bagging_seed\" : 42,\n#             \"verbosity\" : -1,\n#             \"bagging_frequency\" : 5,\n#             'lambda_l2': 0.5,\n#             'lambda_l1': 0.5,\n#             'min_child_samples': 36\n           }\n\n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    evals_result = {}\n    model = lgb.train(params, lgtrain, 10000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=100, evals_result=evals_result)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    return pred_test_y, model, evals_result\n \n\npred_test = 0\nkf = model_selection.KFold(n_splits=5, random_state=42, shuffle=True)\nfor fold_, (dev_index, val_index) in enumerate(kf.split(train_df, train_y)):\n    dev_X, val_X = train_X.loc[dev_index,:], train_X.loc[val_index,:]\n    dev_y, val_y = train_y[dev_index], train_y[val_index]\n    print(\"-\" * 20 +\"LGB Fold:\"+str(fold_)+ \"-\" * 20)    \n    pred_test_tmp, model, evals_result = run_lgb(dev_X, dev_y, val_X, val_y, test_X)\n    pred_test += model.predict(test_X, num_iteration=clf.best_iteration)\npred_test /= 5","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"29133f353bbef488b88871762486469f139bf2b2"},"cell_type":"markdown","source":"## 10.Feature Importance\n---"},{"metadata":{"trusted":true,"_uuid":"716ba7fa7b63cffe7bd7893510c1125aa07afa71","_kg_hide-input":true},"cell_type":"code","source":"###--------LightGBM1 feature Importance--------------\nprint(\"Feature Importance For LGB Model1\")\ncols = (feature_importance_df_lgb[[\"feature\", \"importance\"]]\n        .groupby(\"feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df_lgb.loc[feature_importance_df_lgb.feature.isin(cols)]\n\nplt.figure(figsize=(14,14))\nsns.barplot(x=\"importance\",\n            y=\"feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')\n\n###--------LightGBM2 feature Importance--------------\n\nprint(\"Feature Importance For LGB Model2\")\nfig, ax = plt.subplots(figsize=(20,10))\nlgb.plot_importance(model, max_num_features=50, height=0.9, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=25)\nplt.show()\nplt.savefig('lgbm_importances1.png')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b9fa180ccac5e345df29daa53c6a74c084c8eb64"},"cell_type":"markdown","source":"## 11.Ensembling\n---"},{"metadata":{"trusted":true,"_uuid":"9ec8ed24c53ff9fc9418e2a562c4d6feaaf51a26"},"cell_type":"code","source":"temp_df = pd.DataFrame()\ntemp_df[\"target1\"] = predictions_lgb\ntemp_df[\"target2\"] = pred_test\ntemp_df[\"target3\"] = temp_df[\"target1\"] * 0.5 + temp_df[\"target2\"] * 0.5","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c46450151829167da3036e5aef8e4d975161e9b"},"cell_type":"markdown","source":"## 12.Final Submission"},{"metadata":{"_uuid":"34cf66d2f1fde5e4885c03ef3dd9978f4a967a35"},"cell_type":"markdown","source":"#### 1. Blending Result"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f2b3176652c2d3623faa6c28191fc770b5506c34"},"cell_type":"code","source":"sub = pd.read_csv(\"../input/sample_submission.csv\")\nsub['target'] = temp_df[\"target3\"]\nsub.to_csv(\"ELO_LGB_Blend.csv\", index=False)\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c105bd72db5cb3e7e325ee6145318269e03f12f"},"cell_type":"code","source":"sub = pd.read_csv(\"../input/sample_submission.csv\")\nsub['target'] = temp_df[\"target3\"] * 0.6 + sub['target'] * 0.4\n# sub['target'] = sub['target'].apply(lambda x : 0 if x < 0 else x)\nsub.to_csv(\"ELO_LGB_Sample.csv\", index=False)\nsub.head()\n# sub[sub['target'] == 0].count()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bac17bd8632546fc8cb49e26e8b633e4acc5693e"},"cell_type":"markdown","source":"#### 2. LightGBM 1"},{"metadata":{"trusted":true,"_uuid":"9f054ba53ff1457615e9c622ca89dc277488f1ea"},"cell_type":"code","source":"sub = pd.read_csv(\"../input/sample_submission.csv\")\nsub['target'] = temp_df[\"target1\"] \n# sub['target'] = sub['target'].apply(lambda x : 0 if x < 0 else x)\nsub.to_csv(\"ELO_LGB1.csv\", index=False)\nsub.head()\n# sub[sub['target'] == 0].count()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8633599fe95b56eff19c3694047686e80f3616d8"},"cell_type":"markdown","source":"#### 3. LightGBM 2"},{"metadata":{"trusted":true,"_uuid":"32af7d2e9c43930a4b229ca4d8d9f3e7d9657c92"},"cell_type":"code","source":"sub = pd.read_csv(\"../input/sample_submission.csv\")\nsub['target'] = temp_df[\"target2\"] \n# sub['target'] = sub['target'].apply(lambda x : 0 if x < 0 else x)\nsub.to_csv(\"ELO_LGB2.csv\", index=False)\nsub.head()\n# sub[sub['target'] == 0].count()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}