{"cells":[{"metadata":{"_uuid":"26b420cf6fe13c2db4fba62ed5c9c9a7903614d8"},"cell_type":"markdown","source":"Data Description : Imagine being hungry in an unfamiliar part of town and getting restaurant recommendations served up, based on your personal preferences, at just the right moment. The recommendation comes with an attached discount from your credit card provider for a local place around the corner!\n\nRight now, Elo, one of the largest payment brands in Brazil, has built partnerships with merchants in order to offer promotions or discounts to cardholders. But do these promotions work for either the consumer or the merchant? Do customers enjoy their experience? Do merchants see repeat business? Personalization is key.\n\nElo has built machine learning models to understand the most important aspects and preferences in their customers’ lifecycle, from food to shopping. But so far none of them is specifically tailored for an individual or profile. This is where you come in.\n\nIn this competition, Kagglers will develop algorithms to identify and serve the most relevant opportunities to individuals, by uncovering signal in customer loyalty. Your input will improve customers’ lives and help Elo reduce unwanted campaigns, to create the right experience for customers."},{"metadata":{"_uuid":"2aac67ce394801d03beb0bc469f2608fbde1a7dd"},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/10445/logos/thumb76_76.png?t=2018-10-24-17-14-05)\n\nReference \n- https://www.kaggle.com/youhanlee/hello-elo-ensemble-will-help-you\n- https://www.kaggle.com/ashishpatel26/beginner-guide-of-elo-eda-kfold-lightgbm\n- https://www.kaggle.com/kailex/tidy-elo-starter-3-813\n- https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-elo"},{"metadata":{"_uuid":"27147b15332ddffc145a9c9e1857dca7b668db7b"},"cell_type":"markdown","source":"### Simple Exploration"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Loading packages\nimport pandas as pd #Analysis \nimport matplotlib.pyplot as plt #Visulization\nimport seaborn as sns #Visulization\nimport numpy as np #Analysis \nfrom scipy.stats import norm #Analysis \nfrom sklearn.preprocessing import StandardScaler #Analysis \nfrom scipy import stats #Analysis \nimport warnings \nwarnings.filterwarnings('ignore')\n%matplotlib inline\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"647c5a50c48a8fc2a2c44f53356bc6d7419f303d"},"cell_type":"markdown","source":"- train.csv - the training set\n- test.csv - the test set\n- sample_submission.csv - a sample submission file in the correct format - contains all card_ids you are expected to predict for.\n- historical_transactions.csv - up to 3 months' worth of historical transactions for each card_id\n- merchants.csv - additional information about all merchants / merchant_ids in the dataset.\n- new_merchant_transactions.csv - two months' worth of data for each card_id containing ALL purchases that card_id made at merchant_ids that were not visited in the historical data."},{"metadata":{"trusted":true,"_uuid":"2f3e05a20db0fd849a870e5712af2d69b4d7fafe"},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\", parse_dates=[\"first_active_month\"])\nprint(\"shape of train : \",train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ac6d0d218fca07658d220db99ed198d2557e790"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f1c8c454b4b391ca2a4251e26cfda4e6f10f8a2"},"cell_type":"markdown","source":"- first_active_month : ''YYYY-MM', month of first purchase\n- feature_1,2,3 : Anonymized card categorical feature\n- target : Loyalty numerical score calculated 2 months after historical and evaluation period"},{"metadata":{"trusted":true,"_uuid":"fe102452329416e9902cffb336e9a04a307a3f5d"},"cell_type":"code","source":"test = pd.read_csv(\"../input/test.csv\", parse_dates=[\"first_active_month\"])\nprint(\"shape of test : \",test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d11d126922f8379fb7dfba876ab51e1a2fdd5e6f"},"cell_type":"code","source":"# Code in https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-elo\n# SRK - Simple Exploration Notebook \n\ncnt_srs = train['first_active_month'].dt.date.value_counts()\ncnt_srs = cnt_srs.sort_index()\nplt.figure(figsize=(14,6))\nsns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8, color='green')\nplt.xticks(rotation='vertical')\nplt.xlabel('First active month', fontsize=12)\nplt.ylabel('Number of cards', fontsize=12)\nplt.title(\"First active month count in train set\")\nplt.show()\n\ncnt_srs = test['first_active_month'].dt.date.value_counts()\ncnt_srs = cnt_srs.sort_index()\nplt.figure(figsize=(14,6))\nsns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8, color='green')\nplt.xticks(rotation='vertical')\nplt.xlabel('First active month', fontsize=12)\nplt.ylabel('Number of cards', fontsize=12)\nplt.title(\"First active month count in test set\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea0872d482f2db9b119a9417effe99610ac8830a"},"cell_type":"markdown","source":"First_active_month of Train and Test looks similiar"},{"metadata":{"trusted":true,"_uuid":"21405d328ec46627af90da11963d7b7dd31d07bc"},"cell_type":"code","source":"train.corr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c2ea0ea4f05539fcb77212d42b00ab3017ab783"},"cell_type":"markdown","source":"`feature_1 - feature_3` has 0.58 but `target - feature` low correaltion value"},{"metadata":{"trusted":true,"_uuid":"64fbeef117751887d13f5c6d3f7d38bcdab4963b"},"cell_type":"code","source":"#histogram\nf, ax = plt.subplots(figsize=(14, 6))\nsns.distplot(train['target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d64d4dd69abf506725556c7373dd23a020e16894"},"cell_type":"code","source":"train['target'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18ddcad378f3ea7178c6370130e281c364e7002b"},"cell_type":"code","source":"fig, ((ax1, ax2)) = plt.subplots(nrows=1, ncols=2, figsize=(14,8))\n\ndata = pd.concat([train['target'], train['feature_1']], axis=1)\nfig = sns.boxplot(x='feature_1', y=\"target\", data=data,ax=ax1)\n\n# feature 1\nsns.violinplot(x=\"feature_1\", y=\"target\", data=data,ax=ax2)\nplt.xticks(rotation='vertical')\nplt.xlabel('Feature 1', fontsize=12)\nplt.ylabel('target', fontsize=12)\nplt.title(\"Feature 1 distribution\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02f4034f88b893467e90bf07a223ee2cbcc31201"},"cell_type":"code","source":"fig, ((ax1, ax2)) = plt.subplots(nrows=1, ncols=2, figsize=(14,8))\n\ndata = pd.concat([train['target'], train['feature_2']], axis=1)\nfig = sns.boxplot(x='feature_2', y=\"target\", data=data,ax=ax1)\n\n# feature 1\nsns.violinplot(x=\"feature_2\", y=\"target\", data=data,ax=ax2)\nplt.xticks(rotation='vertical')\nplt.xlabel('Feature 2', fontsize=12)\nplt.ylabel('target', fontsize=12)\nplt.title(\"Feature 2 distribution\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"925ae8b126513c7fc9fd05c68813ea70e4bde8da"},"cell_type":"code","source":"fig, ((ax1, ax2)) = plt.subplots(nrows=1, ncols=2, figsize=(14,8))\n\ndata = pd.concat([train['target'], train['feature_3']], axis=1)\nfig = sns.boxplot(x='feature_3', y=\"target\", data=data,ax=ax1)\n\n# feature 1\nsns.violinplot(x=\"feature_3\", y=\"target\", data=data,ax=ax2)\nplt.xticks(rotation='vertical')\nplt.xlabel('feature_3', fontsize=12)\nplt.ylabel('target', fontsize=12)\nplt.title(\"feature_3 distribution\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67060443cd83eeb522326836339d512362178c32"},"cell_type":"code","source":"fig, ((ax1, ax2)) = plt.subplots(nrows=1, ncols=2, figsize=(14,8))\n\nsns.boxplot(x=\"feature_1\", y=\"target\", hue=\"feature_3\",\n               data=train, palette=\"Set3\",ax=ax1)\n\nsns.boxplot(x=\"feature_3\", y=\"target\", hue=\"feature_1\",\n               data=train, palette=\"Set3\",ax=ax2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5dcb3806cbf073a20918dd8965071dd515054310"},"cell_type":"markdown","source":"feature_3 has 1 when feautre_1 high than 3"},{"metadata":{"trusted":true,"_uuid":"14e7990f95a49f9117a09d298f9958b8a050a24e"},"cell_type":"code","source":"fig, ((ax1, ax2)) = plt.subplots(nrows=1, ncols=2, figsize=(14,8))\n\nsns.boxplot(x=\"feature_1\", y=\"target\", hue=\"feature_2\",\n               data=train, palette=\"Set3\",ax=ax1)\n\nsns.boxplot(x=\"feature_2\", y=\"target\", hue=\"feature_1\",\n               data=train, palette=\"Set3\",ax=ax2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"665f42f65c363e593f04cee4b5ef6d4ab92a7e3b"},"cell_type":"markdown","source":"feature_2 has not 3 when feature_1 == 5\nbut what is target low than -30???"},{"metadata":{"trusted":true,"_uuid":"1ee03daa34f4ba317c229a6554a5bf34cb620339"},"cell_type":"code","source":"# Code in https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-elo\n# SRK - Simple Exploration Notebook \ntrain_low_m30 = train[train['target']<-30]\ncnt_srs = train_low_m30['first_active_month'].dt.date.value_counts()\ncnt_srs = cnt_srs.sort_index()\nplt.figure(figsize=(14,6))\nsns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8, color='green')\nplt.xticks(rotation='vertical')\nplt.xlabel('First active month', fontsize=12)\nplt.ylabel('Number of cards', fontsize=12)\nplt.title(\"First active month count in target low than -30\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"466cc95fa0c9882b95853fd2b5f0d7d916af08e8"},"cell_type":"code","source":"train_low_m30.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1affc2621e0ce520718471541fce720693be6e02"},"cell_type":"markdown","source":"### Missing value"},{"metadata":{"trusted":true,"_uuid":"61c9063db006d0023706da2fd6a68bcbea3f88c5"},"cell_type":"code","source":"# checking missing data\ntotal = train.isnull().sum().sort_values(ascending = False)\npercent = (train.isnull().sum()/train.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afb3ee6f143762755fad62310cb6eccd3558bec6"},"cell_type":"code","source":"# checking missing data\ntotal = test.isnull().sum().sort_values(ascending = False)\npercent = (test.isnull().sum()/test.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"001883df0e6d41154a96cbc146b95966726d92e8"},"cell_type":"code","source":"import datetime\n\nfor df in [train,test]:\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['year'] = df['first_active_month'].dt.year\n    df['month'] = df['first_active_month'].dt.month\n    df['elapsed_time'] = (datetime.date(2018, 2, 1) - df['first_active_month'].dt.date).dt.days\n\ntarget = train['target']\ndel train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b1dc9c508229c4ca8ec0bc7cb49e7f3c6e46c247"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f16f59b07c85cdcd397c28ebf01f9b4cca5ec0c"},"cell_type":"markdown","source":"### Simple Exploration : historical_transactions"},{"metadata":{"trusted":true,"_uuid":"6d123379e6664ef9cdd31bdf8f67a148ba1fe97b"},"cell_type":"code","source":"ht = pd.read_csv(\"../input/historical_transactions.csv\")\nprint(\"shape of historical_transactions : \",ht.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21161a1f1031e00d7fd76e7572e7b25e9e1ff316"},"cell_type":"code","source":"ht.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aaccb032fb6dbc53e49250e3dd957998a8b92907"},"cell_type":"markdown","source":"- card_id\t: Card identifier\n- month_lag\t: month lag to reference date\n- purchase_date\t: Purchase date\n- authorized_flag\t: Y' if approved, 'N' if denied\n- category_3\t: anonymized category\n- installments\t: number of installments of purchase\n- category_1 : \tanonymized category\n- merchant_category_id\t: Merchant category identifier (anonymized )\n- subsector_id\t: Merchant category group identifier (anonymized )\n- merchant_id\t: Merchant identifier (anonymized)\n- purchase_amount\t: Normalized purchase amount\n- city_id\t: City identifier (anonymized )\n- state_id\t: State identifier (anonymized )\n- category_2 : anonymized category"},{"metadata":{"trusted":true,"_uuid":"fe0841fbf9ebc067b90eab239274c41254326468"},"cell_type":"code","source":"temp = ht[\"authorized_flag\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('authorized_flag - Y or N')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"919d00fb45608ff8e81cb2a900211ed38c00430f"},"cell_type":"code","source":"temp = ht[\"category_1\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('category_1 - Y or N')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d7dc6aff52dd158901a8c731308783abf21f7da"},"cell_type":"code","source":"temp = ht[\"category_3\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('category_3 - A B C')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1952d9a7c4ca289ffe7a54ffd5cb82af323dd7c6"},"cell_type":"code","source":"temp = ht[\"category_2\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('category_2 - 1,2,3,4,5')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c03c5558fb327bd0bd5578f1a30fa45fd40d65d"},"cell_type":"code","source":"#histogram\nf, ax = plt.subplots(figsize=(14, 6))\nsns.distplot(ht['month_lag'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a1aa040ea9312646f4217fac59bf4508c1f29ef5"},"cell_type":"code","source":"#histogram\nf, ax = plt.subplots(figsize=(14, 6))\nsns.distplot(ht['purchase_amount'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9b4ccbdfc590fb06ebf76ad4b75f2e821279c65c"},"cell_type":"code","source":"#histogram\nf, ax = plt.subplots(figsize=(14, 6))\nsns.distplot(ht['installments'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b695bf1693d7abb7dc3161e497318fbd896e940b"},"cell_type":"code","source":"temp = ht[\"city_id\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (18,8))\nplt.title('city_id')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2d468a8c461d18cfc8bb2d36308f0c4f453f3e4e"},"cell_type":"code","source":"ht['authorized_flag'] = ht['authorized_flag'].map({'Y':1, 'N':0})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"45dacfcc7a14e680520f4637847534b569b521be"},"cell_type":"code","source":"def aggregate_historical_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n        }\n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['hist_' + '_'.join(col).strip() \n                           for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='hist_transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history\n\nhistory = aggregate_historical_transactions(ht)\ndel ht\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd2e038c44e568ce88a3dbbe2cab6fad03922527"},"cell_type":"code","source":"train = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f741be784db9431bdc920153e24912c879bf690"},"cell_type":"markdown","source":"### Simple Exploration : merchants.csv"},{"metadata":{"trusted":true,"_uuid":"aeb29c9260afa31aad9c62ecfb3c262112267a71"},"cell_type":"code","source":"merchant = pd.read_csv(\"../input/merchants.csv\")\nprint(\"shape of merchant : \",merchant.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26fdb80ab34d243c96b8146da9e523a8e2d92383"},"cell_type":"code","source":"merchant.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f247ab68d5427c7a5a3d598273a68d79b87ac7a"},"cell_type":"markdown","source":"- merchant_id : Unique merchant identifier\n- merchant_group_id\t: Merchant group (anonymized )\n- merchant_category_id\t: Unique identifier for merchant category (anonymized )\n- subsector_id\t: Merchant category group (anonymized )\n- numerical_1\t: anonymized measure\n- numerical_2\t: anonymized measure\n- category_1\t: anonymized category\n- most_recent_sales_range\t: Range of revenue (monetary units) in last active month --> A > B > C > D > E\n- most_recent_purchases_range\t: Range of quantity of transactions in last active month --> A > B > C > D > E\n- avg_sales_lag3\t: Monthly average of revenue in last 3 months divided by revenue in last active month\n- avg_purchases_lag3\t: Monthly average of transactions in last 3 months divided by transactions in last active month\n- active_months_lag3\t: Quantity of active months within last 3 months\n- avg_sales_lag6\t: Monthly average of revenue in last 6 months divided by revenue in last active month\n- avg_purchases_lag6\t: Monthly average of transactions in last 6 months divided by transactions in last active month\n- active_months_lag6\t: Quantity of active months within last 6 months\n- avg_sales_lag12\t: Monthly average of revenue in last 12 months divided by revenue in last active month\n- avg_purchases_lag12\t: Monthly average of transactions in last 12 months divided by transactions in last active month\n- active_months_lag12\t: Quantity of active months within last 12 months\n- category_4\t: anonymized category\n- city_id\t: City identifier (anonymized )\n- state_id\t: State identifier (anonymized )\n- category_2\t: anonymized category"},{"metadata":{"trusted":true,"_uuid":"23f953e61328d419ab84bb29cac71a6483b9af31"},"cell_type":"code","source":"# checking missing data\ntotal = merchant.isnull().sum().sort_values(ascending = False)\npercent = (merchant.isnull().sum()/merchant.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10d685239d1abd2d684bfb529cceb572a5e05b82"},"cell_type":"code","source":"#histogram\nf, ax = plt.subplots(figsize=(14, 6))\nsns.distplot(merchant['numerical_1'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92bd8d11ad495c41663dda7b665a4b603d975d96"},"cell_type":"code","source":"#histogram\nf, ax = plt.subplots(figsize=(14, 6))\nsns.distplot(merchant['numerical_2'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c4b69956e70de2b57dd6e50ffb861879b31f0e11"},"cell_type":"code","source":"temp = merchant[\"category_1\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('category_1 in merchant')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"13a8b47401de40ebe3a9718005fad5cc55347438"},"cell_type":"code","source":"#most_recent_sales_range, most_recent_purchases_range, category_4\ntemp = merchant[\"category_2\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('category_2 in merchant')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1966faf302dfff2c8c14dbcef3930eef6556874b"},"cell_type":"code","source":"#most_recent_sales_range, most_recent_purchases_range, category_4\ntemp = merchant[\"most_recent_sales_range\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('most_recent_sales_range in merchant')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da80613a136fe1a8c08ab7439078461e9cf5334b"},"cell_type":"code","source":"#most_recent_sales_range, most_recent_purchases_range, category_4\ntemp = merchant[\"most_recent_purchases_range\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('most_recent_purchases_range in merchant')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e7ef65e003988c445bd23fce72500bc286047f0e"},"cell_type":"code","source":"#most_recent_sales_range, most_recent_purchases_range, category_4\ntemp = merchant[\"category_4\"].value_counts()\ndf = pd.DataFrame({'labels': temp.index,\n                   'values': temp.values\n                  })\nplt.figure(figsize = (6,6))\nplt.title('category_4 in merchant')\nsns.set_color_codes(\"pastel\")\nsns.barplot(x = 'labels', y=\"values\", data=df)\nlocs, labels = plt.xticks()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e42513b79fa0a9ee74667fc19dc39024a00d976"},"cell_type":"markdown","source":"### Simple Exploration : new_merchants.csv"},{"metadata":{"trusted":true,"_uuid":"2158daadfff2f35ca36e0d1c5b5cbe795dc411f2"},"cell_type":"code","source":"new_merchant = pd.read_csv(\"../input/new_merchant_transactions.csv\")\nprint(\"shape of new_merchant_transactions : \",new_merchant.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b046368f582985ea824ec3c1242a1e4e45f71e9a"},"cell_type":"code","source":"new_merchant.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"141a8598508a27db2d7acd354c5849c0d498f768"},"cell_type":"markdown","source":"- card_id\t: Card identifier\n- month_lag\t: month lag to reference date\n- purchase_date\t: Purchase date\n- authorized_flag\t: Y' if approved, 'N' if denied\n- category_3\t: anonymized category\n- installments\t: number of installments of purchase\n- category_1\t: anonymized category\n- merchant_category_id\t: Merchant category identifier (anonymized )\n- subsector_id\t: Merchant category group identifier (anonymized )\n- merchant_id\t: Merchant identifier (anonymized)\n- purchase_amount\t: Normalized purchase amount\n- city_id\t: City identifier (anonymized )\n- state_id\t: State identifier (anonymized )\n- category_2\t: anonymized category\n"},{"metadata":{"trusted":true,"_uuid":"14841d7b310526d610c9499b477d08d27aab01ca"},"cell_type":"code","source":"# checking missing data\ntotal = new_merchant.isnull().sum().sort_values(ascending = False)\npercent = (new_merchant.isnull().sum()/new_merchant.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d20375c23b9528dd3125127939ff559e7520eb2"},"cell_type":"code","source":"new_merchant['authorized_flag'] = new_merchant['authorized_flag'].map({'Y':1, 'N':0})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d226c0360094b87654c9aec5677f76164836c6cd"},"cell_type":"code","source":"def aggregate_new_transactions(new_trans):    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'month_lag': ['min', 'max']\n        }\n    agg_new_trans = new_trans.groupby(['card_id']).agg(agg_func)\n    agg_new_trans.columns = ['new_' + '_'.join(col).strip() \n                           for col in agg_new_trans.columns.values]\n    agg_new_trans.reset_index(inplace=True)\n    \n    df = (new_trans.groupby('card_id')\n          .size()\n          .reset_index(name='new_transactions_count'))\n    \n    agg_new_trans = pd.merge(df, agg_new_trans, on='card_id', how='left')\n    \n    return agg_new_trans\n\nnew_trans = aggregate_new_transactions(new_merchant)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31612aef5e2d693a899776a2dfcd0f4af8e38ea2"},"cell_type":"code","source":"train = pd.merge(train, new_trans, on='card_id', how='left')\ntest = pd.merge(test, new_trans, on='card_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8cb85391a726126ae42d9c6508a031d2658df9fc"},"cell_type":"markdown","source":"### Make a Baseline model\nkernel : https://www.kaggle.com/youhanlee/hello-elo-ensemble-will-help-you"},{"metadata":{"trusted":true,"_uuid":"923477910006486e5dd0a8e395a5c361e1419ef2"},"cell_type":"code","source":"use_cols = [col for col in train.columns if col not in ['card_id', 'first_active_month']]\n\ntrain = train[use_cols]\ntest = test[use_cols]\n\nfeatures = list(train[use_cols].columns)\ncategorical_feats = [col for col in features if 'feature_' in col]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a1f1e1aeccbcf3e0049fdb1f465e0670f6b91c2"},"cell_type":"code","source":"for col in categorical_feats:\n    print(col, 'have', train[col].value_counts().shape[0], 'categories.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba4b71e85aa8063f9dbbe5e8a3e8f6bd498cfdc3"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfor col in categorical_feats:\n    print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(train[col].values.astype('str')) + list(test[col].values.astype('str')))\n    train[col] = lbl.transform(list(train[col].values.astype('str')))\n    test[col] = lbl.transform(list(test[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad4ed61eaef4f37206043da70067007a394c0aac"},"cell_type":"code","source":"df_all = pd.concat([train, test])\ndf_all = pd.get_dummies(df_all, columns=categorical_feats)\n\nlen_train = train.shape[0]\n\ntrain = df_all[:len_train]\ntest = df_all[len_train:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"44a765e67afd943349311fe4a079827178efe840"},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\n\n\nlgb_params = {\"objective\" : \"regression\", \"metric\" : \"rmse\", \n               \"max_depth\": 7, \"min_child_samples\": 20, \n               \"reg_alpha\": 1, \"reg_lambda\": 1,\n               \"num_leaves\" : 64, \"learning_rate\" : 0.005, \n               \"subsample\" : 0.8, \"colsample_bytree\" : 0.8, \n               \"verbosity\": -1}\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=1989)\n\noof_lgb = np.zeros(len(train))\npredictions_lgb = np.zeros(len(test))\n\nfeatures_lgb = list(train.columns)\nfeature_importance_df_lgb = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train)):\n    trn_data = lgb.Dataset(train.iloc[trn_idx], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train.iloc[val_idx], label=target.iloc[val_idx])\n\n    print(\"LGB \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    clf = lgb.train(lgb_params, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=1000, early_stopping_rounds = 50)\n    oof_lgb[val_idx] = clf.predict(train.iloc[val_idx], num_iteration=clf.best_iteration)\n\n    fold_importance_df_lgb = pd.DataFrame()\n    fold_importance_df_lgb[\"feature\"] = features_lgb\n    fold_importance_df_lgb[\"importance\"] = clf.feature_importance()\n    fold_importance_df_lgb[\"fold\"] = fold_ + 1\n    feature_importance_df_lgb = pd.concat([feature_importance_df_lgb, fold_importance_df_lgb], axis=0)\n    predictions_lgb += clf.predict(test, num_iteration=clf.best_iteration) / FOLDs.n_splits\n    \n\nprint(np.sqrt(mean_squared_error(oof_lgb, target)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e3d4406d0998a4bf720e95f290d982cdfbe5979"},"cell_type":"code","source":"cols = (feature_importance_df_lgb[[\"feature\", \"importance\"]]\n        .groupby(\"feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df_lgb.loc[feature_importance_df_lgb.feature.isin(cols)]\n\nplt.figure(figsize=(14,14))\nsns.barplot(x=\"importance\",\n            y=\"feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"95a2bc22b3efcf1fa4eeb4b008f6ccc616eb0b6b"},"cell_type":"code","source":"sub_df = pd.read_csv('../input/sample_submission.csv')\nsub_df[\"target\"] = predictions_lgb \nsub_df.to_csv(\"submission_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fee8aa65e71e571769007b1934572ce8cf808494"},"cell_type":"code","source":"import xgboost as xgb\n\nxgb_params = {'eta': 0.005, 'max_depth': 7, 'subsample': 0.8, 'colsample_bytree': 0.8, \n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True}\n\n\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=1989)\n\noof_xgb = np.zeros(len(train))\npredictions_xgb = np.zeros(len(test))\n\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train)):\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb += xgb_model.predict(xgb.DMatrix(test), ntree_limit=xgb_model.best_ntree_limit+50) / FOLDs.n_splits\n\nnp.sqrt(mean_squared_error(oof_xgb, target))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7fc3eb5bcd9b9b13ea95f242715b56f1a0c48ac"},"cell_type":"code","source":"print('lgb', np.sqrt(mean_squared_error(oof_lgb, target)))\nprint('xgb', np.sqrt(mean_squared_error(oof_xgb, target)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a04ddcfb08e5dcdb8ed81e4ead947345310f4765"},"cell_type":"code","source":"total_sum = 0.5 * oof_lgb + 0.5 * oof_xgb\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(total_sum, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96b23345fabb64c3bf6879620222a57e3d465c22"},"cell_type":"code","source":"sub_df = pd.read_csv('../input/sample_submission.csv')\nsub_df[\"target\"] = 0.5 * predictions_lgb + 0.5 * predictions_xgb\nsub_df.to_csv(\"submission_ensemble.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}