{"cells":[{"metadata":{"_uuid":"8c086a072e6bcd89a82a777876021de8f2ea994b"},"cell_type":"markdown","source":"### Introductory comments:\n\nThis is my first time using python for other than data wrangling and committing a kernel. I tried to arrange the kernel and comment so that beginners should be able to understand and run the individual steps fairly easy. If something is unclear i suggest running the code bit by bit and inspecting the output. In order to not be biased by previous kernels i tried to limit the time spend in discussions and reading existing kernels, although i had a look at few (if steps taken from existing kernels, this is outlined in the code). The kernel only takes the *'train.csv'*, *'test.csv'* and *'historical_transactions.csv'* into consideration, why features from *'new_merchants_transaction.csv'* and *'merchants.csv'* is not introduced. Although *'new_merchants_transaction.csv' *can be largely introduced by the same approach as *'hitorical_transactions.csv'*, i chose not to do this as i believe a different approach would be better.\n\nThe purpose of the notebook is more leaned towards providing a **framework** and giving ideas for **further feature engineering** than performance itself.\n\n\n## Notebook content\n\n## [1. Data Preperation and Exploration for *train.csv* and *train.csv*](#1)\n\n## [2. Data Preperation and Exploration for *historical_transactions.csv*](#2)\n\n## [3. Merging and Preparing Data for Modelling](#3)\n\n## [4. GridSearch](#4)\n\n## [5. Light Gradient Boosting and Feature Importance](#5)\n\n## [6. Submission](#6)\n\n"},{"metadata":{"_uuid":"7be37abd57347d91d9e46bc1e28314e62b2d6062"},"cell_type":"markdown","source":"<a id=\"1\"></a>\n## 1. Data Preperation and Exploration for *train.csv* and *train.csv*\n"},{"metadata":{"_uuid":"4b6f2f8c3682c4197f50ea9b9be81bc8ec03c7be"},"cell_type":"markdown","source":"Load numpy and pandas:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":false,"_kg_hide-input":true},"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nprint(os.listdir(\"../input\"))\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\npd.set_option('display.max_columns', 400)\n%matplotlib inline\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bacba8d014bfece4c82efce3ddbefc99742635ad"},"cell_type":"markdown","source":"Importing train and test data:\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":false,"scrolled":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv', parse_dates=[\"first_active_month\"])\ntest = pd.read_csv('../input/test.csv', parse_dates=[\"first_active_month\"])\n\ntrain.head(5)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d2eaaaac83179f08b94f12b962a4ce4dfecc329"},"cell_type":"markdown","source":"Variable description:\n    * first_active_month = 'YYYY-MM', month of first purchase\n    * feature_1\t= Anonymized card categorical feature\n    * feature_2\t= Anonymized card categorical feature\n    * feature_3\t= Anonymized card categorical feature\n    * target = Loyalty numerical score calculated 2 months after historical and evaluation period\n\n\n"},{"metadata":{"_uuid":"0cb0c7b1a1c1a584be88d850aca6cfda88e08041"},"cell_type":"markdown","source":"Concatenate train and test for easier merge later:"},{"metadata":{"trusted":true,"_uuid":"44b0838af2d9fc279cef32e192618f3aac3c9b2a"},"cell_type":"code","source":"train_test = pd.concat([train, test], ignore_index=False, keys=['train', 'test'], sort=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3be353cc72d937781000a4611b96e6acd278be84"},"cell_type":"markdown","source":"Checking for duplicates on card_id:"},{"metadata":{"trusted":true,"_uuid":"4689a80c18fe9832dd86e051dcf559bc057f1a6a","_kg_hide-input":true},"cell_type":"code","source":"print('Rows in train: {}'.format(len(train_test[['card_id']])))\nprint('Unique card_ids in train: {}'.format(len(train_test.card_id.value_counts())))\n\n# no issues regarding duplicate card_id's","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4129ce604df7fcc1a699a5b322a18ee7085203f"},"cell_type":"markdown","source":"Checking for missing values:"},{"metadata":{"trusted":true,"_uuid":"50e12464989c9260c5b2834c97d5fdcca5b39238"},"cell_type":"code","source":"print('train missing values:')\nprint(train_test.loc['train'].isnull().sum())\n\nprint('test missing values:')\nprint(train_test.loc['test'].isnull().sum())\n\nprint('missing value:')\nprint(train_test[train_test['first_active_month'].isnull()])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"acccfeb194c72d639c02f06296ac814d1e49ed89"},"cell_type":"markdown","source":"Calculate mean of datetime series and substitute missing value:"},{"metadata":{"trusted":true,"_uuid":"2c6e11ac70e03be7aae2d3db34dc60fdec711d66"},"cell_type":"code","source":"# calculate mean of datetime series\n\nmean_date = (train_test.first_active_month - train_test.first_active_month.min()).mean() + train_test.first_active_month.min()\n\n# fill with mean\ntrain_test['first_active_month'] = train_test['first_active_month'].fillna(mean_date)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08be5cbc183a52f1b3c5cdf1c8e411114d7e4028"},"cell_type":"markdown","source":"Adding 'months since' active feature:"},{"metadata":{"trusted":true,"_uuid":"7be8bb90b0942b8ed5030a3e4fe4cd3317a523e3"},"cell_type":"code","source":"# data pull seems to be 2018-04-30 as this is max in new_merchant_transactions\n\ntrain_test['Days_since_first_active'] = (pd.to_datetime('2018-04-30') - train_test['first_active_month']).dt.days\n\ntrain_test['Months_since_first_active'] = train_test['Days_since_first_active'] // 30 #floor division\n\n#clean dataset\n\nkeep_columns = ['card_id', 'first_active_month', 'feature_1', 'feature_2', 'feature_3', 'target', \n                'Months_since_first_active']\n\ntrain_test = train_test[keep_columns]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a489e6efb096b2c1e6d73ba6bc7993156cb8e01d"},"cell_type":"markdown","source":"### Explanatory data analysis:"},{"metadata":{"trusted":true,"_uuid":"6fe07cdc9f348f99f6acd1226be92e692dc5cb57"},"cell_type":"code","source":"import seaborn as sns; sns.set() #set plot theme\nimport matplotlib.pyplot as plt\n\nprint(train_test.loc['train'].describe())\nprint(train_test.loc['test'].describe())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b16376ae9471d6b337da54d7dfdc2b6d70074d17"},"cell_type":"markdown","source":"Setting up for barplots:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"47fdf25d3fbb7eb50e0aa1ac875316e67bcc9f1a"},"cell_type":"code","source":"#feature_1\nfeature_1_train = train_test.loc['train'].feature_1.value_counts().sort_index()\nfeature_1_train = feature_1_train / len(train_test.loc['train'])\n\nfeature_1_test = train_test.loc['test'].feature_1.value_counts().sort_index()\nfeature_1_test = feature_1_test / len(train_test.loc['test'])\n\n#feature_2\nfeature_2_train = train_test.loc['train'].feature_2.value_counts().sort_index()\nfeature_2_train = feature_2_train / len(train_test.loc['train'])\n\nfeature_2_test = train_test.loc['test'].feature_2.value_counts().sort_index()\nfeature_2_test = feature_2_test / len(train_test.loc['test'])\n\n#feature_3\nfeature_3_train = train_test.loc['train'].feature_3.value_counts().sort_index()\nfeature_3_train = feature_3_train / len(train_test.loc['train'])\n\nfeature_3_test = train_test.loc['test'].feature_3.value_counts().sort_index()\nfeature_3_test = feature_3_test / len(train_test.loc['test'])\n\n#Months_since_first_active\nMonths_since_first_active_train = train_test.loc['train'].Months_since_first_active.value_counts().sort_index()\nMonths_since_first_active_train = Months_since_first_active_train / len(train_test.loc['train'])\n\nMonths_since_first_active_test = train_test.loc['test'].Months_since_first_active.value_counts().sort_index()\nMonths_since_first_active_test = Months_since_first_active_test / len(train_test.loc['test'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a27007e9633b6259e2004cbf3399720838f44e83"},"cell_type":"markdown","source":"Plotting feature_1 to Months_since_first_active: "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d7356fb815fbb4e2f7055e81b45c956c8588f029"},"cell_type":"code","source":"fig, ax = plt.subplots(2, 2, figsize=(14,14))\n\n# =============================================================================\n# #plotting independt variables checking for consistency vs train and test data\n# =============================================================================\n\n#plotting feature_1\nax[0,0].bar(feature_1_train.index, feature_1_train, label='train', alpha=0.3)\nax[0,0].bar(feature_1_test.index, feature_1_test, label='test', alpha=0.3)\nax[0,0].legend()\nax[0,0].set_title('feature_1')\n\n#plotting feature_2\nax[0,1].bar(feature_2_train.index, feature_2_train, label='train', alpha=0.3)\nax[0,1].bar(feature_2_test.index, feature_2_test, label='test', alpha=0.3)\nax[0,1].legend()\nax[0,1].set_title('feature_2')\n\n#plotting feature_3\nax[1,0].bar(feature_3_train.index, feature_3_train, label='train', alpha=0.3)\nax[1,0].bar(feature_3_test.index, feature_3_test, label='test', alpha=0.3)\nax[1,0].legend()\nax[1,0].set_title('feature_3')\n\n#plotting Months_since_first_active\nax[1,1].bar(Months_since_first_active_train.index, Months_since_first_active_train, label='train', alpha=0.3)\nax[1,1].bar(Months_since_first_active_test.index, Months_since_first_active_test, label='test', alpha=0.3)\nax[1,1].legend()\nax[1,1].set_title('Months_since_first_active');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2489d593b2608eaeee775360d6ac888fbcaa0b3"},"cell_type":"markdown","source":"We see distribution is rougly the same between train and test data, why we should no be to concerned about differences between training and test data."},{"metadata":{"_uuid":"eaf8445d5e65bd5d408c2281ecf0312f38e50f2f"},"cell_type":"markdown","source":"Plotting target variable:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"521c5bb0adf3998a1fd510a1704af627a59549ab"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(14,7))\nsns.distplot(train_test.loc['train'].target)\nplt.title('target distribution');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d4793cbd7ae9a00dfa17f27a0cab537f28298ec"},"cell_type":"markdown","source":"Plotting dependent variable vs independent (full sample):"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7edf0c74621103f8aa7e3e528e32776f32a0d994"},"cell_type":"code","source":"#plotting feature_1 - feature_2 vs target\nfig, ((ax1, ax2)) = plt.subplots(nrows=1, ncols=2, figsize=(14,7))\n\nfig = sns.boxplot(x='feature_1', y='target', data=train_test.loc['train'],ax=ax1)\nax1.set_title('target distribution [feature_1]')\n\nsns.boxplot(x='feature_2', y='target', data=train_test.loc['train'],ax=ax2)\nax2.set_title('target distribution [feature_2]')\n\n#plotting feature_1 - feature_2 vs target\nfig = plt.subplots(nrows=1, ncols=1, figsize=(7,7))\nfig = sns.boxplot(x='feature_3', y='target', data=train_test.loc['train'])\nplt.title('target distribution [feature_3]')\n\n#plotting jointplot, taking every 60th observation to lower load time\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nfig = sns.swarmplot(x='Months_since_first_active', y='target', data=train_test.loc['train'::60])\nplt.title('Months_since_first_active vs. target')\nplt.xticks(rotation='90');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c831a7acf849e5522f5c66737b0f0f028d7e954b"},"cell_type":"markdown","source":"<a id=\"2\"></a>\n# 2. Data Preperation and Exploration for *historical_transactions.csv*"},{"metadata":{"_uuid":"a2d17550c1ec7f8176ebe9d81b379dabf7b60dd6"},"cell_type":"markdown","source":"Load data and inspect missing values:"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"6e345ae56071d9f9a040d2628788a5a656ca6aec"},"cell_type":"code","source":"hist_transactions = pd.read_csv('../input/historical_transactions.csv', parse_dates=['purchase_date'])\n\n#function found in https://www.kaggle.com/fabiendaniel/elo-world\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n\nhist_transactions = reduce_mem_usage(hist_transactions)\n                                \nprint('missing values %:') \nprint(hist_transactions.isnull().sum() / len(hist_transactions))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d3cb8de863f45afcf746a23662cb91c2b815193"},"cell_type":"markdown","source":"We see some missing values, we will have this in consideration when inspecting the variables."},{"metadata":{"_uuid":"9fbe835c1d4d507ea81b1863001701e9112c445d"},"cell_type":"markdown","source":"Visualizing variables in historical_transactions dataset:"},{"metadata":{"_uuid":"c243a915637b65a9cce1d7e9dced0e21f7e1c72a"},"cell_type":"markdown","source":"    *authorized_flag:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6bd42ac842fba2cf957f584d7ea5cfb42e7161cd"},"cell_type":"code","source":"##### authorized_flag\nhist_transactions.authorized_flag.head()\nhist_transactions['authorized_flag'] = hist_transactions['authorized_flag'].map({'Y' : 1, 'N' : 0})\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(7,7))\nsns.barplot(x = hist_transactions['authorized_flag'].value_counts().index,\n            y = hist_transactions['authorized_flag'].value_counts() / len(hist_transactions),\n            order = hist_transactions['authorized_flag'].value_counts().index)\nplt.ylabel('freq %')\nplt.title('authorized_flag');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"669416ec54c183d14d7bdc312e83a4948356617e"},"cell_type":"markdown","source":"    *city_id:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"402139c867be10e79310cccac07cec3c80050cd0"},"cell_type":"code","source":"##### city_id\nhist_transactions.city_id.head(10)\nprint(hist_transactions.city_id.value_counts().head()) #-1 probalbly missing values, we treet them as a seperate group for now\n\nshow = 20\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = hist_transactions['city_id'].value_counts().index[:show],\n            y = hist_transactions['city_id'].value_counts()[:show] / len(hist_transactions['city_id']),\n            order = hist_transactions['city_id'].value_counts().index[:show])\nplt.ylabel('freq %')\nplt.title('Most {}th frequent city_ids'.format(show))\n\n#cumulative barplot\n\ncity_id_freq = hist_transactions['city_id'].value_counts() / len(hist_transactions['city_id'])\n\ncity_id_cum = pd.DataFrame(city_id_freq)\ncity_id_cum.columns = ['city_id_freq']\n \ncity_id_cum = city_id_cum.reset_index()\n\ncum_var = np.empty((0,1),float)\n\nfor i in city_id_cum.index:\n    if i == 0:\n        cum_var = np.append(cum_var,(city_id_cum.loc[i,'city_id_freq']))\n    else: \n        cum_var = np.append(cum_var, (cum_var[i-1] + city_id_cum.loc[i,'city_id_freq']))\n\ncity_id_cum['city_id_cum'] = cum_var    \n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = city_id_cum['index'][:show],\n            y = city_id_cum['city_id_cum'][:show],\n            order = city_id_cum['index'][:show])\nplt.ylabel('freq %')\nplt.title('Most {}th frequent city_ids (cumulative)'.format(show)); #we see that a relatively small number of cities make up a large part of the transactions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"85ad1161766ead3d5b887be184b06810dc709468"},"cell_type":"markdown","source":"We see that a relatively small number of cities make up a large part of the transactions. Also -1 is probably missing values, we will treat them as a seperate group for now."},{"metadata":{"_uuid":"88cc7bcb1ad17c792409cc71aee8a2da24a811b1"},"cell_type":"markdown","source":"    *state_id:"},{"metadata":{"trusted":true,"_uuid":"485f2bf85a5b8d9c2b2de5825c1f8df13fcea4d6","_kg_hide-input":true},"cell_type":"code","source":"##### state_id\nhist_transactions.state_id.head(10)\n\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = hist_transactions['state_id'].value_counts().index,\n            y = hist_transactions['state_id'].value_counts() / len(hist_transactions['state_id']),\n            order = hist_transactions['state_id'].value_counts().index)\nplt.ylabel('freq %')\nplt.title('state_id')\n\n#cumulative barplot\n\nstate_id_freq = hist_transactions['state_id'].value_counts() / len(hist_transactions['state_id'])\n\nstate_id_cum = pd.DataFrame(state_id_freq)\nstate_id_cum.columns = ['state_id_freq']\n \nstate_id_cum = state_id_cum.reset_index()\n\ncum_var = np.empty((0,1),float)\n\nfor i in state_id_cum.index:\n    if i == 0:\n        cum_var = np.append(cum_var,(state_id_cum.loc[i,'state_id_freq']))\n    else: \n        cum_var = np.append(cum_var, (cum_var[i-1] + state_id_cum.loc[i,'state_id_freq']))\n\nstate_id_cum['state_id_cum'] = cum_var    \n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = state_id_cum['index'],\n            y = state_id_cum['state_id_cum'],\n            order = state_id_cum['index'])\nplt.ylabel('freq %')\nplt.title('state_id cummulative dist'); #we see that a relatively small number of states make up a large part of the transactions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fdba02242549ed16964be65c15c601fe4c465233"},"cell_type":"markdown","source":"    *category_1, category_2, category_3:"},{"metadata":{"trusted":true,"_uuid":"c2de0a7075231cd6e84131dbb4b98e855676c035","_kg_hide-input":true},"cell_type":"code","source":"##### category_1\nhist_transactions.category_1.head()\nhist_transactions.category_1.unique()\n\nhist_transactions['category_1'] = hist_transactions['category_1'].map({'Y' : 1, 'N' : 0})\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(7,7))\nsns.barplot(x = hist_transactions['category_1'].value_counts().index,\n            y = hist_transactions['category_1'].value_counts() / len(hist_transactions['category_1']),\n            order = hist_transactions['category_1'].value_counts().index)\nplt.ylabel('freq %')\nplt.title('category_1')\n\n##### category_2\nhist_transactions.category_2.head()\nhist_transactions.category_2.value_counts(dropna=False)\n\nhist_transactions['category_2'] = hist_transactions.category_2.fillna(6.0)\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(7,7))\nsns.barplot(x = hist_transactions['category_2'].value_counts().index,\n            y = hist_transactions['category_2'].value_counts() / len(hist_transactions['category_2']),\n            order = hist_transactions['category_2'].value_counts().index)\nplt.ylabel('freq %')\nplt.title('category_2')\n\n\n##### category_3\nhist_transactions.category_3.head()\nhist_transactions.category_3.value_counts(dropna=False)\n\n#we see relatively small number of NAs. We handle them as a seperate group.\nhist_transactions['category_3'] = hist_transactions.category_3.fillna('NA')\n\nhist_transactions.category_3.value_counts(dropna=False)\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(7,7))\nsns.barplot(x = hist_transactions['category_3'].value_counts().index,\n            y = hist_transactions['category_3'].value_counts() / len(hist_transactions['category_3']),\n            order = hist_transactions['category_3'].value_counts().index)\nplt.ylabel('% freq')\nplt.title('category_3');\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d89ceb1370edc97190f17a81111d91ba78c8703a"},"cell_type":"markdown","source":"We see relatively small number of NAs in category_3. We handle them as a seperate group ('NA')."},{"metadata":{"_uuid":"58a072a85c9ebd6c45c7196e76d2f875b370b573"},"cell_type":"markdown","source":"    *installments:"},{"metadata":{"trusted":true,"_uuid":"3e7b2438626d39bfcc3ed98a149479467af11636","_kg_hide-input":true},"cell_type":"code","source":"##### installments\nhist_transactions.installments.unique()\nprint('Values:')\nprint(hist_transactions.installments.value_counts()) \n\nhist_transactions['installments'] = hist_transactions.installments.replace({-1 : 0, 999 : 0})\nhist_transactions.installments.value_counts()\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(7,7))\nsns.barplot(x = hist_transactions['installments'].value_counts().index,\n            y = hist_transactions['installments'].value_counts() / len(hist_transactions['installments']),\n            order = hist_transactions['installments'].value_counts().index)\nplt.xlabel('# of installments')\nplt.ylabel('% freq')\nplt.title('installments');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"176c431bb32ef1413090c021e026ae10e1b056ca"},"cell_type":"markdown","source":"999  and -1 could be missing values, given installments should be known by the client we will assume 0 installments for these cases."},{"metadata":{"_uuid":"0119ea870733c666b11cf637a839b63f1fcf9ab1"},"cell_type":"markdown","source":"    *merchant_category_id"},{"metadata":{"trusted":true,"_uuid":"5a208461c143a8d61fc00465bdb40a8808a27682","_kg_hide-input":true},"cell_type":"code","source":"##### merchant_category_id\nhist_transactions.merchant_category_id.value_counts().head(15)\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = hist_transactions['merchant_category_id'].value_counts().index[:30],\n            y = hist_transactions['merchant_category_id'].value_counts()[:30] / len(hist_transactions['merchant_category_id']),\n            order = hist_transactions['merchant_category_id'].value_counts().index[:30])\nplt.ylabel('freq %')\nplt.xlabel('merchant_category_id')\nplt.title('merchant_category_id')\n\n#cumulative barplot\n\nmerchant_category_id_freq = hist_transactions['merchant_category_id'].value_counts() / len(hist_transactions['merchant_category_id'])\n\nmerchant_category_id_cum = pd.DataFrame(merchant_category_id_freq)\nmerchant_category_id_cum.columns = ['merchant_category_id_freq']\n \nmerchant_category_id_cum = merchant_category_id_cum.reset_index()\n\ncum_var = np.empty((0,1),float)\n\nfor i in merchant_category_id_cum.index:\n    if i == 0:\n        cum_var = np.append(cum_var,(merchant_category_id_cum.loc[i,'merchant_category_id_freq']))\n    else: \n        cum_var = np.append(cum_var, (cum_var[i-1] + merchant_category_id_cum.loc[i,'merchant_category_id_freq']))\n\nmerchant_category_id_cum['merchant_category_id_cum'] = cum_var    \n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = merchant_category_id_cum['index'][:30],\n            y = merchant_category_id_cum['merchant_category_id_cum'][:30],\n            order = merchant_category_id_cum['index'][:30])\nplt.xlabel('merchant_category_id')\nplt.ylabel('freq %')\nplt.title('merchant_category_id cummulative dist'); #we see that a relatively small number of merchant_category_id's make up a large part of the transactions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92c2b9208c83044db7ed316959c36f7645de941c"},"cell_type":"markdown","source":"Again we see that a relatively small number of merchant_category_id's make up a large part of the transactions."},{"metadata":{"_uuid":"1110c1e4d86de27e74721e43b48aa3c4abb209e3"},"cell_type":"markdown","source":"    *month_lag / purchase_date:"},{"metadata":{"trusted":true,"_uuid":"4d9b28b616a1ea0992d9d92008edc53bc6c9ee81","_kg_hide-input":true},"cell_type":"code","source":"##### month_lag / purchase_date\nhist_transactions['month_lag'].head()\nhist_transactions['month_lag'].value_counts()\n\n#calculate so it is consistent with the formula used on the training set\n\nhist_transactions['Days_since_trans'] = (pd.to_datetime('2018-03-01') - hist_transactions['purchase_date']).dt.days\n\nhist_transactions['MonthsSince_trans'] = hist_transactions['Days_since_trans'] // 30 #floor division\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(7,7))\nsns.barplot(x = hist_transactions['MonthsSince_trans'].value_counts().index,\n            y = hist_transactions['MonthsSince_trans'].value_counts() / len(hist_transactions['MonthsSince_trans']),)\nplt.ylabel('% freq')\nplt.xlabel('# months since transaction')\nplt.title('MonthsSince_trans');\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"feaff878505b0daf8fa0325d1b035047aae29368"},"cell_type":"markdown","source":"Quite irrational distributiion, we would expect the number of transactions to increase with time it seems. Lets have a another look at it."},{"metadata":{"trusted":true,"_uuid":"d699b8de92dba09652de96c9a0ed5810776903f2"},"cell_type":"code","source":"show = 20\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = hist_transactions['purchase_date'].dt.date.value_counts().index[:show],\n            y = hist_transactions['purchase_date'].dt.date.value_counts()[:show] / len(hist_transactions['purchase_date']),\n            order = hist_transactions['purchase_date'].dt.date.value_counts().index[:show])\nplt.xticks(rotation='45')\nplt.ylabel('% freq')\nplt.title('Most {}th frequent purchase_dates'.format(show));","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f463624a475b7d4fd154f0143fa86c15cc26b93"},"cell_type":"markdown","source":"We see christmas time is popular, why monthsince 2 peaks."},{"metadata":{"_uuid":"bd513b67aae18b2c2c610c48c160fd85862dae26"},"cell_type":"markdown","source":"    *purchase_amount"},{"metadata":{"trusted":true,"_uuid":"76cae9cf45e8826f77a82097214e2fb41911f7f9"},"cell_type":"code","source":"##### purchase_amount\n\nprint(hist_transactions['purchase_amount'].describe())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56f8af0752afc22abf2ca9f07b0d3d0c0384c4f6"},"cell_type":"markdown","source":"    *subsector_id:"},{"metadata":{"trusted":true,"_uuid":"eb66f93d177e99f50c6d53bfb2e98c3b4ef14d4c","_kg_hide-input":true},"cell_type":"code","source":"##### subsector_id\n\nhist_transactions.subsector_id.value_counts().head(15)\n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = hist_transactions['subsector_id'].value_counts().index[:30],\n            y = hist_transactions['subsector_id'].value_counts()[:30] / len(hist_transactions['subsector_id']),\n            order = hist_transactions['subsector_id'].value_counts().index[:30])\nplt.xlabel('subsector_id')\nplt.ylabel('freq %')\nplt.title('subsector_id')\n\n#cumulative barplot\n\nsubsector_id_freq = hist_transactions['subsector_id'].value_counts() / len(hist_transactions['subsector_id'])\n\nsubsector_id_cum = pd.DataFrame(subsector_id_freq)\nsubsector_id_cum.columns = ['subsector_id_freq']\n \nsubsector_id_cum = subsector_id_cum.reset_index()\n\ncum_var = np.empty((0,1),float)\n\nfor i in subsector_id_cum.index:\n    if i == 0:\n        cum_var = np.append(cum_var,(subsector_id_cum.loc[i,'subsector_id_freq']))\n    else: \n        cum_var = np.append(cum_var, (cum_var[i-1] + subsector_id_cum.loc[i,'subsector_id_freq']))\n\nsubsector_id_cum['subsector_id_cum'] = cum_var    \n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = subsector_id_cum['index'][:30],\n            y = subsector_id_cum['subsector_id_cum'][:30],\n            order = subsector_id_cum['index'][:30])\nplt.xlabel('subsector_id')\nplt.ylabel('freq %')\nplt.title('subsector_id cummulative dist');\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08b1fa41ab336e93fccd2dfd849e36b452b43d51"},"cell_type":"markdown","source":"    *merchant_id:"},{"metadata":{"trusted":true,"_uuid":"12126fc376577024ba04972ae8c2453a21de6db4","_kg_hide-input":true},"cell_type":"code","source":"#### merchant_id\n\nhist_transactions.merchant_id.value_counts().head(15)\nprint('Number of unique merchant_id~s: {}'.format(len(hist_transactions.merchant_id.unique())))\n\nshow = 20\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = hist_transactions['merchant_id'].value_counts().index[:show],\n            y = hist_transactions['merchant_id'].value_counts()[:show] / len(hist_transactions['merchant_id']),\n            order = hist_transactions['merchant_id'].value_counts().index[:show])\nplt.xticks(rotation='60')\nplt.ylabel('freq %')\nplt.xlabel('merchant_id')\nplt.title('Most {}th frequent merchant_id~s'.format(show))\n\n#cumulative barplot\n\nmerchant_id_freq = hist_transactions['merchant_id'].value_counts() / len(hist_transactions['merchant_id'])\n\nmerchant_id_cum = pd.DataFrame(merchant_id_freq)\nmerchant_id_cum.columns = ['merchant_id_freq']\n \nmerchant_id_cum = merchant_id_cum.reset_index()\n\ncum_var = np.empty((0,1),float)\n\nfor i in merchant_id_cum.index:\n    if i == 0:\n        cum_var = np.append(cum_var,(merchant_id_cum.loc[i,'merchant_id_freq']))\n    else: \n        cum_var = np.append(cum_var, (cum_var[i-1] + merchant_id_cum.loc[i,'merchant_id_freq']))\n\nmerchant_id_cum['merchant_id_cum'] = cum_var    \n\nfig = plt.subplots(nrows=1, ncols=1, figsize=(14,7))\nsns.barplot(x = merchant_id_cum['index'][:show],\n            y = merchant_id_cum['merchant_id_cum'][:show],\n            order = merchant_id_cum['index'][:show])\nplt.xticks(rotation='60')\nplt.ylabel('freq %')\nplt.xlabel('merchant_id')\nplt.title('merchant_id cummulative dist'); ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9e73b6dee3b674bf3b23ec7603c90f34a93e2a6"},"cell_type":"markdown","source":"<a id=\"3\"></a>\n## 3. Merging and Preparing Data for Modelling"},{"metadata":{"trusted":true,"_uuid":"5e65706a2fbfb61b8b552ce020b27b0014be249c","_kg_hide-input":true},"cell_type":"code","source":"agg_func = {\n        'MonthsSince_trans' : ['min', 'max', 'mean', 'std'],\n        'purchase_date' : ['count'],\n        'authorized_flag': ['min', 'max', 'sum', 'mean'],\n        'category_3': ['nunique'],\n        'installments': ['min', 'max', 'mean', 'sum', 'std'],\n        'category_1' : ['min', 'max', 'mean'],\n        'merchant_category_id' : ['nunique'],\n        'subsector_id' : ['nunique'],\n        'merchant_id' : ['nunique'],\n        'purchase_amount' : ['min', 'max', 'sum', 'mean', 'std'],\n        'city_id' : ['nunique'],\n        'state_id' : ['nunique'],\n        'category_2' : ['nunique', 'min', 'max', 'mean']\n        }\n\nhist_trans_agg = hist_transactions.groupby(['card_id']).agg(agg_func)\n\nhist_trans_agg.columns = ['hist_' + '_'.join(col).strip() \n                           for col in hist_trans_agg.columns.values]\n\nhist_trans_agg.reset_index(inplace=True)\n\ndel hist_transactions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4085e22f03140d128cffcb22096ac6a121b07c4"},"cell_type":"markdown","source":"Merging:"},{"metadata":{"trusted":true,"_uuid":"da249e20da70005d1adc0beaae40e3bd671f1ff7"},"cell_type":"code","source":"train = train_test.loc['train'].set_index('card_id').drop('first_active_month', axis=1)\ntest = train_test.loc['test'].set_index('card_id').drop('first_active_month', axis=1)\n\ntrain = train.merge(hist_trans_agg, left_on='card_id', right_on='card_id',\n                              how='left').set_index('card_id')\n\ntest = test.merge(hist_trans_agg, left_on='card_id', right_on='card_id',\n                              how='left').set_index('card_id')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51233deb87aa630996b12de49782aabdaaad9991"},"cell_type":"markdown","source":"Transforming categorical features:"},{"metadata":{"trusted":true,"_uuid":"2a8559b85b6e2e50ef7aa0ad5b27410a3d557b47"},"cell_type":"code","source":"# =============================================================================\n# #transforming categorical features\n# =============================================================================\nfeatures = list(train.columns)\ncategorical_feats = [col for col in features if 'feature_' in col]\nfor col in categorical_feats:\n    print(col, 'have', train[col].value_counts().shape[0], 'categories.')\n\nfrom sklearn.preprocessing import LabelEncoder\nfor col in categorical_feats:\n    print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(train[col].values.astype('str')) + list(test[col].values.astype('str')))\n    train[col] = lbl.transform(list(train[col].values.astype('str')))\n    test[col] = lbl.transform(list(test[col].values.astype('str')))\n\ndf_all = pd.concat([train, test])\ndf_all = pd.get_dummies(df_all, columns=categorical_feats)\n\nlen_train = train.shape[0]\n\ntrain = df_all[:len_train]\ntest = df_all[len_train:]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56b1babca58daaa86eaec580fc71a51556418670"},"cell_type":"markdown","source":"<a id=\"4\"></a>\n## 4. GridSearch:"},{"metadata":{"_uuid":"5769e7edbd52d21da84d61b43b428412a750423e"},"cell_type":"markdown","source":"We create a 'grid' to test for most optimal tuning parameters.\n\nPrepering parameters for Light Gradient Boosting."},{"metadata":{"trusted":true,"_uuid":"0e2087c7b490c3276da24b5bdad2fc2ff92aa361"},"cell_type":"code","source":"#inspired by https://www.kaggle.com/garethjns/microsoft-lightgbm-with-parameter-tuning-0-823\n\nimport lightgbm as lgb\n\n#set params\n\nlgb_params = {'max_depth' : 6,\n          'objective': 'regression',\n          'num_leaves': 55,\n          'max_bin' : 60,\n          'learning_rate': 0.05,\n          'subsample': 1,\n          'subsample_freq': 1,\n          'colsample_bytree': 0.8,\n          'subsample_for_bin': 200,\n          'reg_alpha': 1,\n          'reg_lambda': 1,    \n          'min_child_weight': 1,\n          'min_child_samples': 12,\n          'min_split_gain': 0.5,\n          'scale_pos_weight': 1,\n          'metric' : 'rmse'}\n\ngridParams = {\n    'learning_rate': [0.05],\n    'n_estimators': [40],\n    'num_leaves': [8,16,32,64],\n    'objective' : ['regression'],\n    'random_state' : [501], # Updated from 'seed'\n    'colsample_bytree' : [0.6, 0.8],\n    'subsample' : [0.7,0.75],\n    'reg_alpha' : [1,1.5],\n    'reg_lambda' : [1,1.2,1.5],\n    }\n\n\nmdl = lgb.LGBMRegressor(boosting_type= 'gbdt',\n          n_jobs = 3, # Updated from 'nthread'\n          silent = True,\n          **lgb_params)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b340bccc2c938592e3676fc576abc439fe3cee66"},"cell_type":"markdown","source":"Setting up GridSearch:"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"86118cfd161dfb5e5f864492bb1fabb7e5707f62"},"cell_type":"code","source":"# =============================================================================\n# #grid search\n# =============================================================================\n\n#dropping 10 features with low importance from first run\nto_drop = ['hist_authorized_flag_max', 'hist_category_1_min', 'hist_authorized_flag_min', \n            'feature_3_1', 'feature_1_3', 'hist_category_1_max', 'feature_3_0', 'feature_1_2', \n            'hist_category_2_max', 'feature_1_0']\n\ntarget = train['target']\n\ntrain = train.drop(to_drop + ['target'], axis=1)\ntest = test.drop(to_drop + ['target'], axis=1)\n\nfrom sklearn.model_selection import GridSearchCV \nfrom sklearn.metrics import make_scorer\nfrom sklearn.metrics import mean_squared_error\n\n       \nmean_squared_error_ = make_scorer(mean_squared_error, greater_is_better=False)\n\n#setting up GridSearchCV\ngrid = GridSearchCV(mdl, gridParams,\n                    verbose=4,\n                    cv=4,\n                    n_jobs=2,  #paralel computing\n                    scoring=mean_squared_error_) \n\ngrid.fit(train, target)\n    \n# Print the best parameters found\n\nprint('mean square error for best params {}'.format(np.abs(grid.best_score_)))\nprint('root mean square error for best params {}'.format(np.sqrt(np.abs(grid.best_score_))))\n\nlgb_params['colsample_bytree'] = grid.best_params_['colsample_bytree']\nlgb_params['learning_rate'] = grid.best_params_['learning_rate']\n# params['max_bin'] = grid.best_params_['max_bin']\nlgb_params['num_leaves'] = grid.best_params_['num_leaves']\nlgb_params['reg_alpha'] = grid.best_params_['reg_alpha']\nlgb_params['reg_lambda'] = grid.best_params_['reg_lambda']\nlgb_params['subsample'] = grid.best_params_['subsample']\n\n\nprint('Fitting with params: ')\nprint(lgb_params)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41f6ae36a608a2045e24d7916d5b1892f6c07519"},"cell_type":"markdown","source":"<a id=\"5\"></a>\n## 5. Light Gradient Boosting and Feature Importance"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5353638f6f71efcf473910ad71db9f03c0cd0eba"},"cell_type":"code","source":"# =============================================================================\n# Model Build / Best Params\n# =============================================================================\n\n# inspiration from https://www.kaggle.com/chocozzz/simple-data-exploration-with-python-lb-3-764\n\nfrom sklearn.model_selection import KFold\n\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=1987)\n\noof_lgb = np.zeros(len(train))\npredictions_lgb = np.zeros(len(test))\n\nfeatures_lgb = list(train.columns)\nfeature_importance_df_lgb = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train)):\n    trn_data = lgb.Dataset(train.iloc[trn_idx], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train.iloc[val_idx], label=target.iloc[val_idx])\n\n    print(\"LGB \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    clf = lgb.train(lgb_params, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=1000, early_stopping_rounds = 60)\n    oof_lgb[val_idx] = clf.predict(train.iloc[val_idx], num_iteration=clf.best_iteration)\n\n    fold_importance_df_lgb = pd.DataFrame()\n    fold_importance_df_lgb[\"feature\"] = features_lgb\n    fold_importance_df_lgb[\"importance\"] = clf.feature_importance()\n    fold_importance_df_lgb[\"fold\"] = fold_ + 1\n    feature_importance_df_lgb = pd.concat([feature_importance_df_lgb, fold_importance_df_lgb], axis=0)\n    predictions_lgb += clf.predict(test, num_iteration=clf.best_iteration) / FOLDs.n_splits\n\nprint(np.sqrt(mean_squared_error(oof_lgb, target)))\n\n# =============================================================================\n# inspecting var importance\n# =============================================================================\n\n\ncols = (feature_importance_df_lgb[[\"feature\", \"importance\"]]\n        .groupby(\"feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df_lgb.loc[feature_importance_df_lgb.feature.isin(cols)]\n\nplt.figure(figsize=(14,14))\nsns.barplot(x=\"importance\",\n            y=\"feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2092793695125e0b448bb23a622bc01ed37dca13"},"cell_type":"markdown","source":"<a id=\"6\"></a>\n## 6. Submission"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a9c1d9bb6291a6991d8ac1744c84a8d0b718745b"},"cell_type":"code","source":"# =============================================================================\n# best_features dataframe\n# =============================================================================\n\n#best_features = best_features.drop('fold', axis=1).groupby(\"feature\").mean().sort_values(by=\"importance\", ascending=True)\n\n#print(best_features.index[:10])\n\n# =============================================================================\n# submission\n# =============================================================================\n\nsub_df = pd.read_csv('../input/sample_submission.csv')\nsub_df[\"target\"] = predictions_lgb \n#sub_df.to_csv(\"submission_lgb.csv\", index=False)\n\nprint(sub_df.head(10))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}