{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Elo world \n\nIn this kernel, I build a LGBM model that aggregates the `new_merchant_transactions.csv` and `historical_transactions.csv` tables to the main train table. New features are built by successive grouping on`card_id` and `month_lag`, in order to recover some information from the time serie.\n\nDuring the competition, I took into account the enlightments provided by others kernels, and included a few features that appeared to be important. In particular, I closely looked at the following kernels (ordered by release time):\n1.  [You're Going to Want More Categories [LB 3.737] by Peter Hurford](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737)\n2. [EloDA with Feature Engineering and Stacking by Bojan Tunguz](https://www.kaggle.com/tunguz/eloda-with-feature-engineering-and-stacking)\n3. [A Closer Look at Date Variables by Robin Denz](https://www.kaggle.com/denzo123/a-closer-look-at-date-variables)\n4. [LGB + FE (LB 3.707) by Konrad Banachewicz](https://www.kaggle.com/konradb/lgb-fe-lb-3-707)\n5. [My first kernel (3.699) by Chau Ngoc Huynh](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/)\n\n## Notebook  Content\n1. [Loading the data](#1)\n1. [Feature engineering](#2)\n1. [Training the model](#3)\n1. [Feature importance](#4)\n1. [Submission](#5)","metadata":{"_uuid":"332fa118684da9a9278d2caf565620686fd73a68","trusted":true}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold\nimport warnings\nimport time\nimport sys\nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc\nfrom sklearn.metrics import mean_squared_error\nwarnings.simplefilter(action='ignore', category=FutureWarning)\npd.set_option('display.max_columns', 500)","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2023-11-07T04:54:21.856781Z","iopub.status.busy":"2023-11-07T04:54:21.856460Z","iopub.status.idle":"2023-11-07T04:54:22.969803Z","shell.execute_reply":"2023-11-07T04:54:22.968888Z","shell.execute_reply.started":"2023-11-07T04:54:21.856707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n## 1. Loading the data\n\nFirst, we load the `new_merchant_transactions.csv` and `historical_transactions.csv`. In practice, these two files contain the same variables and the difference between the two tables only concern the position with respect to a reference date.  Also, booleans features are made numeric:","metadata":{"_uuid":"6a059a8dcf9d93a650f1ccaa8e2bfa3e087219f3"}},{"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","metadata":{"_kg_hide-input":true,"_uuid":"7a7877dff5c337c09ca111cdcbf527362c9217c7","execution":{"iopub.execute_input":"2023-11-07T04:54:22.971552Z","iopub.status.busy":"2023-11-07T04:54:22.971135Z","iopub.status.idle":"2023-11-07T04:54:22.981574Z","shell.execute_reply":"2023-11-07T04:54:22.980702Z","shell.execute_reply.started":"2023-11-07T04:54:22.971464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fillna_with_mode(df):\n    \"\"\"\n    Fill NaN values in a DataFrame with the respective column mode (assuming mode is always available).\n\n    Parameters:\n    - df: pandas DataFrame\n\n    Returns:\n    - DataFrame with NaN values filled with mode\n    \"\"\"\n    return df.apply(lambda col: col.fillna(col.mode()[0]), axis=0)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"e = pd.read_excel('../input/Data Dictionary.xlsx', sheet_name='train')\ne","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:54:22.983284Z","iopub.status.busy":"2023-11-07T04:54:22.982712Z","iopub.status.idle":"2023-11-07T04:54:23.261903Z","shell.execute_reply":"2023-11-07T04:54:23.261209Z","shell.execute_reply.started":"2023-11-07T04:54:22.983230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_transactions = pd.read_csv('../input/new_merchant_transactions.csv',\n                               parse_dates=['purchase_date'])\n\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv',\n                                      parse_dates=['purchase_date'])\n\n\ndef binarize(df):\n    for col in ['authorized_flag', 'category_1']:\n        df[col] = df[col].map({'Y':1, 'N':0})\n    return df\n\nhistorical_transactions = binarize(historical_transactions)\nhistorical_transactions = fillna_with_mode(historical_transactions )\nnew_transactions = binarize(new_transactions)\nnew_transactions = fillna_with_mode(new_transactions )","metadata":{"_uuid":"b6b5a505bfecabd70c6dac0ee30386d074b03953","execution":{"iopub.execute_input":"2023-11-07T04:54:23.263460Z","iopub.status.busy":"2023-11-07T04:54:23.263078Z","iopub.status.idle":"2023-11-07T04:56:23.621663Z","shell.execute_reply":"2023-11-07T04:56:23.621098Z","shell.execute_reply.started":"2023-11-07T04:54:23.263413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We then load the main files, formatting the dates and extracting the target:","metadata":{"_uuid":"156acd1e83dfba2c2896561b75a6a5a7782cab1d"}},{"cell_type":"code","source":"def read_data(input_file):\n    df = pd.read_csv(input_file)\n    df = fillna_with_mode(df)\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'], format='%Y-%m')\n    reference_date = pd.to_datetime('2018-11-27', format='%Y-%m-%d')\n    df['elapsed_time'] = (reference_date - df['first_active_month']).dt.days\n    return df\n#_________________________________________\ntrain = read_data('../input/train.csv')\ntest = read_data('../input/test.csv')\n","metadata":{"_uuid":"a24cf99ad6b0785b5af2b101e06b400e26360d1e","execution":{"iopub.execute_input":"2023-11-07T04:56:23.622839Z","iopub.status.busy":"2023-11-07T04:56:23.622542Z","iopub.status.idle":"2023-11-07T04:56:25.240089Z","shell.execute_reply":"2023-11-07T04:56:25.239116Z","shell.execute_reply.started":"2023-11-07T04:56:23.622806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:25.241473Z","iopub.status.busy":"2023-11-07T04:56:25.241227Z","iopub.status.idle":"2023-11-07T04:56:25.255232Z","shell.execute_reply":"2023-11-07T04:56:25.253870Z","shell.execute_reply.started":"2023-11-07T04:56:25.241422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**EDA**","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=[4,3])\nplt.bar([0, 1], [train.shape[0], test.shape[0]], edgecolor=[0.2]*3, color=(1,0,0,0.5))\nplt.xticks([0,1], ['train rows', 'test rows'], fontsize=13)\nplt.title('Number of rows in train.csv and test.csv', fontsize=15)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:25.256865Z","iopub.status.busy":"2023-11-07T04:56:25.256626Z","iopub.status.idle":"2023-11-07T04:56:25.378629Z","shell.execute_reply":"2023-11-07T04:56:25.377688Z","shell.execute_reply.started":"2023-11-07T04:56:25.256810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:25.380150Z","iopub.status.busy":"2023-11-07T04:56:25.379875Z","iopub.status.idle":"2023-11-07T04:56:25.434685Z","shell.execute_reply":"2023-11-07T04:56:25.432194Z","shell.execute_reply.started":"2023-11-07T04:56:25.380099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.describe()","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:25.437139Z","iopub.status.busy":"2023-11-07T04:56:25.436728Z","iopub.status.idle":"2023-11-07T04:56:25.514478Z","shell.execute_reply":"2023-11-07T04:56:25.513354Z","shell.execute_reply.started":"2023-11-07T04:56:25.437079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,3, figsize = (16, 6));\ntrain['feature_1'].value_counts().sort_index().plot(kind='bar', ax=ax[0], title='feature_1');\ntrain['feature_2'].value_counts().sort_index().plot(kind='bar', ax=ax[1], title='feature_2');\ntrain['feature_3'].value_counts().sort_index().plot(kind='bar', ax=ax[2], title='feature_3');\nplt.suptitle('Counts of categiories for features');","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:25.515912Z","iopub.status.busy":"2023-11-07T04:56:25.515657Z","iopub.status.idle":"2023-11-07T04:56:25.956019Z","shell.execute_reply":"2023-11-07T04:56:25.955236Z","shell.execute_reply.started":"2023-11-07T04:56:25.515862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.hist(train.target.values, bins=200)\nplt.title('Histogram target counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()\nprint(f'There are {train.loc[train.target  == min(train.target)].shape[0]} samples in the target equal to {min(train.target)}.')","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:25.957584Z","iopub.status.busy":"2023-11-07T04:56:25.957350Z","iopub.status.idle":"2023-11-07T04:56:26.407574Z","shell.execute_reply":"2023-11-07T04:56:26.406930Z","shell.execute_reply.started":"2023-11-07T04:56:25.957547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplt.subplot(131)\nsns.kdeplot(x ='target',data = train,hue = 'feature_1',palette='rainbow')\nplt.title('distribution of target over different categories of Feature_1')\nplt.subplot(132)\nsns.kdeplot(x ='target',data = train,hue = 'feature_2',palette='Dark2_r')\nplt.title('distribution of target over different categories of Feature_2')\nplt.subplot(133)\nsns.kdeplot(x ='target',data = train,hue = 'feature_3',palette='Dark2_r')\nplt.title('distribution of target over different categories of Feature_3')\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Group the DataFrame by 'first_active_month' and count the number of rows in each group\ndate_counts_train = train.groupby('first_active_month').size()\ndate_counts_test = test.groupby('first_active_month').size()\n\n# Create a time series plot\nplt.figure(figsize=(12, 6))\ndate_counts_train.plot(label='train')\ndate_counts_test.plot(label='test')\nplt.title('Total Rows on Date')\nplt.xlabel('Date')\nplt.ylabel('Total Rows')\nplt.grid(True)\nplt.legend()\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = train['first_active_month'].dt.month.value_counts().sort_index()\nax = temp.plot()\n_ = ax.set_xticklabels(range(-1, 15, 2))\n_ = ax.set_title(\"Distribution across months\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.lineplot(x = train['first_active_month'], y= train['target'])\nplt.title(\"Distribution of target over first_active_month\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_data = pd.merge(historical_transactions, train[['card_id','target']], how = 'outer', on = 'card_id')\n\n# merging target value of card_id for each transction in new_merchants_transactions Data\nnew_data =  pd.merge(new_transactions, train[['card_id','target']], how = 'outer', on = 'card_id')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_data.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_data.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_numeric_df = historical_data.select_dtypes(include=['number'])\nnew_numeric_df = new_data.select_dtypes(include=['number'])\n\nfig, axes = plt.subplots(1, 2, figsize=(20, 8))\n\n# First subplot\nhistorical_correlation_matrix = historical_numeric_df .corr()\nsns.heatmap(historical_correlation_matrix, annot=True, cmap='coolwarm', fmt=\".2f\", linewidths=.5, ax=axes[0])\naxes[0].set_title('Historical Transactions Correlation Matrix')\n\n# Second subplot\nnew_correlation_matrix = new_numeric_df.corr()\nsns.heatmap(new_correlation_matrix, annot=True, cmap='coolwarm', fmt=\".2f\", linewidths=.5, ax=axes[1])\naxes[1].set_title('New Transactions Correlation Matrix')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#delete variables for EDA to save momories\ndel historical_numeric_df,new_numeric_df,fig,axes ,historical_correlation_matrix,new_correlation_matrix,historical_data,new_data ,temp,ax,date_counts_train,date_counts_test","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## Feature engineering\nFollowing [Chau Ngoc Huynh's kernel](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/), I add the following features:","metadata":{"_uuid":"b3c2a5853f29c314dca418e1ad8420358fbf74fe"}},{"cell_type":"code","source":"e = pd.read_excel('../input/Data_Dictionary.xlsx', sheet_name='history')\ne","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:26.409265Z","iopub.status.busy":"2023-11-07T04:56:26.408812Z","iopub.status.idle":"2023-11-07T04:56:26.432139Z","shell.execute_reply":"2023-11-07T04:56:26.430755Z","shell.execute_reply.started":"2023-11-07T04:56:26.409216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = train['target']\ndel train['target']","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:26.435979Z","iopub.status.busy":"2023-11-07T04:56:26.435687Z","iopub.status.idle":"2023-11-07T04:56:26.441704Z","shell.execute_reply":"2023-11-07T04:56:26.440738Z","shell.execute_reply.started":"2023-11-07T04:56:26.435916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datetime.datetime.today()\ndatetime.datetime(2018, 11, 27)","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:26.443450Z","iopub.status.busy":"2023-11-07T04:56:26.443102Z","iopub.status.idle":"2023-11-07T04:56:26.456167Z","shell.execute_reply":"2023-11-07T04:56:26.455030Z","shell.execute_reply.started":"2023-11-07T04:56:26.443371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_transactions['month_diff'] = ((datetime.datetime(2018, 11, 27) - historical_transactions['purchase_date']).dt.days)//30\nhistorical_transactions['month_diff'] += historical_transactions['month_lag']\n\nnew_transactions['month_diff'] = ((datetime.datetime(2018, 11, 27) - new_transactions['purchase_date']).dt.days)//30\nnew_transactions['month_diff'] += new_transactions['month_lag']\n\n#hoidays source https://www.officeholidays.com/countries/brazil/2017 , https://www.officeholidays.com/countries/brazil/2018 \n#including Christmas Eve, New Year's Eve and Black Awareness Day\nholiday_dates = pd.to_datetime( ['01-01-2017','27-02-2017','28-02-2017','01-02-2017','14-04-2017','21-04-2017','01-05-2017','15-06-2017','07-09-2017',\n                    '12-10-2017','01-11-2017','02-11-2017','15-11-2017','20-11-2017','24-12-2017','25-12-2017','31-12-2017',\n                    '01-01-2018', '12-02-2018', '13-02-2018','30-03-2018','21-04-2018','01-05-2018','31-05-2018','07-09-2018',\n                    '12-10-2018','02-11-2018','15-11-2018','20-11-2018','24-12-2018','25-12-2018','31-12-2018'] ,format=\"%d-%m-%Y\")\n\nhistorical_transactions[\"holiday_purchase\"] = np.where(historical_transactions[\"purchase_date\"].isin(holiday_dates) , 1, 0)\nnew_transactions[\"holiday_purchase\"] = np.where(new_transactions[\"purchase_date\"].isin(holiday_dates) , 1, 0)","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:26.458137Z","iopub.status.busy":"2023-11-07T04:56:26.457768Z","iopub.status.idle":"2023-11-07T04:56:34.161131Z","shell.execute_reply":"2023-11-07T04:56:34.159834Z","shell.execute_reply.started":"2023-11-07T04:56:26.458073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"f787e3f81121b236528b59d0e0a723bcef510e4b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_transactions.head()","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:56:34.164753Z","iopub.status.busy":"2023-11-07T04:56:34.163831Z","iopub.status.idle":"2023-11-07T04:56:34.199225Z","shell.execute_reply":"2023-11-07T04:56:34.197542Z","shell.execute_reply.started":"2023-11-07T04:56:34.164696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_transactions[:5]","metadata":{"_uuid":"9db35828b8cb62fbe95a9d30e63b30b49966a304","execution":{"iopub.execute_input":"2023-11-07T04:56:34.201663Z","iopub.status.busy":"2023-11-07T04:56:34.201232Z","iopub.status.idle":"2023-11-07T04:56:34.245473Z","shell.execute_reply":"2023-11-07T04:56:34.243255Z","shell.execute_reply.started":"2023-11-07T04:56:34.201551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#Convert categorical variable into dummy/indicator variables.\nhistorical_transactions = pd.get_dummies(historical_transactions, columns=['category_2', 'category_3'])\nnew_transactions = pd.get_dummies(new_transactions, columns=['category_2', 'category_3'])\n\n\nhistorical_transactions = reduce_mem_usage(historical_transactions)\nnew_transactions = reduce_mem_usage(new_transactions)\n\n\nagg_fun = {'authorized_flag': ['mean']}\nauth_mean = historical_transactions.groupby(['card_id']).agg(agg_fun)\nauth_mean.columns = ['_'.join(col).strip() for col in auth_mean.columns.values]\nauth_mean.reset_index(inplace=True)\n\n\nauthorized_transactions = historical_transactions[historical_transactions['authorized_flag'] == 1]\nhistorical_transactions = historical_transactions[historical_transactions['authorized_flag'] == 0]","metadata":{"_uuid":"b1da58e20da9fbb6d200119b242c168ae8fc5843","execution":{"iopub.execute_input":"2023-11-07T04:56:34.247452Z","iopub.status.busy":"2023-11-07T04:56:34.247208Z","iopub.status.idle":"2023-11-07T04:57:16.698161Z","shell.execute_reply":"2023-11-07T04:57:16.697008Z","shell.execute_reply.started":"2023-11-07T04:56:34.247400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"historical_transactions[:5] ","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:57:16.701014Z","iopub.status.busy":"2023-11-07T04:57:16.700652Z","iopub.status.idle":"2023-11-07T04:57:16.724180Z","shell.execute_reply":"2023-11-07T04:57:16.722447Z","shell.execute_reply.started":"2023-11-07T04:57:16.700940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"authorized_transactions.head()","metadata":{"execution":{"iopub.execute_input":"2023-11-07T04:57:16.725892Z","iopub.status.busy":"2023-11-07T04:57:16.725654Z","iopub.status.idle":"2023-11-07T04:57:16.753758Z","shell.execute_reply":"2023-11-07T04:57:16.752742Z","shell.execute_reply.started":"2023-11-07T04:57:16.725858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, following [Robin Denz](https://www.kaggle.com/denzo123/a-closer-look-at-date-variables) analysis, I define a few dates features:","metadata":{"_uuid":"73e0936c181e4cec16da05d6987cd96b99ed87d1"}},{"cell_type":"code","source":"historical_transactions['purchase_month'] = historical_transactions['purchase_date'].dt.month\nauthorized_transactions['purchase_month'] = authorized_transactions['purchase_date'].dt.month\nnew_transactions['purchase_month'] = new_transactions['purchase_date'].dt.month","metadata":{"_uuid":"76c41ff3af21c7bfa9194f873f6f47c5849eb4d8","execution":{"iopub.execute_input":"2023-11-07T04:57:16.755484Z","iopub.status.busy":"2023-11-07T04:57:16.755215Z","iopub.status.idle":"2023-11-07T04:57:18.288072Z","shell.execute_reply":"2023-11-07T04:57:18.286862Z","shell.execute_reply.started":"2023-11-07T04:57:16.755413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then I define two functions that aggregate the info contained in these two tables. The first function aggregates the function by grouping on `card_id`:","metadata":{"_uuid":"7493b32cb783d6fb6afdad60964eb41c9e42c2e3"}},{"cell_type":"code","source":"def aggregate_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n    'category_1': ['count','sum', 'mean'],\n    'category_2_1.0': ['count','sum','mean'],\n    'category_2_2.0': ['count','sum','mean'],\n    'category_2_3.0': ['count','sum','mean'],\n    'category_2_4.0': ['count','sum','mean'],\n    'category_2_5.0': ['count','sum','mean'],\n    'category_3_A': ['count','sum','mean'],\n    'category_3_B': ['count','sum','mean'],\n    'category_3_C': ['count','sum','mean'],\n    'merchant_id': ['nunique'],\n    'merchant_category_id': ['nunique'],\n    'state_id': ['nunique'],\n    'city_id': ['nunique'],\n    'subsector_id': ['nunique'],\n    'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n    'installments': ['sum', 'mean', 'max', 'min', 'std'],\n    'purchase_month': ['mean', 'max', 'min', 'std'],\n    'purchase_date': [np.ptp, 'min', 'max'],\n    'month_lag': ['mean', 'max', 'min', 'std'],\n    'month_diff': ['mean','max', 'min', 'std'],\n    }\n    \n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['_'.join(col).strip() for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history\n\n","metadata":{"_uuid":"82c25c7cd0d075195fb7bd63211c66f0dac9304b","execution":{"iopub.execute_input":"2023-11-07T04:57:18.291204Z","iopub.status.busy":"2023-11-07T04:57:18.290944Z","iopub.status.idle":"2023-11-07T04:57:18.308175Z","shell.execute_reply":"2023-11-07T04:57:18.306892Z","shell.execute_reply.started":"2023-11-07T04:57:18.291162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = aggregate_transactions(historical_transactions)\nhistory.columns = ['hist_' + c if c != 'card_id' else c for c in history.columns]\nhistory[:5]","metadata":{"_uuid":"44ea0524d0af0001c5ea57a6593be35e2402d0b6","execution":{"iopub.execute_input":"2023-11-07T04:57:18.310237Z","iopub.status.busy":"2023-11-07T04:57:18.310007Z","iopub.status.idle":"2023-11-07T04:58:00.041439Z","shell.execute_reply":"2023-11-07T04:58:00.040098Z","shell.execute_reply.started":"2023-11-07T04:57:18.310204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"authorized = aggregate_transactions(authorized_transactions)\nauthorized.columns = ['auth_' + c if c != 'card_id' else c for c in authorized.columns]\nauthorized[:5]","metadata":{"_uuid":"be09e1a7d4f307beb59c18bf61261239877da4e3","execution":{"iopub.execute_input":"2023-11-07T04:58:00.043222Z","iopub.status.busy":"2023-11-07T04:58:00.042917Z","iopub.status.idle":"2023-11-07T05:00:52.719526Z","shell.execute_reply":"2023-11-07T05:00:52.718009Z","shell.execute_reply.started":"2023-11-07T04:58:00.043174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new = aggregate_transactions(new_transactions)\nnew.columns = ['new_' + c if c != 'card_id' else c for c in new.columns]\nnew[:5]","metadata":{"_uuid":"78b87a7863b62fa3758ef3167504d3ad8c279f3d","execution":{"iopub.execute_input":"2023-11-07T05:00:52.721478Z","iopub.status.busy":"2023-11-07T05:00:52.721171Z","iopub.status.idle":"2023-11-07T05:01:36.680631Z","shell.execute_reply":"2023-11-07T05:01:36.679910Z","shell.execute_reply.started":"2023-11-07T05:00:52.721425Z"},"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The second function first aggregates on the two variables `card_id` and `month_lag`. Then a second grouping is performed to aggregate over time:","metadata":{"_uuid":"843a929ddea5e317f882c58c3b69f5e5a4476a38"}},{"cell_type":"code","source":"def aggregate_per_month(history):\n    grouped = history.groupby(['card_id', 'month_lag'])\n\n    agg_func = {\n            'purchase_amount': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            'installments': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            }\n\n    intermediate_group = grouped.agg(agg_func)\n    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n    intermediate_group.reset_index(inplace=True)\n\n    final_group = intermediate_group.groupby('card_id').agg(['mean', 'std'])\n    final_group.columns = ['_'.join(col).strip() for col in final_group.columns.values]\n    final_group.reset_index(inplace=True)\n    \n    return final_group\n#___________________________________________________________\nfinal_group =  aggregate_per_month(authorized_transactions) \nfinal_group[:10]\n","metadata":{"_uuid":"f210acd074326ea74c1b9316eec3136f4ab73855","execution":{"iopub.execute_input":"2023-11-07T05:01:36.682521Z","iopub.status.busy":"2023-11-07T05:01:36.682047Z","iopub.status.idle":"2023-11-07T05:01:49.711583Z","shell.execute_reply":"2023-11-07T05:01:49.709671Z","shell.execute_reply.started":"2023-11-07T05:01:36.682472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def successive_aggregates(df, field1, field2):\n    t = df.groupby(['card_id', field1])[field2].mean()\n    u = pd.DataFrame(t).reset_index().groupby('card_id')[field2].agg(['mean', 'min', 'max', 'std'])\n    u.columns = [field1 + '_' + field2 + '_' + col for col in u.columns.values]\n    u.reset_index(inplace=True)\n    return u\n\n","metadata":{"_uuid":"95bfafc86d846204d7defa8015d9d2b936a72844","execution":{"iopub.execute_input":"2023-11-07T05:01:49.713429Z","iopub.status.busy":"2023-11-07T05:01:49.713142Z","iopub.status.idle":"2023-11-07T05:01:49.720327Z","shell.execute_reply":"2023-11-07T05:01:49.719246Z","shell.execute_reply.started":"2023-11-07T05:01:49.713372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"additional_fields = successive_aggregates(new_transactions, 'category_1', 'purchase_amount')\nadditional_fields = additional_fields.merge(successive_aggregates(new_transactions, 'installments', 'purchase_amount'),\n                                            on = 'card_id', how='left')\nadditional_fields = additional_fields.merge(successive_aggregates(new_transactions, 'city_id', 'purchase_amount'),\n                                            on = 'card_id', how='left')\nadditional_fields = additional_fields.merge(successive_aggregates(new_transactions, 'category_1', 'installments'),\n                                            on = 'card_id', how='left')\n\n","metadata":{"_uuid":"5a85689aef5f58b6af8d8617b0d8b97ee5e85a43","execution":{"iopub.execute_input":"2023-11-07T05:01:49.721839Z","iopub.status.busy":"2023-11-07T05:01:49.721558Z","iopub.status.idle":"2023-11-07T05:01:55.535743Z","shell.execute_reply":"2023-11-07T05:01:55.534319Z","shell.execute_reply.started":"2023-11-07T05:01:49.721780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Grouping month lag data (auth month-13, auth month-12 and etc.) for count purchases, and sum adjusted purchase_amount, std, min, max)\n\ndef aggregate_date(history):\n\n    grouped = history.groupby(['card_id','purchase_date'])\n\n    agg_func = {\n            'purchase_amount': ['sum', 'mean', 'min', 'max', 'std'],\n            'installments': ['sum', 'mean', 'min', 'max', 'std'],\n             }\n \n\n    intermediate_group = grouped.agg(agg_func)\n    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n\n    intermediate_group.reset_index(inplace=True)\n    final_group = intermediate_group.groupby('card_id').agg(['mean','std'])\n    final_group.columns = ['_'.join(col).strip() for col in final_group.columns.values]\n    final_group.reset_index(inplace=True)\n    \n    return final_group\n#___________________________________________________________\nx =  aggregate_date(historical_transactions) \nx[:10]\n","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:01:55.537271Z","iopub.status.busy":"2023-11-07T05:01:55.537087Z","iopub.status.idle":"2023-11-07T05:02:02.232789Z","shell.execute_reply":"2023-11-07T05:02:02.231754Z","shell.execute_reply.started":"2023-11-07T05:01:55.537240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y =  aggregate_date(authorized_transactions) ","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:02:02.234283Z","iopub.status.busy":"2023-11-07T05:02:02.234018Z","iopub.status.idle":"2023-11-07T05:03:20.492665Z","shell.execute_reply":"2023-11-07T05:03:20.491227Z","shell.execute_reply.started":"2023-11-07T05:02:02.234226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 3. Training the model\nWe now train the model with the features we previously defined. A first step consists in merging all the dataframes:","metadata":{"_uuid":"dcf0403c10b8ee817257a51e5edf8f1f81fcd593"}},{"cell_type":"code","source":"train = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')\n\ntrain = pd.merge(train, x, on='card_id', how='left')\ntest = pd.merge(test,x, on='card_id', how='left')\n\ntrain = pd.merge(train, y, on='card_id', how='left')\ntest = pd.merge(test,y, on='card_id', how='left')\n\ntrain = pd.merge(train, authorized, on='card_id', how='left')\ntest = pd.merge(test, authorized, on='card_id', how='left')\n\ntrain = pd.merge(train, new, on='card_id', how='left')\ntest = pd.merge(test, new, on='card_id', how='left')\n\ntrain = pd.merge(train, final_group, on='card_id', how='left')\ntest = pd.merge(test, final_group, on='card_id', how='left')\n\ntrain = pd.merge(train, auth_mean, on='card_id', how='left')\ntest = pd.merge(test, auth_mean, on='card_id', how='left')\n\ntrain = pd.merge(train, additional_fields, on='card_id', how='left')\ntest = pd.merge(test, additional_fields, on='card_id', how='left')","metadata":{"_uuid":"d27264c6c7f0af6af7ba141177bfd38f7a68dec3","execution":{"iopub.execute_input":"2023-11-07T05:03:20.494559Z","iopub.status.busy":"2023-11-07T05:03:20.494288Z","iopub.status.idle":"2023-11-07T05:03:31.050659Z","shell.execute_reply":"2023-11-07T05:03:31.048892Z","shell.execute_reply.started":"2023-11-07T05:03:20.494509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"f48c931c326bf1a7ef21614f6e10b2b538059ab5","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['transactions_ratio'] = train['new_transactions_count'] / train['hist_transactions_count']\ntest['transactions_ratio'] = test['new_transactions_count'] / test['hist_transactions_count']","metadata":{"_uuid":"6a96a54f9f074363064723cdadc6d1f93a5a4f73","execution":{"iopub.execute_input":"2023-11-07T05:03:31.053052Z","iopub.status.busy":"2023-11-07T05:03:31.052652Z","iopub.status.idle":"2023-11-07T05:03:31.530409Z","shell.execute_reply":"2023-11-07T05:03:31.528754Z","shell.execute_reply.started":"2023-11-07T05:03:31.052971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"d93d5177600acd75fc4b3b2a35df6208850aeab5","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"and to define the features we want to keep to train the model. For that purpose, I use the results obtained in the [Selecting features kernel](https://www.kaggle.com/fabiendaniel/selecting-features/notebook):","metadata":{"_uuid":"e06ddc8d941360b1b3e32bc59221e5f6fe729763"}},{"cell_type":"code","source":"\nfeatures = [c for c in train.columns if c not in ['card_id', 'first_active_month']]\ncols_remove = ['card_id', 'first_active_month']\nfor col in cols_remove:  \n    del train[col]\n\ndel test['first_active_month']","metadata":{"_uuid":"d121b2bf5c6adf83495edc63c08662899fd39e0b","execution":{"iopub.execute_input":"2023-11-07T05:03:31.532185Z","iopub.status.busy":"2023-11-07T05:03:31.531889Z","iopub.status.idle":"2023-11-07T05:03:31.541125Z","shell.execute_reply":"2023-11-07T05:03:31.540004Z","shell.execute_reply.started":"2023-11-07T05:03:31.532141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"numerical = [var for var in train.columns if train[var].dtype!='O']\nprint('There are {} numerical variables'.format(len(numerical)))\n\n# Get discrete var\ndiscrete= []\nfor var in numerical:\n    if len(train[var].unique())<8:\n        discrete.append(var)\n        \nprint('There are {} discrete variables'.format(len(discrete)))\n\ncontinuous = [var for var in numerical if var not in discrete and var not in ['card_id', 'first_active_month','target']]\nprint('There are {} continuous variables'.format(len(continuous)))","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:03:31.542521Z","iopub.status.busy":"2023-11-07T05:03:31.542245Z","iopub.status.idle":"2023-11-07T05:03:32.634322Z","shell.execute_reply":"2023-11-07T05:03:32.633462Z","shell.execute_reply.started":"2023-11-07T05:03:31.542464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_feats =discrete","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:03:32.635550Z","iopub.status.busy":"2023-11-07T05:03:32.635336Z","iopub.status.idle":"2023-11-07T05:03:32.638731Z","shell.execute_reply":"2023-11-07T05:03:32.638237Z","shell.execute_reply.started":"2023-11-07T05:03:32.635502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in features:\n    if col in continuous:\n        train[col] = train[col].fillna(value = train[col].mean())\n        test[col] = test[col].fillna(value = test[col].mean())\n","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:03:32.639946Z","iopub.status.busy":"2023-11-07T05:03:32.639658Z","iopub.status.idle":"2023-11-07T05:03:33.348395Z","shell.execute_reply":"2023-11-07T05:03:33.346848Z","shell.execute_reply.started":"2023-11-07T05:03:32.639892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.impute import SimpleImputer\nimport numpy as np\ntrain_null = train.columns[train.isnull().any()].tolist()\ntest_null = test.columns[test.isnull().any()].tolist()\n\nimp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')\ntrain[train_null] =imp.fit_transform(train[train_null])\ntest[test_null] =imp.fit_transform(test[test_null])","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:03:33.350471Z","iopub.status.busy":"2023-11-07T05:03:33.350085Z","iopub.status.idle":"2023-11-07T05:03:50.640911Z","shell.execute_reply":"2023-11-07T05:03:50.639190Z","shell.execute_reply.started":"2023-11-07T05:03:33.350371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We then set the hyperparameters of the LGBM model, these parameters are obtained by an [bayesian optimization done in another kernel](https://www.kaggle.com/fabiendaniel/hyperparameter-tuning/edit):","metadata":{"_uuid":"e67399b25b3886e306b274069ab240af7d060397"}},{"cell_type":"code","source":"from sklearn.model_selection import RepeatedKFold,StratifiedKFold\nfrom sklearn.linear_model import Ridge, BayesianRidge\nfolds = RepeatedKFold(n_splits=5, n_repeats=2, random_state=2023)\noof_ridge = np.zeros(train.shape[0])\npredictions_ridge = np.zeros(test.shape[0])\n\ntst_data = test.copy()\ntst_data.fillna((tst_data.mean()), inplace=True)\n\ntst_data = tst_data[features].values\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train, target)):\n    print(\"fold n°{}\".format(fold_+1))\n    trn_data, trn_y = train.iloc[trn_idx][features], target.iloc[trn_idx].values\n    val_data, val_y = train.iloc[val_idx][features], target.iloc[val_idx].values\n    \n    trn_data.fillna((trn_data.mean()), inplace=True)\n    val_data.fillna((val_data.mean()), inplace=True)\n    \n    trn_data = trn_data.values\n    val_data = val_data.values\n\n    clf = BayesianRidge()\n    clf.fit(trn_data, trn_y)\n    \n    oof_ridge[val_idx] = clf.predict(val_data)\n    predictions_ridge += clf.predict(tst_data) / 10\n\nnp.save('oof_ridge', oof_ridge)\nnp.save('predictions_ridge', predictions_ridge)\nnp.sqrt(mean_squared_error(target.values, oof_ridge))","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:03:50.643477Z","iopub.status.busy":"2023-11-07T05:03:50.643150Z","iopub.status.idle":"2023-11-07T05:05:44.539621Z","shell.execute_reply":"2023-11-07T05:05:44.538933Z","shell.execute_reply.started":"2023-11-07T05:03:50.643417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[](http://)","metadata":{}},{"cell_type":"code","source":"import xgboost as xgb\n\nxgb_params = {'eta': 0.005, 'max_depth': 8, 'subsample': 0.8, 'colsample_bytree': 0.8, 'alpha':0.1,\n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True, 'random_state':folds}\n\n\nfolds = RepeatedKFold(n_splits=3, n_repeats=1, random_state=2023)\noof_xgb = np.zeros(len(train))\npredictions_xgb = np.zeros(len(test))\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print(\"fold n°{}\".format(fold_ + 1))\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx][features], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 6000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx][features]), ntree_limit=xgb_model.best_ntree_limit+50)\n    predictions_xgb += xgb_model.predict(xgb.DMatrix(test[features]), ntree_limit=xgb_model.best_ntree_limit+50) / 10\n\nnp.save('oof_xgb', oof_xgb)\nnp.save('predictions_xgb', predictions_xgb)\nprint(\"RMSE : \",np.sqrt(mean_squared_error(target.values, oof_xgb)))\n\n","metadata":{"execution":{"iopub.execute_input":"2023-11-07T05:05:44.544721Z","iopub.status.busy":"2023-11-07T05:05:44.543021Z","iopub.status.idle":"2023-11-07T06:02:01.022837Z","shell.execute_reply":"2023-11-07T06:02:01.021767Z","shell.execute_reply.started":"2023-11-07T05:05:44.544673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"param = {'num_leaves': 111,\n         'min_data_in_leaf': 149, \n         'objective':'regression',\n         'max_depth': 9,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.7522,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.7083 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.2634,\n         \"random_state\": 133,\n         \"verbosity\": -1}","metadata":{"_uuid":"c610f51450145101732f4e9ed3247f2a9fa0b091","execution":{"iopub.execute_input":"2023-11-07T06:02:01.024668Z","iopub.status.busy":"2023-11-07T06:02:01.024348Z","iopub.status.idle":"2023-11-07T06:02:01.031713Z","shell.execute_reply":"2023-11-07T06:02:01.030698Z","shell.execute_reply.started":"2023-11-07T06:02:01.024605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We now train the model. Here, we use a standard KFold split of the dataset in order to validate the results and to stop the training. Interstingly, during the writing of this kernel, the model was enriched adding new features, which improved the CV score. **The variations observed on the CV were found to be quite similar to the variations on the LB**: it seems that the current competition won't give us headaches to define the correct validation scheme:","metadata":{"_uuid":"b7a7377cd7401f2cbd13ea707fbc2a2bebe229a6"}},{"cell_type":"code","source":"folds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n                           label=target.iloc[trn_idx],\n                           categorical_feature=categorical_feats\n                          )\n    val_data = lgb.Dataset(train.iloc[val_idx][features],\n                           label=target.iloc[val_idx],\n                           categorical_feature=categorical_feats\n                          )\n\n    num_round = 10000\n    clf = lgb.train(param,\n                    trn_data,\n                    num_round,\n                    valid_sets = [trn_data, val_data],\n                    verbose_eval=100,\n                    early_stopping_rounds = 200)\n    \n    oof[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(test[features], num_iteration=clf.best_iteration) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof, target)**0.5))","metadata":{"_uuid":"b23550968ef3fb49ae0fcc5533551d702297c990","execution":{"iopub.execute_input":"2023-11-07T06:02:01.033216Z","iopub.status.busy":"2023-11-07T06:02:01.032979Z","iopub.status.idle":"2023-11-07T06:15:43.873991Z","shell.execute_reply":"2023-11-07T06:15:43.872925Z","shell.execute_reply.started":"2023-11-07T06:02:01.033169Z"},"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from catboost import CatBoostRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.neighbors import KNeighborsRegressor\nfrom sklearn.linear_model import  Lasso\nfrom mlxtend.regressor import StackingCVRegressor\nfrom sklearn.model_selection import cross_val_predict\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.metrics import make_scorer\nfrom tqdm import tqdm\n\ncatboost_model = CatBoostRegressor(iterations=130)\nrf_model = RandomForestRegressor(max_depth = 20,n_estimators=20, min_samples_leaf=20, min_samples_split=20)\nknn_model = KNeighborsRegressor(n_neighbors=15, weights='distance')\nridge_model = Ridge(alpha=75.0)\nlasso_model = Lasso(alpha=0.7)\n\n","metadata":{"execution":{"iopub.execute_input":"2023-11-07T06:15:43.875538Z","iopub.status.busy":"2023-11-07T06:15:43.875323Z","iopub.status.idle":"2023-11-07T06:15:44.110932Z","shell.execute_reply":"2023-11-07T06:15:44.109837Z","shell.execute_reply.started":"2023-11-07T06:15:43.875495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models = [\n     ('CatBoost', catboost_model),\n     ('RandomForest', rf_model),\n     ('Lasso', lasso_model),\n     ('KNN', knn_model),\n]\n\ndef rmse_fun(predicted, actual):\n    return mean_squared_error(predicted, actual)**0.5\n\nrmse = make_scorer(rmse_fun, greater_is_better=False)\nscores = []\nfor _,model in tqdm(models):\n    print(model)\n    score = -1* cross_val_score(model, train.values, target.values,cv=4, scoring=rmse).mean()\n    print(score)\n    scores.append(score)","metadata":{"execution":{"iopub.execute_input":"2023-11-07T06:15:44.112867Z","iopub.status.busy":"2023-11-07T06:15:44.112562Z","iopub.status.idle":"2023-11-07T07:01:44.265629Z","shell.execute_reply":"2023-11-07T07:01:44.264319Z","shell.execute_reply.started":"2023-11-07T06:15:44.112821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## 4. Feature importance\nFinally, we can have a look at the features that were used by the model:","metadata":{"_uuid":"9a1f0a866e05f8a450960e2d787a641fc35991a1"}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"cols = (feature_importance_df[[\"feature\", \"importance\"]]\n        .groupby(\"feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:50].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.feature.isin(cols)]\n\nplt.figure(figsize=(14,25))\nsns.barplot(x=\"importance\",\n            y=\"feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","metadata":{"_uuid":"d479e83032448481b40c216264a039cacdb2f9a1","execution":{"iopub.execute_input":"2023-11-07T07:01:44.267392Z","iopub.status.busy":"2023-11-07T07:01:44.267120Z","iopub.status.idle":"2023-11-07T07:01:46.468120Z","shell.execute_reply":"2023-11-07T07:01:46.467417Z","shell.execute_reply.started":"2023-11-07T07:01:44.267339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_stack = np.vstack([oof_ridge, oof, oof_xgb]).transpose()\ntest_stack = np.vstack([predictions_ridge, predictions,predictions_xgb]).transpose()\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_stack = np.zeros(train_stack.shape[0])\npredictions_stack = np.zeros(test_stack.shape[0])\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_stack, target)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data, trn_y = train_stack[trn_idx], target.iloc[trn_idx].values\n    val_data, val_y = train_stack[val_idx], target.iloc[val_idx].values\n\n    print(\"-\" * 10 + \"Ridge Regression\" + str(fold_) + \"-\" * 10)\n    clf = Ridge(alpha=75)\n    clf.fit(trn_data, trn_y)\n    \n    oof_stack[val_idx] = clf.predict(val_data)\n    predictions_stack += clf.predict(test_stack) / 5\n","metadata":{"execution":{"iopub.execute_input":"2023-11-07T07:01:46.469745Z","iopub.status.busy":"2023-11-07T07:01:46.469348Z","iopub.status.idle":"2023-11-07T07:01:46.626276Z","shell.execute_reply":"2023-11-07T07:01:46.625569Z","shell.execute_reply.started":"2023-11-07T07:01:46.469696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models = [\n    'CatBoost',\n    'RandomForest',\n    'Lasso',\n    'KNN',\n    'LGB',\n    'Ridge',\n    'XGB',\n    'Stack']\nr = [np.sqrt(mean_squared_error(oof, target)),np.sqrt(mean_squared_error(oof_ridge, target)),np.sqrt(mean_squared_error(oof_xgb, target)),np.sqrt(mean_squared_error(oof_stack, target))]\nscores+= r                                                                                         \n                                                                                        ","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2023-11-07T07:01:46.628197Z","iopub.status.busy":"2023-11-07T07:01:46.627678Z","iopub.status.idle":"2023-11-07T07:01:46.637087Z","shell.execute_reply":"2023-11-07T07:01:46.636427Z","shell.execute_reply.started":"2023-11-07T07:01:46.628143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.DataFrame({ 'Model': models, 'Error': scores })\ndf","metadata":{"execution":{"iopub.execute_input":"2023-11-07T07:01:46.638727Z","iopub.status.busy":"2023-11-07T07:01:46.638319Z","iopub.status.idle":"2023-11-07T07:01:46.656320Z","shell.execute_reply":"2023-11-07T07:01:46.655273Z","shell.execute_reply.started":"2023-11-07T07:01:46.638683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\nplt.barh(df['Model'], df['Error'])\nplt.title('Model vs RMSE')\nplt.ylabel('Model')\nplt.xlabel('Error')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-11-07T07:01:46.658058Z","iopub.status.busy":"2023-11-07T07:01:46.657702Z","iopub.status.idle":"2023-11-07T07:01:46.820014Z","shell.execute_reply":"2023-11-07T07:01:46.819107Z","shell.execute_reply.started":"2023-11-07T07:01:46.657989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test1 = test.copy()\ndel test1['card_id']","metadata":{"execution":{"iopub.execute_input":"2023-11-07T07:01:46.821580Z","iopub.status.busy":"2023-11-07T07:01:46.821276Z","iopub.status.idle":"2023-11-07T07:01:46.964750Z","shell.execute_reply":"2023-11-07T07:01:46.963287Z","shell.execute_reply.started":"2023-11-07T07:01:46.821517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ncatboost_model.fit(train.values, target.values) \npredictions_cat = catboost_model.predict(test1.values)","metadata":{"execution":{"iopub.execute_input":"2023-11-07T07:01:46.967139Z","iopub.status.busy":"2023-11-07T07:01:46.966730Z","iopub.status.idle":"2023-11-07T07:02:28.287537Z","shell.execute_reply":"2023-11-07T07:02:28.286101Z","shell.execute_reply.started":"2023-11-07T07:01:46.967063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## 5. Submission\nNow, we just need to prepare the submission file:","metadata":{"_uuid":"003ae1b1bd522b1b0d992ff220ed98d2a6d7477a"}},{"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions\nsub_df.to_csv(\"submit.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_xgb\nsub_df.to_csv(\"submit1.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_stack\nsub_df.to_csv(\"submit2.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_cat\nsub_df.to_csv(\"submit3.csv\", index=False)","metadata":{"_uuid":"82d5ac08a13603b2a66c59d98584c4b709daee2d","execution":{"iopub.execute_input":"2023-11-07T07:02:28.288940Z","iopub.status.busy":"2023-11-07T07:02:28.288717Z","iopub.status.idle":"2023-11-07T07:02:29.400742Z","shell.execute_reply":"2023-11-07T07:02:29.399795Z","shell.execute_reply.started":"2023-11-07T07:02:28.288891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#[1050]\ttraining's rmse: 3.47865\tvalid_1's rmse: 3.61203     CV score: 3.65527 ","metadata":{"_uuid":"c1c6c1d40dbb2137ddd6404670b5f3dd3d381207","execution":{"iopub.execute_input":"2023-11-07T07:02:29.402177Z","iopub.status.busy":"2023-11-07T07:02:29.401819Z","iopub.status.idle":"2023-11-07T07:02:29.406256Z","shell.execute_reply":"2023-11-07T07:02:29.404664Z","shell.execute_reply.started":"2023-11-07T07:02:29.402111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"3900aa85af1f706015c7e8ed219395348f9852ae","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"e155324a5fd4a7e07b06a5e00b7fb4a02afc38d9","trusted":true},"execution_count":null,"outputs":[]}]}