{"cells":[{"metadata":{"_uuid":"28a5acac45c8d82058dbec99d999e2749e1d4b8e"},"cell_type":"markdown","source":"Inspired by **@raddar** with his great kernel ([https://www.kaggle.com/raddar/target-true-meaning-revealed/)] reveals that the 'target' is actually a ratio between historical purchase behaviour and further purchase behavour.\n\nTherefore, as the transction data literally includes the time series info of each customer purchasing behaviour (from -13 to 2 month in 'month_lag' columns ), I try to do some feature engineering to use time series model (LSTM) and convolutional networks (SqueezeNet) for predicting the target.\n\nThe logic is to conduct feature engineering to describe the customer purchasing behaviour by month_lag.\neg. month -13,  [purchase mean, std, mode], [city_code], [ratio of feature 1]  etc\n\nAnd because the preprocessing takes times to finish, I make my dataset public for better use in this kernel.\n \n This is my first Kaggle Kernel and hope you enjoy it! Feel free to leave any comment to make it better.\n Thanks.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport scipy.stats as sts\nimport datetime\nimport gc\nfrom keras.models import Model\nfrom keras.layers import LSTM,Dropout,Input,Dense,BatchNormalization,Reshape,concatenate\nfrom keras.layers import Conv2D, MaxPooling2D,GlobalAveragePooling2D,AveragePooling2D,ConvLSTM2D, Concatenate,MaxPool2D,GRU\nimport gc\nfrom keras.callbacks import EarlyStopping\nfrom keras.optimizers import Adam, RMSprop\nimport keras.backend as K\nfrom sklearn.model_selection import StratifiedKFold\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"86233f65f9ce97eaa17dd262e64bba5bdb4a5876"},"cell_type":"markdown","source":"With some exploration, we can find there's lots of 'purchase_amount' is too large to be a reasonable data point in transaction but can also find out those transactions did not valid as their 'authorized_flag' is false.\n\nTherefore, the following preprocessing considered only the entry with 'authorized_flag' == Y and also introduce an ratio of vaid transaction per month.\n\n\nDetail columns for each customer feature for each month:\n\n\n\ncols = ['_authorized_ratio',  \n'_city_id_mean', \n'_city_id_count', \n'_category_1_mean',\n '_installments_mean',\n '_installments_std',\n '_installments_min',\n '_installments_max',\n '_installments_median', \n '_merchant_category_id_nunique',\n '_merchant_id_nunique',         \n '_purchase_amount_mean',\n '_purchase_amount_std',          \n '_purchase_amount_min',\n '_purchase_amount_max',\n '_purchase_amount_median',\n '_state_id_nunique',\n '_state_id_mode',\n '_subsector_id_nunique',\n '_subsector_id_mode',\n '__category_20_mean',\n '__category_21_mean',\n '__category_22_mean',\n '__category_23_mean',\n '__category_24_mean',\n '__category_25_mean',\n '__category_3A_mean',\n '__category_3B_mean',\n '__category_3C_mean']\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"\n'''df_timefeature = pd.read_csv('historical_transactions.csv')\n#df_timefeature = pd.read_csv('new_merchant_transactions.csv')\ndf_timefeature.authorized_flag = df_timefeature.authorized_flag.map({'Y':1,'N':0})\ndf_timefeature.category_1 = df_timefeature.category_1.map({'Y':1,'N':0})\ndf_timefeature.category_2 = df_timefeature.category_2.fillna('0').astype('int')\ndf_timefeature.category_3 = df_timefeature.category_3.fillna('D')\ndf_timefeature.merchant_id = df_timefeature.merchant_id.fillna('NO')\ndf_timefeature = df_timefeature[['authorized_flag', 'card_id', 'city_id', \n                                  'category_1', 'installments','category_3', 'merchant_category_id', 'merchant_id', 'month_lag',\n                                  'purchase_amount', 'purchase_date', 'category_2', 'state_id',\n                                  'subsector_id']] \n\nfor ft in ['category_2','category_3']:\n    ft_ = pd.get_dummies(df_timefeature[ft])\n    ft_.columns = [ft.join('_%s'%(cols)) for cols in ft_.columns]\n    df_timefeature = df_timefeature.join(ft_, how = 'left')\n    \ndf_timefeature = df_timefeature[['authorized_flag', 'card_id', 'city_id', 'category_1', 'installments',\n       'merchant_category_id', 'merchant_id', 'month_lag',\n       'purchase_amount', 'state_id','subsector_id', \n       '_category_20', '_category_21', '_category_22',\n       '_category_23', '_category_24', '_category_25', '_category_3A',\n       '_category_3B', '_category_3C']]\naggs = {'city_id':['mean','count']\n , 'category_1':['mean']\n , 'installments':['mean','std','min','max','median']\n , 'merchant_category_id':['nunique']\n , 'merchant_id':['nunique']\n ,'purchase_amount':['mean','std','min','max','median']\n , 'state_id':['nunique', lambda x: x.value_counts().index[0]]\n ,'subsector_id':['nunique', lambda x: x.value_counts().index[0]]\n ,'_category_20':['mean']\n ,'_category_21':['mean']\n ,'_category_22':['mean']\n ,'_category_23':['mean']\n ,'_category_24':['mean']\n ,'_category_25':['mean']\n ,'_category_3A':['mean']\n ,'_category_3B':['mean']\n ,'_category_3C':['mean']}\n\n\ndf_timefeature = df_timefeature.groupby(['card_id','month_lag']).agg(aggs)\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68be40d4dc275d3eb089e2d7dd5a579c537f61a6"},"cell_type":"code","source":"df_all= pd.read_pickle('../input/elo-timeseries-fe/df_all_Time.pkl')\ndf_all = df_all.set_index('card_id')\ndf_all = df_all.reset_index()\n\ndrop_columns = ['Train','card_id','target','outlier']\ncate_columns = ['_feature_12','_feature_13','_feature_14','_feature_15','_feature_22','_feature_23','_feature_30',\n               'active_year','active_month']\ndf_all.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"869256fabf6a05166a02ba7580c84c6c0df86581"},"cell_type":"markdown","source":"As the outliers matter in this competition , I construct Train and Valid set both includes outliers using random sampling with the same ratio of outliers."},{"metadata":{"trusted":true,"_uuid":"d17fafdf0c0b1145b06e2a422eea886eb1fbbd5b"},"cell_type":"code","source":"train = df_all[df_all.Train == True]\ny = train.target.values\n#y = 10**(train['target'].values*np.log10(2))\n#y_std = y.std()\n#y_mean = y.mean()\n#y = (y-y_mean)/y_std\n\n\nval_idx = train[train.outlier == 1].sample(450).index.tolist()\nval_idx = train[train.outlier == 0].sample(int(train[train.outlier == 0].shape[0]/5)).index.tolist()+val_idx\n\ntrn_idx = train[~train.index.isin(val_idx)].index.tolist()\n\nx_train = train[train.index.isin(trn_idx)].drop(drop_columns,axis = 1)\nx_train = dict(cate = x_train[cate_columns].values,\n              mem = x_train.drop(cate_columns,axis = 1).values)\n#x_train = x_train.drop(cate_columns,axis = 1).values\n#y_train =  train[train.index.isin(trn_idx)].target.values\ny_train =  y[trn_idx]\n\n\n\nx_val = train[train.index.isin(val_idx)].drop(drop_columns,axis = 1)\nx_val = dict(cate = x_val[cate_columns].values,\n              mem = x_val.drop(cate_columns,axis = 1).values)\n#x_val = x_val.drop(cate_columns,axis = 1).values\n#y_val=  train[train.index.isin(val_idx)].target.values\ny_val =  y[val_idx]\n\nprint('done!')\n#print('Trn shape: ', x_train.shape)\n#print('Val shape: ', x_val.shape)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de56fdf372a1927617a8045142230a7871f8a123"},"cell_type":"code","source":"plt.plot(np.sort(y),label= 'y')\nplt.plot(np.sort(np.array(y_train).reshape(-1,)), label = 'y_train')\nplt.plot(np.sort(np.array(y_val).reshape(-1,)), label = 'y_val')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e292dc8aae43b2566063af9a057a46335b589776"},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.distplot(np.sort(y),label= 'y')\nsns.distplot(np.sort(np.array(y_train).reshape(-1,)), label = 'y_train')\nsns.distplot(np.sort(np.array(y_val).reshape(-1,)), label = 'y_val')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3958cf626029acd0d4eb658f7fcaadf1f8f1245b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bcd263b461e9541d99211ccd0b8e357105b57055"},"cell_type":"markdown","source":"I did some experiences on the following four models and in this kernel I used the model with one LSTM layer for time series data  and dense layers for categorical data."},{"metadata":{"trusted":true,"_uuid":"f384e50d484fecf854ab6a8faebf42d5b49e86f5"},"cell_type":"code","source":"np.random.seed(0)\n\ndef root_mean_squared_error(y_true, y_pred):\n        return K.sqrt(K.mean((K.square(y_pred - y_true))))\n    \ndef get_model(cate_shape, lstm_shape):\n    Category = Input(cate_shape, name = 'cate')\n    Lstm_Input = Input(lstm_shape, name = 'mem')\n    Norm_1 = BatchNormalization()(Lstm_Input)\n    C_D1 = Dense(64, activation= 'relu')(Category)\n    C_D1 = Dense(32, activation= 'relu')(C_D1)\n    C_D1 = Dense(16, activation= 'relu')(C_D1)\n    \n    #R_1 = Reshape((-1,464))(Lstm_Input)\n    R_1 = Reshape((-1,29))(Lstm_Input)\n    L_1 = LSTM(256)(R_1)\n    \n    main = concatenate([C_D1,L_1])\n    main = Dense(128, activation= 'relu')(main)\n    main = Dense(64, activation= 'relu')(main)\n    main = Dense(32, activation= 'relu')(main)\n    out = Dense(1, activation= 'linear')(main)\n    \n    model = Model([Category,Lstm_Input],out)\n    model.summary()\n    \n    return model\n\n\n\nprint('Finish Construct Models')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e39b38760bb7c678644e97617c2e371d84638c4f"},"cell_type":"code","source":"def get_model_lstmonly(lstm_shape):\n    #Category = Input(cate_shape, name = 'cate')\n    Lstm_Input = Input(lstm_shape, name = 'mem')\n    Norm_1 = BatchNormalization()(Lstm_Input)\n    R_1 = Reshape((-1,29))(Norm_1)\n    L_1 = LSTM(256)(R_1)\n    #L_1 = LSTM(64)(L_1)\n    \n    #main = concatenate([C_D1,L_1])\n    main = Dense(128, activation= 'relu')(L_1)\n    main = Dense(64, activation= 'relu')(main)\n    main = Dense(32, activation= 'relu')(main)\n    #main = Dense(16, activation= 'relu')(main)\n    out = Dense(1, activation= 'linear')(main)\n    \n    model = Model(Lstm_Input,out)\n    model.summary()\n    \n    return model\n\n\ndef SqueezeNet(lstm_shape):\n    def fire(np_filters, name=\"fire\"):\n        def layer(x):\n            sq_filters, ex1_filters, ex2_filters = np_filters\n            squeeze = Conv2D(sq_filters, (1, 1), activation='relu', padding='same', name=name + \"/squeeze1x1\")(x)\n            expand1 = Conv2D(ex1_filters, (1, 1), activation='relu', padding='same', name=name + \"/expand1x1\")(squeeze)\n            expand2 = Conv2D(ex2_filters, (3, 3), activation='relu', padding='same', name=name + \"/expand3x3\")(squeeze)\n            out = Concatenate(axis=-1, name=name+'/concat')([expand1, expand2])\n            return out\n        return layer\n    Lstm_Input = Input(lstm_shape, name = 'mem')\n    R_1 = Reshape((-1,29,1))(Lstm_Input)\n    Norm_1 = BatchNormalization()(R_1)\n    Conv1 = Conv2D(128, (3, 3), padding='same', activation='relu', name='conv1')(Norm_1)\n    Max_pool1 =MaxPool2D((2, 2), strides=(2, 2), name='max_pool1')(Conv1) #(10,10)\n    fire1 = fire((32,64,64),name = 'fire1')(Max_pool1)\n    fire2 = fire((32,64,64),name = 'fire2')(fire1)\n    Max_pool2 = MaxPool2D((2, 2), strides=(2, 2), name='max_pool2')(fire2)#(5,5)\n    Drop_1 = Dropout(0.25, name = 'dropout_1')(Max_pool2)\n    fire3 = fire((32,128,128),name = 'fire3')(Drop_1)\n    fire4 = fire((32,128,128),name = 'fire4')(fire3)\n    Max_pool3 = MaxPool2D((2, 2), strides=(2, 2), name='max_pool3')(fire4)#(2,2)\n    Drop_2 = Dropout(0.25, name = 'dropout_2')(Max_pool3)\n    fire5 = fire((16,32,32),name = 'fire5')(Drop_2)\n    fire6 = fire((16,64,64),name = 'fire6')(fire5)\n    fire7 = fire((16,128,128),name = 'fire7')(fire6)\n    Gl_avg_pooling = GlobalAveragePooling2D(name = 'gl_avg')(fire7)\n    D_1= Dense(32,activation='linear', name='predictions')(Gl_avg_pooling)\n    D_1= Dense(16,activation='linear', name='predictions')(D_1)\n    Output_layer = Dense(1,activation='linear', name='predictions')(D_1)\n    model = Model(Lstm_Input,Output_layer, name = 'SqueezeNet')\n    model.summary()\n        \n\n    return model\n\ndef get_model_cnnlstm(lstm_shape):\n     \n    #Category = Input(cate_shape, name = 'cate')\n    Lstm_Input = Input(lstm_shape, name = 'mem')\n    Norm_1 = BatchNormalization()(Lstm_Input)\n    R_1 = Reshape((-1,29,1))(Norm_1)\n    \n    C_1 = Conv2D(32,(3,3),strides=(1,1),data_format=\"channels_last\",padding='same',activation='relu')(R_1)\n    C_1 = Conv2D(32,(3,3),strides=(1,1),data_format=\"channels_last\",padding='same',activation='relu')(C_1)\n    C_1 = Conv2D(32,(3,3),strides=(1,1),data_format=\"channels_last\",padding='same',activation='relu')(C_1)\n    Pool = MaxPool2D(pool_size=(2,2),strides=(2,2))(C_1)\n    R_2 = Reshape(((8, 14*32)))(Pool)\n    L_1 = LSTM(256)(R_2)\n    #L_1 = LSTM(64)(L_1)\n    \n    #main = concatenate([C_D1,L_1])\n    main = Dense(32, activation= 'relu')(L_1)\n    main = Dense(16, activation= 'relu')(main)\n    out = Dense(1, activation= 'linear')(main)\n    \n    model = Model(Lstm_Input,out)\n    model.summary()\n    \n    return model\n\n\ndef get_model_Squeezelstm(lstm_shape):\n    def fire(np_filters, name=\"fire\"):\n        def layer(x):\n            sq_filters, ex1_filters, ex2_filters = np_filters\n            squeeze = Conv2D(sq_filters, (1, 1), activation='relu', padding='same', name=name + \"/squeeze1x1\")(x)\n            expand1 = Conv2D(ex1_filters, (1, 1), activation='relu', padding='same', name=name + \"/expand1x1\")(squeeze)\n            expand2 = Conv2D(ex2_filters, (3, 3), activation='relu', padding='same', name=name + \"/expand3x3\")(squeeze)\n            out = Concatenate(axis=-1, name=name+'/concat')([expand1, expand2])\n            return out\n        return layer\n    #Category = Input(cate_shape, name = 'cate')\n    Lstm_Input = Input(lstm_shape, name = 'mem')\n    #Norm_1 = BatchNormalization()(Lstm_Input)\n    R_1 = Reshape((-1,29,1))(Lstm_Input)\n    Conv1 = Conv2D(64, (3, 3), padding='same', activation='relu', name='conv1')(R_1)\n    Max_pool1 =MaxPool2D((2, 2), strides=(2, 2), name='max_pool1')(Conv1) #(10,10)\n    fire1 = fire((16,32,32),name = 'fire1')(Max_pool1)\n    fire2 = fire((16,64,64),name = 'fire2')(fire1)\n    fire3 = fire((16,64,64),name = 'fire3')(fire2)\n    Max_pool2 = MaxPool2D((2, 2), strides=(2, 2), name='max_pool2')(fire3)#(5,5)\n    R_2 = Reshape(((4, 7*128)))(Max_pool2)\n    L_1 = LSTM(128)(R_2)\n    main = Dense(32, activation= 'relu')(L_1)\n    main = Dense(16, activation= 'relu')(main)\n    main = Dense(8, activation= 'relu')(main)\n    out = Dense(1, activation= 'linear')(main)\n    \n    model = Model(Lstm_Input,out)\n    model.summary()\n    \n    return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e70540ee7c0ea239cd1fe8200f2e83a5e04ecd5b"},"cell_type":"code","source":"model = get_model((9,), (464,))\n#model = get_model_lstmonly((464,))\n#model = SqueezeNet((464,))\n#model =  get_model_cnnlstm((464,))\n#model = get_model_Squeezelstm((464,))\n\nearly_stopping = EarlyStopping(monitor='val_loss', patience=5)\nmodel.compile(loss='mse', optimizer=Adam(),  metrics=[root_mean_squared_error])\nhistory = model.fit(x_train, y_train, epochs=20,verbose=1, \n                    batch_size=32,validation_data =(x_val,y_val), \n                    callbacks=[early_stopping])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac38a66224976cd74bca53574f725b4da0447039"},"cell_type":"code","source":"#%%\n\nplt.plot(history.history['root_mean_squared_error'])\nplt.plot(history.history['val_root_mean_squared_error'])\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2d74d5b131c08beaf8c6210745ef1e0420a3ac7"},"cell_type":"code","source":"\nsubmission = pd.read_csv('../input/elo-merchant-category-recommendation/sample_submission.csv')\nx_ts = df_all[df_all.Train == False]\nx_ts = x_ts.drop(drop_columns,axis = 1)\nx_ts = dict(cate = x_ts[cate_columns].values,\n              mem = x_ts.drop(cate_columns,axis = 1).values)\n\ny_pre = model.predict(x_ts)\n#y_pre = (y_pre*y_std)+y_mean\nsubmission['target'] = y_pre\nsubmission.to_csv('LSTM.csv',index = False)\n \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea9391b1c83f91b0b25c411b4ec76924b7e4bedc"},"cell_type":"code","source":"plt.plot(np.sort(y),label= 'y')\nplt.plot(np.sort(np.array(y_pre).reshape(-1,)), label = 'y_pre')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bf9094163b71bccb680f53f77f86eb174a1016e0"},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.distplot(np.sort(y),label= 'y')\nsns.distplot(np.sort(np.array(y_pre).reshape(-1,)), label = 'y_pre')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"afb53a2a39f3b8d0203c8288cb334ba79185dfee"},"cell_type":"markdown","source":"The result might not be very outstanding as other boosting method and great kernels because of the sample of outliers are too less for a DL model. Some upsampling method might be required for further improment. \n\nHowever, it's a good practice to use different method other than boosting for this competiton.\nI'll keep improve the the kernel in the following days.\n\nThanks for reading my kernel! Hope you enjoy it.\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}