{"cells":[{"metadata":{"_uuid":"4343a60cf5439f4fc3f4b3967cfd83e5824f3df3"},"cell_type":"markdown","source":"I started by developing ther **baseline model** using the features available in the training dataset- achieved score of around 3.98, implying that the features in training dataset were not very useful. Next, I did some **manual feature engineering by aggregating purchase amount** feature from historical and new merchant transactions getting an accuracy of roughly 3.8. \n\nThen I decided to try **featuretools for automated feature engineering and use Permutation Importance to select the relevant features.**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport math\nfrom matplotlib import pyplot as plt\nimport gc\nprint(os.listdir(\"../input\"))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train=pd.read_csv('../input/train.csv')\ntest= pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d11db125279432de4863c6081703ca08ca85961"},"cell_type":"markdown","source":"Now comes the interesting part of running featuretools to get automated feature engineering. This library might take a little while to understand completely but an excellent and elegant solution. While using featuretools for automated feature engineering, below are some of the challenges I encountered:\n\n1. Most of the features are categorical in datasets but marked as numerical, so manually I had to change the variable type of many features to Id and Categorical to avoid getting rubbish features.\n\n2. The biggest problem by far at least in this competition was the large size of historical transactions file. Everytime, I was running out of RAM and the kernel died. Tackling this, I had to take some redundant steps but yes they worked.\n    * Performed featuretools feature engineering twice on train and test dataset that I could have done in 1 go by combining the two sets.\n    * Performed operations on each dataset and then immediately deleted them to retain precious memory.\n    * Turned off GPU to increase RAM from 14GB to 17GB."},{"metadata":{"trusted":true,"_uuid":"25fd40464e0fd101e7e484952ca1f8177fbefc5e"},"cell_type":"code","source":"import featuretools as ft\n\nes= ft.EntitySet(id= 'train')\n\nvariable_types={'feature_1':ft.variable_types.Categorical,'feature_2':ft.variable_types.Categorical, \n                'feature_3':ft.variable_types.Categorical, 'target':ft.variable_types.Id}\nes= es.entity_from_dataframe(entity_id='train',dataframe= train, index= 'card_id',variable_types= variable_types)\n\nmerchants= pd.read_csv('../input/merchants.csv')\nmerchants= merchants.drop_duplicates(['merchant_id'])\nvariable_types={'merchant_group_id':ft.variable_types.Id, 'merchant_category_id':ft.variable_types.Id, \n                'subsector_id':ft.variable_types.Categorical,\n               'city_id':ft.variable_types.Id,'state_id':ft.variable_types.Id,'category_2':ft.variable_types.Categorical}\nes= es.entity_from_dataframe(entity_id='merchants', dataframe= merchants, index='merchant_id',variable_types= variable_types)\ndel merchants\ngc.collect()\n\nnew_merchant_transactions= pd.read_csv('../input/new_merchant_transactions.csv')\nnew_merchant_transactions= new_merchant_transactions[(new_merchant_transactions['card_id']).isin(train['card_id'])]\nvariable_types={'card_id':ft.variable_types.Id, 'state_id':ft.variable_types.Id, 'city_id':ft.variable_types.Id,\n               'merchant_category_id':ft.variable_types.Id,'merchant_id':ft.variable_types.Id,'subsector_id':ft.variable_types.Id,\n              'category_2':ft.variable_types.Categorical}\nes= es.entity_from_dataframe(entity_id='new_merchant_transactions',dataframe= new_merchant_transactions, make_index= True,\n                             index='new_merchants_id',time_index='purchase_date',variable_types= variable_types)\ndel new_merchant_transactions\ngc.collect()\n\nhistorical_transactions= pd.read_csv('../input/historical_transactions.csv')\nhistorical_transactions= historical_transactions[(historical_transactions['card_id']).isin(train['card_id'])]\nvariable_types={'card_id':ft.variable_types.Id, 'state_id':ft.variable_types.Id, 'city_id':ft.variable_types.Id,\n               'merchant_category_id':ft.variable_types.Id,'merchant_id':ft.variable_types.Id,'subsector_id':ft.variable_types.Id,\n              'category_2':ft.variable_types.Categorical}\nes= es.entity_from_dataframe(entity_id='historical_transactions',dataframe= historical_transactions, make_index= True,index='historical_id',\n                             time_index='purchase_date',variable_types= variable_types)\ndel historical_transactions\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"262de2b257c3e1a9600b2e789da7c93301f11afa"},"cell_type":"markdown","source":"Adding relationships to the entity set is so natural in featuretools just like anybody would be dealing with databases."},{"metadata":{"trusted":true,"_uuid":"93fbde0cbf35fcad6da7764fd3657f0efdebb088"},"cell_type":"code","source":"r_cards_historical= ft.Relationship(es['train']['card_id'],es['historical_transactions']['card_id'])\nes= es.add_relationship(r_cards_historical)\n\nr_cards_new_merchants= ft.Relationship(es['train']['card_id'],es['new_merchant_transactions']['card_id'])\nes= es.add_relationship(r_cards_new_merchants)\n\nr_merchants_historical= ft.Relationship(es['merchants']['merchant_id'], es['historical_transactions']['merchant_id'])\nes= es.add_relationship(r_merchants_historical)\n\nr_merchants_new_merchants= ft.Relationship(es['merchants']['merchant_id'],es['new_merchant_transactions']['merchant_id'])\nes= es.add_relationship(r_merchants_new_merchants)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"013cbf5ebf15e0f5fb11bb69aa96e5b572c28a59"},"cell_type":"markdown","source":"Comes the final part of feature engineering- DFS or Deep Feature Synthesis where different features get stacked up (that's mean the term deep comes from). I have kept the maximum depth as 1. With more depth, you get transformation of transformations that can be useful at times but are very hard to interpret, so avoiding that."},{"metadata":{"trusted":true,"_uuid":"5e4bd64b14558d73fccd5edea972406e0bd619c5"},"cell_type":"code","source":"features_train, feature_names_train= ft.dfs(entityset= es, target_entity= 'train', max_depth= 1)\n\ndel es\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"593d7d0ca9ea3b901e2fe6d992e7505d38b7fba5"},"cell_type":"code","source":"features_train","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"866f65696905fef60945d21281dae3948b8856d9"},"cell_type":"markdown","source":"Same featuretools engineering for the test set. As I mentioned above, due to memory constraints I had to do this twice by deleting the intermediary files in between."},{"metadata":{"trusted":true,"_uuid":"91195bb7481f0a87f018def6333a5bcd87b6d7cc"},"cell_type":"code","source":"import featuretools as ft\n\nes= ft.EntitySet(id= 'test')\n\nvariable_types={'feature_1':ft.variable_types.Categorical,'feature_2':ft.variable_types.Categorical, \n                'feature_3':ft.variable_types.Categorical}\nes= es.entity_from_dataframe(entity_id='test',dataframe= test, index= 'card_id',variable_types= variable_types)\n\nmerchants= pd.read_csv('../input/merchants.csv')\nmerchants= merchants.drop_duplicates(['merchant_id'])\nvariable_types={'merchant_group_id':ft.variable_types.Id, 'merchant_category_id':ft.variable_types.Id, \n                'subsector_id':ft.variable_types.Categorical,\n               'city_id':ft.variable_types.Id,'state_id':ft.variable_types.Id,'category_2':ft.variable_types.Categorical}\nes= es.entity_from_dataframe(entity_id='merchants', dataframe= merchants, index='merchant_id',variable_types= variable_types)\ndel merchants\ngc.collect()\n\nnew_merchant_transactions= pd.read_csv('../input/new_merchant_transactions.csv')\nnew_merchant_transactions= new_merchant_transactions[(new_merchant_transactions['card_id']).isin(test['card_id'])]\nvariable_types={'card_id':ft.variable_types.Id, 'state_id':ft.variable_types.Id, 'city_id':ft.variable_types.Id,\n               'merchant_category_id':ft.variable_types.Id,'merchant_id':ft.variable_types.Id,'subsector_id':ft.variable_types.Id,\n              'category_2':ft.variable_types.Categorical}\nes= es.entity_from_dataframe(entity_id='new_merchant_transactions',dataframe= new_merchant_transactions, make_index= True,\n                             index='new_merchants_id',time_index='purchase_date',variable_types= variable_types)\ndel new_merchant_transactions\ngc.collect()\n\nhistorical_transactions= pd.read_csv('../input/historical_transactions.csv')\nhistorical_transactions= historical_transactions[(historical_transactions['card_id']).isin(test['card_id'])]\nvariable_types={'card_id':ft.variable_types.Id, 'state_id':ft.variable_types.Id, 'city_id':ft.variable_types.Id,\n               'merchant_category_id':ft.variable_types.Id,'merchant_id':ft.variable_types.Id,'subsector_id':ft.variable_types.Id,\n              'category_2':ft.variable_types.Categorical}\nes= es.entity_from_dataframe(entity_id='historical_transactions',dataframe= historical_transactions, make_index= True,index='historical_id',\n                             time_index='purchase_date',variable_types= variable_types)\ndel historical_transactions\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98040579bd11764bef5698f5a9ad5ba9b1c06543"},"cell_type":"code","source":"r_cards_historical= ft.Relationship(es['test']['card_id'],es['historical_transactions']['card_id'])\nes= es.add_relationship(r_cards_historical)\n\nr_cards_new_merchants= ft.Relationship(es['test']['card_id'],es['new_merchant_transactions']['card_id'])\nes= es.add_relationship(r_cards_new_merchants)\n\nr_merchants_historical= ft.Relationship(es['merchants']['merchant_id'], es['historical_transactions']['merchant_id'])\nes= es.add_relationship(r_merchants_historical)\n\nr_merchants_new_merchants= ft.Relationship(es['merchants']['merchant_id'],es['new_merchant_transactions']['merchant_id'])\nes= es.add_relationship(r_merchants_new_merchants)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e91a6555bf3158c54a9f21ce3e98384ba86377d6"},"cell_type":"code","source":"features_test, feature_names_test= ft.dfs(entityset= es, target_entity= 'test', max_depth= 1)\n\ndel es\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6439c45094b28aac3e030a1a2971d699c41752c3"},"cell_type":"markdown","source":"These are some features that contained string data, so label encoding them below. A better approach could be to label encode all the features before performing, will make that optimization going forward."},{"metadata":{"trusted":true,"_uuid":"2ad985a5e21ba3d200dfc7cc6476e89f320be5da"},"cell_type":"code","source":"columns_categorical=['MODE(new_merchant_transactions.authorized_flag)', 'MODE(new_merchant_transactions.category_1)', \n              'MODE(new_merchant_transactions.category_3)', 'MODE(new_merchant_transactions.merchant_id)', \n              'MODE(historical_transactions.authorized_flag)', 'MODE(historical_transactions.category_1)', \n              'MODE(historical_transactions.category_3)', 'MODE(historical_transactions.merchant_id)']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"32492cd6fb381f68f9ba5f94f4b6545d0578b5c0"},"cell_type":"code","source":"Y=features_train['target']\n\nfeatures_train= features_train.drop(columns=['target'])\nfeatures_train= features_train.fillna(method= 'bfill')\nfeatures_test= features_test.fillna(method= 'bfill')\n\nX= features_train.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f611ea7c29fb236d74b634c66b948fe5e5391a10"},"cell_type":"code","source":"features_train.to_csv('features_train.csv',index= False)\nfeatures_test.to_csv('features_test.csv',index= False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ee614e439af8da0d12b0ab707a436c9198ae0a2"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\ntemp= features_train.append(features_test)\nfor i in columns_categorical:\n    le= LabelEncoder()\n    temp[i]= temp[i].astype('str')\n    X[i]= X[i].astype('str')\n    features_test[i]= features_test[i].astype('str')\n    le.fit(temp[i])\n    X[i]= le.transform(X[i])\n    features_test[i]= le.transform(features_test[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85a6437611a642387dc1c83e3ba18fb01ff7361d"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nxtrain,xval,ytrain,yval= train_test_split(X,Y,test_size=0.1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab4d685a8962521f732fb64915b30c273ebe6941"},"cell_type":"markdown","source":"Using Light GBM, generally I prefer XGBoost but they provide similar accuracy."},{"metadata":{"trusted":true,"_uuid":"ed39d71e4b0eb2b4687a1f04d179a09be0eeea10"},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\n\nmodel= lgb.LGBMRegressor()\nmodel.fit(xtrain,ytrain)\n\nprint(\"RMSE of Validation Data using Light GBM: %.2f\" % math.sqrt(mean_squared_error(yval,model.predict(xval))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ece76da14ce5a60cd2b9978c17176fda3a56c60"},"cell_type":"code","source":"fig, ax= plt.subplots(figsize=(14,14))\nlgb.plot_importance(model, ax= ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"337f0be221beb1cf715393dde437b2e9c6a96ce9"},"cell_type":"markdown","source":"To select features, I am using the 3 methods:\n\n1. In-built feature importance of Light GBM\n2. Permutation Importance\n3. SHAP values\n\nI prefer to perform feature selection based on Permutation Importance simply because that's the best I understand. Possible, that Feature importances and SHAP values can give better results but I do not understand their maths very well (especially the SHAP values). I will try learning them better.\n\nPermutation Importance is easy to comprehend and a natural way to remove useless features- if you randomly shuffle a feature and it doesn't reduce your accuracy, then that feature is not a good indicator. If someone had to do all this manually, this is the way to go about it."},{"metadata":{"trusted":true,"_uuid":"51cad3275b062adffd6df2406b10724f91eea73f"},"cell_type":"code","source":"import eli5\nfrom eli5.sklearn import PermutationImportance\n \nperm = PermutationImportance(model).fit(xval,yval)\neli5.show_weights(perm, feature_names = xval.columns.tolist())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad3279434427cc5656f244000d8fdc2b339b4123"},"cell_type":"code","source":"import shap\n\nexplainer = shap.TreeExplainer(model)\nshap_values = explainer.shap_values(xval)\nshap.summary_plot(shap_values, xval)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"632d5f857e811d47fbd506d7fd92f38563868986"},"cell_type":"markdown","source":"This is simply selecting the features based on a threshold from Permutation Importance. One problem that I am facing (not able to resolve completely)- due to the fact that automtaed feature engineering has generated so many junk features that any single feature is having very less impact on overall accuracy. So, even with a low threshold of 0.001, I remove almost 70% of the features generated from featuretools."},{"metadata":{"trusted":true,"_uuid":"1730e3adbeec9a427a1d962d6602b753e6af9969"},"cell_type":"code","source":"from sklearn.feature_selection import SelectFromModel\n\nsubmission= pd.read_csv('../input/sample_submission.csv')\nfeatures_test= features_test.fillna(0)\nfeatures_test= features_test.reindex(index= submission['card_id'])\n\nsel= SelectFromModel(perm, threshold= 0.002, prefit= True)\nX= sel.transform(X)\nfeatures_test= sel.transform(features_test)\n\nprint(\"Modified shape:\", X.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6a888cb63d1ad585ba9bb2ec9204f24f490eaf9d"},"cell_type":"code","source":"model_1= lgb.LGBMRegressor(learning_rate= 0.1, gamma=1)\nmodel_1.fit(X,Y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"608a3020328600d27b1a6ed9e04d7ef5934feea0"},"cell_type":"code","source":"ypred= model_1.predict(features_test)\n\nsubmission['target']=ypred\nsubmission.to_csv('submission.csv', index= False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5eac38ebf70f66edd57be07784abb8848ff1f6a3"},"cell_type":"markdown","source":"To increase accuracy further, I see two options (please let me know if you have other ideas as well):\n\n1. Perform manual feature engineering based on domain knowledge. One, I need to thoroughly understand how cards business work but then lot of features are anonymized and it is difficult to comprehend them.\n\n2. Hyper tune the paramters in Light GBM/ XGBoost model."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}