{"cells":[{"metadata":{"_uuid":"bafe2feb4ccc182faa9bd323b55ba7c40e86a24e"},"cell_type":"markdown","source":"This is one more attempt to find robust feature selection strategy for Elo data.\n\nI spent days exctracting features from the data we have. When the number of features reached 500 I realized that none of those ten feature selection algorithms I used gave me firm advice on what features are important. I noticed that features selected by those strategies affect CV even if the have close to zero importance. From the other side - very important feature (consensus by multiple votes) being excluded gave improvement.\nI tried recursive search (out of the box solutions like RFECV) but again it works for a long time and gives a result that doesn't give rubust improvement. Though I agree it allows to decrease number of features from 500 to 70 with great speed boost and minimal CV increase. But in order to move up on the leader board I had to find the best composition of features.\n\nI implemented recursive feature selection with report on each cycle. It works slow, but i can stop it in the middle, add selected features to the start list and restart again. I even work in parallel adding new features to train dataset and I do not need to start the search from the scartch.\n\nThe sequence of actions is:\n1. Start training with minimal set of features (start set) and N features to test. Add features one by one and measure CV improvement (remove added feauture before take the next).\n2. Sort features by CV and take best one. Add the feature to start feature set and run step 1 with N-1 features. Save added features to a file.\n\nWhen the process doesn't give improvement for multiple cycles, turn back to the last improvement and fix the feature set.\nDo the reverse operation with this feature set: remove features one at a time and measure CV. If it is improved - exclude the feature with best improvement and repeat the reversed run again untill all redundant features are excluded (no mo CV improvement).\n\nIt is nice to run your best model with features selected by the logic above time to time and submit results to get LB. It can tell if thereis an overfit.\n\nBelow is an example of a few steps of the feature selection process (300 features). This time it was backward removal of features. For each removed feature the model CV is measured.\n\nIt is interesting that features change their role from one cycle to another. The same feature removed at step 1 can give worse CV while 2=3 steps later it gives the best CV.\nAnother observation is that the feature importance provided by LightGBM (split and gain) doesn't correlate with the results of this selection.\n"},{"metadata":{"_uuid":"47be1ae1c1474186803be65d57eb2b29cf7c57e1"},"cell_type":"markdown","source":"![CV changes for a few steps of the Feature selection](https://storage.googleapis.com/kagglesdsdata/datasets/116892/279885/FeatureRemovalCV-300first.png?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1549811090&Signature=hm1XQyCzNfubVcPL9wKPADJWTCSQsZB0gIN9lBUa7ybnjHpZpWGYfl6ppD0xyiEklzs5nWUQMbbnrCLOquWjeSp%2BMrjo1dR%2FSk140YciMR2oGCeT3%2F6GL%2BwcwF3tCNmkn2K0nk15iYGa%2BSnh1OJlA6Eo5VKGWzx%2BK1xHnDACQkLPV2Wz4RYgVerVmaW4s%2B6F2AuHMJVE32mF87UI9A2HrFlFOurQgQIvICp4pX5kEVM1%2FWhTIw3%2B8qj6NDtPKZ8LqSbwdp0zeQoBFiZcsK%2F0FbKrisTTCKPIuHp%2BagaevZ5tAc%2Fv%2BMPePCyX2Vc1s%2Btozfi8mbAZFY2w548ppPBMpg%3D%3D)"},{"metadata":{"_uuid":"165d58ae932542bb064c9bb5d0e7e0f0a3222a71"},"cell_type":"markdown","source":"The same picture for CV improvements against median CV for the step (500+ features)\nHere you can see that the algorithm selected worse features three times (x=4,5,6), removed them and it did not change CV for other features.\n\n![CV diff against median](https://storage.googleapis.com/kagglesdsdata/datasets/116892/279885/FeatureRemovalImprovements.png?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1549811457&Signature=JaYpbayU%2FNOrZ5s02LgU1abKNHaerriEHXoyYk8vXGTZnLoONMn42wcMOecGrvROZuIWFXTa8CJF%2FOZ8fNBsJhfzg%2BHEOTpfy67q8lbAov1HnLyxEoUbFPPONajQXgD8qkowJSQ8QW2k42%2B6DtLmhZQBsNQyoNtv7XkScY11GjfCBSkJOv1DjFPV1S8EQ%2BclScggwki1uz6urbulUGRmTre8dnvHhHImhaKW%2BPBSPqB4QB6CboDWGHDuTXq9QWcxKxVtw9oEI0LkcErqFynV6vinQEXMUgtjCaU%2FmelyWLMseKf%2F845VBuS53KJiVI4CxB6yLiv01JiFMpSktVLuFw%3D%3D)"},{"metadata":{"_uuid":"bfbc558fdf729e4f57d0034861b362fe0fb1ddf4"},"cell_type":"markdown","source":"Assuming that you have already finished your feature engineering and you have train dataset:\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport lightgbm as lgb\nimport warnings\nimport time\nimport sys\nimport datetime\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import mean_squared_error\n\nwarnings.simplefilter(action='ignore', category=FutureWarning)\npd.set_option('display.max_columns', 500)\n\nimport os\nprint(os.listdir(\"../input\"))\nprint(os.listdir(\"../input/elo-merchant-category-recommendation\"))\n\ndef save_dict_to_file(dic, fname):\n    f = open(fname,'w')\n    f.write(str(dic))\n    f.close()\n\ndef load_dict_from_file(fname):\n    f = open(fname,'r')\n    data=f.read()\n    f.close()\n    return eval(data)\n\n\nprint(\"Loading\")\nt0 = time.time()\n\n\ntrain=pd.read_csv(\"../input/elo-merchant-category-recommendation/train.csv\")\n\n#test=pd.read_csv(\"test.csv\")\nprint(\"Number of records loaded \", len(train.index))\n\n# dummy features for demo purposes\ntrain['demo1']=train['feature_1']*train['feature_2']\ntrain['demo2']=train['feature_2']*train['feature_3']\ntrain['demo3']=train['feature_1']*train['feature_3']\ntrain['demo4']=train['feature_1']*train['feature_3']*train['feature_2']\ntrain['demo5']=train['feature_1']+train['feature_3']+train['feature_2']\ntrain.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad6f8fbfe9ecd16a7a6657d18cc21671e2b5d451"},"cell_type":"code","source":"split=80000 # part of the data for training. There will be no folds for speedup\n\ntrain=train.sample(frac=0.50, random_state=2019).reset_index(drop=True)\ntarget=train[\"target\"]\ndel(train[\"target\"])\n\n\n#import cartegorical features prepared in advance\n#with open(\"categorical_features\"+ver+\".l\", 'r') as fp:\n#\t#categorical_features=pickle.load(fp)\n#\tcategorical_feats=[line.rstrip('\\n') for line in fp]\n#\t\n\nfeatures = [c for c in train.columns]# take all features from the file\n\ncategorical_feats=['city_id','authorized_flag','category_1','merchant_category_id','state_id','subsector_id','merchant_group_id']\ncategorical_feats = categorical_feats + [c for c in features if 'feature_' in c]# \"calculated_merchant_category\", \"calculated_merchant_group\",\"calculated_city\",\n\n\n# remove categorical features if they do not match column names (to preven exceptions)\ncategorical_feats=[i for i in categorical_feats if i in train.columns] # for a case if we lost a column to prevent exception later\n\nif \"target\" in features:\n    features.remove(\"target\") # do not add target to the train or test\n\nif \"card_id\" in features:\n    features.remove(\"card_id\") # do not add target to the train or test\n    \nif \"first_active_month\" in features:\n    features.remove(\"first_active_month\") # do not add target to the train or test\n    \n    \nprint(\"Number of features\", len(features))\n\nparam = {'num_leaves': 120,\n         'min_data_in_leaf': 90, \n         'objective':'regression',\n         'max_depth': 9,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.4,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.92 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 14.5,\n         \"random_state\": 133,\n         \"verbosity\": -1}\n\nprint(f'{time.time() - t0:.1f} seconds')\nprint(\"Start looping\")\nt0 = time.time()\n\nadded_feats=dict()\nadded_feats_cv=dict()\nstep_n=0\n\nfeatures_history = pd.DataFrame(columns=['feature','cv','improvement','step'])\n\n# Prepare feature list to start\n# features from previous run\nfeat = {512: 'new_purchase_amount_max', 1023: 'month_lag_std', 1533: 'hist_coocrnc_merchant_group_id_merchant_group_id_4', 2042: 'new_month_lag_mean', 2550: 'trade_count', 3057: 'installments_1000_count', 3563: 'elapsed_time', 4068: 'auth_month_lag_max', 4572: 'latest_merchant_id_1', 5075: 'new_merchant_id_nunique', 5577: 'hist_coocrnc_city_id_city_id_5', 6078: 'hist_coocrnc_city_id_city_id_1', 6578: 'most_frequent_merchant_category_id_count_pct', 7077: 'new_purchase_date_ptp', 7575: 'hist_purchase_amount_sum', 8072: 'auth_numerical_2_sum', 8568: 'purchase_amount_max', 9063: 'auth_purchase_amount_min', 9557: 'new_purchase_weekofyear_mean'}\nfeatures_out=list()\nfor f in feat.keys():\n    features_out.append(feat[f])\n\n# features to start with added manually\nfeatures_out=features_out+['installments_sum_mean','auth_category_1_sum','hist_coocrnc_city_id__1','hist_month_lag_std', 'feature_3','feature_2','feature_1','new_purchase_date_max','latest_purchase_date_1','auth_purchase_date_max','hist_purchase_date_max','new_installments_mean','hist_installments_mean','auth_installments_mean']\n\nfeatures_out=list(set(features_out)) # remove duplicates\n\nprint(\"Configuration for the  features list has \",len(features_out), \"records\")\n\n# cleanup if made a mistake\nfeatures_out = [c for c in features_out if c in features] \n\nprint(\"Cleaned features list has \",len(features_out), \"records\")\n\n# save the list for later use (if process fails or to run concurent process)\n#with open(\"features_selection_starting_with_\"+ver+\".l\", 'w') as fp:\n#    for s in features_out:\n#        fp.write(s + '\\n')\n\nprint(\"Starting features list\",len(features_out))\nprint(\"Total  features list\",len(features))\n\ncategorical_feats_out=categorical_feats.copy()\n\npass_no=0\n\nwhile len(features_out)<300:\n    cv_dic=dict()\n    pass_no=pass_no+1\n    is_excluded=0\n    print(\"Pass ---- \" , pass_no)\n    for fet in features:\n        \n        if fet not in features_out:\n            is_excluded=1\n            print(\"################ \",step_n,\" #####################\")\n            step_n=step_n+1\n            print(\"feature added >>>> \", fet)\n            oof = np.zeros(len(train)-split)\n            \n            features_in=features_out.copy() + [fet]\n\n            categorical_in=[i for i in categorical_feats if i in features_in]\n\n\n            trn_data = lgb.Dataset(train.iloc[:split][features_in],\n                                   label=target.iloc[:split],\n                                   categorical_feature=categorical_in\n                                  )\n            val_data = lgb.Dataset(train.iloc[split:][features_in],\n                                   label=target.iloc[split:],\n                                   categorical_feature=categorical_in\n                                  )\n        \n            start = time.time()\n            feature_importance_df = pd.DataFrame()\n        \n        \n            num_round = 10000\n            clf = lgb.train(param,\n                            trn_data,\n                            num_round,\n                            valid_sets = [trn_data, val_data],\n                            verbose_eval=100,\n                            early_stopping_rounds = 200)\n            \n            oof = clf.predict(train.iloc[split:][features_in], num_iteration=clf.best_iteration)\n            cv=mean_squared_error(oof, target[split:])**0.5\n            cv_dic[fet]=cv\n            print(\"CV score: {:<8.5f}\".format(cv))\n            print(f'{time.time() - t0:.1f} seconds')\n            t0 = time.time()\n    if is_excluded==0:\n        print(\"No more features to add\")\n        break\n        #fold_importance_df = pd.DataFrame()\n        #fold_importance_df[\"feature\"] = features_in\n        #fold_importance_df[\"importance-split\"] = clf.feature_importance(importance_type='split') # split levels\n        #fold_importance_df[\"importance-gain\"] = clf.feature_importance(importance_type='gain') # split levels\n        #fold_importance_df.to_csv(\"importance-\"+fet+\".csv\")\n        \n    cv_df=pd.DataFrame.from_dict(cv_dic, orient='index', columns=[ 'cv'])\n    mean_cv=cv_df['cv'].mean()\n    cv_df['improvement']=mean_cv-cv_df['cv']\n    \n    cv_df.reset_index(inplace=True)\n    cv_df=cv_df[[\"index\",'cv','improvement']]\n    cv_df.columns=['feature','cv','improvement']\n    cv_df['step']=step_n\n    features_history=features_history.append(cv_df)\n    cv_df=cv_df.sort_values(by=\"improvement\", ascending=False).reset_index()\n    column_to_add=cv_df.head(1)['feature'].values[0]\n    \n    #features_history.to_csv(\"importance_direct.csv\")\n\n    features_out.append(column_to_add)\n    #with open(\"full_features_list\"+ver+\".l\", 'w') as fp:\n    #    fp.write(\"'\"+\"','\".join(features_out)+\"'\" + '\\n')\n\n    cv=cv_df.head(1)['cv'].values[0]\n    \n    features_tmp=list(cv_df['feature']) # reset order of features. Most improving go first\n    \n    features=features_tmp + [c for c in features if c not in features_tmp] # adding non rated features for more longer run\n\n        \n    print('>>>>>>>>>>',column_to_add, 'removed with cv ', cv)\n    added_feats[step_n]=column_to_add\n    added_feats_cv[step_n]=cv\n    #save_dict_to_file(added_feats, 'added_feats.txt')\n    #save_dict_to_file(added_feats_cv, 'added_feats_cv.txt')\n    #save_dict_to_file(added_feats, 'added_feats_full_list.txt')\n    print(f'{time.time() - t0:.1f} seconds')\n    print(datetime.datetime.now().strftime(\"%a, %d %B %Y %H:%M:%S\"))\n    print(\"Cycle \", step_n)\n    t0 = time.time()\n    \n    \nprint(added_feats)\nprint(added_feats_cv)\n\n    \n\n\nprint(f'{time.time() - t0:.1f} seconds')\nprint(\"Start looping\")\nt0 = time.time()\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df=features_history.copy()\ndic=dict()\nncount=0\nfor st in list(df[['step']].drop_duplicates()['step']):\n    dic[st]=ncount\n    ncount=ncount+1\n    \ndf['step']=df['step'].apply(lambda x: dic[x])\n\nfrom matplotlib.pyplot import cm\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n#variable n should be number of curves to plot (I skipped this earlier thinking that it is obvious when looking at picture - sorry my bad mistake xD): n=len(array_of_curves_to_plot)\n#version 1:\n\nfeatures=list(df[['feature']].drop_duplicates()['feature'])\nn=len(features)\n\n\ncolor=iter(cm.rainbow(np.linspace(0,1,n)))\nfig, ax1 = plt.subplots(figsize=(14,14))\nfor i in range(n):\n    x=df[df['feature']==features[i]]['step'].values\n\n    y=df[df['feature']==features[i]]['cv'].values\n    c=next(color)\n    ax1.plot(x, y,c=c)\n#ax1.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}