{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn the Chalange Description it si stated that: \"This challenge adds the additional complexity of ***feature interactions***, as well as missing data.\"\n\nI am new to Kaggle and this is my first public kernal. In this kernal i will try to find some ***feature interactions*** using **Rule Fit** model by **Friedman and Popescu (2008)**.\n\nI used **christophM** python implementation of RuleFit model: https://christophm.github.io/interpretable-ml-book/rulefit.html\n**christophM** has also a very nice book on \"Interpretable Machine Learing\" and a chapter dedicated to the RuleFit model: https://christophm.github.io/interpretable-ml-book/rulefit.html. \n\n\"The RuleFit algorithm by Friedman and Popescu (2008) learns sparse linear models that include automatically detected interaction effects in the form of decision rules.\n\nThe linear regression model does not account for interactions between features. Would it not be convenient to have a model that is as simple and interpretable as linear models, but also integrates feature interactions? RuleFit fills this gap. RuleFit learns a sparse linear model with the original features and also a number of new features that are decision rules. These new features capture interactions between the original features. RuleFit automatically generates these features from decision trees. Each path through a tree can be transformed into a decision rule by combining the split decisions into a rule. The node predictions are discarded and only the splits are used in the decision rules.\"\n\n# Conclusions:\n\nThe RuleFit model found some interations:\n\n'# nom_8 <= 0.2058642879128456 & ord_3 <= 0.2574392706155777\nX_train_rule['rule1'] = np.where((X_train_rule['nom_8']<= 0.2058642879128456) & (X_train_rule['ord_3'] <= 0.2574392706155777), 1, 0)\n'# nom_8 <= 0.15247249603271484\nX_train_rule['rule2'] = np.where(X_train_rule['nom_8']<= 0.15247249603271484, 1, 0)\n'# nom_8 > 0.2058642879128456 & ord_3 <= 0.2574392706155777\nX_train_rule['rule3'] = np.where((X_train_rule['nom_8']> 0.2058642879128456) & (X_train_rule['ord_3'] <= 0.2574392706155777), 1, 0)\n'# day_Saturday > 0.5 & ord_1 <= 0.18961423635482788 & day_Monday <= 0.5\nX_train_rule['rule4'] = np.where((X_train_rule['day_Saturday'] == 1) & (X_train_rule['ord_1'] <= 0.18961423635482788), 1, 0)\n'# ord_5 > 0.11862009018659592 & ord_0 > 2.5 & nom_5 > 0.17176222056150436\nX_train_rule['rule5'] = np.where((X_train_rule['ord_5'] > 0.11862009018659592) & (X_train_rule['ord_0'] > 2.5) & (X_train_rule['nom_5'] > 0.17176222056150436), 1, 0)\n'# ord_3 > 0.13002829253673553 & nom_8 > 0.15247249603271484 & bin_0 <= 0.5\nX_train_rule['rule6'] = np.where((X_train_rule['ord_3'] > 0.13002829253673553) & (X_train_rule['nom_8'] > 0.15247249603271484) & (X_train_rule['bin_0']  <= 0.5), 1, 0)\n'# nom_9 <= 0.17295265197753906\nX_train_rule['rule7'] = np.where(X_train_rule['nom_9']<= 0.17295265197753906, 1, 0)\n'# month_Sep <= 0.5 & nom_9 <= 0.17295265197753906\nX_train_rule['rule8'] = np.where((X_train_rule['month_Sep'] < 1) & (X_train_rule['nom_9'] <= 0.17295265197753906), 1, 0)\n'# nom_9 > 0.13000915944576263 & month_Jul > 0.5\nX_train_rule['rule9'] = np.where((X_train_rule['nom_9'] > 0.13000915944576263) & (X_train_rule['month_Jul'] == 1 ), 1, 0)\n'# nom_8 > 0.2521960437297821 & nom_6 > 0.24674799293279648 & ord_4 > 0.2339029535651207\nX_train_rule['rule10'] = np.where((X_train_rule['nom_8'] > 0.2521960437297821) & (X_train_rule['nom_6'] > 0.24674799293279648) & (X_train_rule['ord_4']  > 0.2339029535651207), 1, 0)\n'# month_Apr > 0.5 & ord_1 > 0.15924452245235443 & nom_8 <= 0.2626989334821701\nX_train_rule['rule11'] = np.where((X_train_rule['month_Apr'] > 0.5) & (X_train_rule['ord_1'] > 0.15924452245235443) & (X_train_rule['nom_8']  <= 0.2626989334821701), 1, 0)\n'# nom_5 > 0.2712095379829407 & nom_7 > 0.2467493638396263 & bin_1 > 0.5\nX_train_rule['rule12'] = np.where((X_train_rule['nom_5'] > 0.2712095379829407) & (X_train_rule['nom_7'] > 0.2467493638396263 ) & (X_train_rule['bin_1']  == 1), 1, 0)\n\nThe competition dataset looks synthetically created and feature interactions seems to be quiet low in my opinion. The feature interactions that the RuleFit model found seems to add kind of little information to the linear model.\n\nAll aside,  RuleFit is great at finding interactions in my experience, i used this framework to other projects with good results. If interpretability is what you desire then a linear model with RuleFit derived features is a good way to go.\n\n**Thank you and please share other ideas of detecting feature interactions!**\n\n\n\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"\"\"\"\nCreated on Thu Feb  6 19:15:03 2020\n\n@author: Radu\n\"\"\"\n\n\nimport numpy as np\nimport pandas as pd\nfrom sklearn.preprocessing import OneHotEncoder\nfrom category_encoders import TargetEncoder\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\nfrom datetime import datetime\n\nimport warnings; warnings.simplefilter(action='ignore', category=FutureWarning)\n\ndef read_data():\n    print(f'Reading data')\n    path = r\"C:\\Users\\Radu\\Desktop\\ML Projects\\Categorical Feature Encoding Challenge II/\"\n    train_df = pd.read_csv(f'{path}train.csv')\n    test_df = pd.read_csv(f'{path}test.csv')\n    sample_submission_df = pd.read_csv(f'{path}sample_submission.csv')\n\n    return train_df, test_df, sample_submission_df\n\ntrain_df, test_df, sample_submission_df = read_data()\n\ny_train = train_df['target'].copy()\nX_train = train_df.drop(['target', 'id'], axis=1)\nX_test = test_df.copy()\n\n############################################# Missing Variable Imputation ###############################################\n\ndef replace_nan(data):\n    for column in data.columns:\n        print(column)\n        if data[column].isna().sum() > 0:\n            data[column] = data[column].fillna(data[column].mode()[0])\n\n\nreplace_nan(X_train)\nreplace_nan(X_test)\n\n###################################################### Varaible encoding ###############################################\n\nsee = X_train.head(1000)\n\nX_train.loc[X_train['bin_3'] ==  'F', 'bin_3'] = 0\nX_train.loc[X_train['bin_3'] ==  'T', 'bin_3'] = 1\n\nX_train.loc[X_train['bin_4'] ==  'N', 'bin_4'] = 0\nX_train.loc[X_train['bin_4'] ==  'Y', 'bin_4'] = 1\n\nX_train.loc[X_train['day'] ==  1, 'day'] = 'Monday'\nX_train.loc[X_train['day'] ==  2, 'day'] = 'Thusday'\nX_train.loc[X_train['day'] ==  3, 'day'] = 'wednesday'\nX_train.loc[X_train['day'] ==  4, 'day'] = 'Thursday'\nX_train.loc[X_train['day'] ==  5, 'day'] = 'Friday'\nX_train.loc[X_train['day'] ==  6, 'day'] = 'Saturday'\nX_train.loc[X_train['day'] ==  7, 'day'] = 'Sunday'\n\nX_train.loc[X_train['month'] ==  1, 'month'] = 'Jan'\nX_train.loc[X_train['month'] ==  2, 'month'] = 'Feb'\nX_train.loc[X_train['month'] ==  3, 'month'] = 'Mar'\nX_train.loc[X_train['month'] ==  4, 'month'] = 'Apr'\nX_train.loc[X_train['month'] ==  5, 'month'] = 'May'\nX_train.loc[X_train['month'] ==  6, 'month'] = 'Jun'\nX_train.loc[X_train['month'] ==  7, 'month'] = 'Jul'\nX_train.loc[X_train['month'] ==  8, 'month'] = 'Aug'\nX_train.loc[X_train['month'] ==  9, 'month'] = 'Sep'\nX_train.loc[X_train['month'] ==  10, 'month'] = 'Oct'\nX_train.loc[X_train['month'] ==  11, 'month'] = 'Nov'\nX_train.loc[X_train['month'] ==  12, 'month'] = 'Dec'\n\nX_test.loc[X_test['bin_3'] ==  'F', 'bin_3'] = 0\nX_test.loc[X_test['bin_3'] ==  'T', 'bin_3'] = 1\n\nX_test.loc[X_test['bin_4'] ==  'N', 'bin_4'] = 0\nX_test.loc[X_test['bin_4'] ==  'Y', 'bin_4'] = 1\n\nX_test.loc[X_test['day'] ==  1, 'day'] = 'Monday'\nX_test.loc[X_test['day'] ==  2, 'day'] = 'Thusday'\nX_test.loc[X_test['day'] ==  3, 'day'] = 'wednesday'\nX_test.loc[X_test['day'] ==  4, 'day'] = 'Thursday'\nX_test.loc[X_test['day'] ==  5, 'day'] = 'Friday'\nX_test.loc[X_test['day'] ==  6, 'day'] = 'Saturday'\nX_test.loc[X_test['day'] ==  7, 'day'] = 'Sunday'\n\nX_test.loc[X_test['month'] ==  1, 'month'] = 'Jan'\nX_test.loc[X_test['month'] ==  2, 'month'] = 'Feb'\nX_test.loc[X_test['month'] ==  3, 'month'] = 'Mar'\nX_test.loc[X_test['month'] ==  4, 'month'] = 'Apr'\nX_test.loc[X_test['month'] ==  5, 'month'] = 'May'\nX_test.loc[X_test['month'] ==  6, 'month'] = 'Jun'\nX_test.loc[X_test['month'] ==  7, 'month'] = 'Jul'\nX_test.loc[X_test['month'] ==  8, 'month'] = 'Aug'\nX_test.loc[X_test['month'] ==  9, 'month'] = 'Sep'\nX_test.loc[X_test['month'] ==  10, 'month'] = 'Oct'\nX_test.loc[X_test['month'] ==  11, 'month'] = 'Nov'\nX_test.loc[X_test['month'] ==  12, 'month'] = 'Dec'\n\n\nbin_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4']\n\nohe_features = ['nom_0', 'nom_1', 'nom_2', 'nom_3', 'nom_4', 'day', 'month']\n\ntarget_features = ['nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9', 'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\n\n\n# Label encoding columns \ncol_list = []\nfrom sklearn.preprocessing import LabelEncoder\nlabel_x_train = X_train[ohe_features]\nlabel_x_test = X_test[ohe_features]\nfor j in label_x_train.columns.values:\n    print(j)\n    le = LabelEncoder()\n    ### fit with the desired col, col in position 0 for this ###example\n    fit_by = pd.Series([i for i in label_x_train[j].unique() if type(i) == str])\n    le.fit(fit_by)\n    ### Set transformed col leaving np.NaN as they are\n    label_x_train[j] = label_x_train[j].apply(lambda x: le.transform([x])[0] if type(x) == str else x)\n    label_x_test[j] = label_x_test[j].apply(lambda x: le.transform([x])[0] if type(x) == str else x)\n    col_list.extend([ j + \"_\" + s  for s in le.classes_])\n    \n\n# Variable OH encoding \nohe = OneHotEncoder( handle_unknown=\"ignore\", sparse = False )\nohe_x_train = ohe.fit_transform(label_x_train)\nohe_x_test = ohe.transform(label_x_test)\n#ohe_x_test = ohe.transform(X_test[ohe_features])\n\n\n# Target encoding columns 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9'\ndef transform(transformer, x_train, y_train, cv):\n    oof = pd.DataFrame(index=x_train.index, columns=x_train.columns)\n    for train_idx, valid_idx in cv.split(x_train, y_train):\n        x_train_train = x_train.loc[train_idx]\n        y_train_train = y_train.loc[train_idx]\n        x_train_valid = x_train.loc[valid_idx]\n        transformer.fit(x_train_train, y_train_train)\n        oof_part = transformer.transform(x_train_valid)\n        oof.loc[valid_idx] = oof_part\n    return oof\n\ntarget = TargetEncoder(drop_invariant=True, smoothing=0.2)\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\ntarget_x_train = transform(target, X_train[target_features], y_train, cv).astype('float')\n\ntarget.fit(X_train[target_features], y_train)\ntarget_x_test = target.transform(X_test[target_features]).astype('float')\n\n# Putting it all toghether\nX_train = np.hstack([X_train[bin_features], ohe_x_train, target_x_train])\nX_test = np.hstack([X_test[bin_features], ohe_x_test, target_x_test])\n\nsee = pd.DataFrame(X_train[:1000,:])\n\n##################################################### Rule Fit Model ##################################################\n\nfrom sklearn.ensemble import GradientBoostingClassifier\nfrom rulefit import RuleFit\n\nmia_label_x_train = X_train[:200000,:]\nmia_y_train = y_train[:200000]\n\nfeature_names = []\nfeature_names.extend(bin_features)\nfeature_names.extend(col_list)\nfeature_names.extend(target_features)\n\n\nprint(\"Training started at {}\".format(datetime.now()))\nclf = GradientBoostingClassifier(n_estimators=10, learning_rate=0.1, max_depth=3, random_state=0)\nrf = RuleFit(tree_generator= clf, rfmode= 'classify')\nrf.fit(mia_label_x_train, mia_y_train, feature_names=feature_names)\nprint(\"Training ended at {}\".format(datetime.now()))\n\nrules = rf.get_rules()\n\n#rules.to_csv('rules_RuleFit.csv')\n\n# save the model to disk\nimport pickle\nfilename = 'RuleFit_model.sav'\n#pickle.dump(rf, open(filename, 'wb'))\n \n# load the model from disk\nloaded_model = pickle.load(open(filename, 'rb'))\n\nmia_label_x_test = X_train[200000:,:]\nmia_y_test = y_train[200000:]\n\ny_pred = rf.predict(mia_label_x_test)\nprint( 'Accuracy score is: {}'.format(roc_auc_score(mia_y_test, y_pred)))\n\n##################################################### Recreate Rules ##################################################\n\nX_train_rule = pd.DataFrame(X_train, columns = feature_names )\n\nsee1= X_train_rule.head(1000)\n\n# nom_8 <= 0.2058642879128456 & ord_3 <= 0.2574392706155777\nX_train_rule['rule1'] = np.where((X_train_rule['nom_8']<= 0.2058642879128456) & (X_train_rule['ord_3'] <= 0.2574392706155777), 1, 0)\n# nom_8 <= 0.15247249603271484\nX_train_rule['rule2'] = np.where(X_train_rule['nom_8']<= 0.15247249603271484, 1, 0)\n# nom_8 > 0.2058642879128456 & ord_3 <= 0.2574392706155777\nX_train_rule['rule3'] = np.where((X_train_rule['nom_8']> 0.2058642879128456) & (X_train_rule['ord_3'] <= 0.2574392706155777), 1, 0)\n# day_Saturday > 0.5 & ord_1 <= 0.18961423635482788 & day_Monday <= 0.5\nX_train_rule['rule4'] = np.where((X_train_rule['day_Saturday'] == 1) & (X_train_rule['ord_1'] <= 0.18961423635482788), 1, 0)\n# ord_5 > 0.11862009018659592 & ord_0 > 2.5 & nom_5 > 0.17176222056150436\nX_train_rule['rule5'] = np.where((X_train_rule['ord_5'] > 0.11862009018659592) & (X_train_rule['ord_0'] > 2.5) & (X_train_rule['nom_5'] > 0.17176222056150436), 1, 0)\n# ord_3 > 0.13002829253673553 & nom_8 > 0.15247249603271484 & bin_0 <= 0.5\nX_train_rule['rule6'] = np.where((X_train_rule['ord_3'] > 0.13002829253673553) & (X_train_rule['nom_8'] > 0.15247249603271484) & (X_train_rule['bin_0']  <= 0.5), 1, 0)\n# nom_9 <= 0.17295265197753906\nX_train_rule['rule7'] = np.where(X_train_rule['nom_9']<= 0.17295265197753906, 1, 0)\n# month_Sep <= 0.5 & nom_9 <= 0.17295265197753906\nX_train_rule['rule8'] = np.where((X_train_rule['month_Sep'] < 1) & (X_train_rule['nom_9'] <= 0.17295265197753906), 1, 0)\n# nom_9 > 0.13000915944576263 & month_Jul > 0.5\nX_train_rule['rule9'] = np.where((X_train_rule['nom_9'] > 0.13000915944576263) & (X_train_rule['month_Jul'] == 1 ), 1, 0)\n# nom_8 > 0.2521960437297821 & nom_6 > 0.24674799293279648 & ord_4 > 0.2339029535651207\nX_train_rule['rule10'] = np.where((X_train_rule['nom_8'] > 0.2521960437297821) & (X_train_rule['nom_6'] > 0.24674799293279648) & (X_train_rule['ord_4']  > 0.2339029535651207), 1, 0)\n# month_Apr > 0.5 & ord_1 > 0.15924452245235443 & nom_8 <= 0.2626989334821701\nX_train_rule['rule11'] = np.where((X_train_rule['month_Apr'] > 0.5) & (X_train_rule['ord_1'] > 0.15924452245235443) & (X_train_rule['nom_8']  <= 0.2626989334821701), 1, 0)\n# nom_5 > 0.2712095379829407 & nom_7 > 0.2467493638396263 & bin_1 > 0.5\nX_train_rule['rule12'] = np.where((X_train_rule['nom_5'] > 0.2712095379829407) & (X_train_rule['nom_7'] > 0.2467493638396263 ) & (X_train_rule['bin_1']  == 1), 1, 0)\n\n\nprint(\"Training started at {}\".format(datetime.now()))\noof = pd.DataFrame(index=train_df.index, columns= ['y_prob'])\ncv = StratifiedKFold(n_splits=5, random_state = 62)\nfor train_idx, valid_idx in cv.split(X_train_rule, y_train):\n    x_train_train = X_train_rule.iloc[train_idx, :]\n    y_train_train = y_train[train_idx]\n    x_train_valid = X_train_rule.iloc[valid_idx, :]\n    y_train_valid = y_train[valid_idx]\n        \n    # Fitting Logistic Regression to the Training set\n    from sklearn.linear_model import LogisticRegression\n    classifier = LogisticRegression(solver='sag', random_state = 0)\n    classifier.fit(x_train_train, y_train_train)\n    \n    # Predicting the Test set results\n    #y_pred = classifier.predict(X_train)\n    y_prob_train=classifier.predict_proba(x_train_train)   \n    y_prob_valid=classifier.predict_proba(x_train_valid)         \n    oof.loc[valid_idx] = y_prob_valid[:,1:]\n    \n    # AUROC and AR\n    ROC_Train = roc_auc_score(y_train_train, y_prob_train[:,1])\n    ROC_Valid = roc_auc_score(y_train_valid, y_prob_valid[:,1])\n\n    print(\"AUROC Train Set : {:2.2f} %\".format(ROC_Train*100))\n    print(\"AUROC Valid Set : {:2.2f} %\".format(ROC_Valid*100))\n#y_prob_test=classifier.predict_proba(X_test) \nprint(\"Training ended at {}\".format(datetime.now()))\n\ncoef = pd.merge(pd.DataFrame(X_train_rule.columns), pd.DataFrame(np.transpose(classifier.coef_)), left_index = True, right_index = True)\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}