{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":77585,"databundleVersionId":8490368,"sourceType":"competition"},{"sourceId":178708444,"sourceType":"kernelVersion"},{"sourceId":178885269,"sourceType":"kernelVersion"},{"sourceId":179070149,"sourceType":"kernelVersion"}],"dockerImageVersionId":30698,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Only XGBoost for the Tabular Data Benchmark - 2nd Edition\n\nThis notebook submits the predictions of 59 XGBoost models which were tuned individually.\n\nReferences:\n- [TDB2 EDA which makes sense ⭐️⭐️⭐️⭐️⭐️](https://www.kaggle.com/code/ambrosm/tdb2-eda-which-makes-sense) (EDA notebook)\n- [TDB2 Only HGB](https://www.kaggle.com/code/ambrosm/tdb2-only-hgb), [TDB2 Only LightGBM](https://www.kaggle.com/code/ambrosm/tdb2-only-lightgbm), [TDB2 Only CatBoost](https://www.kaggle.com/code/ambrosm/tdb2-only-catboost) (three notebooks to which we'll compare our scores)\n- [TDB2 Feature importance](https://www.kaggle.com/code/ambrosm/tdb2-feature-importance) (notebook which determines the important features)\n- [Tabular Data: are gbdt still outperforming DL?](https://www.kaggle.com/competitions/bench-tab-v1) (the predecessor to this competition)\n- [Tabular Data Benchmark - 2nd Edition](https://www.kaggle.com/competitions/bench-tab-v2) (this competition)\n- [Why do tree-based models still outperform deep learning on tabular data?](https://arxiv.org/pdf/2207.08815)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport json\nimport matplotlib.pyplot as plt\nfrom collections import namedtuple, defaultdict, OrderedDict\nimport warnings\nimport xgboost\nimport pickle\nfrom tqdm import tqdm\nimport datetime\n\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.ensemble import BaggingClassifier, BaggingRegressor\nfrom sklearn.metrics import log_loss, mean_squared_error","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-04T02:48:22.180061Z","iopub.execute_input":"2024-06-04T02:48:22.180831Z","iopub.status.idle":"2024-06-04T02:48:23.175860Z","shell.execute_reply.started":"2024-06-04T02:48:22.180784Z","shell.execute_reply":"2024-06-04T02:48:23.174808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Configuration\n# number of repeats for ensembling\nn_repeats = 5\n","metadata":{"execution":{"iopub.status.busy":"2024-06-04T02:48:23.177836Z","iopub.execute_input":"2024-06-04T02:48:23.178276Z","iopub.status.idle":"2024-06-04T02:48:23.183075Z","shell.execute_reply.started":"2024-06-04T02:48:23.178247Z","shell.execute_reply":"2024-06-04T02:48:23.181900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reading the data\n\nWe read all the data files into the dictionary `datasets`.","metadata":{}},{"cell_type":"code","source":"path = '/kaggle/input/bench-tab-v2/'\ntask_types = json.load(open(f'{path}train/dict_task.json'))\nsuits = json.load(open(f'{path}train/dict_suit.json'))\n\nDataset = namedtuple('Dataset', field_names=['X', 'X_te', 'y', 'names', 'cat_indicator', 'var_type', 'target_type'])\n\nsize_list = []\ndatasets = {}\nfor task_type_num in suits: # 4 entries\n    for suit in suits[task_type_num]: # 59 entries\n        \n        identifier = f'{task_type_num}_{suit}'\n        task_type = task_types[task_type_num]\n        var_type = task_type['features'] # 'num' or 'num+cat'\n        target_type = task_type['target'] # 'num' or 'cat'\n\n        # Read the three data files\n        X = pd.read_parquet(f'{path}train/{identifier}/data_train.parquet')\n        X_te = pd.read_parquet(f'{path}train/{identifier}/data_test.parquet')\n        y = pd.read_parquet(f'{path}train/{identifier}/target_train.parquet')\n        assert len(X) == len(y)\n        \n        # Read the two metadata files\n        names = json.load(open(f'{path}train/{identifier}/attribute_names.json'))\n        X.columns = [s.replace(':', '') for s in X.columns] # LightGBM doesn't like colons\n        X_te.columns = [s.replace(':', '') for s in X_te.columns]\n        names = [s.replace(':', '') for s in names]\n        assert (names == X.columns).all()\n        assert (names == X_te.columns).all()\n        cat_indicator = json.load(open(f'{path}train/{identifier}/categorical_indicator.json'))\n        \n        size_list.append((identifier, var_type, target_type, len(X), len(X_te), len(names)))\n        datasets[identifier] = Dataset(X, X_te, y, names, cat_indicator, var_type, target_type)\nsizes = pd.DataFrame(size_list, columns=['Identifier', 'var_type', 'target_type', 'n_samples_tr', 'n_samples_te', 'n_features'])\ndel X, X_te, y\n","metadata":{"execution":{"iopub.status.busy":"2024-06-04T02:48:23.184234Z","iopub.execute_input":"2024-06-04T02:48:23.184539Z","iopub.status.idle":"2024-06-04T02:48:26.963350Z","shell.execute_reply.started":"2024-06-04T02:48:23.184506Z","shell.execute_reply":"2024-06-04T02:48:26.962528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hyperparameters\n\nHyperparameters were determined with 59 Optuna studies. Features were selected in a [separate notebook](https://www.kaggle.com/code/ambrosm/tdb2-feature-importance) by permutation feature importance.\n\nThe hyperparameters and the selected features are hidden in the following cell.","metadata":{}},{"cell_type":"code","source":"xgb_params = {'334_361110': ({'grow_policy': 'depthwise', 'learning_rate': 0.09803921568627451, 'n_estimators': 179, 'max_depth': 16, 'reg_lambda': 1.1788661081419183, 'min_child_weight': 1.386786755164374, 'subsample': 0.9665348430982745, 'colsample_bytree': 0.9917007041724829, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '334_361111': ({'grow_policy': 'depthwise', 'learning_rate': 0.019998835022321016, 'n_estimators': 177, 'max_depth': 19, 'reg_lambda': 0.35564367098234634, 'min_child_weight': 0.2030092288879782, 'subsample': 0.9940485323642211, 'colsample_bytree': 0.6145374109643057, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '334_361113': ({'grow_policy': 'depthwise', 'learning_rate': 0.02717606583833126, 'n_estimators': 1208, 'max_depth': 17, 'reg_lambda': 1.8376068294781829, 'min_child_weight': 0.10491508978876575, 'subsample': 0.9086069508775807, 'colsample_bytree': 0.8682866508273515, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '334_361282': ({'grow_policy': 'depthwise', 'learning_rate': 0.04258871228769527, 'n_estimators': 200, 'max_depth': 5, 'reg_lambda': 0.6971698518331078, 'min_child_weight': 34.40322475282661, 'subsample': 0.8596225404764983, 'colsample_bytree': 0.7319935995092385, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '334_361283': ({'grow_policy': 'depthwise', 'learning_rate': 0.15243996575242308, 'n_estimators': 35, 'max_depth': 7, 'reg_lambda': 14.868122455472054, 'min_child_weight': 31.57853390083707, 'subsample': 0.8320271805147836, 'colsample_bytree': 0.7598738893505773, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '334_361285': ({'grow_policy': 'depthwise', 'learning_rate': 0.03804523270361819, 'n_estimators': 348, 'max_depth': 20, 'reg_lambda': 4.782274336030839, 'min_child_weight': 0.1037599944965347, 'subsample': 1.0, 'colsample_bytree': 0.5124151235817014, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '334_361286': ({'grow_policy': 'depthwise', 'learning_rate': 0.10546017429605506, 'n_estimators': 45, 'max_depth': 14, 'reg_lambda': 2.939725686067998, 'min_child_weight': 41.85287590815656, 'subsample': 0.9773768608665152, 'colsample_bytree': 0.9796517732094898, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361093': ({'grow_policy': 'depthwise', 'learning_rate': 0.016053053274229193, 'n_estimators': 1404, 'max_depth': 21, 'reg_lambda': 27.63641152751634, 'min_child_weight': 0.38790167039869616, 'subsample': 1.0, 'colsample_bytree': 0.5658473698773195, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361094': ({'grow_policy': 'depthwise', 'learning_rate': 0.06492607094793966, 'n_estimators': 3001, 'max_depth': 19, 'reg_lambda': 10.050938151951025, 'min_child_weight': 126.54229615359571, 'subsample': 0.8993060838157452, 'colsample_bytree': 0.9744508005150527, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361096': ({'grow_policy': 'depthwise', 'learning_rate': 0.05110645474523433, 'n_estimators': 707, 'max_depth': 7, 'reg_lambda': 0.9618474667766557, 'min_child_weight': 0.15745107864557142, 'subsample': 0.9333999026412813, 'colsample_bytree': 0.7989588213587075, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361097': ({'grow_policy': 'depthwise', 'learning_rate': 0.26064291920069504, 'n_estimators': 55, 'max_depth': 3, 'reg_lambda': 87.63995410456923, 'min_child_weight': 8.019354536172127, 'subsample': 1.0, 'colsample_bytree': 0.9625562385404955, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361098': ({'grow_policy': 'depthwise', 'learning_rate': 0.1, 'n_estimators': 1861, 'max_depth': 15, 'reg_lambda': 67.16479827303331, 'min_child_weight': 0.17380685571712795, 'subsample': 1.0, 'colsample_bytree': 0.7630079741223814, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361099': ({'grow_policy': 'depthwise', 'learning_rate': 0.10214193259418691, 'n_estimators': 198, 'max_depth': 9, 'reg_lambda': 17.614920886475794, 'min_child_weight': 0.15574903340807444, 'subsample': 0.8340037236000688, 'colsample_bytree': 0.9850962443076297, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361101': ({'grow_policy': 'depthwise', 'learning_rate': 0.06430727568656247, 'n_estimators': 484, 'max_depth': 12, 'reg_lambda': 0.10971329061985424, 'min_child_weight': 26.7717003238729, 'subsample': 0.9070433772877273, 'colsample_bytree': 0.9561600895281308, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361102': ({'grow_policy': 'depthwise', 'learning_rate': 0.0425, 'n_estimators': 661, 'max_depth': 7, 'reg_lambda': 11.692643867125259, 'min_child_weight': 1.1575222960346814, 'subsample': 0.880721554809894, 'colsample_bytree': 0.5564101656921882, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361103': ({'grow_policy': 'depthwise', 'learning_rate': 0.02186263479227019, 'n_estimators': 1013, 'max_depth': 13, 'reg_lambda': 66.37898801514287, 'min_child_weight': 112.19587964015439, 'subsample': 0.901361278051163, 'colsample_bytree': 0.9699795416614746, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361104': ({'grow_policy': 'depthwise', 'learning_rate': 0.04413118057711528, 'n_estimators': 229, 'max_depth': 21, 'reg_lambda': 18.688849909421737, 'min_child_weight': 2.85087265734882, 'subsample': 0.8774994702745162, 'colsample_bytree': 0.810788923625925, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361287': ({'grow_policy': 'depthwise', 'learning_rate': 0.04774042980608309, 'n_estimators': 67, 'max_depth': 8, 'reg_lambda': 27.60816706206881, 'min_child_weight': 9.988388152564616, 'subsample': 0.905923601684435, 'colsample_bytree': 0.6757940730126286, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361288': ({'grow_policy': 'depthwise', 'learning_rate': 0.08333333333333334, 'n_estimators': 111, 'max_depth': 9, 'reg_lambda': 61.520195958602564, 'min_child_weight': 29.582109702761365, 'subsample': 0.8004771239917164, 'colsample_bytree': 0.928126129419622, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361289': ({'grow_policy': 'depthwise', 'learning_rate': 0.10890853609644333, 'n_estimators': 46, 'max_depth': 5, 'reg_lambda': 0.7300940387071545, 'min_child_weight': 0.12531041713852725, 'subsample': 0.9861807177220242, 'colsample_bytree': 0.6145774851782221, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361291': ({'grow_policy': 'depthwise', 'learning_rate': 0.045249246777479275, 'n_estimators': 139, 'max_depth': 15, 'reg_lambda': 0.20270358684481918, 'min_child_weight': 479.5224043220384, 'colsample_bytree': 0.7754303572006538, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361292': ({'grow_policy': 'depthwise', 'learning_rate': 0.02176663854153232, 'n_estimators': 868, 'max_depth': 11, 'reg_lambda': 44.5101877878101, 'min_child_weight': 133.10325205196733, 'subsample': 0.9404237748101288, 'colsample_bytree': 0.7819964884327284, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361293': ({'grow_policy': 'depthwise', 'learning_rate': 0.02606073175971321, 'n_estimators': 660, 'max_depth': 9, 'reg_lambda': 83.28880564264729, 'min_child_weight': 28.689455576076025, 'subsample': 0.9373245888751729, 'colsample_bytree': 0.6651869001166119, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '335_361294': ({'grow_policy': 'depthwise', 'learning_rate': 0.07787215986844828, 'n_estimators': 359, 'max_depth': 5, 'reg_lambda': 0.26639773846276643, 'min_child_weight': 125.7010828847326, 'subsample': 0.9394970388599023, 'colsample_bytree': 0.6856892345186947, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361072': ({'grow_policy': 'depthwise', 'learning_rate': 0.04115791477026276, 'n_estimators': 754, 'max_depth': 6, 'reg_lambda': 5.310441777259473, 'min_child_weight': 5.974996480991818, 'subsample': 0.807423706315165, 'colsample_bytree': 0.6882220363118072, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361073': ({'grow_policy': 'depthwise', 'learning_rate': 0.01734382379716523, 'n_estimators': 3002, 'max_depth': 21, 'reg_lambda': 70.20308523544557, 'min_child_weight': 4.3415005547579, 'subsample': 0.9081097254777662, 'colsample_bytree': 0.9872773835763772, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361074': ({'grow_policy': 'depthwise', 'learning_rate': 0.07595184549801942, 'n_estimators': 1763, 'max_depth': 5, 'reg_lambda': 55.561895654956594, 'min_child_weight': 20.92246641788384, 'subsample': 0.9099943416233877, 'colsample_bytree': 0.9940825104731721, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361076': ({'grow_policy': 'depthwise', 'learning_rate': 0.026851996675744002, 'n_estimators': 454, 'max_depth': 18, 'reg_lambda': 7.0409626981461, 'min_child_weight': 2.837376129219773, 'subsample': 0.8570624922830694, 'colsample_bytree': 0.6564559955728486, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361077': ({'grow_policy': 'depthwise', 'learning_rate': 0.050787258835305105, 'n_estimators': 329, 'max_depth': 17, 'reg_lambda': 0.27196808236201875, 'min_child_weight': 0.10055347245238712, 'subsample': 0.8353257813348268, 'colsample_bytree': 0.8410060266589178, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361078': ({'grow_policy': 'depthwise', 'learning_rate': 0.04258871228769527, 'n_estimators': 543, 'max_depth': 19, 'reg_lambda': 56.51573366168422, 'min_child_weight': 34.67118012241337, 'subsample': 0.9832588526825666, 'colsample_bytree': 0.8073238067419066, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361079': ({'grow_policy': 'depthwise', 'learning_rate': 0.03492850101405994, 'n_estimators': 421, 'max_depth': 18, 'reg_lambda': 95.90621739852615, 'min_child_weight': 12.415489315504539, 'subsample': 0.8087746297243519, 'colsample_bytree': 0.8282136648205202, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361080': ({'grow_policy': 'depthwise', 'learning_rate': 0.058030581080855535, 'n_estimators': 167, 'max_depth': 7, 'reg_lambda': 3.2117342075475035, 'min_child_weight': 0.12623826186030002, 'subsample': 0.858578833762581, 'colsample_bytree': 0.9739612732419958, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361081': ({'grow_policy': 'depthwise', 'learning_rate': 0.08878495663075141, 'n_estimators': 960, 'max_depth': 17, 'reg_lambda': 51.593246673896346, 'min_child_weight': 2.2435876238484203, 'subsample': 1.0, 'colsample_bytree': 0.7939602579843413, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361082': ({'grow_policy': 'depthwise', 'learning_rate': 0.021459503374243363, 'n_estimators': 421, 'max_depth': 10, 'reg_lambda': 61.99583527426335, 'min_child_weight': 1.1591122373785887, 'subsample': 0.8173195993164948, 'colsample_bytree': 0.9369944723173184, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361083': ({'grow_policy': 'depthwise', 'learning_rate': 0.029613184822163508, 'n_estimators': 2282, 'max_depth': 9, 'reg_lambda': 38.73340572730249, 'min_child_weight': 0.14145171459631856, 'subsample': 0.9609102396358797, 'colsample_bytree': 0.976895574200906, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361084': ({'grow_policy': 'depthwise', 'learning_rate': 0.038028047769931204, 'n_estimators': 571, 'max_depth': 19, 'reg_lambda': 57.22010826009982, 'min_child_weight': 42.23541474368538, 'subsample': 1.0, 'colsample_bytree': 0.5789730542387836, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361085': ({'grow_policy': 'depthwise', 'learning_rate': 0.030693914656931585, 'n_estimators': 1073, 'max_depth': 15, 'reg_lambda': 0.21683462762094782, 'min_child_weight': 2.0723902039314512, 'subsample': 0.8064491554140762, 'colsample_bytree': 0.9575843508373756, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361086': ({'grow_policy': 'depthwise', 'learning_rate': 0.1130913217641098, 'n_estimators': 706, 'max_depth': 6, 'reg_lambda': 10.722779730502275, 'min_child_weight': 0.6387833429175424, 'subsample': 0.9447885871283807, 'colsample_bytree': 0.6598329556187964, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361087': ({'grow_policy': 'depthwise', 'learning_rate': 0.026599250735294674, 'n_estimators': 1752, 'max_depth': 6, 'reg_lambda': 25.708506674969087, 'min_child_weight': 3.460592600560825, 'subsample': 0.8077279044929102, 'colsample_bytree': 0.588259754830604, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361088': ({'grow_policy': 'depthwise', 'learning_rate': 0.06221459818482387, 'n_estimators': 644, 'max_depth': 8, 'reg_lambda': 11.921922933783106, 'min_child_weight': 8.283949098750478, 'subsample': 0.8357756380961748, 'colsample_bytree': 0.8357722611232801, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361279': ({'grow_policy': 'depthwise', 'learning_rate': 0.018705325811215374, 'n_estimators': 333, 'max_depth': 14, 'reg_lambda': 80.79695777074564, 'min_child_weight': 0.23908653188657955, 'subsample': 0.8396330427672827, 'colsample_bytree': 0.5006922096322873, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361280': ({'grow_policy': 'depthwise', 'learning_rate': 0.17869097765114716, 'n_estimators': 86, 'max_depth': 13, 'reg_lambda': 80.14411481632857, 'min_child_weight': 45.95007419374126, 'subsample': 0.9618729814207517, 'colsample_bytree': 0.9680369409626184, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '336_361281': ({'grow_policy': 'depthwise', 'learning_rate': 0.11479481048660284, 'n_estimators': 57, 'max_depth': 11, 'reg_lambda': 22.540420804405997, 'min_child_weight': 2.1388019914923535, 'colsample_bytree': 0.894099182587581, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361055': ({'grow_policy': 'depthwise', 'learning_rate': 0.04825770992678326, 'n_estimators': 287, 'max_depth': 3, 'reg_lambda': 0.5571184267656458, 'min_child_weight': 0.19354262886524104, 'subsample': 0.859110298651137, 'colsample_bytree': 0.6525429223910152, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361060': ({'grow_policy': 'depthwise', 'learning_rate': 0.05, 'n_estimators': 241, 'max_depth': 19, 'reg_lambda': 3.916496371210128, 'min_child_weight': 0.8907506610384007, 'subsample': 0.9726990121910088, 'colsample_bytree': 0.8752410635150819, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361061': ({'grow_policy': 'depthwise', 'learning_rate': 0.030369955587453402, 'n_estimators': 1484, 'max_depth': 20, 'reg_lambda': 1.45702985150414, 'min_child_weight': 0.4978499284386523, 'subsample': 0.8827472220565113, 'colsample_bytree': 0.9856147184714976, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361062': ({'grow_policy': 'depthwise', 'learning_rate': 0.23180589022483086, 'n_estimators': 1118, 'max_depth': 22, 'reg_lambda': 38.5285015257885, 'min_child_weight': 1.2386782639426828, 'subsample': 0.8448177155374993, 'colsample_bytree': 0.9463039819009798, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361063': ({'grow_policy': 'depthwise', 'learning_rate': 0.045690981341141315, 'n_estimators': 262, 'max_depth': 18, 'reg_lambda': 10.355763946101618, 'min_child_weight': 13.563211651265869, 'subsample': 0.8204862435579364, 'colsample_bytree': 0.680625917921166, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361065': ({'grow_policy': 'depthwise', 'learning_rate': 0.08047022030341551, 'n_estimators': 133, 'max_depth': 15, 'reg_lambda': 11.080290434703066, 'min_child_weight': 0.28293003757713014, 'subsample': 0.9879132035065519, 'colsample_bytree': 0.7060401696073733, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361066': ({'grow_policy': 'depthwise', 'learning_rate': 0.17652410004695757, 'n_estimators': 75, 'max_depth': 5, 'reg_lambda': 46.51902417256478, 'min_child_weight': 1.240630963929647, 'subsample': 0.9605025829241703, 'colsample_bytree': 0.6153682562894907, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361068': ({'grow_policy': 'depthwise', 'learning_rate': 0.0338529411764706, 'n_estimators': 1116, 'max_depth': 8, 'reg_lambda': 45.434409086830804, 'min_child_weight': 4.202232516069805, 'subsample': 0.8035588143385307, 'colsample_bytree': 0.5407030955110903, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361069': ({'grow_policy': 'depthwise', 'learning_rate': 0.015, 'n_estimators': 2782, 'max_depth': 17, 'reg_lambda': 84.34995689164684, 'min_child_weight': 10.68021233276754, 'subsample': 0.8789647890200629, 'colsample_bytree': 0.9161768092634928, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361070': ({'grow_policy': 'depthwise', 'learning_rate': 0.02087892140250743, 'n_estimators': 231, 'max_depth': 10, 'reg_lambda': 1.1506342299958567, 'min_child_weight': 0.20444718815352694, 'subsample': 0.8979688127124417, 'colsample_bytree': 0.6068187459411174, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361273': ({'grow_policy': 'depthwise', 'learning_rate': 0.2193791717598261, 'n_estimators': 102, 'max_depth': 3, 'reg_lambda': 19.65335287669972, 'min_child_weight': 5.134993870507477, 'subsample': 1.0, 'colsample_bytree': 0.5488511717184272, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361274': ({'grow_policy': 'depthwise', 'learning_rate': 0.017072727174134546, 'n_estimators': 476, 'max_depth': 12, 'reg_lambda': 0.5155749009485664, 'min_child_weight': 13.46047978030167, 'subsample': 0.8329523201883192, 'colsample_bytree': 0.9564987712492004, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361275': ({'grow_policy': 'depthwise', 'learning_rate': 0.05505784171929854, 'n_estimators': 142, 'max_depth': 13, 'reg_lambda': 61.22873290390628, 'min_child_weight': 18.32426682017544, 'subsample': 0.8338157508869586, 'colsample_bytree': 0.5668361426706398, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361276': ({'grow_policy': 'depthwise', 'learning_rate': 0.02397535247041261, 'n_estimators': 376, 'max_depth': 21, 'reg_lambda': 5.483906549598641, 'min_child_weight': 5.6028488093741995, 'subsample': 0.8409449046756831, 'colsample_bytree': 0.899448379697916, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361277': ({'grow_policy': 'depthwise', 'learning_rate': 0.10917896407610642, 'n_estimators': 191, 'max_depth': 10, 'reg_lambda': 20.878010089319773, 'min_child_weight': 0.874122398241735, 'subsample': 0.9941425104289101, 'colsample_bytree': 0.8219013382182635, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},), '337_361278': ({'grow_policy': 'depthwise', 'learning_rate': 0.036845153525549276, 'n_estimators': 146, 'max_depth': 16, 'reg_lambda': 6.085634766573271, 'min_child_weight': 25.273978947690413, 'subsample': 0.9131042670323976, 'colsample_bytree': 0.5993753883046248, 'tree_method': 'hist', 'enable_categorical': True, 'gamma': 0},)}\nimportant_features = {'334_361110': ['date', 'day', 'period', 'nswprice', 'nswdemand', 'vicprice', 'vicdemand', 'transfer'], '334_361111': ['lineNo', 'assgNo', 'P1stFixation', 'P2stFixation', 'prevFixDur', 'firstfixDur', 'firstPassFixDur', 'nextFixDur', 'firstSaccLen', 'lastSaccLen', 'prevFixPos', 'landingPos', 'totalFixDur', 'regressLen', 'nextWordRegress', 'regressDur', 'pupilDiamMax', 'pupilDiamLag', 'timePrtctg', 'titleNo', 'wordNo'], '334_361113': ['Elevation', 'Aspect', 'Slope', 'Horizontal_Distance_To_Hydrology', 'Vertical_Distance_To_Hydrology', 'Horizontal_Distance_To_Roadways', 'Hillshade_9am', 'Hillshade_Noon', 'Hillshade_3pm', 'Horizontal_Distance_To_Fire_Points', 'Wilderness_Area1', 'Wilderness_Area2', 'Wilderness_Area3', 'Wilderness_Area4', 'Soil_Type2', 'Soil_Type3', 'Soil_Type4', 'Soil_Type7', 'Soil_Type8', 'Soil_Type9', 'Soil_Type10', 'Soil_Type11', 'Soil_Type12', 'Soil_Type13', 'Soil_Type16', 'Soil_Type17', 'Soil_Type18', 'Soil_Type19', 'Soil_Type20', 'Soil_Type21', 'Soil_Type22', 'Soil_Type23', 'Soil_Type24', 'Soil_Type25', 'Soil_Type26', 'Soil_Type27', 'Soil_Type28', 'Soil_Type29', 'Soil_Type30', 'Soil_Type31', 'Soil_Type32', 'Soil_Type33', 'Soil_Type34', 'Soil_Type35', 'Soil_Type36', 'Soil_Type38', 'Soil_Type39', 'Soil_Type40'], '334_361282': ['V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9', 'V11', 'V13', 'V19', 'V30', 'V33', 'V35', 'V36', 'V41', 'V43', 'V47', 'V59', 'V63', 'V72', 'V75'], '334_361283': ['x1', 'x2', 'x5', 'x6', 'x7', 'x8', 'x9', 'x10', 'x11', 'x12', 'x13', 'x14', 'x15', 'x16', 'x18', 'x19', 'x20', 'x21', 'x22', 'x23'], '334_361285': ['Vehicle_Reference_df_res', 'Vehicle_Type', 'Vehicle_Manoeuvre', 'Hit_Object_in_Carriageway', 'Hit_Object_off_Carriageway', 'Was_Vehicle_Left_Hand_Drive?', 'Age_of_Driver', 'Age_Band_of_Driver', 'Engine_Capacity_(CC)', 'Propulsion_Code', 'Age_of_Vehicle', 'Location_Easting_OSGR', 'Location_Northing_OSGR', 'Longitude', 'Latitude', 'Police_Force', 'Number_of_Vehicles', 'Number_of_Casualties', 'Local_Authority_(District)', '1st_Road_Number', '2nd_Road_Number', 'Urban_or_Rural_Area', 'Vehicle_Reference_df', 'Casualty_Reference', 'Sex_of_Casualty', 'Age_of_Casualty', 'Age_Band_of_Casualty', 'Pedestrian_Location', 'Pedestrian_Movement', 'Casualty_Type', 'Casualty_IMD_Decile'], '334_361286': ['sex', 'age', 'priors_count', 'age_cat_25-45', 'age_cat_Greaterthan45', 'age_cat_Lessthan25', 'race_African-American', 'race_Caucasian', 'c_charge_degree_F', 'c_charge_degree_M'], '335_361093': ['Actions_taken', 'Liberal', 'Unconstitutional', 'Precedent_alteration', 'Unanimous', 'Year_of_decision', 'Lower_court_disagreement'], '335_361094': ['northing', 'easting', 'resistivity', 'isns'], '335_361096': ['carat', 'cut', 'color', 'clarity', 'depth', 'table', 'x', 'y', 'z'], '335_361097': ['X3', 'X14', 'X19', 'X29', 'X31', 'X35', 'X37', 'X46', 'X47', 'X48', 'X49', 'X50', 'X52', 'X54', 'X61', 'X65', 'X75', 'X76', 'X80', 'X82', 'X96', 'X98', 'X100', 'X104', 'X111', 'X113', 'X115', 'X117', 'X118', 'X119', 'X120', 'X127', 'X131', 'X132', 'X134', 'X136', 'X139', 'X141', 'X147', 'X148', 'X150', 'X152', 'X156', 'X158', 'X164', 'X168', 'X176', 'X179', 'X181', 'X182', 'X187', 'X189', 'X191', 'X201', 'X208', 'X213', 'X215', 'X218', 'X221', 'X225', 'X226', 'X232', 'X234', 'X238', 'X240', 'X261', 'X263', 'X264', 'X265', 'X267', 'X273', 'X275', 'X283', 'X294', 'X311', 'X313', 'X314', 'X315', 'X316', 'X321', 'X322', 'X324', 'X327', 'X334', 'X340', 'X345', 'X349', 'X350', 'X354', 'X368', 'X376', 'X377'], '335_361098': ['city', 'area', 'rooms', 'bathroom', 'parking_spaces', 'animal', 'furniture', 'hoa_(BRL)', 'rent_amount_(BRL)', 'property_tax_(BRL)', 'fire_insurance_(BRL)'], '335_361099': ['season', 'year', 'month', 'hour', 'holiday', 'workingday', 'weather', 'temp', 'feel_temp', 'humidity', 'windspeed'], '335_361101': ['VendorID', 'store_and_fwd_flag', 'RatecodeID', 'passenger_count', 'extra', 'mta_tax', 'tolls_amount', 'improvement_surcharge', 'total_amount', 'trip_type', 'lpep_pickup_datetime_day', 'lpep_pickup_datetime_hour', 'lpep_pickup_datetime_minute', 'lpep_dropoff_datetime_day', 'lpep_dropoff_datetime_hour', 'lpep_dropoff_datetime_minute'], '335_361102': ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot', 'waterfront', 'grade', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated', 'lat', 'long', 'sqft_living15', 'sqft_lot15', 'date_year', 'date_month', 'date_day'], '335_361103': ['Hour', 'Month', 'DayofWeek', 'Environment.Type', 'Altitude..m.', 'PM.sub.2.5..sub..particulate.matter..Hourly.measured.'], '335_361104': ['KWG', 'KWI', 'STRM', 'STRN', 'SA', 'SB', 'Run2', 'Run3', 'Run4'], '335_361287': ['oz2', 'oz3', 'oz4', 'oz5', 'oz8', 'oz10', 'oz11', 'oz13', 'oz14', 'oz15', 'oz16', 'oz17', 'oz18', 'oz19', 'oz21', 'oz30', 'oz31', 'oz36', 'oz39', 'oz40', 'oz41', 'oz42', 'oz43', 'oz44', 'oz45', 'oz46', 'oz47', 'oz48', 'oz50', 'oz52', 'oz53', 'oz56', 'oz59', 'oz61', 'oz62', 'oz64', 'oz65', 'oz66', 'oz71', 'oz72', 'oz73', 'oz74', 'oz75', 'oz76', 'oz77', 'oz78', 'oz80', 'oz82', 'oz84', 'oz86', 'oz89', 'oz90', 'oz91', 'oz92', 'oz93', 'oz94', 'oz102', 'oz104', 'oz106', 'oz107', 'oz108', 'oz110', 'oz115', 'oz116', 'oz117', 'oz118', 'oz120', 'oz121', 'oz122', 'oz123', 'oz124', 'oz125', 'oz126', 'oz128', 'oz129', 'oz131', 'oz133', 'oz134', 'oz135', 'oz137', 'oz138', 'oz139', 'oz140', 'oz141', 'oz142', 'oz143', 'oz144', 'oz145', 'oz147', 'oz149', 'oz150', 'oz151', 'oz152', 'oz154', 'oz155', 'oz156', 'oz161', 'oz162', 'oz163', 'oz164', 'oz170', 'oz171', 'oz173', 'oz176', 'oz177', 'oz181', 'oz182', 'oz185', 'oz188', 'oz189', 'oz190', 'oz195', 'oz196', 'oz197', 'oz201', 'oz202', 'oz204', 'oz205', 'oz206', 'oz207', 'oz208', 'oz209', 'oz210', 'oz211', 'oz213', 'oz214', 'oz216', 'oz217', 'oz218', 'oz220', 'oz223', 'oz226', 'oz227', 'oz228', 'oz231', 'oz232', 'oz233', 'oz235', 'oz236', 'oz239', 'oz240', 'oz244', 'oz246', 'oz261'], '335_361288': ['Sex', 'Length', 'Diameter', 'Height', 'Whole_weight', 'Shucked_weight', 'Viscera_weight', 'Shell_weight'], '335_361289': ['Precinct', 'Sector', 'Occurred_hour', 'Occurred_min'], '335_361291': ['vehicle_type', 'direction', 'weekday', 'temp', 'windspeed_max', 'windspeed_avg', 'precipitation', 'dew_point', 'humidity', 'hour', 'dayminute'], '335_361292': ['cat1', 'cat2', 'cat3', 'cat4', 'cat5', 'cat6', 'cat7', 'cat8', 'cat9', 'cat10', 'cat11', 'cat12', 'cat13', 'cat16', 'cat17', 'cat18', 'cat19', 'cat20', 'cat21', 'cat22', 'cat23', 'cat24', 'cat25', 'cat26', 'cat27', 'cat28', 'cat29', 'cat30', 'cat31', 'cat32', 'cat33', 'cat35', 'cat36', 'cat37', 'cat38', 'cat39', 'cat40', 'cat41', 'cat42', 'cat43', 'cat44', 'cat45', 'cat46', 'cat47', 'cat48', 'cat49', 'cat50', 'cat51', 'cat52', 'cat53', 'cat54', 'cat55', 'cat56', 'cat57', 'cat59', 'cat64', 'cat65', 'cat66', 'cat67', 'cat71', 'cat72', 'cat73', 'cat74', 'cat75', 'cat76', 'cat77', 'cat78', 'cat79', 'cat80', 'cat81', 'cat82', 'cat83', 'cat84', 'cat85', 'cat86', 'cat87', 'cat88', 'cat89', 'cat90', 'cat91', 'cat92', 'cat93', 'cat94', 'cat95', 'cat96', 'cat97', 'cat98', 'cat99', 'cat100', 'cat101', 'cat102', 'cat103', 'cat104', 'cat105', 'cat106', 'cat107', 'cat108', 'cat111', 'cat114', 'cont1', 'cont2', 'cont3', 'cont4', 'cont5', 'cont6', 'cont7', 'cont8', 'cont9', 'cont10', 'cont11', 'cont12', 'cont13', 'cont14'], '335_361293': ['Month', 'DayofMonth', 'CRSDepTime', 'CRSArrTime', 'Distance'], '335_361294': ['Total_Discharges', 'Average_Covered_Charges', 'Average_Medicare_Payments'], '336_361072': ['lread', 'lwrite', 'scall', 'sread', 'swrite', 'fork', 'exec', 'rchar', 'wchar', 'pgout', 'ppgout', 'pgfree', 'pgscan', 'atch', 'pgin', 'ppgin', 'pflt', 'vflt', 'runqsz', 'freemem', 'freeswap'], '336_361073': ['f5', 'f6', 'f7', 'f8', 'f9', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f20', 'f21', 'f22', 'f23', 'f24', 'f25', 'f26', 'f27', 'f28', 'f29', 'f30', 'f31', 'f32', 'f33'], '336_361074': ['climbRate', 'Sgz', 'p', 'q', 'curRoll', 'absRoll', 'diffClb', 'diffRollRate', 'diffDiffClb', 'SaTime1', 'SaTime2', 'SaTime3', 'SaTime4', 'diffSaTime1', 'diffSaTime3', 'Sa'], '336_361076': ['fixed.acidity', 'volatile.acidity', 'citric.acid', 'residual.sugar', 'chlorides', 'free.sulfur.dioxide', 'total.sulfur.dioxide', 'density', 'pH', 'sulphates', 'alcohol'], '336_361077': ['climbRate', 'Sgz', 'p', 'q', 'curPitch', 'curRoll', 'absRoll', 'diffClb', 'diffRollRate', 'SeTime1', 'SeTime2', 'SeTime3', 'SeTime4', 'SeTime5', 'SeTime6', 'SeTime7', 'SeTime8', 'SeTime9', 'SeTime10', 'SeTime11', 'SeTime12', 'SeTime13', 'SeTime14', 'diffSeTime1', 'diffSeTime3', 'diffSeTime5', 'diffSeTime7', 'diffSeTime9', 'diffSeTime11', 'diffSeTime13', 'alpha', 'Se'], '336_361078': ['median_income', 'housing_median_age', 'total_rooms', 'total_bedrooms', 'population', 'households', 'latitude', 'longitude'], '336_361079': ['P1', 'P5p1', 'P6p2', 'P11p4', 'P14p9', 'P15p1', 'P15p3', 'P16p2', 'P18p2', 'P27p4', 'H2p2', 'H8p2', 'H10p1', 'H13p1', 'H18pA', 'H40p4'], '336_361080': ['carat', 'depth', 'table', 'x', 'y', 'z'], '336_361081': ['area', 'rooms', 'bathroom', 'parking_spaces', 'hoa_(BRL)', 'rent_amount_(BRL)', 'property_tax_(BRL)', 'fire_insurance_(BRL)'], '336_361082': ['month', 'hour', 'temp', 'feel_temp', 'humidity', 'windspeed'], '336_361083': ['passenger_count', 'tolls_amount', 'total_amount', 'lpep_pickup_datetime_day', 'lpep_pickup_datetime_hour', 'lpep_pickup_datetime_minute', 'lpep_dropoff_datetime_day', 'lpep_dropoff_datetime_hour', 'lpep_dropoff_datetime_minute'], '336_361084': ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot', 'grade', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated', 'lat', 'long', 'sqft_living15', 'sqft_lot15', 'date_month', 'date_day'], '336_361085': ['a1', 'a2', 'a3', 'a4', 'a5', 'y2'], '336_361086': ['Total_Discharges', 'Average_Covered_Charges', 'Average_Medicare_Payments'], '336_361087': ['LATITUDE', 'LONGITUDE', 'LND_SQFOOT', 'TOT_LVG_AREA', 'SPEC_FEAT_VAL', 'RAIL_DIST', 'OCEAN_DIST', 'WATER_DIST', 'CNTR_DIST', 'SUBCNTR_DI', 'HWY_DIST', 'age', 'month_sold'], '336_361088': ['mean_atomic_mass', 'wtd_mean_atomic_mass', 'gmean_atomic_mass', 'wtd_gmean_atomic_mass', 'entropy_atomic_mass', 'wtd_entropy_atomic_mass', 'range_atomic_mass', 'wtd_range_atomic_mass', 'std_atomic_mass', 'wtd_std_atomic_mass', 'mean_fie', 'wtd_mean_fie', 'gmean_fie', 'wtd_gmean_fie', 'entropy_fie', 'wtd_entropy_fie', 'range_fie', 'wtd_range_fie', 'std_fie', 'wtd_std_fie', 'mean_atomic_radius', 'wtd_mean_atomic_radius', 'gmean_atomic_radius', 'wtd_gmean_atomic_radius', 'entropy_atomic_radius', 'wtd_entropy_atomic_radius', 'range_atomic_radius', 'wtd_range_atomic_radius', 'std_atomic_radius', 'wtd_std_atomic_radius', 'mean_Density', 'wtd_mean_Density', 'gmean_Density', 'wtd_gmean_Density', 'entropy_Density', 'wtd_entropy_Density', 'range_Density', 'wtd_range_Density', 'std_Density', 'wtd_std_Density', 'mean_ElectronAffinity', 'wtd_mean_ElectronAffinity', 'gmean_ElectronAffinity', 'wtd_gmean_ElectronAffinity', 'entropy_ElectronAffinity', 'wtd_entropy_ElectronAffinity', 'range_ElectronAffinity', 'wtd_range_ElectronAffinity', 'std_ElectronAffinity', 'wtd_std_ElectronAffinity', 'mean_FusionHeat', 'wtd_mean_FusionHeat', 'gmean_FusionHeat', 'wtd_gmean_FusionHeat', 'entropy_FusionHeat', 'wtd_entropy_FusionHeat', 'range_FusionHeat', 'wtd_range_FusionHeat', 'std_FusionHeat', 'wtd_std_FusionHeat', 'mean_ThermalConductivity', 'wtd_mean_ThermalConductivity', 'gmean_ThermalConductivity', 'wtd_gmean_ThermalConductivity', 'entropy_ThermalConductivity', 'wtd_entropy_ThermalConductivity', 'range_ThermalConductivity', 'wtd_range_ThermalConductivity', 'std_ThermalConductivity', 'wtd_std_ThermalConductivity', 'mean_Valence', 'wtd_mean_Valence', 'gmean_Valence', 'wtd_gmean_Valence', 'entropy_Valence', 'wtd_entropy_Valence', 'wtd_range_Valence', 'std_Valence', 'wtd_std_Valence'], '336_361279': ['oz1', 'oz2', 'oz3', 'oz4', 'oz5', 'oz6', 'oz9', 'oz12', 'oz13', 'oz83', 'oz124', 'oz125', 'oz126', 'oz127', 'oz128', 'oz131', 'oz149', 'oz150', 'oz151', 'oz165', 'oz171', 'oz172', 'oz173', 'oz175', 'oz176', 'oz177', 'oz183', 'oz185', 'oz197', 'oz246', 'oz247', 'oz248', 'oz249', 'oz250', 'oz251'], '336_361280': ['Length', 'Diameter', 'Height', 'Whole_weight', 'Shucked_weight', 'Viscera_weight', 'Shell_weight'], '336_361281': ['temp', 'windspeed_max', 'windspeed_avg', 'precipitation', 'dew_point', 'humidity', 'hour', 'dayminute'], '337_361055': ['RevolvingUtilizationOfUnsecuredLines', 'age', 'NumberOfTime30-59DaysPastDueNotWorse', 'DebtRatio', 'MonthlyIncome', 'NumberOfOpenCreditLinesAndLoans', 'NumberOfTimes90DaysLate', 'NumberRealEstateLoansOrLines', 'NumberOfTime60-89DaysPastDueNotWorse', 'NumberOfDependents'], '337_361060': ['date', 'period', 'nswprice', 'nswdemand', 'vicprice', 'vicdemand', 'transfer'], '337_361061': ['X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9', 'X10'], '337_361062': ['f5', 'f6', 'f7', 'f8', 'f9', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f20', 'f21', 'f22', 'f23', 'f24', 'f25', 'f28', 'f29', 'f30', 'f31', 'f32', 'f33'], '337_361063': ['P1', 'P5p1', 'P6p2', 'P11p4', 'P14p9', 'P15p1', 'P15p3', 'P16p2', 'P18p2', 'P27p4', 'H2p2', 'H8p2', 'H10p1', 'H13p1', 'H18pA', 'H40p4'], '337_361065': ['fLength', 'fWidth', 'fSize', 'fConc', 'fConc1', 'fAsym', 'fM3Long', 'fM3Trans', 'fAlpha', 'fDist'], '337_361066': ['V1', 'V6', 'V10', 'V12', 'V13', 'V14', 'V15'], '337_361068': ['ParticleID_0', 'ParticleID_1', 'ParticleID_2', 'ParticleID_3', 'ParticleID_4', 'ParticleID_5', 'ParticleID_6', 'ParticleID_7', 'ParticleID_8', 'ParticleID_9', 'ParticleID_10', 'ParticleID_11', 'ParticleID_12', 'ParticleID_13', 'ParticleID_14', 'ParticleID_15', 'ParticleID_16', 'ParticleID_17', 'ParticleID_18', 'ParticleID_19', 'ParticleID_20', 'ParticleID_21', 'ParticleID_22', 'ParticleID_23', 'ParticleID_24', 'ParticleID_25', 'ParticleID_26', 'ParticleID_27', 'ParticleID_28', 'ParticleID_29', 'ParticleID_30', 'ParticleID_31', 'ParticleID_32', 'ParticleID_33', 'ParticleID_34', 'ParticleID_35', 'ParticleID_36', 'ParticleID_37', 'ParticleID_38', 'ParticleID_39', 'ParticleID_40', 'ParticleID_41', 'ParticleID_42', 'ParticleID_43', 'ParticleID_44', 'ParticleID_45', 'ParticleID_46', 'ParticleID_47', 'ParticleID_48', 'ParticleID_49'], '337_361069': ['lepton_pT', 'lepton_eta', 'lepton_phi', 'missing_energy_magnitude', 'missing_energy_phi', 'jet_1_pt', 'jet_1_eta', 'jet_1_phi', 'jet_2_pt', 'jet_2_eta', 'jet_2_phi', 'jet_3_pt', 'jet_3_eta', 'jet_3_phi', 'jet_4_pt', 'jet_4_eta', 'jet_4_phi', 'm_jj', 'm_jjj', 'm_lv', 'm_jlv', 'm_bb', 'm_wbb', 'm_wwbb'], '337_361070': ['lineNo', 'assgNo', 'prevFixDur', 'firstfixDur', 'firstPassFixDur', 'nextFixDur', 'firstSaccLen', 'lastSaccLen', 'prevFixPos', 'leavingPos', 'regressLen', 'regressDur', 'pupilDiamMax', 'pupilDiamLag', 'timePrtctg', 'titleNo', 'wordNo'], '337_361273': ['time_in_hospital', 'num_lab_procedures', 'num_medications', 'number_outpatient', 'number_emergency', 'number_inpatient', 'number_diagnoses'], '337_361274': ['V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9', 'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19', 'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V30', 'V31', 'V35', 'V37', 'V38', 'V41', 'V42', 'V43', 'V44', 'V47', 'V49', 'V51'], '337_361275': ['x1', 'x5', 'x6', 'x7', 'x8', 'x9', 'x10', 'x11', 'x12', 'x13', 'x14', 'x15', 'x16', 'x17', 'x18', 'x19', 'x20', 'x21', 'x22', 'x23'], '337_361276': ['D2', 'D3', 'D5', 'D6', 'D7', 'D8', 'D10', 'D11', 'D12', 'D13', 'D14', 'D15', 'D16', 'D17', 'D18', 'D19', 'D21', 'D25', 'D26', 'D31', 'D32', 'D33', 'D35', 'D36', 'D37', 'D38', 'D39', 'D40', 'D41', 'D42', 'D43', 'D44', 'D45', 'D46', 'D48', 'D49', 'D52', 'D54', 'D55', 'D56', 'D57', 'D58', 'D59', 'D60', 'D61', 'D62', 'D63', 'D64', 'D65', 'D66', 'D68', 'D70', 'D71', 'D73', 'D74', 'D75', 'D78', 'D84', 'D85', 'D86', 'D89', 'D90', 'D91', 'D95', 'D99', 'D101', 'D102', 'D103', 'D104', 'D105', 'D106', 'D107', 'D117', 'D119', 'D123', 'D125', 'D128', 'D131', 'D132', 'D137', 'D140', 'D142', 'D143', 'D144', 'D145', 'D146', 'D149', 'D150', 'D153', 'D154', 'D157', 'D160', 'D172', 'D173', 'D174', 'D175', 'D177', 'D178', 'D180', 'D181', 'D182', 'D183', 'D187', 'D189', 'D191', 'D194', 'D196', 'D197', 'D200', 'D201', 'D202', 'D204', 'D205', 'D207', 'D209', 'D210', 'D211', 'D212', 'D213', 'D215', 'D216', 'D217', 'D218', 'D221', 'D223', 'D224', 'D239', 'D243', 'D249', 'D250', 'D261', 'D312', 'D337', 'D338', 'D374', 'D375', 'D415', 'D417', 'D419', 'D420', 'D422', 'D423', 'D441', 'D442', 'D443', 'D444', 'D446', 'D447', 'D451', 'D454', 'D461', 'D469', 'D471', 'D473', 'D474', 'D477', 'D479', 'D504', 'D505', 'D512', 'D513', 'D514', 'D518', 'D519', 'D520', 'D521', 'D526', 'D546', 'D548', 'D559', 'D560', 'D582', 'D583', 'D586', 'D607', 'D608', 'D621', 'D622', 'D625', 'D649', 'D659', 'D660', 'D661', 'D663', 'D664', 'D672', 'D697', 'D698', 'D700', 'D701', 'D702', 'D741', 'D742', 'D743', 'D749', 'D751', 'D763', 'D764', 'D767', 'D769', 'D770', 'D802', 'D806', 'D809', 'D810', 'D811', 'D814', 'D815', 'D817', 'D889', 'D891', 'D892', 'D897', 'D911', 'D912', 'D926', 'D927', 'D933', 'D935', 'D937', 'D947', 'D948', 'D950', 'D951'], '337_361277': ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude'], '337_361278': ['ExternalRiskEstimate', 'MSinceOldestTradeOpen', 'MSinceMostRecentTradeOpen', 'AverageMInFile', 'NumSatisfactoryTrades', 'NumTrades90Ever2DerogPubRec', 'PercentTradesNeverDelq', 'MSinceMostRecentDelq', 'MaxDelq2PublicRecLast12M', 'NumTotalTrades', 'PercentInstallTrades', 'MSinceMostRecentInqexcl7days', 'NumInqLast6M', 'NumInqLast6Mexcl7days', 'NetFractionRevolvingBurden', 'NetFractionInstallBurden', 'NumRevolvingTradesWBalance', 'NumBank2NatlTradesWHighUtilization', 'PercentTradesWBalance']}\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-04T02:48:26.964639Z","iopub.execute_input":"2024-06-04T02:48:26.964963Z","iopub.status.idle":"2024-06-04T02:48:27.058299Z","shell.execute_reply.started":"2024-06-04T02:48:26.964935Z","shell.execute_reply":"2024-06-04T02:48:27.057246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Validation\n\nWe validate the tuned hyperparameters using the same split as in the EDA notebook.\n\nFor every task, XGBoost creates a sequence of decision trees. We plot a diagram showing the size (number of leaves) of every decision tree in the sequence hoping to see something which helps us optimize the models.","metadata":{}},{"cell_type":"code","source":"%%time\n# Validation with the same split as in the EDA\nkf = StratifiedKFold(shuffle=True, random_state=1)\nscores = defaultdict(lambda: [])\noof = {}\nfor ds_num, identifier in enumerate(datasets.keys()):\n    start_time_task = datetime.datetime.now()\n    scores['identifier'].append(identifier)\n    data = datasets[identifier]\n    \n    # Prepare stratification and train-test split\n    if data.target_type == 'num': # regression\n        assert data.y.shape[1] == 1\n        le = LabelEncoder()\n        stratification = le.fit_transform(pd.qcut(data.y.iloc[:,0], 10, duplicates='drop'))\n    else: # classification\n        stratification = data.y.target\n    idx_tr, idx_va = next(kf.split(data.X, stratification))\n    X_tr, X_va = data.X.iloc[idx_tr], data.X.iloc[idx_va] # DataFrame\n    y_tr, y_va = data.y.iloc[idx_tr, 0], data.y.iloc[idx_va, 0] # Series\n    \n    # Select features\n    X_tr, X_va = X_tr[important_features[identifier]], X_va[important_features[identifier]]\n    \n    # Fit and predict with XGBoost\n    if data.target_type == 'num': # regression\n        model = BaggingRegressor(xgboost.XGBRegressor(**xgb_params[identifier][0]),\n                                 n_estimators=n_repeats,\n                                 bootstrap=False,\n                                 random_state=0)\n        model.fit(X_tr, y_tr)\n        y_va_pred = model.predict(X_va)\n        score = mean_squared_error(y_va, y_va_pred)\n    else: # classification\n        model = BaggingClassifier(xgboost.XGBClassifier(**xgb_params[identifier][0]),\n                                 n_estimators=n_repeats,\n                                 bootstrap=False,\n                                 random_state=0)\n        model.fit(X_tr, y_tr)\n        y_va_pred = model.predict_proba(X_va)[:, 1]\n        score = log_loss(y_va, y_va_pred)\n    duration_task = datetime.datetime.now() - start_time_task\n    print(f\"{ds_num:2} {identifier} {data.target_type}: xgb {duration_task.seconds:5} s\"\n          f\"    {score:12.4e}\")\n    scores['xgb'].append(score)\n    oof[identifier] = y_va_pred.astype(np.float32)\n    \n    # Plot the tree sizes\n    df = model.estimators_[0].get_booster().trees_to_dataframe()\n    n_leaves = df.query(\"Feature == 'Leaf'\").groupby('Tree').Node.size()\n    plt.figure(figsize=(15, 3))\n    plt.title(f'Task {ds_num} ({identifier}): leaves per tree')\n    plt.xlabel('tree')\n    plt.ylabel('leaves')\n    plt.bar(n_leaves.index, n_leaves, color='seagreen')\n    plt.show()\n\n# Save the out-of-fold predictions for ensembling\nwith open('oof1.pickle', 'wb') as f:\n    pickle.dump(oof, f)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-06-04T02:48:27.060537Z","iopub.execute_input":"2024-06-04T02:48:27.060931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can now compare the XGBoost scores to the scores in the other notebooks. The second-to-last column (`winner`) indicates which is the best model.","metadata":{}},{"cell_type":"code","source":"cv1_df = pd.DataFrame(scores).set_index('identifier')\nwith open('/kaggle/input/tdb2-only-lightgbm/cv1.pickle', 'rb') as f:\n    cv1_df['lightgbm'] = pickle.load(f)['lightgbm'].values\nwith open('/kaggle/input/tdb2-only-catboost/cv1.pickle', 'rb') as f:\n    cv1_df['catboost'] = pickle.load(f)['cb'].values\nwith open('/kaggle/input/tdb2-only-hgb/cv1.pickle', 'rb') as f:\n    cv1_df['hgb'] = pickle.load(f)['hgb'].values\n\nresult_df = cv1_df.copy().reset_index()\nresult_df['winner'] = cv1_df.columns[np.argmin(cv1_df.fillna(1e12), axis=1)]\nresult_df['best'] = cv1_df.fillna(1e12).min(axis=1).values\n\nwith open('cv1.pickle', 'wb') as f:\n    pickle.dump(result_df, f)\nresult_df.style.format(precision=5).map(lambda s: 'background-color:lightgreen;' if 'xgb' in str(s) else None)\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission\n\nWe retrain the models on the full datasets and concatenate the predictions to a submission file.","metadata":{}},{"cell_type":"code","source":"%%time\nsubmission = []\nfor ds_num, identifier in tqdm(list(enumerate(datasets.keys()))):\n    data = datasets[identifier]\n    X_tr = data.X # DataFrame\n    y_tr = data.y.iloc[:, 0] # Series\n    \n    # Select features\n    X_tr, X_te = X_tr[important_features[identifier]], data.X_te[important_features[identifier]]\n        \n    # Fit and predict with XGBoost\n    if data.target_type == 'num': # regression\n        model = BaggingRegressor(xgboost.XGBRegressor(**xgb_params[identifier][0]),\n                                 n_estimators=n_repeats,\n                                 bootstrap=False,\n                                 random_state=0)\n        model.fit(X_tr, y_tr)\n        y_pred = model.predict(X_te)\n    else: # classification\n        model = BaggingClassifier(xgboost.XGBClassifier(**xgb_params[identifier][0]),\n                                 n_estimators=n_repeats,\n                                 bootstrap=False,\n                                 random_state=0)\n        model.fit(X_tr, y_tr)\n        y_pred = model.predict_proba(X_te)[:, 1]\n\n    # Add index for submission\n    preds = pd.Series(y_pred,\n                      index=pd.Index(identifier + '_' + data.X_te.index.astype(str), name='ID'),\n                      name='target')\n    submission.append(preds)\n    \nprint()\nsubmission = pd.concat(submission)\nsubmission.to_csv('submission.csv', index=True)\n!head submission.csv","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}