{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Motivation\nIn this competition locally I generated 200+ features, and it is too much for my RAM (I think, you have the same problem) <br>\nI wanted to reduce usage of memory and it can be done in following ways: <br>\n- `reduce_mem_usage` function\n- split dataset by type and fit different models separately <br>\n- feature reduction <br><br>\nIn this kernel I will cover method `Permutation importance` that the case of feature selection."},{"metadata":{},"cell_type":"markdown","source":"# The Idea\nWe train one model on all generated features. After that you can permutate column one by one and look at the change of score on hold-out validation.\nIf it is getting much worse, then this column contains useful information about target. If it is doesn't change or getting better, most probably this feature is useless. After that you return the column to its normal state and go to next column. <br><br>\nBenefits of this method:\n - It is fast. Model is trained once. it is better than greedy algorithm, because you don't need to re-fit your model each time\n - When column is permutated, it has the same distribution. It means - no bias towards some classes or modes in target distribution.\n - Experiment can be ran several times so mean and std of error change can be measured."},{"metadata":{},"cell_type":"markdown","source":"# Imports and utils"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport time\nimport datetime\nimport gc\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import GroupKFold\nfrom sklearn.metrics import mean_absolute_error\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm import tqdm_notebook as tqdm\n\nfrom catboost import CatBoostRegressor, Pool\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        \n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max and c_prec == np.finfo(np.float16).precision:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max and c_prec == np.finfo(np.float32).precision:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n\ndef group_mean_log_mae(y_true, y_pred, types, floor=1e-9):\n    \"\"\"\n    Fast metric computation for this competition: https://www.kaggle.com/c/champs-scalar-coupling\n    Code is from this kernel: https://www.kaggle.com/uberkinder/efficient-metric\n    \"\"\"\n    maes = (y_true-y_pred).abs().groupby(types).mean()\n    return np.log(maes.map(lambda x: max(x, floor))).mean()\n\ndef encode_categoric(df):\n    lbl = LabelEncoder()\n    cat_cols=[]\n    try:\n        cat_cols = df.describe(include=['O']).columns.tolist()\n        for cat in cat_cols:\n            df[cat] = lbl.fit_transform(list(df[cat].values))\n    except Exception as e:\n        print('error: ', str(e) )\n\n    return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load train data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\nstructures = pd.read_csv('../input/structures.csv')\n\nprint('Train dataset shape is -> rows: {} cols:{}'.format(train.shape[0],train.shape[1]))\nprint('Structures dataset shape is  -> rows: {} cols:{}'.format(structures.shape[0],structures.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"unique_molecules = train['molecule_name'].unique()\n\nprint(\"Few examples of molecule's names: \", '  '.join(unique_molecules[:3]), end='\\n\\n')\nprint('Amount of unique molecules in train: ', len(unique_molecules))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's use subset of training data due kaggle kernel power constraints"},{"metadata":{"trusted":true},"cell_type":"code","source":"molecules_fraction = 0.1\nmolecules_amount = int(molecules_fraction * len(unique_molecules))\n\nnp.random.shuffle(unique_molecules)\ntrain_molecules = unique_molecules[:molecules_amount]\n\ntrain = train[train['molecule_name'].isin(train_molecules)]\n\nprint(f'Amount of molecules in the subset of train: {molecules_amount}, samples: {train.shape[0]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All these feture generation functions based on public kernels:\n - https://www.kaggle.com/artgor/using-meta-features-to-improve-model\n - https://www.kaggle.com/aekoch95/bonds-from-structure-data\n - https://www.kaggle.com/adrianoavelar/bond-calculation-lb-0-82\n - https://www.kaggle.com/kmat2019/effective-feature\n \n**Please upvote them!**"},{"metadata":{},"cell_type":"markdown","source":"# Create features based on structures.csv"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def atomic_radius_electonegativety(structures):\n    atomic_radius = {'H':0.38, 'C':0.77, 'N':0.75, 'O':0.73, 'F':0.71} # Without fudge factor\n    fudge_factor = 0.05\n    atomic_radius = {k:v + fudge_factor for k,v in atomic_radius.items()}\n\n    electronegativity = {'H':2.2, 'C':2.55, 'N':3.04, 'O':3.44, 'F':3.98}\n\n    atoms = structures['atom'].values\n    atoms_en = [electronegativity[x] for x in atoms]\n    atoms_rad = [atomic_radius[x] for x in atoms]\n\n    structures['EN'] = atoms_en\n    structures['rad'] = atoms_rad\n    \n    return structures\n\n\ndef create_bonds(structures):\n    i_atom = structures['atom_index'].values\n    p = structures[['x', 'y', 'z']].values\n    p_compare = p\n    m = structures['molecule_name'].values\n    m_compare = m\n    r = structures['rad'].values\n    r_compare = r\n\n    source_row = np.arange(len(structures))\n    max_atoms = 28\n\n    bonds = np.zeros((len(structures)+1, max_atoms+1), dtype=np.int8)\n    bond_dists = np.zeros((len(structures)+1, max_atoms+1), dtype=np.float32)\n\n#     print('Calculating bonds')\n\n    for i in range(max_atoms-1):\n        p_compare = np.roll(p_compare, -1, axis=0)\n        m_compare = np.roll(m_compare, -1, axis=0)\n        r_compare = np.roll(r_compare, -1, axis=0)\n\n        mask = np.where(m == m_compare, 1, 0) #Are we still comparing atoms in the same molecule?\n        dists = np.linalg.norm(p - p_compare, axis=1) * mask\n        r_bond = r + r_compare\n\n        bond = np.where(np.logical_and(dists > 0.0001, dists < r_bond), 1, 0)\n\n        source_row = source_row\n        target_row = source_row + i + 1 #Note: Will be out of bounds of bonds array for some values of i\n        target_row = np.where(np.logical_or(target_row > len(structures), mask==0), len(structures), target_row) #If invalid target, write to dummy row\n\n        source_atom = i_atom\n        target_atom = i_atom + i + 1 #Note: Will be out of bounds of bonds array for some values of i\n        target_atom = np.where(np.logical_or(target_atom > max_atoms, mask==0), max_atoms, target_atom) #If invalid target, write to dummy col\n\n        bonds[(source_row, target_atom)] = bond\n        bonds[(target_row, source_atom)] = bond\n        bond_dists[(source_row, target_atom)] = dists\n        bond_dists[(target_row, source_atom)] = dists\n\n    bonds = np.delete(bonds, axis=0, obj=-1) #Delete dummy row\n    bonds = np.delete(bonds, axis=1, obj=-1) #Delete dummy col\n    bond_dists = np.delete(bond_dists, axis=0, obj=-1) #Delete dummy row\n    bond_dists = np.delete(bond_dists, axis=1, obj=-1) #Delete dummy col\n\n#     print('Counting and condensing bonds')\n\n    bonds_numeric = [[i for i,x in enumerate(row) if x] for row in bonds]\n    bond_lengths = [[dist for i,dist in enumerate(row) if i in bonds_numeric[j]] for j,row in enumerate(bond_dists)]\n    bond_lengths_mean = [ np.mean(x) for x in bond_lengths]\n    n_bonds = [len(x) for x in bonds_numeric]\n\n\n    bond_data = {'n_bonds':n_bonds, 'bond_lengths_mean': bond_lengths_mean }\n    bond_df = pd.DataFrame(bond_data)\n    structures = structures.join(bond_df)\n    \n    return structures\n\ndef map_atom_info(df, atom_idx):\n    df = pd.merge(df, structures, how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    \n    df = df.drop('atom_index', axis=1)\n    df = df.rename(columns={'atom': f'atom_{atom_idx}',\n                            'x': f'x_{atom_idx}',\n                            'y': f'y_{atom_idx}',\n                            'z': f'z_{atom_idx}',\n                            'EN': f'EN_{atom_idx}',\n                            'rad': f'rad_{atom_idx}',\n                            'n_bonds': f'n_bonds_{atom_idx}',\n                            'bond_lengths_mean': f'bond_lengths_mean_{atom_idx}',\n                           })\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"structures = atomic_radius_electonegativety(structures)\nstructures = create_bonds(structures)\n\ntrain = map_atom_info(train, 0)\ntrain = map_atom_info(train, 1)\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature generation"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def distances(df):\n    df_p_0 = df[['x_0', 'y_0', 'z_0']].values\n    df_p_1 = df[['x_1', 'y_1', 'z_1']].values\n    \n    df['dist'] = np.linalg.norm(df_p_0 - df_p_1, axis=1)\n    df['dist_x'] = (df['x_0'] - df['x_1']) ** 2\n    df['dist_y'] = (df['y_0'] - df['y_1']) ** 2\n    df['dist_z'] = (df['z_0'] - df['z_1']) ** 2\n    \n    df['type_0'] = df['type'].apply(lambda x: x[0])\n    \n    return df\n\ndef map_atom_info(df_1,df_2, atom_idx):\n    df = pd.merge(df_1, df_2, how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    df = df.drop('atom_index', axis=1)\n\n    return df\n\ndef create_closest(df):\n    df_temp=df.loc[:,[\"molecule_name\",\"atom_index_0\",\"atom_index_1\",\"dist\",\"x_0\",\"y_0\",\"z_0\",\"x_1\",\"y_1\",\"z_1\"]].copy()\n    df_temp_=df_temp.copy()\n    df_temp_= df_temp_.rename(columns={'atom_index_0': 'atom_index_1',\n                                       'atom_index_1': 'atom_index_0',\n                                       'x_0': 'x_1',\n                                       'y_0': 'y_1',\n                                       'z_0': 'z_1',\n                                       'x_1': 'x_0',\n                                       'y_1': 'y_0',\n                                       'z_1': 'z_0'})\n    df_temp=pd.concat(objs=[df_temp,df_temp_],axis=0)\n\n    df_temp[\"min_distance\"]=df_temp.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('min')\n    df_temp= df_temp[df_temp[\"min_distance\"]==df_temp[\"dist\"]]\n\n    df_temp=df_temp.drop(['x_0','y_0','z_0','min_distance', 'dist'], axis=1)\n    df_temp= df_temp.rename(columns={'atom_index_0': 'atom_index',\n                                     'atom_index_1': 'atom_index_closest',\n                                     'distance': 'distance_closest',\n                                     'x_1': 'x_closest',\n                                     'y_1': 'y_closest',\n                                     'z_1': 'z_closest'})\n\n    for atom_idx in [0,1]:\n        df = map_atom_info(df,df_temp, atom_idx)\n        df = df.rename(columns={'atom_index_closest': f'atom_index_closest_{atom_idx}',\n                                        'distance_closest': f'distance_closest_{atom_idx}',\n                                        'x_closest': f'x_closest_{atom_idx}',\n                                        'y_closest': f'y_closest_{atom_idx}',\n                                        'z_closest': f'z_closest_{atom_idx}'})\n    return df\n\ndef add_cos_features(df):\n    df[\"distance_0\"]=((df['x_0']-df['x_closest_0'])**2+(df['y_0']-df['y_closest_0'])**2+(df['z_0']-df['z_closest_0'])**2)**(1/2)\n    df[\"distance_1\"]=((df['x_1']-df['x_closest_1'])**2+(df['y_1']-df['y_closest_1'])**2+(df['z_1']-df['z_closest_1'])**2)**(1/2)\n    df[\"vec_0_x\"]=(df['x_0']-df['x_closest_0'])/df[\"distance_0\"]\n    df[\"vec_0_y\"]=(df['y_0']-df['y_closest_0'])/df[\"distance_0\"]\n    df[\"vec_0_z\"]=(df['z_0']-df['z_closest_0'])/df[\"distance_0\"]\n    df[\"vec_1_x\"]=(df['x_1']-df['x_closest_1'])/df[\"distance_1\"]\n    df[\"vec_1_y\"]=(df['y_1']-df['y_closest_1'])/df[\"distance_1\"]\n    df[\"vec_1_z\"]=(df['z_1']-df['z_closest_1'])/df[\"distance_1\"]\n    df[\"vec_x\"]=(df['x_1']-df['x_0'])/df[\"dist\"]\n    df[\"vec_y\"]=(df['y_1']-df['y_0'])/df[\"dist\"]\n    df[\"vec_z\"]=(df['z_1']-df['z_0'])/df[\"dist\"]\n    df[\"cos_0_1\"]=df[\"vec_0_x\"]*df[\"vec_1_x\"]+df[\"vec_0_y\"]*df[\"vec_1_y\"]+df[\"vec_0_z\"]*df[\"vec_1_z\"]\n    df[\"cos_0\"]=df[\"vec_0_x\"]*df[\"vec_x\"]+df[\"vec_0_y\"]*df[\"vec_y\"]+df[\"vec_0_z\"]*df[\"vec_z\"]\n    df[\"cos_1\"]=df[\"vec_1_x\"]*df[\"vec_x\"]+df[\"vec_1_y\"]*df[\"vec_y\"]+df[\"vec_1_z\"]*df[\"vec_z\"]\n    df=df.drop(['vec_0_x','vec_0_y','vec_0_z','vec_1_x','vec_1_y','vec_1_z','vec_x','vec_y','vec_z'], axis=1)\n    return df\n\ndef create_features(df):\n    df['molecule_couples'] = df.groupby('molecule_name')['id'].transform('count')\n    df['molecule_dist_mean'] = df.groupby('molecule_name')['dist'].transform('mean')\n    df['molecule_dist_min'] = df.groupby('molecule_name')['dist'].transform('min')\n    df['molecule_dist_max'] = df.groupby('molecule_name')['dist'].transform('max')\n    df['atom_0_couples_count'] = df.groupby(['molecule_name', 'atom_index_0'])['id'].transform('count')\n    df['atom_1_couples_count'] = df.groupby(['molecule_name', 'atom_index_1'])['id'].transform('count')\n    df[f'molecule_atom_index_0_x_1_std'] = df.groupby(['molecule_name', 'atom_index_0'])['x_1'].transform('std')\n    df[f'molecule_atom_index_0_y_1_mean'] = df.groupby(['molecule_name', 'atom_index_0'])['y_1'].transform('mean')\n    df[f'molecule_atom_index_0_y_1_mean_diff'] = df[f'molecule_atom_index_0_y_1_mean'] - df['y_1']\n    df[f'molecule_atom_index_0_y_1_mean_div'] = df[f'molecule_atom_index_0_y_1_mean'] / df['y_1']\n    df[f'molecule_atom_index_0_y_1_max'] = df.groupby(['molecule_name', 'atom_index_0'])['y_1'].transform('max')\n    df[f'molecule_atom_index_0_y_1_max_diff'] = df[f'molecule_atom_index_0_y_1_max'] - df['y_1']\n    df[f'molecule_atom_index_0_y_1_std'] = df.groupby(['molecule_name', 'atom_index_0'])['y_1'].transform('std')\n    df[f'molecule_atom_index_0_z_1_std'] = df.groupby(['molecule_name', 'atom_index_0'])['z_1'].transform('std')\n    df[f'molecule_atom_index_0_dist_mean'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('mean')\n    df[f'molecule_atom_index_0_dist_mean_diff'] = df[f'molecule_atom_index_0_dist_mean'] - df['dist']\n    df[f'molecule_atom_index_0_dist_mean_div'] = df[f'molecule_atom_index_0_dist_mean'] / df['dist']\n    df[f'molecule_atom_index_0_dist_max'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('max')\n    df[f'molecule_atom_index_0_dist_max_diff'] = df[f'molecule_atom_index_0_dist_max'] - df['dist']\n    df[f'molecule_atom_index_0_dist_max_div'] = df[f'molecule_atom_index_0_dist_max'] / df['dist']\n    df[f'molecule_atom_index_0_dist_min'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('min')\n    df[f'molecule_atom_index_0_dist_min_diff'] = df[f'molecule_atom_index_0_dist_min'] - df['dist']\n    df[f'molecule_atom_index_0_dist_min_div'] = df[f'molecule_atom_index_0_dist_min'] / df['dist']\n    df[f'molecule_atom_index_0_dist_std'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('std')\n    df[f'molecule_atom_index_0_dist_std_diff'] = df[f'molecule_atom_index_0_dist_std'] - df['dist']\n    df[f'molecule_atom_index_0_dist_std_div'] = df[f'molecule_atom_index_0_dist_std'] / df['dist']\n    df[f'molecule_atom_index_1_dist_mean'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('mean')\n    df[f'molecule_atom_index_1_dist_mean_diff'] = df[f'molecule_atom_index_1_dist_mean'] - df['dist']\n    df[f'molecule_atom_index_1_dist_mean_div'] = df[f'molecule_atom_index_1_dist_mean'] / df['dist']\n    df[f'molecule_atom_index_1_dist_max'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('max')\n    df[f'molecule_atom_index_1_dist_max_diff'] = df[f'molecule_atom_index_1_dist_max'] - df['dist']\n    df[f'molecule_atom_index_1_dist_max_div'] = df[f'molecule_atom_index_1_dist_max'] / df['dist']\n    df[f'molecule_atom_index_1_dist_min'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('min')\n    df[f'molecule_atom_index_1_dist_min_diff'] = df[f'molecule_atom_index_1_dist_min'] - df['dist']\n    df[f'molecule_atom_index_1_dist_min_div'] = df[f'molecule_atom_index_1_dist_min'] / df['dist']\n    df[f'molecule_atom_index_1_dist_std'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('std')\n    df[f'molecule_atom_index_1_dist_std_diff'] = df[f'molecule_atom_index_1_dist_std'] - df['dist']\n    df[f'molecule_atom_index_1_dist_std_div'] = df[f'molecule_atom_index_1_dist_std'] / df['dist']\n    df[f'molecule_atom_1_dist_mean'] = df.groupby(['molecule_name', 'atom_1'])['dist'].transform('mean')\n    df[f'molecule_atom_1_dist_min'] = df.groupby(['molecule_name', 'atom_1'])['dist'].transform('min')\n    df[f'molecule_atom_1_dist_min_diff'] = df[f'molecule_atom_1_dist_min'] - df['dist']\n    df[f'molecule_atom_1_dist_min_div'] = df[f'molecule_atom_1_dist_min'] / df['dist']\n    df[f'molecule_atom_1_dist_std'] = df.groupby(['molecule_name', 'atom_1'])['dist'].transform('std')\n    df[f'molecule_atom_1_dist_std_diff'] = df[f'molecule_atom_1_dist_std'] - df['dist']\n    df[f'molecule_type_0_dist_std'] = df.groupby(['molecule_name', 'type_0'])['dist'].transform('std')\n    df[f'molecule_type_0_dist_std_diff'] = df[f'molecule_type_0_dist_std'] - df['dist']\n    df[f'molecule_type_dist_mean'] = df.groupby(['molecule_name', 'type'])['dist'].transform('mean')\n    df[f'molecule_type_dist_mean_diff'] = df[f'molecule_type_dist_mean'] - df['dist']\n    df[f'molecule_type_dist_mean_div'] = df[f'molecule_type_dist_mean'] / df['dist']\n    df[f'molecule_type_dist_max'] = df.groupby(['molecule_name', 'type'])['dist'].transform('max')\n    df[f'molecule_type_dist_min'] = df.groupby(['molecule_name', 'type'])['dist'].transform('min')\n    df[f'molecule_type_dist_std'] = df.groupby(['molecule_name', 'type'])['dist'].transform('std')\n    df[f'molecule_type_dist_std_diff'] = df[f'molecule_type_dist_std'] - df['dist']\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"start_time = time.time()\n\ntrain = distances(train)\n\nprint('Create closest features')\n\ntrain = create_closest(train)\n\nprint('Create cos features')\n\ntrain = add_cos_features(train)\n\nprint('Create groupby features', end='\\n\\n')\n\ntrain = create_features(train)\n\ntrain = reduce_mem_usage(train, verbose=False)\n\nprint('Train dataset shape is -> rows: {} cols:{}'.format(train.shape[0],train.shape[1]))\nprint('Structures dataset shape is  -> rows: {} cols:{}'.format(structures.shape[0],structures.shape[1]), end='\\n\\n')\nprint(f'Exe time: {(time.time() - start_time)/60:.2} min')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"molecules_id = train['molecule_name']\nX = train.drop(['id', 'scalar_coupling_constant', 'molecule_name'], axis=1)\ny = train['scalar_coupling_constant']\n\nX = encode_categoric(X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('X size', X.shape)\n\ndel train\ngc.collect()\nX.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's make group hold-out validation with 25% validation size\nI used GroupKFold because train/test split has no intersection in molecules, so this validation will be more veridical."},{"metadata":{"trusted":true},"cell_type":"code","source":"kf = GroupKFold(4)\nfor tr_idx, val_idx in kf.split(X, groups=molecules_id):\n    tr_X = X.iloc[tr_idx]; val_X = X.iloc[val_idx]\n    tr_y = y.iloc[tr_idx]; val_y = y.iloc[val_idx]\n    \n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Permutation importance implementation\n\n - `metric` - score function that have arguments: true_y and preds\n - `theshold` - threshold of score changing, confirming that score is useful\n - `minimize` - metric should be minimized or maximazed. In this competition `minimize=True`"},{"metadata":{"trusted":true},"cell_type":"code","source":"def permutation_importance(model, X_val, y_val, metric, threshold=0.005,\n                           minimize=True, verbose=True):\n    results = {}\n    \n    y_pred = model.predict(X_val)\n    \n    results['base_score'] = metric(y_val, y_pred)\n    if verbose:\n        print(f'Base score {results[\"base_score\"]:.5}')\n\n    \n    for col in tqdm(X_val.columns):\n        freezed_col = X_val[col].copy()\n\n        X_val[col] = np.random.permutation(X_val[col])\n        preds = model.predict(X_val)\n        results[col] = metric(y_val, preds)\n\n        X_val[col] = freezed_col\n        \n        if verbose:\n            print(f'column: {col} - {results[col]:.5}')\n    \n    if minimize:\n        bad_features = [k for k in results if results[k] < results['base_score'] + threshold]\n    else:\n        bad_features = [k for k in results if results[k] > results['base_score'] + threshold]\n    bad_features.remove('base_score')\n    \n    return results, bad_features","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Usage"},{"metadata":{},"cell_type":"markdown","source":"Fit model on all generated features"},{"metadata":{"trusted":true},"cell_type":"code","source":"def catboost_fit(model, X_train, y_train, X_val, y_val):\n    train_pool = Pool(X_train, y_train)\n    val_pool = Pool(X_val, y_val)\n    model.fit(train_pool, eval_set=val_pool)\n    \n    return model\n\nmodel = CatBoostRegressor(iterations=20000, \n                          max_depth=9,\n                          objective='MAE',\n                          task_type='GPU',\n                          verbose=False)\nmodel = catboost_fit(model, tr_X, tr_y, val_X, val_y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This wrapper for function is used for passing `types=val_X['type']` like it was defined by default: <br>\n`def group_mean_log_mae(y_true, y_pred, types=val_X['type']): ...`"},{"metadata":{"trusted":true},"cell_type":"code","source":"from functools import partial\nmetric = partial(group_mean_log_mae, types=val_X['type'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results, bad_features = permutation_importance(model=model,\n                                               X_val=val_X,\n                                               y_val=val_y,\n                                               metric=metric,\n                                               verbose=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`result` values contains score after permutatation of key column"},{"metadata":{"trusted":true},"cell_type":"code","source":"results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bad_features","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check what the score without `bad_features`"},{"metadata":{"trusted":true},"cell_type":"code","source":"tr_X_reduced = tr_X.drop(bad_features, axis=1).copy()\nval_X_reduced = val_X.drop(bad_features, axis=1).copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_reduced = CatBoostRegressor(iterations=20000, \n                          max_depth=9,\n                          objective='MAE',\n                          task_type='GPU',\n                          verbose=False)\nmodel_reduced = catboost_fit(model, tr_X_reduced, tr_y, val_X_reduced, val_y)\n\ny_pred = model_reduced.predict(val_X_reduced)\nnew_score = metric(val_y, y_pred)\n\nprint(f'Original score: {results[\"base_score\"]:.3}, amount of features: {len(results)-1}')\nprint(f'Score after removing bad_features: {new_score:.3}, amount of features: {tr_X_reduced.shape[1]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Also let's check implementation from eli5\n`get_score_importances` has some parameters:\n - `score_func`: your function with model inference and scoring.\n - `X`: features\n - `y`: target\n - `n_iter=5`: how many times columns will be permuted.\n - `columns_to_shuffle=None`: subset of columns to shuffle. If None, then all columns will be checked.\n - `random_state=None`"},{"metadata":{},"cell_type":"markdown","source":"Function returns:\n - `base_score`: score on original features\n - `score_decreases`: list of length `n_iter` with feature importance arrays"},{"metadata":{"trusted":true},"cell_type":"code","source":"from eli5.permutation_importance import get_score_importances\n\ndef score(X, y):\n    y_pred = model.predict(X)\n    return metric(y, y_pred)\n\nbase_score, score_decreases = get_score_importances(score, np.array(val_X), val_y, n_iter=1)\n\nthreshold = 0.001\nbad_features = val_X.columns[score_decreases[0] > -threshold]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr_X_reduced = tr_X.drop(bad_features, axis=1).copy()\nval_X_reduced = val_X.drop(bad_features, axis=1).copy()\n\nmodel_reduced = CatBoostRegressor(iterations=20000, \n                          max_depth=9,\n                          objective='MAE',\n                          task_type='GPU',\n                          verbose=False)\nmodel_reduced = catboost_fit(model_reduced, tr_X_reduced, tr_y, val_X_reduced, val_y)\n\ny_pred = model_reduced.predict(val_X_reduced)\nnew_score = metric(val_y, y_pred)\n\nprint(f'Original score: {base_score:.3}, amount of features: {len(results)-1}')\nprint(f'Score after removing bad_features: {new_score:.3}, amount of features: {val_X_reduced.shape[1]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\nThis method improves CV and removes redundant features from your data. You can try it on your data and models."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.4"}},"nbformat":4,"nbformat_minor":1}