{"cells":[{"metadata":{},"cell_type":"markdown","source":"# A big thanks to all kaggleer out there"},{"metadata":{},"cell_type":"markdown","source":"## Core Idea\n\nDespite a lot of creeping Physics and Chemistry knowledge introduced in the description, this competition is more about Geometry and pattern matching.\n\nThe hypothesis of this kernel is next:\n1. If we have two similar sets of atoms with the same distances between them and the same types - the scalar coupling constant should be very close.\n2. More closest atoms to the pair of atoms under prediction have higher influence on scalar coupling constant then those with higher distance\n\nSo, basically, this problem could be dealt with some kind of K-Nearest Neighbor algorithm or any tree-based - e.g. LightGBM, in case we can find some representation which would describe similar configurations with similar feature sets.\n\nEach atom is described with 3 cartesian coordinates. This representation is not stable. Each coupling pair is located in a different point in space and two similar coupling sets would have very different X,Y,Z.\n\nSo, instead of using coordinates let's consider next system:\n1. Take each pair of atoms as two first core atoms\n2. Calculate the center between the pair\n3. Find all n-nearest atoms to the center (excluding first two atoms)\n4. Take two closest atoms from step 3 - they will be 3rd and 4th core atoms\n5. Calculate the distances from 4 core atoms to the rest of the atoms and to the core atoms as well\n\nUsing this representation each atom position can be described by 4 distances from the core atoms. This representation is stable to rotation and translation. And it's suitable for pattern-matching. So, we can take a sequence of atoms, describe each by 4 distances + atom type(H,O,etc) and looking up for the same pattern we can find similar configurations and detect scalar coupling constant.\n\nHere I used LightGBM, because sklearn KNN can't deal with the amount of data. My blind guess is that hand-crafted KNN can outperform LightGBM.\n\nLet's code the solution!"},{"metadata":{},"cell_type":"markdown","source":"## Load Everything"},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nimport pandas as pd\nimport numpy as np\n\nimport math\nimport gc\nimport copy\n\nfrom sklearn.model_selection import KFold, train_test_split\nfrom sklearn.metrics import mean_absolute_error\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom lightgbm import LGBMRegressor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = '../input'\nSUBMISSIONS_PATH = './'\n# use atomic numbers to recode atomic names\nATOMIC_NUMBERS = {\n    'H': 1,\n    'C': 6,\n    'N': 7,\n    'O': 8,\n    'F': 9\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('display.max_colwidth', -1)\npd.set_option('display.max_rows', 120)\npd.set_option('display.max_columns', 120)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Dataset"},{"metadata":{},"cell_type":"markdown","source":"By default all data is read as `float64` and `int64`. We can trade this uneeded precision for memory and higher prediction speed. So, let's read with Pandas all the data in the minimal representation: "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dtypes = {\n    'molecule_name': 'category',\n    'atom_index_0': 'int8',\n    'atom_index_1': 'int8',\n    'type': 'category',\n    'scalar_coupling_constant': 'float32'\n}\ntrain_csv = pd.read_csv(f'{DATA_PATH}/train.csv', index_col='id', dtype=train_dtypes)\ntrain_csv['molecule_index'] = train_csv.molecule_name.str.replace('dsgdb9nsd_', '').astype('int32')\ntrain_csv = train_csv[['molecule_index', 'atom_index_0', 'atom_index_1', 'type', 'scalar_coupling_constant']]\ntrain_csv.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Shape: ', train_csv.shape)\nprint('Total: ', train_csv.memory_usage().sum())\ntrain_csv.memory_usage()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_csv = pd.read_csv(f'{DATA_PATH}/sample_submission.csv', index_col='id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_csv = pd.read_csv(f'{DATA_PATH}/test.csv', index_col='id', dtype=train_dtypes)\ntest_csv['molecule_index'] = test_csv['molecule_name'].str.replace('dsgdb9nsd_', '').astype('int32')\ntest_csv = test_csv[['molecule_index', 'atom_index_0', 'atom_index_1', 'type']]\ntest_csv.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"structures_dtypes = {\n    'molecule_name': 'category',\n    'atom_index': 'int8',\n    'atom': 'category',\n    'x': 'float32',\n    'y': 'float32',\n    'z': 'float32'\n}\nstructures_csv = pd.read_csv(f'{DATA_PATH}/structures.csv', dtype=structures_dtypes)\nstructures_csv['molecule_index'] = structures_csv.molecule_name.str.replace('dsgdb9nsd_', '').astype('int32')\nstructures_csv = structures_csv[['molecule_index', 'atom_index', 'atom', 'x', 'y', 'z']]\nstructures_csv['atom'] = structures_csv['atom'].replace(ATOMIC_NUMBERS).astype('int8')\nstructures_csv.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Shape: ', structures_csv.shape)\nprint('Total: ', structures_csv.memory_usage().sum())\nstructures_csv.memory_usage()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build Distance Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_type_dataframes(base, structures, coupling_type):\n    base = base[base['type'] == coupling_type].drop('type', axis=1).copy()\n    base = base.reset_index()\n    base['id'] = base['id'].astype('int32')\n    structures = structures[structures['molecule_index'].isin(base['molecule_index'])]\n    return base, structures","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_coordinates(base, structures, index):\n    df = pd.merge(base, structures, how='inner',\n                  left_on=['molecule_index', f'atom_index_{index}'],\n                  right_on=['molecule_index', 'atom_index']).drop(['atom_index'], axis=1)\n    df = df.rename(columns={\n        'atom': f'atom_{index}',\n        'x': f'x_{index}',\n        'y': f'y_{index}',\n        'z': f'z_{index}'\n    })\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_atoms(base, atoms):\n    df = pd.merge(base, atoms, how='inner',\n                  on=['molecule_index', 'atom_index_0', 'atom_index_1'])\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def merge_all_atoms(base, structures):\n    df = pd.merge(base, structures, how='left',\n                  left_on=['molecule_index'],\n                  right_on=['molecule_index'])\n    df = df[(df.atom_index_0 != df.atom_index) & (df.atom_index_1 != df.atom_index)]\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_center(df):\n    df['x_c'] = ((df['x_1'] + df['x_0']) * np.float32(0.5))\n    df['y_c'] = ((df['y_1'] + df['y_0']) * np.float32(0.5))\n    df['z_c'] = ((df['z_1'] + df['z_0']) * np.float32(0.5))\n\ndef add_distance_to_center(df):\n    df['d_c'] = ((\n        (df['x_c'] - df['x'])**np.float32(2) +\n        (df['y_c'] - df['y'])**np.float32(2) + \n        (df['z_c'] - df['z'])**np.float32(2)\n    )**np.float32(0.5))\n\ndef add_distance_between(df, suffix1, suffix2):\n    df[f'd_{suffix1}_{suffix2}'] = ((\n        (df[f'x_{suffix1}'] - df[f'x_{suffix2}'])**np.float32(2) +\n        (df[f'y_{suffix1}'] - df[f'y_{suffix2}'])**np.float32(2) + \n        (df[f'z_{suffix1}'] - df[f'z_{suffix2}'])**np.float32(2)\n    )**np.float32(0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_distances(df):\n    n_atoms = 1 + max([int(c.split('_')[1]) for c in df.columns if c.startswith('x_')])\n    \n    for i in range(1, n_atoms):\n        for vi in range(min(4, i)):\n            add_distance_between(df, i, vi)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_n_atoms(base, structures):\n    dfs = structures['molecule_index'].value_counts().rename('n_atoms').to_frame()\n    return pd.merge(base, dfs, left_on='molecule_index', right_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_couple_dataframe(some_csv, structures_csv, coupling_type, n_atoms=10):\n    base, structures = build_type_dataframes(some_csv, structures_csv, coupling_type)\n    base = add_coordinates(base, structures, 0)\n    base = add_coordinates(base, structures, 1)\n    \n    base = base.drop(['atom_0', 'atom_1'], axis=1)\n    atoms = base.drop('id', axis=1).copy()\n    if 'scalar_coupling_constant' in some_csv:\n        atoms = atoms.drop(['scalar_coupling_constant'], axis=1)\n        \n    add_center(atoms)\n    atoms = atoms.drop(['x_0', 'y_0', 'z_0', 'x_1', 'y_1', 'z_1'], axis=1)\n\n    atoms = merge_all_atoms(atoms, structures)\n    \n    add_distance_to_center(atoms)\n    \n    atoms = atoms.drop(['x_c', 'y_c', 'z_c', 'atom_index'], axis=1)\n    atoms.sort_values(['molecule_index', 'atom_index_0', 'atom_index_1', 'd_c'], inplace=True)\n    atom_groups = atoms.groupby(['molecule_index', 'atom_index_0', 'atom_index_1'])\n    atoms['num'] = atom_groups.cumcount() + 2\n    atoms = atoms.drop(['d_c'], axis=1)\n    atoms = atoms[atoms['num'] < n_atoms]\n\n    atoms = atoms.set_index(['molecule_index', 'atom_index_0', 'atom_index_1', 'num']).unstack()\n    atoms.columns = [f'{col[0]}_{col[1]}' for col in atoms.columns]\n    atoms = atoms.reset_index()\n    \n    # downcast back to int8\n    for col in atoms.columns:\n        if col.startswith('atom_'):\n            atoms[col] = atoms[col].fillna(0).astype('int8')\n            \n    atoms['molecule_index'] = atoms['molecule_index'].astype('int32')\n    \n    full = add_atoms(base, atoms)\n    add_distances(full)\n    \n    full.sort_values('id', inplace=True)\n    \n    return full","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def take_n_atoms(df, n_atoms, four_start=4):\n    labels = []\n    for i in range(2, n_atoms):\n        label = f'atom_{i}'\n        labels.append(label)\n\n    for i in range(n_atoms):\n        num = min(i, 4) if i < four_start else 4\n        for j in range(num):\n            labels.append(f'd_{i}_{j}')\n    if 'scalar_coupling_constant' in df:\n        labels.append('scalar_coupling_constant')\n    return df[labels]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check LightGBM with the smallest type"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndef type_select(types = '1JHN'):\n    full = build_couple_dataframe(train_csv, structures_csv, types, n_atoms=10)\n    print(full.shape)\n\n\n    df = take_n_atoms(full, 7)\n    # LightGBM performs better with 0-s then with NaN-s\n    df = df.fillna(0)\n    # df.columns\n\n    X_data = df.drop(['scalar_coupling_constant'], axis=1).values.astype('float32')\n    y_data = df['scalar_coupling_constant'].values.astype('float32')\n\n    X_train, X_val, y_train, y_val = train_test_split(X_data, y_data, test_size=0.2, random_state=128)\n    print(X_train.shape, X_val.shape, y_train.shape, y_val.shape)\n    \n    return X_train, X_val, y_train, y_val\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We don't calculate distances for `d_0_x`, `d_1_1`, `d_2_2`, `d_2_3`, `d_3_3` because we already have them in later atoms(`d_0_1` == `d_1_0`) or they are equal to zeros(e.g. `d_1_1`, `d_2_2`)."},{"metadata":{},"cell_type":"markdown","source":"For experiments, full dataset can be built with higher number of atoms, and for building a training/validation sets we can trim them:"},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# LGB_PARAMS = {\n#     'objective': 'regression',\n#     'metric': 'mae',\n#     'verbosity': -1,\n#     'boosting_type': 'gbdt',\n#     'learning_rate': 0.1455,\n#     'num_leaves': 128,\n#     'min_child_samples': 79,\n#     'max_depth': 11,\n#     'subsample_freq': 1,\n#     'subsample': 0.88,\n#     'bagging_seed': 14,\n#     'reg_alpha': 0.1,\n#     'reg_lambda': 0.3,\n#     'colsample_bytree': 1.0\n# }\n\n# -1.0147180396368805\n\n# model = LGBMRegressor(**LGB_PARAMS, n_estimators=3000, n_jobs = -1)\n# model.fit(X_train, y_train, \n#         eval_set=[(X_train, y_train), (X_val, y_val)], eval_metric='mae',\n#         verbose=400, early_stopping_rounds=500)\n\n# y_pred = model.predict(X_val)\n# np.log(mean_absolute_error(y_val, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# LGB_PARAMS = {\n#     'objective': 'regression',\n#     'metric': 'mae',\n#     'verbosity': -1,\n#     'boosting_type': 'gbdt',\n#     'learning_rate': 0.1455,\n#     'num_leaves': 129,\n#     'min_child_samples': 79,\n#     'max_depth': 13,\n#     'subsample_freq': 1,\n#     'subsample': 0.88,\n#     'bagging_seed': 15,\n#     'reg_alpha': 0.10108,\n#     'reg_lambda': 0.30013,\n#     'colsample_bytree': 1.0\n# }\n# #-1.0178287146032219 #r_a .101\n# -1.0179811056684678 #'learning_rate': 0.1455, 'max_depth': 12, 'subsample': 0.88, \n\n# model = LGBMRegressor(**LGB_PARAMS, n_estimators=3000, n_jobs = -1)\n# model.fit(X_train, y_train, \n#         eval_set=[(X_train, y_train), (X_val, y_val)], eval_metric='mae',\n#         verbose=400, early_stopping_rounds=500)\n\n# y_pred = model.predict(X_val)\n# np.log(mean_absolute_error(y_val, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_params = {\n    '1JHN': 7,\n    '1JHC': 10,\n    '2JHH': 9,\n    '2JHN': 9,\n    '2JHC': 9,\n    '3JHH': 9,\n    '3JHC': 10,\n    '3JHN': 10\n}\n\nmodel_params.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = type_select(types = '3JHN')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nLGB_PARAMS = {\n    'objective': 'regression',\n    'metric': 'mae',\n    'verbosity': -1,\n    'boosting_type': 'gbdt',\n    'learning_rate': 0.1455,\n    'num_leaves': 129,\n    'min_child_samples': 78,\n    'max_depth': 13,\n    'subsample_freq': 1,\n    'subsample': 0.88,\n    'bagging_seed': 15,\n    'reg_alpha': 0.10107001,\n    'reg_lambda': 0.300132,\n    'colsample_bytree': 1.0\n}\n# -1.0327442550828587\n\nLGB_PARAMS={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.14848931924611134, 'max_depth': -1, 'min_child_samples': 80, 'categorical_feature':categorical_feature, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-2.181925527723545\n\n\nmodel = LGBMRegressor(**LGB_PARAMS, n_estimators=3000, n_jobs = -1)\nmodel.fit(X_train, y_train, \n        eval_set=[(X_train, y_train), (X_val, y_val)], eval_metric='mae',\n        verbose=400, early_stopping_rounds=500)\n\ny_pred = model.predict(X_val)\nnp.log(mean_absolute_error(y_val, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_feature=[0,1,2,3,4]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# LGB_PARAMS_3JHN={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.15848931924611134, 'max_depth': 14, 'min_child_samples': 80,  'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.23, 'subsample': 0.9, 'subsample_freq': 1, 'verbosity': -1}\n\n# LGB_PARAMS_3JHN={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.15848931924611134, 'max_depth': 14, 'min_child_samples': 80, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1}  #-2.1795166793513037\n\n# LGB_PARAMS_3JHN={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.15848931924611134, 'max_depth': 14, 'min_child_samples': 80, 'categorical_feature':categorical_feature, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-2.1795305847801667\n\n# LGB_PARAMS_3JHN={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.14848931924611134, 'max_depth': 14, 'min_child_samples': 80, 'categorical_feature':categorical_feature, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-2.181925527723545\n\n\nLGB_PARAMS_3JHN={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.14548931924611134, 'max_depth': 14, 'min_child_samples': 80,  'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-2.181925527723545","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# from sklearn.preprocessing import CategoricalEncoder\n\n# cat_encoder = CategoricalEncoder()\n\n# pd.DataFrame(X_train).columns[:4]\n\n# housing_cat_reshaped = pd.DataFrame(X_train).values.reshape(-1, 1)\n# housing_cat_1hot = cat_encoder.fit_trasform(housing_cat_reshaped)\n# housing_cat_1hot","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n\n# model = LGBMRegressor(**LGB_PARAMS_3JHN, n_estimators=3000, n_jobs = -1)\n# model.fit(X_train, y_train, \n#         eval_set=[(X_train, y_train), (X_val, y_val)], eval_metric='mae',\n#         verbose=400, early_stopping_rounds=500, **categorical_feature)\n\n# y_pred = model.predict(X_val)\n# np.log(mean_absolute_error(y_val, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_x_y_data(some_csv, coupling_type, n_atoms):\n    full = build_couple_dataframe(some_csv, structures_csv, coupling_type, n_atoms=n_atoms)\n    \n    df = take_n_atoms(full, n_atoms)\n    df = df.fillna(0)\n    print(df.columns)\n    \n    if 'scalar_coupling_constant' in df:\n        X_data = df.drop(['scalar_coupling_constant'], axis=1).values.astype('float32')\n        y_data = df['scalar_coupling_constant'].values.astype('float32')\n    else:\n        X_data = df.values.astype('float32')\n        y_data = None\n    \n    return X_data, y_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_and_predict_for_one_coupling_type(coupling_type, submission, n_atoms, n_folds=4, n_splits=4, random_state=128):\n    print(f'*** Training Model for {coupling_type} ***')\n    \n    X_data, y_data = build_x_y_data(train_csv, coupling_type, n_atoms)\n    X_test, _ = build_x_y_data(test_csv, coupling_type, n_atoms)\n    y_pred = np.zeros(X_test.shape[0], dtype='float32')\n\n    cv_score = 0\n    \n    if n_folds > n_splits:\n        n_splits = n_folds\n    \n    kfold = KFold(n_splits=n_splits, shuffle=True, random_state=random_state)\n    print('fitting {} with {} LGB_PARAMS'.format(coupling_type,LGB_PARAMS))\n    for fold, (train_index, val_index) in enumerate(kfold.split(X_data, y_data)):\n        if fold >= n_folds:\n            break\n\n        X_train, X_val = X_data[train_index], X_data[val_index]\n        y_train, y_val = y_data[train_index], y_data[val_index]\n\n        model = LGBMRegressor(**LGB_PARAMS, n_estimators= 9000, n_jobs = -1)\n        model.fit(X_train, y_train, \n            eval_set=[(X_train, y_train), (X_val, y_val)], eval_metric='mae',\n            verbose=1000, early_stopping_rounds=500)\n\n        y_val_pred = model.predict(X_val)\n        val_score = np.log(mean_absolute_error(y_val, y_val_pred))\n        print(f'{coupling_type} Fold {fold}, logMAE: {val_score}')\n        \n        cv_score += val_score / n_folds\n        y_pred += model.predict(X_test) / n_folds\n        \n    plt.scatter(y_val, y_val_pred)   \n    plt.show()\n    submission.loc[test_csv['type'] == coupling_type, 'scalar_coupling_constant'] = y_pred\n    return cv_score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model"},{"metadata":{},"cell_type":"markdown","source":"Let's build a separate model for each type of coupling. Dataset is split into 5 pieces and in this kernel we will use only 3 folds for speed up.\n\nMain tuning parameter is the number of atoms. I took good numbers, but accuracy can be improved a bit by tuning them for each type."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model_params = {\n    '1JHN': 7,\n    '1JHC': 10,\n    '2JHH': 9,\n    '2JHN': 9,\n    '2JHC': 9,\n    '3JHH': 9,\n    '3JHC': 10,\n    '3JHN': 10\n}\n\n\n\ncat_code = {'1JHN': 5,'1JHC': 7,'2JHH': 7,\n             '2JHN': 7,'2JHC': 7,'3JHH': 7,\n            '3JHC': 8,'3JHN': 7}\n\ncategorical_feature = [0,1,2,3,4,5,6]\n\n\n\nLGB_PARAMS_2JHC={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.14548931824400012, 'max_depth': -1, 'min_child_samples': 81,'categorical_feature':categorical_feature, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.30, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-1.10519978867864\nLGB_PARAMS_3JHN={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.1454893092, 'max_depth': -1, 'min_child_samples': 81,  'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-2.181925527723545\nLGB_PARAMS_2JHN={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.1454893072, 'max_depth': -1, 'min_child_samples': 82,  'num_leaves': 130, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-2.181925527723545\n\n\nN_FOLDS = 5\nsubmission = submission_csv.copy()\n\ncv_scores = {}\nfor coupling_type in model_params.keys():\n    if( coupling_type == '2JHC'):\n        categorical_feature = [0,1,2,3,4,5,6]\n        LGB_PARAMS = {'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.14548931824400012, 'max_depth': -1, 'min_child_samples': 81,'categorical_feature':categorical_feature, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.30, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1}\n    elif(coupling_type == '3JHN'):\n        categorical_feature = [0,1,2,3,4,5,6]\n        LGB_PARAMS = {'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.1454893092, 'max_depth': -1, 'min_child_samples': 81,  'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} \n    elif(coupling_type == '2JHN'):\n        categorical_feature = [0,1,2,3,4,5,6]\n        LGB_PARAMS = {'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.1454893072, 'max_depth': -1, 'min_child_samples': 82,  'num_leaves': 130, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.3, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1}\n    elif(coupling_type == '3JHC'):\n        categorical_feature = [0,1,2,3,4,5,6,7]\n        LGB_PARAMS={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.14548931824400012, 'max_depth': -1, 'min_child_samples': 81,'categorical_feature':categorical_feature, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.30, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-1.10519978867864\n    elif(coupling_type == '1JHN'):\n        categorical_feature = [0,1,2,3,4]\n        LGB_PARAMS={'bagging_seed': 14, 'colsample_bytree': 1.0, 'learning_rate': 0.1454893092, 'max_depth': -1, 'min_child_samples': 81,'categorical_feature':categorical_feature, 'num_leaves': 129, 'random_state': 42, 'reg_alpha': 0.1, 'reg_lambda': 0.30, 'subsample': 0.89, 'subsample_freq': 1, 'verbosity': -1} #-1.10519978867864\n\n    else:\n        LGB_PARAMS = LGB_PARAMS_2JHN\n    cv_score = train_and_predict_for_one_coupling_type(\n        coupling_type, submission, n_atoms=model_params[coupling_type], n_folds=N_FOLDS)\n    cv_scores[coupling_type] = cv_score\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Checking cross-validation scores for each type:"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.mean(list(cv_scores.values()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sanity check for all cells to be filled with predictions:"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission Model"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"submission.to_csv(f'{SUBMISSIONS_PATH}/submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Room for improvement"},{"metadata":{},"cell_type":"markdown","source":"There are many steps, how to improve the score for this kernel:\n* Tune LGB hyperparameters - I did nothing for this   **checked***\n* Tune number of atoms for each type        **checked**\n* Try to add other features\n* Play with categorical features for atom types (one-hot-encoding, CatBoost?)   **checked***\n* Try other tree libraries\n\nAlso, this representation fails badly on `*JHC` coupling types. The main reason for this is that 3rd and 4th atoms are usually located on the same distance and representation starts \"jittering\" randomly picking one of them. So, two similar configurations will have different representation due to usage of 3/4 of 4/3 distances.\n\nThe biggest challenge would be to implement handcrafted KNN with some compiled language(Rust, C++, C).\n\nWould be cool to see this kernel forked and addressed some of the issues with higher LB score."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}