{"cells":[{"metadata":{},"cell_type":"markdown","source":"### **1. Introduction**\n\nIn this kernel I share the pipeline to get a -2LB NN or LGB\n\nThe features used in this kernel and how to get them where shared in [Features for top 5% LB with NN or LGB](https://www.kaggle.com/felipemello/features-for-top-5-lb-with-nn-or-lgb):\n\nThere were a few tricks that boosted my score:\n- Using only features that actually impacted the model, as described in [permutation importance](https://www.kaggle.com/speedwagon/permutation-importance)\n- Using as objective MSE or logcosh, instead of MAE\n- Using as target for the NN the scalar coupling contributions ['fc', 'sd','pso', 'dso'], and not only the sum of them\n- Adding other models predictions/oof as features\n- Setting distances on LGB as yukawa distances, i.e. dist = exp(-dist)/dist\n- Getting the results of a NN layer as features for the LGB model\n- Adding the results as new features and running the model again\n\n**Please, if you find the content here interesting, consider upvoting the kernel to reward my time editing and sharing it. Thank you very much :)**\n\nIn this kernel we will be calculating predictions only for 1JHN and for 1 fold, for speed purposes, but this can be easily changed on section 3 settings"},{"metadata":{},"cell_type":"markdown","source":"### **2. Load libs and utils**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import StandardScaler\nimport tensorflow as tf\nfrom keras.layers import Dense, Input, Activation, Concatenate\nfrom keras.layers import BatchNormalization,Add,Dropout\nfrom keras.optimizers import Adam\nfrom keras.models import Model, load_model\nfrom keras import callbacks\nfrom keras import backend as K\nimport warnings\nimport os\nimport lightgbm as lgb\nimport copy\nimport gc\nimport random\nwarnings.filterwarnings(\"ignore\")\nwarnings.filterwarnings(action=\"ignore\",category=DeprecationWarning)\nwarnings.filterwarnings(action=\"ignore\",category=FutureWarning)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def calc_logmae(y_val, y_pred):\n    \n    return np.log(np.sum(np.abs(y_val - y_pred)/len(y_pred)))\n\n\ndef permutation_importance(model, X_val, y_val, calc_logmae, threshold=0.01,\n                           minimize=True, verbose=True):\n    results = {}\n    \n    y_pred = model.predict(X_val)\n    \n    results['base_score'] = calc_logmae(y_val, y_pred)\n    if verbose:\n        print(f'Base score {results[\"base_score\"]:.5}')\n\n    \n    for col in tqdm(X_val.columns):\n        freezed_col = X_val[col].copy()\n\n        X_val[col] = np.random.permutation(X_val[col])\n        preds = model.predict(X_val)\n        results[col] = calc_logmae(y_val, preds)\n\n        X_val[col] = freezed_col\n        \n        if verbose:\n            print(f'column: {col} - {results[col]:.5}')\n \n            \n    if minimize:\n        bad_features = [k for k in results if results[k] < results['base_score'] + threshold]\n    else:\n        bad_features = [k for k in results if results[k] > results['base_score'] + threshold]\n    bad_features.remove('base_score')\n    \n    return results, bad_features\n\ndef load_lgb_params(mol_type, n_estimators = 2000):\n    \n    seed = 2319\n    param_1J = {\n        'num_leaves': int(0.7*(25**2)), #int(0.7*(25**2))\n        'learning_rate': 0.1,\n        'feature_fraction': 1,\n        'save_binary': True, \n        'seed': seed,\n        'feature_fraction_seed': seed,\n        'bagging_seed': seed,\n        'drop_seed': seed,\n        'data_random_seed': seed,\n        'objective': 'regression_l2',\n        'boosting_type': 'gbdt',\n        'verbosity': -1,\n        'metric': 'mae',\n        'is_unbalance': True,\n        'boost_from_average': 'false',   \n    #    'device': 'gpu',\n    #    'gpu_platform_id': 0,\n    #    'gpu_device_id': 0,\n        'bagging_fraction': 1,\n         'bagging_freq': 0,\n         'lambda_l1': 0.5,\n         'lambda_l2': 1.7244553699717466,\n         'max_bin': 238, #238\n         'max_depth': 25, #int(25)\n    #     'min_data_in_leaf': int(203.49294923362797),\n    #     'min_gain_to_split': 0.0665822332705641,\n    #     'min_sum_hessian_in_leaf': 11.250160554801903,\n         'n_estimators': n_estimators,\n         'sparse_threshold': 1.0,\n          'n_jobs': 6}  \n    \n    param_2J = {\n        'num_leaves': int(1*(20**2)),\n        'learning_rate': 0.1,\n        'feature_fraction': 1,\n        'save_binary': True, \n        'seed': seed,\n        'feature_fraction_seed': seed,\n        'bagging_seed': seed,\n        'drop_seed': seed,\n        'data_random_seed': seed,\n        'objective': 'regression_l2',\n        'boosting_type': 'gbdt',\n        'verbosity': -1,\n        'metric': 'mae',\n        'is_unbalance': True,\n        'boost_from_average': 'false',   \n    #    'device': 'gpu',\n    #    'gpu_platform_id': 0,\n    #    'gpu_device_id': 0,\n        'bagging_fraction': 1,\n         'bagging_freq': int(0),\n         'lambda_l1': 1,\n         'lambda_l2': 1.89,\n         'max_bin': 255, #255\n         'max_depth': 20, #int(20)\n         'min_data_in_leaf': int(10),\n         'min_gain_to_split': 0,\n         'min_sum_hessian_in_leaf': 1/869,\n         'n_estimators': n_estimators,\n         'sparse_threshold': 1.0,\n         'n_jobs':6}  \n    \n    \n    param_3J = {\n        'num_leaves': int(1*(30**2)),\n        'learning_rate': 0.1,\n        'feature_fraction': 1,\n        'save_binary': True, \n        'seed': seed,\n        'feature_fraction_seed': seed,\n        'bagging_seed': seed,\n        'drop_seed': seed,\n        'data_random_seed': seed,\n        'objective': 'regression_l2',\n        'boosting_type': 'gbdt',\n        'verbosity': -1,\n        'metric': 'mae',\n        'is_unbalance': True,\n        'boost_from_average': 'false',   \n    #    'device': 'gpu',\n    #    'gpu_platform_id': 0,\n    #    'gpu_device_id': 0,\n        'bagging_fraction': 1,\n         'bagging_freq': int(0),\n         'lambda_l1': 0.5,\n         'lambda_l2': 1,\n         'max_bin': 50, #50\n         'max_depth': 20, #int(30)\n         'min_data_in_leaf': int(10),\n         'min_gain_to_split': 0,\n         'min_sum_hessian_in_leaf': 1/202,\n         'n_estimators': n_estimators,\n         'sparse_threshold': 1.0,\n         'n_jobs':6}  \n\n    if mol_type[0] == '1':\n        params = param_1J\n    if mol_type[0] == '2':\n        params = param_2J\n    if mol_type[0] == '3':\n        params = param_3J\n        \n    return params\n        \ndef create_nn_model(input_shape):\n    inp = Input(shape=(input_shape,))\n    \n    x = Dense(2048, activation=\"relu\")(inp)\n    x = BatchNormalization()(x)\n    x = Dense(1024, activation=\"relu\")(x)\n    x = BatchNormalization()(x)\n    x = Dense(512, activation=\"relu\")(x)\n    x = BatchNormalization()(x)\n    x = Dense(32, activation=\"relu\")(x)\n    x = BatchNormalization()(x)\n \n    out = Dense(5, activation=\"linear\")(x)  \n\n    model = Model(inputs=inp, outputs=out)\n    return model\n\ndef plot_history(history, label):\n    plt.plot(history.history['loss'])\n    plt.plot(history.history['val_loss'])\n    plt.title('Loss for %s' % label)\n    plt.ylabel('Loss')\n    plt.xlabel('Epoch')\n    _= plt.legend(['Train','Validation'], loc='upper left')\n    plt.show()\n    \ndef change_dists_to_yukawa(df, features):\n    \n    df[features] = np.exp(df[features]) / df[features]\n    df.replace(np.inf, 0, inplace=True)\n\n    return df\n\ndef get_folds(train, k_folds = 5, val_data_ratio_if_no_kfolds = 0.2, verbose = True):\n        \n    #get all molecules names\n    molecules_names = train['molecule_name'].unique()\n    random.shuffle(molecules_names)\n    n_molecules = len(molecules_names)\n\n    molecules_folds = []\n    index_folds = []\n    \n    for k in range(1, k_folds+1):\n        if k_folds > 1:\n            ratio = 1/k_folds\n        elif k_folds == 1:\n            ratio = val_data_ratio_if_no_kfolds\n        start_index = int(np.round(n_molecules*ratio*(k-1)))\n        end_index = int(np.round(n_molecules*ratio*(k)))\n        \n        train_molecules = list(molecules_names[:start_index]) + list(molecules_names[end_index:])\n        val_molecules = list(molecules_names[start_index:end_index])\n        molecules_folds.append([train_molecules, val_molecules])\n        \n        index_folds.append([train[train['molecule_name'].isin(train_molecules)].index, train[train['molecule_name'].isin(val_molecules)].index])\n       \n        if verbose:\n            print('-------------')\n            print(f'fold {k}')\n            print('validation molecules indices go from', start_index, 'to', end_index)\n            print(f'{len(train_molecules)} train molecules and {len(val_molecules)} validation molecules')\n            print(f'{len(index_folds[-1][0])} train samples and {len(index_folds[-1][1])} validation samples')\n\n    return index_folds\n    \ndef preprocess_train_data(mol_type, train, scalar_coupling_contributions, k_folds=5, val_data_ratio_if_no_kfolds = 0.2, verbose = True):\n            \n    seed = 2319\n    random.seed(seed)  \n    \n    #randomize data\n    train = train.sample(frac=1, random_state = seed).reset_index(drop=True)    \n    \n    #create folds based on molecules\n    index_folds = get_folds(train, k_folds, val_data_ratio_if_no_kfolds, verbose)\n    \n        \n    return train, index_folds\n\ndef split_train_and_val_data(train, trn_idx, val_idx, mol_features):\n            \n    X_train = train.loc[trn_idx, mol_features]\n    X_val = train.loc[val_idx, mol_features]\n    y_train = train.loc[trn_idx, ['scalar_coupling_constant', 'fc', 'sd','pso', 'dso']]\n    y_val = train.loc[val_idx, ['scalar_coupling_constant', 'fc', 'sd','pso', 'dso']]\n    \n    std_scaler = StandardScaler().fit(train[mol_features])\n    \n    X_t = std_scaler.transform(X_train.values)\n    X_v = std_scaler.transform(X_val.values)\n    \n    return X_t, X_v, y_train, y_val\n     \ndef load_or_create_nn_model(X_t, k_fold, k_folds, file_folder, load_existing_model = True):\n            \n    if load_existing_model:\n        try:\n            nn_model = load_model(f'nn_model_{mol_type}_{str(k_fold)}')\n        except:\n            nn_model = create_nn_model(X_t.shape[1])\n    else:\n        nn_model = create_nn_model(X_t.shape[1])\n\n    return nn_model\n\ndef select_best_features(file_folder, train, scalar_coupling_contributions):\n        \n    train, index_folds = preprocess_train_data(mol_type, train, scalar_coupling_contributions, \n                                  k_folds=1, val_data_ratio_if_no_kfolds = 0.05, verbose = True)\n    \n    trn_idx, val_idx = index_folds[0]\n    \n    mol_features = [c for c in train.columns if (c not in ['scalar_coupling_constant', 'id', 'molecule_name', 'type', 'aromaticity_vec_1', 'aromaticity_vec_0', 'fc', 'sd','pso', 'dso'])]\n    \n    X_t, X_v, y_t, y_v = split_train_and_val_data(train, trn_idx, val_idx, mol_features)\n    \n    lgb_params = load_lgb_params(mol_type, n_estimators = 1000)\n    \n    lgb_model = lgb.LGBMRegressor(**lgb_params)\n    lgb_model.fit(X_t, y_t['scalar_coupling_constant'],\n            eval_set=[(X_t, y_t['scalar_coupling_constant']),(X_v, y_v['scalar_coupling_constant'])],\n              eval_metric='mae', verbose=100, early_stopping_rounds= 250)\n    \n    lgb_pred = lgb_model.predict(X_v, num_iteration=lgb_model.best_iteration_)\n    lgb_score = calc_logmae(y_v['scalar_coupling_constant'], lgb_pred)\n    \n    print(f'Inital LGB_score --> logmae for {mol_type} is {lgb_score}')\n    \n    results, bad_features = permutation_importance(model = lgb_model, X_val = pd.DataFrame(X_v, columns = mol_features), y_val = y_v['scalar_coupling_constant'],\n                                                   calc_logmae = calc_logmae, threshold=0.01, minimize=True, verbose=True)\n\n    print(f'{len(bad_features)} were removed from {len(mol_features)} initial features')\n    \n    mol_features = [feat for feat in mol_features if (feat not in bad_features)]\n    \n    return mol_features\n         \ndef get_patience_dict(max_number_epochs = 30, min_number_epochs = 3):\n    \n    \"\"\"\n    Patience is used to determine how many epochs can an NN run on PLateau.\n    1JHN is the smallest dataset with 43363 datasamples.\n    The number os epochs is defined for 1JHN, and every other patience is proportional to that\n    \"\"\"\n    \n    patience_dict = {'1JHN':max(min_number_epochs, int(max_number_epochs*43363/43363)), '1JHC':max(min_number_epochs, int(max_number_epochs*43363/709416)),\n                     '2JHN':max(min_number_epochs, int(max_number_epochs*43363/119253)), '2JHC':max(min_number_epochs, int(max_number_epochs*43363/1140674)),\n                     '2JHH':max(min_number_epochs, int(max_number_epochs*43363/378036)), '3JHN':max(min_number_epochs, int(max_number_epochs*43363/166415)),\n                     '3JHC':max(min_number_epochs, int(max_number_epochs*43363/1510379)), '3JHH':max(min_number_epochs, int(max_number_epochs*43363/590611))}\n    \n    return patience_dict           \n\ndef run_nn(load_existing_model, k_fold, trn_idx, val_idx, train, nn_mol_features, k_folds, file_folder, nn_features_for_lgb_test, nn_features_for_lgb_train,\n               scores_nn, test_selected, mol_type, oof_nn, pred_nn, run_number):\n    \n    print(f'fold {k_fold} of the {mol_type}')\n    \n    #model_name_wrt = ('nn_model_{mol_type}_{str(k_fold)}')\n\n    X_t, X_v, y_t, y_v = split_train_and_val_data(train, trn_idx, val_idx, nn_mol_features)\n        \n    nn_model = load_or_create_nn_model(X_t, k_fold, k_folds, file_folder, load_existing_model = load_existing_model)\n    nn_model.compile(loss='mse', optimizer=Adam())\n    \n    #-----CALL BACKS-----\n    # Callback for Early Stopping... May want to raise the min_delta for small numbers of epochs\n    # Callback for Reducing the Learning Rate... when the monitor levels out for 'patience' epochs, then the LR is reduced\n    # Save the best value of the model for future use\n    patience_dict = get_patience_dict(max_number_epochs = 10, min_number_epochs = 3) #change to a higher number to get a better result\n    \n    patience = patience_dict[mol_type]\n    \n    es = callbacks.EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=int(patience*1.7),verbose=1, mode='auto', restore_best_weights=True)\n    rlr = callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1,patience=patience, min_lr=1.001e-6, mode='auto', verbose=1)\n    #sv_mod = callbacks.ModelCheckpoint(model_name_wrt, monitor='val_loss', save_best_only=True, period=1)\n\n    try:    \n        history = nn_model.fit(X_t,y_t,\n                validation_data=(X_v, y_v), \n                callbacks=[es, rlr], epochs=epoch_n, batch_size=batch_size, verbose=verbose)\n    except:\n        nn_model = create_nn_model(X_t.shape[1])\n        nn_model.compile(loss='mse', optimizer=Adam())\n\n        history = nn_model.fit(X_t,y_t,\n                validation_data=(X_v, y_v), \n                callbacks=[es, rlr], epochs=epoch_n, batch_size=batch_size, verbose=verbose)\n     \n    #oof pred\n    nn_oof_pred = nn_model.predict(X_v)\n    \n    nn_features_for_lgb_train.loc[val_idx, ['sc', 'fc', 'sd','pso', 'dso']] = nn_oof_pred\n    \n    nn_oof_pred = 0.5*nn_oof_pred[:,0] + 0.5*nn_oof_pred[:,1:].sum(axis=-1) #blend between 'sc' and ['fc', 'sd','pso', 'dso']\n    oof_nn.iloc[val_idx, 0] = nn_oof_pred\n    \n    #test pred\n    nn_test_pred = nn_model.predict(test_selected)\n    \n    nn_features_for_lgb_test.loc[:, ['sc', 'fc', 'sd','pso', 'dso']] += nn_test_pred\n    \n    nn_test_pred = 0.5*nn_test_pred[:,0] + 0.5*nn_test_pred[:,1:].sum(axis=-1) #blend between 'sc' and ['fc', 'sd','pso', 'dso']\n    pred_nn.loc[:, k_fold] = nn_test_pred\n    \n    #save the generated NN features for the LGB model. They are the 5 predictions + a 32 neurons layer\n    intermediate_layer_model = Model(inputs=nn_model.input, outputs=nn_model.layers[7].output)\n    nn_features_for_lgb_train.loc[val_idx, cols_nn_features_for_lgb[5:(n_features + 5)]] = intermediate_layer_model.predict(X_v)\n    nn_features_for_lgb_test.loc[:, cols_nn_features_for_lgb[5 + n_features*k_fold: 5 + n_features*(k_fold+1)]] = intermediate_layer_model.predict(test_selected)\n    \n    nn_score = calc_logmae(y_v['scalar_coupling_constant'].values, nn_oof_pred)\n    scores_nn[mol_type][k_fold] = nn_score\n    \n    print(f'NN_score --> logmae for {mol_type} fold {k_fold} is {nn_score}')\n\n    return nn_features_for_lgb_test, nn_features_for_lgb_train, pred_nn, oof_nn, scores_nn\n\ndef get_oofs_and_preds(df_type, mol_type):\n    \n    #this function is used for stacking models\n    if df_type == 'oof':\n        df = pd.read_csv(f'{original_data_folder}/train.csv', usecols = ['id', 'type'])\n        file_names = ['OOF_FELIPE_LGB_1944.csv',\n                      'OOF_FELIPE_NN_1917.csv',\n                         'harsh_oof_1.688.csv',\n                         'oof_lolstart_lgb_1720.csv',\n                         'oof_yassine_lgb_5_folds_-1.295.csv',\n                         'harsh_10fold_oof_1.670.csv']\n    \n    if df_type == 'pred':\n        df = pd.read_csv(f'{original_data_folder}/test.csv', usecols = ['id', 'type'])\n        file_names = ['PRED_FELIPE_LGB_1944.csv',\n                      'PRED_FELIPE_NN_1917.csv',\n                         'harsh_pred_1.688.csv',\n                         'pred_lolstart_lgb_1720.csv',\n                         'pred_yassine_lgb_5_folds-1.295.csv',\n                         'harsh_10fold_pred_1.670.csv']\n    \n    sc_columns = []\n    for i, file_name in enumerate(file_names):\n        data = pd.read_csv(f'{preds_and_oofs_folder}/{file_name}')\n        \n        try:\n            data.rename(columns={'oof':'scalar_coupling_constant'}, inplace=True)\n        except:\n            pass\n        \n        try:\n            data.rename(columns={'pred':'scalar_coupling_constant'}, inplace=True)\n        except:\n            pass\n        \n        try:\n            data.rename(columns={'prediction':'scalar_coupling_constant'}, inplace=True)\n        except:\n            pass\n        \n        try:\n            data.rename(columns={'ind':'id'}, inplace=True)\n        except:\n            pass\n        \n        data.sort_values('id', inplace=True)\n        df[f'sc_{i}'] = data['scalar_coupling_constant'].values\n        sc_columns.append(f'sc_{i}')\n    \n    df = df[df['type'] == mol_type]\n    del df['type']\n    \n    df.loc[:, sc_columns] -= df.loc[:, sc_columns].mean().mean()\n    df.loc[:, sc_columns] /= df.loc[:, sc_columns].stack().std() \n              \n    return df, sc_columns\n              \ndef create_or_load_selected_features(preds_and_oofs_folder, train, scalar_coupling_contributions):\n    \n    try:\n        selected_features = np.load(preds_and_oofs_folder + f'/nn_mol_features_{mol_type}.npy', allow_pickle=True)\n        print(f'-------There are {len(selected_features)} features in your model-------')\n    except:\n        selected_features = select_best_features(preds_and_oofs_folder, train, scalar_coupling_contributions)\n        np.save(preds_and_oofs_folder + f'/nn_mol_features_{mol_type}', selected_features)\n    \n    return list(selected_features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **3. Define settings**"},{"metadata":{"trusted":true},"cell_type":"code","source":"original_data_folder = '../input/champs-scalar-coupling'\npreds_and_oofs_folder = '../input/preds-on-oof-and-test'\ntrain_and_test_with_feats_folder = '../input/features-for-top-5-lb-with-nn-or-lgb'\nscalar_coupling_contributions = pd.read_csv(original_data_folder + '/scalar_coupling_contributions.csv')\n\nconfig = tf.ConfigProto( device_count = {'GPU': 1 , 'CPU': 4} ) \nconfig.gpu_options.allow_growth = True\nconfig.gpu_options.per_process_gpu_memory_fraction = 1\nsess = tf.Session(config=config) \nK.set_session(sess)\n\nepoch_n = 500\nverbose = 1\nbatch_size = 2048\n\n#mol_types = ['1JHN', '1JHC', '2JHN', '2JHC', '2JHH', '3JHN', '3JHC', '3JHH']*20\nmol_types = ['1JHN']\n\nrun_number = 0\n\nk_folds = 1\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **4. Run the model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"scores_nn = dict()\n\nfor mol_type_index, mol_type in enumerate(mol_types):\n    \n    print(mol_type, f'- run number {run_number}')\n    \n    ###############\n    #LOAD DATA\n    ###############\n    \n    #scores_nn is a dict used to save the score reached on each mol_type\n    try:\n        scores_nn = np.load(f'run_{run_number}_scores_nn.npy', allow_pickle=True).item()\n    except:\n        scores_nn = dict()\n    \n    train = pd.read_csv(train_and_test_with_feats_folder + '/train_' + mol_type + '.csv').fillna(0)\n    test_full = pd.read_csv(train_and_test_with_feats_folder + '/test_' + mol_type + '.csv').fillna(0)\n    \n    print(f' Working with data of type {mol_type} and shape {train.shape}')\n    \n    #predictions from other models used for stacking\n    df_oofs, sc_columns = get_oofs_and_preds('oof', mol_type)\n    df_pred, sc_columns = get_oofs_and_preds('pred', mol_type)\n\n    #Add scalar contributions to our train df\n    #This will be used by the NN as target values\n    molecules_names = train['molecule_name'].unique()\n    type_scalar_contributions = scalar_coupling_contributions[scalar_coupling_contributions['molecule_name'].isin(molecules_names)]\n    type_scalar_contributions = type_scalar_contributions[type_scalar_contributions['type'] == mol_type]\n    \n    for col in ['fc', 'sd','pso', 'dso']:\n        train[col] = type_scalar_contributions[col].values        \n    \n    del type_scalar_contributions, molecules_names\n    \n    ###############\n    #SELECT BEST FEATURES\n    #Use an LGBM model to selected the best features for this coupling type\n    #The selection method is define as: If by randomizing the values of a feature, the LGB score doesnt change\n    #by a considerable delta, then this feature is not relevant\n    ###############\n\n    nn_mol_features = create_or_load_selected_features(preds_and_oofs_folder, train, scalar_coupling_contributions)\n        \n    for col in sc_columns:\n        nn_mol_features.append(col)\n        \n\n    ###############\n    #SPLIT THE DATA EM PREPROCESS IT\n    #Process the data, randomizing it and selecting a train and validations datasets.\n    #The split is my molecule, so in the train_df we wont see any molecules that are in the val_df\n    ###############\n\n    train, index_folds = preprocess_train_data(mol_type, train, scalar_coupling_contributions, \n                                  k_folds=k_folds, val_data_ratio_if_no_kfolds = 0.2, verbose = True)\n    \n    #add target and oof predictions to our features\n    train = pd.merge(train, df_oofs, on=['id'])\n    test_full = pd.merge(test_full, df_pred, on=['id'])\n    \n    del df_oofs, df_pred    \n    \n    #scale test dataset\n    std_scaler = StandardScaler().fit(train[nn_mol_features])\n    test_selected = std_scaler.transform(test_full[nn_mol_features].values)\n    del test_full\n    \n    ###############\n    #CREATE THE ARRAYS THAT WE WILL FILL WITH OUR RESULTS\n    ###############\n    \n    #create empty arrays for our predictions\n    oof_nn = pd.DataFrame(np.zeros((len(train), 1)))\n    pred_nn = pd.DataFrame(np.zeros((len(test_selected), k_folds)), columns = list(range(k_folds)))\n    \n    n_features = 32\n    cols_nn_features_for_lgb = np.array(['sc', 'fc', 'sd','pso', 'dso'] + [f'nn_feat_{k_fold}_{i}'  for k_fold in range(k_folds) for i in range(1, n_features+1)]).flatten()\n    \n    nn_features_for_lgb_train = pd.DataFrame(np.zeros((train.shape[0], n_features + 5)), columns = cols_nn_features_for_lgb[:n_features + 5])\n    nn_features_for_lgb_test= pd.DataFrame(np.zeros((test_selected.shape[0], n_features*k_folds + 5)), columns = cols_nn_features_for_lgb)\n    \n    #create the dict to save our scores\n    if mol_type not in scores_nn:\n        scores_nn[mol_type] = dict()\n    \n    gc.collect()\n    \n    ###############\n    #START THE TRAINING\n    ###############\n    for k_fold, (trn_idx, val_idx) in enumerate(index_folds):\n        \n        load_existing_model = True\n        nn_features_for_lgb_test, nn_features_for_lgb_train, pred_nn, oof_nn, scores_nn = run_nn(load_existing_model, k_fold, trn_idx, val_idx, train, nn_mol_features, k_folds,\n                                                                                                 preds_and_oofs_folder, nn_features_for_lgb_test, nn_features_for_lgb_train,\n                                                                                                 scores_nn, test_selected, mol_type, oof_nn, pred_nn, run_number)\n        gc.collect()\n        \n    nn_features_for_lgb_test.loc[:, ['sc', 'fc', 'sd','pso', 'dso']] /= k_folds\n    \n    ###############\n    #SAVE OUR RESULTS FOR THIS COUPLING TYPE\n    ###############\n    \n    np.save(f'run_{run_number}_scores_nn', scores_nn)\n    np.save(f'run_{run_number}_{mol_type}_oof_nn', oof_nn)\n    pred_nn.to_pickle(f'run_{run_number}_{mol_type}_pred_nn')\n    nn_features_for_lgb_train.to_pickle(f'run_{run_number}_{mol_type}_nn_features_for_lgb_train')\n    nn_features_for_lgb_test.to_pickle(f'run_{run_number}_{mol_type}_nn_features_for_lgb_test')\n    \n    #del nn_features_for_lgb_test, nn_features_for_lgb_train, pred_nn, oof_nn, train, test_selected\n    \n    gc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **5. Checking features created by the NN**\n\nLet's take a look at the 32 features that we generated with the NN network. They will improve LB score, but for them to be useful, you will have to start each fold of your NN with initial weights  equal to the NN of the last fold."},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(32):\n    nn_feat = nn_features_for_lgb_train.iloc[val_idx, i+5]\n    target = train.loc[val_idx, 'scalar_coupling_constant']\n    plt.scatter(nn_feat, target, s = 0.2)\n    plt.xlabel('scalar coupling constant')\n    plt.ylabel(f'nn_feat_{i}')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Really interesting patterns!**\n"},{"metadata":{"trusted":false,"_kg_hide-input":false},"cell_type":"markdown","source":"### **6. Submiting the model that you can get from this approach**\n\nBelow I load the final model we achieved by blending the NN and LGB generated by this kernel with a -1.7LB schnet and -1.66 gnn. The blending with schnet and gnn improved the score from -2 to -2.1.\n\nTo use this kernel with an LGB, a few modifications have to be done on the function \"run_nn\", which is basically changing the model from an NN to an LGB."},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(f'{preds_and_oofs_folder}/final_model_submission.csv')\nsub.to_csv('final_sub.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Please, if you find the content here interesting, consider upvoting the kernel to reward my time editing and sharing it. Thank you very much :)**"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}