{"cells":[{"metadata":{},"cell_type":"markdown","source":"**TLDR:**\n\n+ train['dist'] = 1/(train['dist']**2)\n+ test['dist'] = 1/(test['dist']**2)\n"},{"metadata":{},"cell_type":"markdown","source":"<img src=\"https://raw.githubusercontent.com/EdsonAvelar/auc/master/molecular_banner.png\" width=1900px height=400px />"},{"metadata":{},"cell_type":"markdown","source":"# Predicting Molecular Properties\n\n<h3 style=\"color:red\">If this Kernel Helps You! Please UP VOTE! 😁</h3>\n\n<h3> Can you measure the magnetic interactions between a pair of atoms? </h3>\n\nThis kernel is a combination of multiple kernels. The goal is to organize and explain the code to beginner competitors like me.<br>\nThis Kernels creates lots of new features and uses lightgbm as model"},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents:\n\n**1. [Problem Definition](#id1)** <br>\n**2. [Get the Data (Collect / Obtain)](#id2)** <br>\n**3. [Load the Dataset](#id3)** <br>\n**4. [Data Pre-processing](#id4)** <br>\n**5. [Model](#id5)** <br>\n**6. [Visualization and Analysis of Results](#id6)** <br>\n**7. [Submittion](#id7)** <br>\n**8. [References](#ref)** <br>"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id1\"></a> <br> \n# **1. Problem Definition:** \n\nThis challenge aims to predict interactions between atoms. The main task is develop an algorithm that can predict the magnetic interaction between two atoms in a molecule (i.e., the scalar coupling constant)<br>\n\nIn this competition, you will be predicting the scalar_coupling_constant between atom pairs in molecules, given the two atom types (e.g., C and H), the coupling type (e.g., 2JHC), and any features you are able to create from the molecule structure (xyz) files.\n\n**Data**\n* **train.csv** - the training set, where the first column (molecule_name) is the name of the molecule where the coupling constant originates, the second (atom_index_0) and third column (atom_index_1) is the atom indices of the atom-pair creating the coupling and the fourth column (**scalar_coupling_constant**) is the scalar coupling constant that we want to be able to predict\n* **test.csv** - the test set; same info as train, without the target variable\n* **sample_submission.csv** - a sample submission file in the correct format\n* **structures.csv** - this file contains the same information as the individual xyz structure files, but in a single file\n\n**Additional Data**<br>\n*NOTE: additional data is provided for the molecules in Train only!*\n* **scalar_coupling_contributions.csv** - The scalar coupling constants in train.csv are a sum of four terms. The first column (**molecule_name**) are the name of the molecule, the second (**atom_index_0**) and third column (**atom_index_1**) are the atom indices of the atom-pair, the fourth column indicates the **type** of coupling, the fifth column (**fc**) is the Fermi Contact contribution, the sixth column (**sd**) is the Spin-dipolar contribution, the seventh column (**pso**) is the Paramagnetic spin-orbit contribution and the eighth column (**dso**) is the Diamagnetic spin-orbit contribution.\n\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id2\"></a> <br> \n# **2. Get the Data (Collect / Obtain):** "},{"metadata":{},"cell_type":"markdown","source":"## All imports used in this kernel"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom tqdm import tqdm_notebook\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.svm import NuSVR, SVR\nfrom sklearn.metrics import mean_absolute_error\npd.options.display.precision = 15\n\nimport lightgbm as lgb\nimport xgboost as xgb\nimport time\nimport datetime\nfrom catboost import CatBoostRegressor\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import StratifiedKFold, KFold, RepeatedKFold\nfrom sklearn import metrics\nfrom sklearn import linear_model\nimport gc\nimport seaborn as sns\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nfrom IPython.display import HTML\nimport json\nimport altair as alt\n\nimport networkx as nx\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport os\nimport time\nimport datetime\nimport json\nimport gc\nfrom numba import jit\n\nimport numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm import tqdm_notebook\n\nimport lightgbm as lgb\nimport xgboost as xgb\nfrom catboost import CatBoostRegressor, CatBoostClassifier\nfrom sklearn import metrics\n\nfrom itertools import product\n\n# import altair as alt\n# from altair.vega import v3\n# from IPython.display import HTML\n# alt.renderers.enable('notebook')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## All function used in this kernel"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# using ideas from this kernel: https://www.kaggle.com/notslush/altair-visualization-2018-stackoverflow-survey\ndef prepare_altair():\n    \"\"\"\n    Helper function to prepare altair for working.\n    \"\"\"\n    vega_url = 'https://cdn.jsdelivr.net/npm/vega@' + v3.SCHEMA_VERSION\n    vega_lib_url = 'https://cdn.jsdelivr.net/npm/vega-lib'\n    vega_lite_url = 'https://cdn.jsdelivr.net/npm/vega-lite@' + alt.SCHEMA_VERSION\n    vega_embed_url = 'https://cdn.jsdelivr.net/npm/vega-embed@3'\n    noext = \"?noext\"\n    \n    paths = {\n        'vega': vega_url + noext,\n        'vega-lib': vega_lib_url + noext,\n        'vega-lite': vega_lite_url + noext,\n        'vega-embed': vega_embed_url + noext\n    }\n    \n    workaround = f\"\"\"    requirejs.config({{\n        baseUrl: 'https://cdn.jsdelivr.net/npm/',\n        paths: {paths}\n    }});\n    \"\"\"\n    \n    return workaround\n    \n\ndef add_autoincrement(render_func):\n    # Keep track of unique <div/> IDs\n    cache = {}\n    def wrapped(chart, id=\"vega-chart\", autoincrement=True):\n        if autoincrement:\n            if id in cache:\n                counter = 1 + cache[id]\n                cache[id] = counter\n            else:\n                cache[id] = 0\n            actual_id = id if cache[id] == 0 else id + '-' + str(cache[id])\n        else:\n            if id not in cache:\n                cache[id] = 0\n            actual_id = id\n        return render_func(chart, id=actual_id)\n    # Cache will stay outside and \n    return wrapped\n           \n\n@add_autoincrement\ndef render(chart, id=\"vega-chart\"):\n    \"\"\"\n    Helper function to plot altair visualizations.\n    \"\"\"\n    chart_str = \"\"\"\n    <div id=\"{id}\"></div><script>\n    require([\"vega-embed\"], function(vg_embed) {{\n        const spec = {chart};     \n        vg_embed(\"#{id}\", spec, {{defaultStyle: true}}).catch(console.warn);\n        console.log(\"anything?\");\n    }});\n    console.log(\"really...anything?\");\n    </script>\n    \"\"\"\n    return HTML(\n        chart_str.format(\n            id=id,\n            chart=json.dumps(chart) if isinstance(chart, dict) else chart.to_json(indent=None)\n        )\n    )\n    \n\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n    \n\n@jit\ndef fast_auc(y_true, y_prob):\n    \"\"\"\n    fast roc_auc computation: https://www.kaggle.com/c/microsoft-malware-prediction/discussion/76013\n    \"\"\"\n    y_true = np.asarray(y_true)\n    y_true = y_true[np.argsort(y_prob)]\n    nfalse = 0\n    auc = 0\n    n = len(y_true)\n    for i in range(n):\n        y_i = y_true[i]\n        nfalse += (1 - y_i)\n        auc += y_i * nfalse\n    auc /= (nfalse * (n - nfalse))\n    return auc\n\n\ndef eval_auc(y_true, y_pred):\n    \"\"\"\n    Fast auc eval function for lgb.\n    \"\"\"\n    return 'auc', fast_auc(y_true, y_pred), True\n\n\ndef group_mean_log_mae(y_true, y_pred, types, floor=1e-9):\n    \"\"\"\n    Fast metric computation for this competition: https://www.kaggle.com/c/champs-scalar-coupling\n    Code is from this kernel: https://www.kaggle.com/uberkinder/efficient-metric\n    \"\"\"\n    maes = (y_true-y_pred).abs().groupby(types).mean()\n    return np.log(maes.map(lambda x: max(x, floor))).mean()\n    \n\ndef train_model_regression(X, X_test, y, params, folds, model_type='lgb', eval_metric='mae', columns=None, plot_feature_importance=False, model=None,\n                               verbose=10000, early_stopping_rounds=200, n_estimators=50000):\n    \"\"\"\n    A function to train a variety of regression models.\n    Returns dictionary with oof predictions, test predictions, scores and, if necessary, feature importances.\n    \n    :params: X - training data, can be pd.DataFrame or np.ndarray (after normalizing)\n    :params: X_test - test data, can be pd.DataFrame or np.ndarray (after normalizing)\n    :params: y - target\n    :params: folds - folds to split data\n    :params: model_type - type of model to use\n    :params: eval_metric - metric to use\n    :params: columns - columns to use. If None - use all columns\n    :params: plot_feature_importance - whether to plot feature importance of LGB\n    :params: model - sklearn model, works only for \"sklearn\" model type\n    \n    \"\"\"\n    columns = X.columns if columns is None else columns\n    X_test = X_test[columns]\n    \n    # to set up scoring parameters\n    metrics_dict = {'mae': {'lgb_metric_name': 'mae',\n                        'catboost_metric_name': 'MAE',\n                        'sklearn_scoring_function': metrics.mean_absolute_error},\n                    'group_mae': {'lgb_metric_name': 'mae',\n                        'catboost_metric_name': 'MAE',\n                        'scoring_function': group_mean_log_mae},\n                    'mse': {'lgb_metric_name': 'mse',\n                        'catboost_metric_name': 'MSE',\n                        'sklearn_scoring_function': metrics.mean_squared_error}\n                    }\n\n    \n    result_dict = {}\n    \n    # out-of-fold predictions on train data\n    oof = np.zeros(len(X))\n    \n    # averaged predictions on train data\n    prediction = np.zeros(len(X_test))\n    \n    # list of scores on folds\n    scores = []\n    feature_importance = pd.DataFrame()\n    \n    # split and train on folds\n    for fold_n, (train_index, valid_index) in enumerate(folds.split(X)):\n        print(f'Fold {fold_n + 1} started at {time.ctime()}')\n        if type(X) == np.ndarray:\n            X_train, X_valid = X[columns][train_index], X[columns][valid_index]\n            y_train, y_valid = y[train_index], y[valid_index]\n        else:\n            X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n            y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n            \n        if model_type == 'lgb':\n            model = lgb.LGBMRegressor(**params, n_estimators = n_estimators, n_jobs = -1)\n            model.fit(X_train, y_train, \n                    eval_set=[(X_train, y_train), (X_valid, y_valid)], eval_metric=metrics_dict[eval_metric]['lgb_metric_name'],\n                    verbose=verbose, early_stopping_rounds=early_stopping_rounds)\n            \n            y_pred_valid = model.predict(X_valid)\n            y_pred = model.predict(X_test, num_iteration=model.best_iteration_)\n            \n        if model_type == 'xgb':\n            train_data = xgb.DMatrix(data=X_train, label=y_train, feature_names=X.columns)\n            valid_data = xgb.DMatrix(data=X_valid, label=y_valid, feature_names=X.columns)\n\n            watchlist = [(train_data, 'train'), (valid_data, 'valid_data')]\n            model = xgb.train(dtrain=train_data, num_boost_round=20000, evals=watchlist, early_stopping_rounds=200, verbose_eval=verbose, params=params)\n            y_pred_valid = model.predict(xgb.DMatrix(X_valid, feature_names=X.columns), ntree_limit=model.best_ntree_limit)\n            y_pred = model.predict(xgb.DMatrix(X_test, feature_names=X.columns), ntree_limit=model.best_ntree_limit)\n        \n        if model_type == 'sklearn':\n            model = model\n            model.fit(X_train, y_train)\n            \n            y_pred_valid = model.predict(X_valid).reshape(-1,)\n            score = metrics_dict[eval_metric]['sklearn_scoring_function'](y_valid, y_pred_valid)\n            print(f'Fold {fold_n}. {eval_metric}: {score:.4f}.')\n            print('')\n            \n            y_pred = model.predict(X_test).reshape(-1,)\n        \n        if model_type == 'cat':\n            model = CatBoostRegressor(iterations=20000,  eval_metric=metrics_dict[eval_metric]['catboost_metric_name'], **params,\n                                      loss_function=metrics_dict[eval_metric]['catboost_metric_name'])\n            model.fit(X_train, y_train, eval_set=(X_valid, y_valid), cat_features=[], use_best_model=True, verbose=False)\n\n            y_pred_valid = model.predict(X_valid)\n            y_pred = model.predict(X_test)\n        \n        oof[valid_index] = y_pred_valid.reshape(-1,)\n        if eval_metric != 'group_mae':\n            scores.append(metrics_dict[eval_metric]['sklearn_scoring_function'](y_valid, y_pred_valid))\n        else:\n            scores.append(metrics_dict[eval_metric]['scoring_function'](y_valid, y_pred_valid, X_valid['type']))\n\n        prediction += y_pred    \n        \n        if model_type == 'lgb' and plot_feature_importance:\n            # feature importance\n            fold_importance = pd.DataFrame()\n            fold_importance[\"feature\"] = columns\n            fold_importance[\"importance\"] = model.feature_importances_\n            fold_importance[\"fold\"] = fold_n + 1\n            feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n\n    prediction /= folds.n_splits\n    \n    print('CV mean score: {0:.4f}, std: {1:.4f}.'.format(np.mean(scores), np.std(scores)))\n    \n    result_dict['oof'] = oof\n    result_dict['prediction'] = prediction\n    result_dict['scores'] = scores\n    \n    if model_type == 'lgb':\n        if plot_feature_importance:\n            feature_importance[\"importance\"] /= folds.n_splits\n            cols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n                by=\"importance\", ascending=False)[:50].index\n\n            best_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n\n            plt.figure(figsize=(16, 12));\n            sns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\n            plt.title('LGB Features (avg over folds)');\n            \n            result_dict['feature_importance'] = feature_importance\n        \n    return result_dict\n    \n\n\ndef train_model_classification(X, X_test, y, params, folds, model_type='lgb', eval_metric='auc', columns=None, plot_feature_importance=False, model=None,\n                               verbose=10000, early_stopping_rounds=200, n_estimators=50000):\n    \"\"\"\n    A function to train a variety of regression models.\n    Returns dictionary with oof predictions, test predictions, scores and, if necessary, feature importances.\n    \n    :params: X - training data, can be pd.DataFrame or np.ndarray (after normalizing)\n    :params: X_test - test data, can be pd.DataFrame or np.ndarray (after normalizing)\n    :params: y - target\n    :params: folds - folds to split data\n    :params: model_type - type of model to use\n    :params: eval_metric - metric to use\n    :params: columns - columns to use. If None - use all columns\n    :params: plot_feature_importance - whether to plot feature importance of LGB\n    :params: model - sklearn model, works only for \"sklearn\" model type\n    \n    \"\"\"\n    columns = X.columns if columns == None else columns\n    X_test = X_test[columns]\n    \n    # to set up scoring parameters\n    metrics_dict = {'auc': {'lgb_metric_name': eval_auc,\n                        'catboost_metric_name': 'AUC',\n                        'sklearn_scoring_function': metrics.roc_auc_score},\n                    }\n    \n    result_dict = {}\n    \n    # out-of-fold predictions on train data\n    oof = np.zeros((len(X), len(set(y.values))))\n    \n    # averaged predictions on train data\n    prediction = np.zeros((len(X_test), oof.shape[1]))\n    \n    # list of scores on folds\n    scores = []\n    feature_importance = pd.DataFrame()\n    \n    # split and train on folds\n    for fold_n, (train_index, valid_index) in enumerate(folds.split(X)):\n        print(f'Fold {fold_n + 1} started at {time.ctime()}')\n        if type(X) == np.ndarray:\n            X_train, X_valid = X[columns][train_index], X[columns][valid_index]\n            y_train, y_valid = y[train_index], y[valid_index]\n        else:\n            X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n            y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n            \n        if model_type == 'lgb':\n            model = lgb.LGBMClassifier(**params, n_estimators=n_estimators, n_jobs = -1)\n            model.fit(X_train, y_train, \n                    eval_set=[(X_train, y_train), (X_valid, y_valid)], eval_metric=metrics_dict[eval_metric]['lgb_metric_name'],\n                    verbose=verbose, early_stopping_rounds=early_stopping_rounds)\n            \n            y_pred_valid = model.predict_proba(X_valid)\n            y_pred = model.predict_proba(X_test, num_iteration=model.best_iteration_)\n            \n        if model_type == 'xgb':\n            train_data = xgb.DMatrix(data=X_train, label=y_train, feature_names=X.columns)\n            valid_data = xgb.DMatrix(data=X_valid, label=y_valid, feature_names=X.columns)\n\n            watchlist = [(train_data, 'train'), (valid_data, 'valid_data')]\n            model = xgb.train(dtrain=train_data, num_boost_round=n_estimators, evals=watchlist, early_stopping_rounds=early_stopping_rounds, verbose_eval=verbose, params=params)\n            y_pred_valid = model.predict(xgb.DMatrix(X_valid, feature_names=X.columns), ntree_limit=model.best_ntree_limit)\n            y_pred = model.predict(xgb.DMatrix(X_test, feature_names=X.columns), ntree_limit=model.best_ntree_limit)\n        \n        if model_type == 'sklearn':\n            model = model\n            model.fit(X_train, y_train)\n            \n            y_pred_valid = model.predict(X_valid).reshape(-1,)\n            score = metrics_dict[eval_metric]['sklearn_scoring_function'](y_valid, y_pred_valid)\n            print(f'Fold {fold_n}. {eval_metric}: {score:.4f}.')\n            print('')\n            \n            y_pred = model.predict_proba(X_test)\n        \n        if model_type == 'cat':\n            model = CatBoostClassifier(iterations=n_estimators, eval_metric=metrics_dict[eval_metric]['catboost_metric_name'], **params,\n                                      loss_function=metrics_dict[eval_metric]['catboost_metric_name'])\n            model.fit(X_train, y_train, eval_set=(X_valid, y_valid), cat_features=[], use_best_model=True, verbose=False)\n\n            y_pred_valid = model.predict(X_valid)\n            y_pred = model.predict(X_test)\n        \n        oof[valid_index] = y_pred_valid\n        scores.append(metrics_dict[eval_metric]['sklearn_scoring_function'](y_valid, y_pred_valid[:, 1]))\n\n        prediction += y_pred    \n        \n        if model_type == 'lgb' and plot_feature_importance:\n            # feature importance\n            fold_importance = pd.DataFrame()\n            fold_importance[\"feature\"] = columns\n            fold_importance[\"importance\"] = model.feature_importances_\n            fold_importance[\"fold\"] = fold_n + 1\n            feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n\n    prediction /= folds.n_splits\n    \n    print('CV mean score: {0:.4f}, std: {1:.4f}.'.format(np.mean(scores), np.std(scores)))\n    \n    result_dict['oof'] = oof\n    result_dict['prediction'] = prediction\n    result_dict['scores'] = scores\n    \n    if model_type == 'lgb':\n        if plot_feature_importance:\n            feature_importance[\"importance\"] /= folds.n_splits\n            cols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n                by=\"importance\", ascending=False)[:50].index\n\n            best_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n\n            plt.figure(figsize=(16, 12));\n            sns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\n            plt.title('LGB Features (avg over folds)');\n            \n            result_dict['feature_importance'] = feature_importance\n        \n    return result_dict\n\n# setting up altair\n# workaround = prepare_altair()\n# HTML(\"\".join((\n#     \"<script>\",\n#     workaround,\n#     \"</script>\",\n# )))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id3\"></a> <br> \n# **3. Load the Dataset** \n\nLet's load all necessary datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\nsub = pd.read_csv('../input/sample_submission.csv')\nstructures = pd.read_csv('../input/structures.csv')\nscalar_coupling_contributions = pd.read_csv('../input/scalar_coupling_contributions.csv')\n\nprint('Train dataset shape is -> rows: {} cols:{}'.format(train.shape[0],train.shape[1]))\nprint('Test dataset shape is  -> rows: {} cols:{}'.format(test.shape[0],test.shape[1]))\nprint('Sub dataset shape is  -> rows: {} cols:{}'.format(sub.shape[0],sub.shape[1]))\nprint('Structures dataset shape is  -> rows: {} cols:{}'.format(structures.shape[0],structures.shape[1]))\nprint('Scalar_coupling_contributions dataset shape is  -> rows: {} cols:{}'.format(scalar_coupling_contributions.shape[0],\n                                                                                   scalar_coupling_contributions.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For an fast model/feature evaluation, get only 10% of dataset. Final submission must remove/coments this code"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_estimators_default = 4000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nsize = round(0.10*train.shape[0])\ntrain = train[:size]\ntest = test[:size]\nsub = sub[:size]\nstructures = structures[:size]\nscalar_coupling_contributions = scalar_coupling_contributions[:size]\n\nprint('Train dataset shape is now rows: {} cols:{}'.format(train.shape[0],train.shape[1]))\nprint('Test dataset shape is now rows: {} cols:{}'.format(test.shape[0],test.shape[1]))\nprint('Sub dataset shape is now rows: {} cols:{}'.format(sub.shape[0],sub.shape[1]))\nprint('Structures dataset shape is now rows: {} cols:{}'.format(structures.shape[0],structures.shape[1]))\nprint('Scalar_coupling_contributions dataset shape is now rows: {} cols:{}'.format(scalar_coupling_contributions.shape[0],\n                                                                                   scalar_coupling_contributions.shape[1]))\n'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The importante things to know is that the scalar coupling constants in train.csv are a sum of four terms. \n```\n* fc is the Fermi Contact contribution\n* sd is the Spin-dipolar contribution\n* pso is the Paramagnetic spin-orbit contribution\n* dso is the Diamagnetic spin-orbit contribution. \n```\nLet's merge this into train"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.merge(train, scalar_coupling_contributions, how = 'left',\n                  left_on  = ['molecule_name', 'atom_index_0', 'atom_index_1', 'type'],\n                  right_on = ['molecule_name', 'atom_index_0', 'atom_index_1', 'type'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scalar_coupling_contributions.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`train['scalar_coupling_constant'] and scalar_coupling_contributions['fc']` quite similar"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.concat(objs=[train['scalar_coupling_constant'],scalar_coupling_contributions['fc'] ],axis=1)[:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based in others ideais we can:<br>\n\n- train a model to predict `fc` feature;\n- add this feature to train and test and train the same model to compare performance;\n- train a better model;"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id4\"></a> <br> \n# **4. Data Pre-processing** "},{"metadata":{},"cell_type":"markdown","source":"## Feature generation"},{"metadata":{},"cell_type":"markdown","source":"I use this great kernel to get x,y,z position. https://www.kaggle.com/seriousran/just-speed-up-calculate-distance-from-benchmark"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm_notebook as tqdm\natomic_radius = {'H':0.38, 'C':0.77, 'N':0.75, 'O':0.73, 'F':0.71} # Without fudge factor\n\nfudge_factor = 0.05\natomic_radius = {k:v + fudge_factor for k,v in atomic_radius.items()}\nprint(atomic_radius)\n\nelectronegativity = {'H':2.2, 'C':2.55, 'N':3.04, 'O':3.44, 'F':3.98}\n\n#structures = pd.read_csv(structures, dtype={'atom_index':np.int8})\n\natoms = structures['atom'].values\natoms_en = [electronegativity[x] for x in tqdm(atoms)]\natoms_rad = [atomic_radius[x] for x in tqdm(atoms)]\n\nstructures['EN'] = atoms_en\nstructures['rad'] = atoms_rad\n\ndisplay(structures.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Chemical Bond Calculation"},{"metadata":{"trusted":true},"cell_type":"code","source":"i_atom = structures['atom_index'].values\np = structures[['x', 'y', 'z']].values\np_compare = p\nm = structures['molecule_name'].values\nm_compare = m\nr = structures['rad'].values\nr_compare = r\n\nsource_row = np.arange(len(structures))\nmax_atoms = 28\n\nbonds = np.zeros((len(structures)+1, max_atoms+1), dtype=np.int8)\nbond_dists = np.zeros((len(structures)+1, max_atoms+1), dtype=np.float32)\n\nprint('Calculating bonds')\n\nfor i in tqdm(range(max_atoms-1)):\n    p_compare = np.roll(p_compare, -1, axis=0)\n    m_compare = np.roll(m_compare, -1, axis=0)\n    r_compare = np.roll(r_compare, -1, axis=0)\n    \n    mask = np.where(m == m_compare, 1, 0) #Are we still comparing atoms in the same molecule?\n    dists = np.linalg.norm(p - p_compare, axis=1) * mask\n    r_bond = r + r_compare\n    \n    bond = np.where(np.logical_and(dists > 0.0001, dists < r_bond), 1, 0)\n    \n    source_row = source_row\n    target_row = source_row + i + 1 #Note: Will be out of bounds of bonds array for some values of i\n    target_row = np.where(np.logical_or(target_row > len(structures), mask==0), len(structures), target_row) #If invalid target, write to dummy row\n    \n    source_atom = i_atom\n    target_atom = i_atom + i + 1 #Note: Will be out of bounds of bonds array for some values of i\n    target_atom = np.where(np.logical_or(target_atom > max_atoms, mask==0), max_atoms, target_atom) #If invalid target, write to dummy col\n    \n    bonds[(source_row, target_atom)] = bond\n    bonds[(target_row, source_atom)] = bond\n    bond_dists[(source_row, target_atom)] = dists\n    bond_dists[(target_row, source_atom)] = dists\n\nbonds = np.delete(bonds, axis=0, obj=-1) #Delete dummy row\nbonds = np.delete(bonds, axis=1, obj=-1) #Delete dummy col\nbond_dists = np.delete(bond_dists, axis=0, obj=-1) #Delete dummy row\nbond_dists = np.delete(bond_dists, axis=1, obj=-1) #Delete dummy col\n\nprint('Counting and condensing bonds')\n\nbonds_numeric = [[i for i,x in enumerate(row) if x] for row in tqdm(bonds)]\nbond_lengths = [[dist for i,dist in enumerate(row) if i in bonds_numeric[j]] for j,row in enumerate(tqdm(bond_dists))]\nbond_lengths_mean = [ np.mean(x) for x in bond_lengths]\nbond_lengths_std = [ np.std(x) for x in bond_lengths]\nn_bonds = [len(x) for x in bonds_numeric]\n\n#bond_data = {'bond_' + str(i):col for i, col in enumerate(np.transpose(bonds))}\n#bond_data.update({'bonds_numeric':bonds_numeric, 'n_bonds':n_bonds})\n\nbond_data = {'n_bonds':n_bonds, 'bond_lengths_mean': bond_lengths_mean,'bond_lengths_std':bond_lengths_std }\nbond_df = pd.DataFrame(bond_data)\nstructures = structures.join(bond_df)\ndisplay(structures.head(20))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def map_atom_info(df, atom_idx):\n    df = pd.merge(df, structures, how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    \n    #df = df.drop('atom_index', axis=1)\n    df = df.rename(columns={'atom': f'atom_{atom_idx}',\n                            'x': f'x_{atom_idx}',\n                            'y': f'y_{atom_idx}',\n                            'z': f'z_{atom_idx}'})\n    return df\n\ntrain = map_atom_info(train, 0)\ntrain = map_atom_info(train, 1)\n\ntest = map_atom_info(test, 0)\ntest = map_atom_info(test, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's get the distance between atoms first."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_p_0 = train[['x_0', 'y_0', 'z_0']].values\ntrain_p_1 = train[['x_1', 'y_1', 'z_1']].values\ntest_p_0 = test[['x_0', 'y_0', 'z_0']].values\ntest_p_1 = test[['x_1', 'y_1', 'z_1']].values\n\ntrain['dist'] = np.linalg.norm(train_p_0 - train_p_1, axis=1)\ntest['dist'] = np.linalg.norm(test_p_0 - test_p_1, axis=1)\ntrain['dist'] = 1/(train['dist']**2)\ntest['dist'] = 1/(test['dist']**2)\n\ntrain['dist_x'] = (train['x_0'] - train['x_1']) ** 2\ntest['dist_x'] = (test['x_0'] - test['x_1']) ** 2\ntrain['dist_y'] = (train['y_0'] - train['y_1']) ** 2\ntest['dist_y'] = (test['y_0'] - test['y_1']) ** 2\ntrain['dist_z'] = (train['z_0'] - train['z_1']) ** 2\ntest['dist_z'] = (test['z_0'] - test['z_1']) ** 2\n\ntrain['type_0'] = train['type'].apply(lambda x: x[0])\ntest['type_0'] = test['type'].apply(lambda x: x[0])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_features(df):\n    df['molecule_couples'] = df.groupby('molecule_name')['id'].transform('count')\n    df['molecule_dist_mean'] = df.groupby('molecule_name')['dist'].transform('mean')\n    df['molecule_dist_min'] = df.groupby('molecule_name')['dist'].transform('min')\n    df['molecule_dist_max'] = df.groupby('molecule_name')['dist'].transform('max')\n    df['atom_0_couples_count'] = df.groupby(['molecule_name', 'atom_index_0'])['id'].transform('count')\n    df['atom_1_couples_count'] = df.groupby(['molecule_name', 'atom_index_1'])['id'].transform('count')\n    df[f'molecule_atom_index_0_x_1_std'] = df.groupby(['molecule_name', 'atom_index_0'])['x_1'].transform('std')\n    df[f'molecule_atom_index_0_y_1_mean'] = df.groupby(['molecule_name', 'atom_index_0'])['y_1'].transform('mean')\n    df[f'molecule_atom_index_0_y_1_mean_diff'] = df[f'molecule_atom_index_0_y_1_mean'] - df['y_1']\n    df[f'molecule_atom_index_0_y_1_mean_div'] = df[f'molecule_atom_index_0_y_1_mean'] / df['y_1']\n    df[f'molecule_atom_index_0_y_1_max'] = df.groupby(['molecule_name', 'atom_index_0'])['y_1'].transform('max')\n    df[f'molecule_atom_index_0_y_1_max_diff'] = df[f'molecule_atom_index_0_y_1_max'] - df['y_1']\n    df[f'molecule_atom_index_0_y_1_std'] = df.groupby(['molecule_name', 'atom_index_0'])['y_1'].transform('std')\n    df[f'molecule_atom_index_0_z_1_std'] = df.groupby(['molecule_name', 'atom_index_0'])['z_1'].transform('std')\n    df[f'molecule_atom_index_0_dist_mean'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('mean')\n    df[f'molecule_atom_index_0_dist_mean_diff'] = df[f'molecule_atom_index_0_dist_mean'] - df['dist']\n    df[f'molecule_atom_index_0_dist_mean_div'] = df[f'molecule_atom_index_0_dist_mean'] / df['dist']\n    df[f'molecule_atom_index_0_dist_max'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('max')\n    df[f'molecule_atom_index_0_dist_max_diff'] = df[f'molecule_atom_index_0_dist_max'] - df['dist']\n    df[f'molecule_atom_index_0_dist_max_div'] = df[f'molecule_atom_index_0_dist_max'] / df['dist']\n    df[f'molecule_atom_index_0_dist_min'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('min')\n    df[f'molecule_atom_index_0_dist_min_diff'] = df[f'molecule_atom_index_0_dist_min'] - df['dist']\n    df[f'molecule_atom_index_0_dist_min_div'] = df[f'molecule_atom_index_0_dist_min'] / df['dist']\n    df[f'molecule_atom_index_0_dist_std'] = df.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('std')\n    df[f'molecule_atom_index_0_dist_std_diff'] = df[f'molecule_atom_index_0_dist_std'] - df['dist']\n    df[f'molecule_atom_index_0_dist_std_div'] = df[f'molecule_atom_index_0_dist_std'] / df['dist']\n    df[f'molecule_atom_index_1_dist_mean'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('mean')\n    df[f'molecule_atom_index_1_dist_mean_diff'] = df[f'molecule_atom_index_1_dist_mean'] - df['dist']\n    df[f'molecule_atom_index_1_dist_mean_div'] = df[f'molecule_atom_index_1_dist_mean'] / df['dist']\n    df[f'molecule_atom_index_1_dist_max'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('max')\n    df[f'molecule_atom_index_1_dist_max_diff'] = df[f'molecule_atom_index_1_dist_max'] - df['dist']\n    df[f'molecule_atom_index_1_dist_max_div'] = df[f'molecule_atom_index_1_dist_max'] / df['dist']\n    df[f'molecule_atom_index_1_dist_min'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('min')\n    df[f'molecule_atom_index_1_dist_min_diff'] = df[f'molecule_atom_index_1_dist_min'] - df['dist']\n    df[f'molecule_atom_index_1_dist_min_div'] = df[f'molecule_atom_index_1_dist_min'] / df['dist']\n    df[f'molecule_atom_index_1_dist_std'] = df.groupby(['molecule_name', 'atom_index_1'])['dist'].transform('std')\n    df[f'molecule_atom_index_1_dist_std_diff'] = df[f'molecule_atom_index_1_dist_std'] - df['dist']\n    df[f'molecule_atom_index_1_dist_std_div'] = df[f'molecule_atom_index_1_dist_std'] / df['dist']\n    df[f'molecule_atom_1_dist_mean'] = df.groupby(['molecule_name', 'atom_1'])['dist'].transform('mean')\n    df[f'molecule_atom_1_dist_min'] = df.groupby(['molecule_name', 'atom_1'])['dist'].transform('min')\n    df[f'molecule_atom_1_dist_min_diff'] = df[f'molecule_atom_1_dist_min'] - df['dist']\n    df[f'molecule_atom_1_dist_min_div'] = df[f'molecule_atom_1_dist_min'] / df['dist']\n    df[f'molecule_atom_1_dist_std'] = df.groupby(['molecule_name', 'atom_1'])['dist'].transform('std')\n    df[f'molecule_atom_1_dist_std_diff'] = df[f'molecule_atom_1_dist_std'] - df['dist']\n    df[f'molecule_type_0_dist_std'] = df.groupby(['molecule_name', 'type_0'])['dist'].transform('std')\n    df[f'molecule_type_0_dist_std_diff'] = df[f'molecule_type_0_dist_std'] - df['dist']\n    df[f'molecule_type_dist_mean'] = df.groupby(['molecule_name', 'type'])['dist'].transform('mean')\n    df[f'molecule_type_dist_mean_diff'] = df[f'molecule_type_dist_mean'] - df['dist']\n    df[f'molecule_type_dist_mean_div'] = df[f'molecule_type_dist_mean'] / df['dist']\n    df[f'molecule_type_dist_max'] = df.groupby(['molecule_name', 'type'])['dist'].transform('max')\n    df[f'molecule_type_dist_min'] = df.groupby(['molecule_name', 'type'])['dist'].transform('min')\n    df[f'molecule_type_dist_std'] = df.groupby(['molecule_name', 'type'])['dist'].transform('std')\n    df[f'molecule_type_dist_std_diff'] = df[f'molecule_type_dist_std'] - df['dist']\n    df = reduce_mem_usage(df)\n    return df\n\ntrain = create_features(train)\ntest = create_features(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def map_atom_info(df_1,df_2, atom_idx):\n    df = pd.merge(df_1, df_2, how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    df = df.drop('atom_index', axis=1)\n\n    return df\n\ndef create_closest(df_train):\n    #I apologize for my poor coding skill. Please make the better one.\n    df_temp=df_train.loc[:,[\"molecule_name\",\"atom_index_0\",\"atom_index_1\",\"dist\",\"x_0\",\"y_0\",\"z_0\",\"x_1\",\"y_1\",\"z_1\"]].copy()\n    df_temp_=df_temp.copy()\n    df_temp_= df_temp_.rename(columns={'atom_index_0': 'atom_index_1',\n                                       'atom_index_1': 'atom_index_0',\n                                       'x_0': 'x_1',\n                                       'y_0': 'y_1',\n                                       'z_0': 'z_1',\n                                       'x_1': 'x_0',\n                                       'y_1': 'y_0',\n                                       'z_1': 'z_0'})\n    df_temp=pd.concat(objs=[df_temp,df_temp_],axis=0)\n\n    df_temp[\"min_distance\"]=df_temp.groupby(['molecule_name', 'atom_index_0'])['dist'].transform('min')\n    df_temp= df_temp[df_temp[\"min_distance\"]==df_temp[\"dist\"]]\n\n    df_temp=df_temp.drop(['x_0','y_0','z_0','min_distance'], axis=1)\n    df_temp= df_temp.rename(columns={'atom_index_0': 'atom_index',\n                                     'atom_index_1': 'atom_index_closest',\n                                     'distance': 'distance_closest',\n                                     'x_1': 'x_closest',\n                                     'y_1': 'y_closest',\n                                     'z_1': 'z_closest'})\n\n    for atom_idx in [0,1]:\n        df_train = map_atom_info(df_train,df_temp, atom_idx)\n        df_train = df_train.rename(columns={'atom_index_closest': f'atom_index_closest_{atom_idx}',\n                                            'distance_closest': f'distance_closest_{atom_idx}',\n                                            'x_closest': f'x_closest_{atom_idx}',\n                                            'y_closest': f'y_closest_{atom_idx}',\n                                            'z_closest': f'z_closest_{atom_idx}'})\n    return df_train\n\n#dtrain = create_closest(train)\n#dtest = create_closest(test)\n#print('dtrain size',dtrain.shape)\n#print('dtest size',dtest.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### cosine angles calculation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_cos_features(df):\n    df[\"distance_0\"]=((df['x_0']-df['x_closest_0'])**2+(df['y_0']-df['y_closest_0'])**2+(df['z_0']-df['z_closest_0'])**2)**(1/2)\n    df[\"distance_1\"]=((df['x_1']-df['x_closest_1'])**2+(df['y_1']-df['y_closest_1'])**2+(df['z_1']-df['z_closest_1'])**2)**(1/2)\n    df[\"vec_0_x\"]=(df['x_0']-df['x_closest_0'])/df[\"distance_0\"]\n    df[\"vec_0_y\"]=(df['y_0']-df['y_closest_0'])/df[\"distance_0\"]\n    df[\"vec_0_z\"]=(df['z_0']-df['z_closest_0'])/df[\"distance_0\"]\n    df[\"vec_1_x\"]=(df['x_1']-df['x_closest_1'])/df[\"distance_1\"]\n    df[\"vec_1_y\"]=(df['y_1']-df['y_closest_1'])/df[\"distance_1\"]\n    df[\"vec_1_z\"]=(df['z_1']-df['z_closest_1'])/df[\"distance_1\"]\n    df[\"vec_x\"]=(df['x_1']-df['x_0'])/df[\"dist\"]\n    df[\"vec_y\"]=(df['y_1']-df['y_0'])/df[\"dist\"]\n    df[\"vec_z\"]=(df['z_1']-df['z_0'])/df[\"dist\"]\n    df[\"cos_0_1\"]=df[\"vec_0_x\"]*df[\"vec_1_x\"]+df[\"vec_0_y\"]*df[\"vec_1_y\"]+df[\"vec_0_z\"]*df[\"vec_1_z\"]\n    df[\"cos_0\"]=df[\"vec_0_x\"]*df[\"vec_x\"]+df[\"vec_0_y\"]*df[\"vec_y\"]+df[\"vec_0_z\"]*df[\"vec_z\"]\n    df[\"cos_1\"]=df[\"vec_1_x\"]*df[\"vec_x\"]+df[\"vec_1_y\"]*df[\"vec_y\"]+df[\"vec_1_z\"]*df[\"vec_z\"]\n    df=df.drop(['vec_0_x','vec_0_y','vec_0_z','vec_1_x','vec_1_y','vec_1_z','vec_x','vec_y','vec_z'], axis=1)\n    return df\n    \n#train = add_cos_features(train)\n#test = add_cos_features(test)\n\n#print('train size',train.shape)\n#print('test size',test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Dropping molecule_name and encode atom_0, atom_1 and type_0.<br>\n**@TODO:** Try others encoders "},{"metadata":{"trusted":true},"cell_type":"code","source":"del_cols_list = ['id','molecule_name','sd','pso','dso']\ndef del_cols(df, cols):\n    del_cols_list_ = [l for l in del_cols_list if l in df]\n    df = df.drop(del_cols_list_,axis=1)\n    return df\n\ntrain = del_cols(train,del_cols_list)\ntest = del_cols(test,del_cols_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def encode_categoric_single(df):\n    lbl = LabelEncoder()\n    cat_cols=[]\n    try:\n        cat_cols = df.describe(include=['O']).columns.tolist()\n        for cat in cat_cols:\n            df[cat] = lbl.fit_transform(list(df[cat].values))\n    except Exception as e:\n        print('error: ', str(e) )\n\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def encode_categoric(dtrain,dtest):\n    lbl = LabelEncoder()\n    objs_n = len(dtrain)\n    dfmerge = pd.concat(objs=[dtrain,dtest],axis=0)\n    cat_cols=[]\n    try:\n        cat_cols = dfmerge.describe(include=['O']).columns.tolist()\n        for cat in cat_cols:\n            dfmerge[cat] = lbl.fit_transform(list(dfmerge[cat].values))\n    except Exception as e:\n        print('error: ', str(e) )\n\n    dtrain = dfmerge[:objs_n]\n    dtest = dfmerge[objs_n:]\n    return dtrain,dtest\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = encode_categoric_single(train)\ntest = encode_categoric_single(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_fc = train['fc']\nX = train.drop(['scalar_coupling_constant','fc'],axis=1)\ny = train['scalar_coupling_constant']\n\nX_test = test.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('X size',X.shape)\nprint('X_test size',X_test.shape)\nprint('dtest size',test.shape)\nprint('y_fc size',y_fc.shape)\n\ndel train, test\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"good_columns = ['type',\n 'bond_lengths_mean_y',\n 'bond_lengths_std_y',\n 'bond_lengths_mean_x',\n 'molecule_atom_index_0_dist_min_div',\n 'molecule_atom_index_0_dist_std_div',\n 'molecule_atom_index_0_dist_mean',\n 'molecule_atom_index_0_dist_max',\n 'dist_y',\n 'molecule_atom_index_1_dist_std_diff',\n 'z_0',\n 'molecule_type_dist_min',\n 'molecule_atom_index_0_y_1_mean_div',\n 'dist_x',\n 'x_0',\n 'y_0',\n 'molecule_type_dist_std',\n 'molecule_atom_index_0_y_1_std',\n 'molecule_dist_mean',\n 'molecule_atom_index_0_dist_std_diff',\n 'dist_z',\n 'molecule_atom_index_0_dist_std',\n 'molecule_atom_index_0_x_1_std',\n 'molecule_type_dist_std_diff',\n 'molecule_type_0_dist_std',\n 'dist',\n 'molecule_atom_index_0_dist_mean_diff',\n 'molecule_atom_index_1_dist_min_div',\n 'molecule_atom_index_1_dist_mean_diff',\n 'y_1',\n 'molecule_type_dist_mean_div',\n 'molecule_dist_max',\n 'molecule_atom_index_0_dist_mean_div',\n 'z_1',\n 'molecule_atom_index_0_z_1_std',\n 'molecule_atom_index_1_dist_mean_div',\n 'molecule_atom_index_1_dist_min_diff',\n 'molecule_atom_index_1_dist_mean',\n 'molecule_atom_index_1_dist_min',\n 'molecule_atom_index_1_dist_max',\n 'molecule_type_0_dist_std_diff',\n 'molecule_atom_index_0_dist_min_diff',\n 'molecule_type_dist_mean_diff',\n 'x_1',\n 'molecule_atom_index_0_y_1_max',\n 'molecule_atom_index_0_y_1_mean_diff',\n 'molecule_atom_1_dist_std_diff',\n 'molecule_atom_index_0_y_1_mean',\n 'molecule_atom_1_dist_std',\n 'molecule_type_dist_max']\n\nX = X[good_columns].copy()\nX_test = X_test[good_columns].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id5\"></a> <br> \n# **5. Model** \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_fold = 5\nfolds = KFold(n_splits=n_fold, shuffle=True, random_state=11)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create out of fold feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {'num_leaves': 50,\n          'min_child_samples': 79,\n          'min_data_in_leaf': 100,\n          'objective': 'regression',\n          'max_depth': 9,\n          'learning_rate': 0.2,\n          \"boosting_type\": \"gbdt\",\n          \"subsample_freq\": 1,\n          \"subsample\": 0.9,\n          \"bagging_seed\": 11,\n          \"metric\": 'mae',\n          \"verbosity\": -1,\n          'reg_alpha': 0.1,\n          'reg_lambda': 0.3,\n          'colsample_bytree': 1.0\n         }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_short = pd.DataFrame({'ind': list(X.index), 'type': X['type'].values, 'oof': [0] * len(X), 'target': y_fc.values})\nX_short_test = pd.DataFrame({'ind': list(X_test.index), 'type': X_test['type'].values, 'prediction': [0] * len(X_test)})\nresult_dict_lgb_oof =  {}\nfor t in X['type'].unique():\n    print(f'Training of type {t}')\n    X_t = X.loc[X['type'] == t]\n    X_test_t = X_test.loc[X_test['type'] == t]\n    y_t = X_short.loc[X_short['type'] == t, 'target']\n    result_dict_lgb_oof = train_model_regression(X=X_t, X_test=X_test_t, y=y_t, params=params, folds=folds, model_type='lgb', eval_metric='group_mae', plot_feature_importance=False,\n                                                      verbose=500, early_stopping_rounds=200, n_estimators=n_estimators_default)\n    X_short.loc[X_short['type'] == t, 'oof'] = result_dict_lgb_oof['oof']\n    X_short_test.loc[X_short_test['type'] == t, 'prediction'] = result_dict_lgb_oof['prediction']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X['oof_fc'] = X_short['oof']\nX_test['oof_fc'] = X_short_test['prediction']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ngood_columns = ['type','oof_fc',\n 'bond_lengths_mean_y',\n 'bond_lengths_std_y',\n 'bond_lengths_mean_x',\n 'molecule_atom_index_0_dist_min_div',\n 'molecule_atom_index_0_dist_std_div',\n 'molecule_atom_index_0_dist_mean',\n 'molecule_atom_index_0_dist_max',\n 'dist_y',\n 'molecule_atom_index_1_dist_std_diff',\n 'z_0',\n 'molecule_type_dist_min',\n 'molecule_atom_index_0_y_1_mean_div',\n 'dist_x',\n 'x_0',\n 'y_0',\n 'molecule_type_dist_std',\n 'molecule_atom_index_0_y_1_std',\n 'molecule_dist_mean',\n 'molecule_atom_index_0_dist_std_diff',\n 'dist_z',\n 'molecule_atom_index_0_dist_std',\n 'molecule_atom_index_0_x_1_std',\n 'molecule_type_dist_std_diff',\n 'molecule_type_0_dist_std',\n 'dist',\n 'molecule_atom_index_0_dist_mean_diff',\n 'molecule_atom_index_1_dist_min_div',\n 'molecule_atom_index_1_dist_mean_diff',\n 'y_1',\n 'molecule_type_dist_mean_div',\n 'molecule_dist_max',\n 'molecule_atom_index_0_dist_mean_div',\n 'z_1',\n 'molecule_atom_index_0_z_1_std',\n 'molecule_atom_index_1_dist_mean_div',\n 'molecule_atom_index_1_dist_min_diff',\n 'molecule_atom_index_1_dist_mean',\n 'molecule_atom_index_1_dist_min',\n 'molecule_atom_index_1_dist_max',\n 'molecule_type_0_dist_std_diff',\n 'molecule_atom_index_0_dist_min_diff',\n 'molecule_type_dist_mean_diff',\n 'x_1',\n 'molecule_atom_index_0_y_1_max',\n 'molecule_atom_index_0_y_1_mean_diff',\n 'molecule_atom_1_dist_std_diff',\n 'molecule_atom_index_0_y_1_mean',\n 'molecule_atom_1_dist_std',\n 'molecule_type_dist_max']\n\nX = X[good_columns].copy()\nX_test = X_test[good_columns].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Checking Best Feature for Final Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {'num_leaves': 50,\n          'min_child_samples': 79,\n          'min_data_in_leaf': 100,\n          'objective': 'regression',\n          'max_depth': 9,\n          'learning_rate': 0.2,\n          \"boosting_type\": \"gbdt\",\n          \"subsample_freq\": 1,\n          \"subsample\": 0.9,\n          \"bagging_seed\": 11,\n          \"metric\": 'mae',\n          \"verbosity\": -1,\n          'reg_alpha': 0.1,\n          'reg_lambda': 0.3,\n          'colsample_bytree': 1.0\n         }\n#result_dict_lgb2 = train_model_regression(X=X, X_test=X_test, y=y, params=params, folds=folds, model_type='lgb', eval_metric='group_mae', plot_feature_importance=True,\n#                                                      verbose=500, early_stopping_rounds=200, n_estimators=n_estimators_default)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Best Features? \n''' \nfeature_importance = result_dict_lgb2['feature_importance']\nbest_features = feature_importance[['feature','importance']].groupby(['feature']).mean().sort_values(\n        by='importance',ascending=False).iloc[:50,0:0].index.tolist()\nbest_features'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id6\"></a> <br> \n# **6. Final Model** "},{"metadata":{},"cell_type":"markdown","source":"## Training models for each type"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_short = pd.DataFrame({'ind': list(X.index), 'type': X['type'].values, 'oof': [0] * len(X), 'target': y.values})\nX_short_test = pd.DataFrame({'ind': list(X_test.index), 'type': X_test['type'].values, 'prediction': [0] * len(X_test)})\nfor t in X['type'].unique():\n    print(f'Training of type {t}')\n    X_t = X.loc[X['type'] == t]\n    X_test_t = X_test.loc[X_test['type'] == t]\n    y_t = X_short.loc[X_short['type'] == t, 'target']\n    result_dict_lgb3 = train_model_regression(X=X_t, X_test=X_test_t, y=y_t, params=params, folds=folds, model_type='lgb', eval_metric='group_mae', plot_feature_importance=False,\n                                                      verbose=500, early_stopping_rounds=200, n_estimators=n_estimators_default)\n    X_short.loc[X_short['type'] == t, 'oof'] = result_dict_lgb3['oof']\n    X_short_test.loc[X_short_test['type'] == t, 'prediction'] = result_dict_lgb3['prediction']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id7\"></a> <br> \n# **7. Submittion** "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Training models for type\nsub['scalar_coupling_constant'] = X_short_test['prediction']\nsub.to_csv('submission_type.csv', index=False)\nsub.head()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"ref\"></a> <br> \n# **8. References** \n\n[1] OOF Model: https://www.kaggle.com/adarshchavakula/out-of-fold-oof-model-cross-validation<br>\n[2] Using Meta Features: https://www.kaggle.com/artgor/using-meta-features-to-improve-model<br>\n[3] Lot of Features: https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b <br>\n[4] Angle Feature: https://www.kaggle.com/kmat2019/effective-feature <br>\n[5] Recovering bonds from structure: https://www.kaggle.com/aekoch95/bonds-from-structure-data <br>\n\n<h3 style=\"color:red\">If this Kernel Helps You! Please UP VOTE! 😁</h3>"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.4"}},"nbformat":4,"nbformat_minor":1}