{"cells":[{"metadata":{},"cell_type":"markdown","source":"w# Keras Multiple Output Solution\nThanks to https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/ and https://www.kaggle.com/kmat2019/neural-network-modeling-with-multiple-outputs for the idea on approaching this problem as a multiple output problem.  Though it doesn't seem to be the favored approach for this competition, I feel that there ought to be a good neural network approach.  This kernel tries a multi-layer, dense neural network implemented in Keras.  The advantage of this approach is that it does not seem to be overfitting, which may pay off against the full dataset.\n\nWays to improve:\n*  I'm not a domain expert in the molecular chem field... I strongly suspect that stronger feature engineering would cause this approach to score higher.  \n*  Network architecture:  I'm putting a simpler variant forward here with some options commented out.  There are tweaks that could be made to this architecture that will improve the score.  Forcing the model to overfit to gain a better score on the leaderboard does not usually pay off in the end...\n*  More epochs.  The more epochs that can be run without overfitting, the better score could be achieved.  My observation is that even after long training epochs, the model seems to still be learning."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler\n\nimport tensorflow as tf\nfrom keras.layers import Dense, Input, Activation\nfrom keras.layers import BatchNormalization,Add,Dropout\nfrom keras.optimizers import Adam\nfrom keras.models import Model, load_model\nfrom keras import callbacks\nfrom keras import backend as K\nfrom keras.layers.advanced_activations import LeakyReLU\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nwarnings.filterwarnings(action=\"ignore\",category=DeprecationWarning)\nwarnings.filterwarnings(action=\"ignore\",category=FutureWarning)\nimport os\n%cd /kaggle/input/champs-scalar-coupling\nprint(os.listdir(\".\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## First grab the data.\nI don't like to clutter up my solution notebooks with my EDA work.  That's usually a separate notebook."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df_train=pd.read_csv('train.csv')\ndf_test=pd.read_csv('test.csv')\ndf_struct=pd.read_csv('structures.csv')\n\n#df_train_sub_potential=pd.read_csv('/content/champs/potential_energy.csv')\n#df_train_sub_moment=pd.read_csv('../input/dipole_moments.csv')\ndf_train_sub_charge=pd.read_csv('mulliken_charges.csv')\ndf_train_sub_tensor=pd.read_csv('magnetic_shielding_tensors.csv')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reduce the Memory Usage\nWithout this call, this kernel definitely can't be run on smaller cloud instances... I always test solutions on CoLaboratory to see if low-resource nodes can process them.  In this case, CoLab can't unless you reduce down.  The results seem similar to when the same network is trained on the full dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\nprint(df_train.shape, df_test.shape, df_struct.shape, df_train_sub_charge.shape, df_train_sub_tensor.shape)\n#df_train = reduce_mem_usage(df_train)\n#df_test = reduce_mem_usage(df_test)\n#df_struct = reduce_mem_usage(df_struct)\n#df_train_sub_charge = reduce_mem_usage(df_train_sub_charge)\n#df_train_sub_tensor = reduce_mem_usage(df_train_sub_tensor)\nprint(df_train.shape, df_test.shape, df_struct.shape, df_train_sub_charge.shape, df_train_sub_tensor.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Map data into a master dataframe\nHere's the code to do mappings.  The drop_duplicates is important, else your test dataset will grow and your predictions will not be output correctly."},{"metadata":{"trusted":true},"cell_type":"code","source":"''' \nMap atom info from the structures.csv into the train/test files\n'''\nimport psutil\nimport os\n\ndef map_atom_info(df_1,df_2, atom_idx):\n    print('Mapping...', df_1.shape, df_2.shape, atom_idx)\n    \n    df = pd.merge(df_1, df_2.drop_duplicates(subset=['molecule_name', 'atom_index']), how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    \n    df = df.drop('atom_index', axis=1)\n\n    return df\n\ndef show_ram_usage():\n    py = psutil.Process(os.getpid())\n    print('RAM usage: {} GB'.format(py.memory_info()[0]/2. ** 30))\n\nshow_ram_usage()\n\nfor atom_idx in [0,1]:\n    df_train = map_atom_info(df_train,df_struct, atom_idx)\n    df_train = map_atom_info(df_train,df_train_sub_charge, atom_idx)\n    df_train = map_atom_info(df_train,df_train_sub_tensor, atom_idx)\n    df_train = df_train.rename(columns={'atom': f'atom_{atom_idx}',\n                                        'x': f'x_{atom_idx}',\n                                        'y': f'y_{atom_idx}',\n                                        'z': f'z_{atom_idx}',\n                                        'mulliken_charge': f'charge_{atom_idx}',\n                                        'XX': f'XX_{atom_idx}',\n                                        'YX': f'YX_{atom_idx}',\n                                        'ZX': f'ZX_{atom_idx}',\n                                        'XY': f'XY_{atom_idx}',\n                                        'YY': f'YY_{atom_idx}',\n                                        'ZY': f'ZY_{atom_idx}',\n                                        'XZ': f'XZ_{atom_idx}',\n                                        'YZ': f'YZ_{atom_idx}',\n                                        'ZZ': f'ZZ_{atom_idx}',})\n    df_test = map_atom_info(df_test,df_struct, atom_idx)\n    df_test = df_test.rename(columns={'atom': f'atom_{atom_idx}',\n                                'x': f'x_{atom_idx}',\n                                'y': f'y_{atom_idx}',\n                                'z': f'z_{atom_idx}'})\n    #add some features\n    \n    df_struct['c_x']=df_struct.groupby('molecule_name')['x'].transform('mean')\n    df_struct['c_y']=df_struct.groupby('molecule_name')['y'].transform('mean')\n    df_struct['c_z']=df_struct.groupby('molecule_name')['z'].transform('mean')\n    df_struct['atom_n']=df_struct.groupby('molecule_name')['atom_index'].transform('max')\n    \n    show_ram_usage()\n    print(df_train.shape, df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Start developing more complex features"},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_features(df):\n    df['dx']=df['x_1']-df['x_0']\n    df['dy']=df['y_1']-df['y_0']\n    df['dz']=df['z_1']-df['z_0']\n    df['distance']=(df['dx']**2+df['dy']**2+df['dz']**2)**(1/2)\n    return df\n\ndf_train=make_features(df_train)\ndf_test=make_features(df_test) \n#df_train = reduce_mem_usage(df_train)\n#df_test = reduce_mem_usage(df_test)\ntest_prediction=np.zeros(len(df_test))\nshow_ram_usage()\nprint(df_train.shape, df_test.shape)\n\ndef get_dist(df):\n    df_temp=df.loc[:,[\"molecule_name\",\"atom_index_0\",\"atom_index_1\",\"distance\",\"x_0\",\"y_0\",\"z_0\",\"x_1\",\"y_1\",\"z_1\"]].copy()\n    df_temp_=df_temp.copy()\n    df_temp_= df_temp_.rename(columns={'atom_index_0': 'atom_index_1',\n                                       'atom_index_1': 'atom_index_0',\n                                       'x_0': 'x_1',\n                                       'y_0': 'y_1',\n                                       'z_0': 'z_1',\n                                       'x_1': 'x_0',\n                                       'y_1': 'y_0',\n                                       'z_1': 'z_0'})\n    df_temp_all=pd.concat((df_temp,df_temp_),axis=0)\n\n    df_temp_all[\"min_distance\"]=df_temp_all.groupby(['molecule_name', 'atom_index_0'])['distance'].transform('min')\n    df_temp_all[\"max_distance\"]=df_temp_all.groupby(['molecule_name', 'atom_index_0'])['distance'].transform('max')\n    \n    df_temp= df_temp_all[df_temp_all[\"min_distance\"]==df_temp_all[\"distance\"]].copy()\n    df_temp=df_temp.drop(['x_0','y_0','z_0','min_distance'], axis=1)\n    df_temp= df_temp.rename(columns={'atom_index_0': 'atom_index',\n                                         'atom_index_1': 'atom_index_closest',\n                                         'distance': 'distance_closest',\n                                         'x_1': 'x_closest',\n                                         'y_1': 'y_closest',\n                                         'z_1': 'z_closest'})\n    \n    for atom_idx in [0,1]:\n        df = map_atom_info(df,df_temp, atom_idx)\n        df = df.rename(columns={'atom_index_closest': f'atom_index_closest_{atom_idx}',\n                                        'distance_closest': f'distance_closest_{atom_idx}',\n                                        'x_closest': f'x_closest_{atom_idx}',\n                                        'y_closest': f'y_closest_{atom_idx}',\n                                        'z_closest': f'z_closest_{atom_idx}'})\n        \n    df_temp= df_temp_all[df_temp_all[\"max_distance\"]==df_temp_all[\"distance\"]].copy()\n    df_temp=df_temp.drop(['x_0','y_0','z_0','max_distance'], axis=1)\n    df_temp= df_temp.rename(columns={'atom_index_0': 'atom_index',\n                                         'atom_index_1': 'atom_index_farthest',\n                                         'distance': 'distance_farthest',\n                                         'x_1': 'x_farthest',\n                                         'y_1': 'y_farthest',\n                                         'z_1': 'z_farthest'})\n        \n    for atom_idx in [0,1]:\n        df = map_atom_info(df,df_temp, atom_idx)\n        df = df.rename(columns={'atom_index_farthest': f'atom_index_farthest_{atom_idx}',\n                                        'distance_farthest': f'distance_farthest_{atom_idx}',\n                                        'x_farthest': f'x_farthest_{atom_idx}',\n                                        'y_farthest': f'y_farthest_{atom_idx}',\n                                        'z_farthest': f'z_farthest_{atom_idx}'})\n    return df\ndf_test=(get_dist(df_test))    \ndf_train=(get_dist(df_train)) \n\nprint(df_train.shape, df_test.shape)\nshow_ram_usage()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Here is where the Cosine Distance features are Created"},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_features(df):\n    df[\"distance_center0\"]=((df['x_0']-df['c_x'])**2+(df['y_0']-df['c_y'])**2+(df['z_0']-df['c_z'])**2)**(1/2)\n    df[\"distance_center1\"]=((df['x_1']-df['c_x'])**2+(df['y_1']-df['c_y'])**2+(df['z_1']-df['c_z'])**2)**(1/2)\n    df[\"distance_c0\"]=((df['x_0']-df['x_closest_0'])**2+(df['y_0']-df['y_closest_0'])**2+(df['z_0']-df['z_closest_0'])**2)**(1/2)\n    df[\"distance_c1\"]=((df['x_1']-df['x_closest_1'])**2+(df['y_1']-df['y_closest_1'])**2+(df['z_1']-df['z_closest_1'])**2)**(1/2)\n    df[\"distance_f0\"]=((df['x_0']-df['x_farthest_0'])**2+(df['y_0']-df['y_farthest_0'])**2+(df['z_0']-df['z_farthest_0'])**2)**(1/2)\n    df[\"distance_f1\"]=((df['x_1']-df['x_farthest_1'])**2+(df['y_1']-df['y_farthest_1'])**2+(df['z_1']-df['z_farthest_1'])**2)**(1/2)\n    df[\"vec_center0_x\"]=(df['x_0']-df['c_x'])/(df[\"distance_center0\"]+1e-10)\n    df[\"vec_center0_y\"]=(df['y_0']-df['c_y'])/(df[\"distance_center0\"]+1e-10)\n    df[\"vec_center0_z\"]=(df['z_0']-df['c_z'])/(df[\"distance_center0\"]+1e-10)\n    df[\"vec_center1_x\"]=(df['x_1']-df['c_x'])/(df[\"distance_center1\"]+1e-10)\n    df[\"vec_center1_y\"]=(df['y_1']-df['c_y'])/(df[\"distance_center1\"]+1e-10)\n    df[\"vec_center1_z\"]=(df['z_1']-df['c_z'])/(df[\"distance_center1\"]+1e-10)\n    df[\"vec_c0_x\"]=(df['x_0']-df['x_closest_0'])/(df[\"distance_c0\"]+1e-10)\n    df[\"vec_c0_y\"]=(df['y_0']-df['y_closest_0'])/(df[\"distance_c0\"]+1e-10)\n    df[\"vec_c0_z\"]=(df['z_0']-df['z_closest_0'])/(df[\"distance_c0\"]+1e-10)\n    df[\"vec_c1_x\"]=(df['x_1']-df['x_closest_1'])/(df[\"distance_c1\"]+1e-10)\n    df[\"vec_c1_y\"]=(df['y_1']-df['y_closest_1'])/(df[\"distance_c1\"]+1e-10)\n    df[\"vec_c1_z\"]=(df['z_1']-df['z_closest_1'])/(df[\"distance_c1\"]+1e-10)\n    df[\"vec_f0_x\"]=(df['x_0']-df['x_farthest_0'])/(df[\"distance_f0\"]+1e-10)\n    df[\"vec_f0_y\"]=(df['y_0']-df['y_farthest_0'])/(df[\"distance_f0\"]+1e-10)\n    df[\"vec_f0_z\"]=(df['z_0']-df['z_farthest_0'])/(df[\"distance_f0\"]+1e-10)\n    df[\"vec_f1_x\"]=(df['x_1']-df['x_farthest_1'])/(df[\"distance_f1\"]+1e-10)\n    df[\"vec_f1_y\"]=(df['y_1']-df['y_farthest_1'])/(df[\"distance_f1\"]+1e-10)\n    df[\"vec_f1_z\"]=(df['z_1']-df['z_farthest_1'])/(df[\"distance_f1\"]+1e-10)\n    df[\"vec_x\"]=(df['x_1']-df['x_0'])/df[\"distance\"]\n    df[\"vec_y\"]=(df['y_1']-df['y_0'])/df[\"distance\"]\n    df[\"vec_z\"]=(df['z_1']-df['z_0'])/df[\"distance\"]\n    df[\"cos_c0_c1\"]=df[\"vec_c0_x\"]*df[\"vec_c1_x\"]+df[\"vec_c0_y\"]*df[\"vec_c1_y\"]+df[\"vec_c0_z\"]*df[\"vec_c1_z\"]\n    df[\"cos_f0_f1\"]=df[\"vec_f0_x\"]*df[\"vec_f1_x\"]+df[\"vec_f0_y\"]*df[\"vec_f1_y\"]+df[\"vec_f0_z\"]*df[\"vec_f1_z\"]\n    df[\"cos_center0_center1\"]=df[\"vec_center0_x\"]*df[\"vec_center1_x\"]+df[\"vec_center0_y\"]*df[\"vec_center1_y\"]+df[\"vec_center0_z\"]*df[\"vec_center1_z\"]\n    df[\"cos_c0\"]=df[\"vec_c0_x\"]*df[\"vec_x\"]+df[\"vec_c0_y\"]*df[\"vec_y\"]+df[\"vec_c0_z\"]*df[\"vec_z\"]\n    df[\"cos_c1\"]=df[\"vec_c1_x\"]*df[\"vec_x\"]+df[\"vec_c1_y\"]*df[\"vec_y\"]+df[\"vec_c1_z\"]*df[\"vec_z\"]\n    df[\"cos_f0\"]=df[\"vec_f0_x\"]*df[\"vec_x\"]+df[\"vec_f0_y\"]*df[\"vec_y\"]+df[\"vec_f0_z\"]*df[\"vec_z\"]\n    df[\"cos_f1\"]=df[\"vec_f1_x\"]*df[\"vec_x\"]+df[\"vec_f1_y\"]*df[\"vec_y\"]+df[\"vec_f1_z\"]*df[\"vec_z\"]\n    df[\"cos_center0\"]=df[\"vec_center0_x\"]*df[\"vec_x\"]+df[\"vec_center0_y\"]*df[\"vec_y\"]+df[\"vec_center0_z\"]*df[\"vec_z\"]\n    df[\"cos_center1\"]=df[\"vec_center1_x\"]*df[\"vec_x\"]+df[\"vec_center1_y\"]*df[\"vec_y\"]+df[\"vec_center1_z\"]*df[\"vec_z\"]\n    df=df.drop(['vec_c0_x','vec_c0_y','vec_c0_z','vec_c1_x','vec_c1_y','vec_c1_z',\n                'vec_f0_x','vec_f0_y','vec_f0_z','vec_f1_x','vec_f1_y','vec_f1_z',\n                'vec_center0_x','vec_center0_y','vec_center0_z','vec_center1_x','vec_center1_y','vec_center1_z',\n                'vec_x','vec_y','vec_z'], axis=1)\n    return df\n    \ndf_train=add_features(df_train)\ndf_test=add_features(df_test)\nprint(df_train.shape, df_test.shape)\nshow_ram_usage()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Neural Network definition\n\nThis neural network is many layers.  In the middle we define our outputs for our two Mullikan charges as well as our Dipole Moment.  The final output is the one we care the most about, the Scalar Coupling Constant.\n\nI think that BatchNormalization at each layer seems superior than small amounts of dropouts.  The network seems to not overfit, even in large numbers of training epochs.  If you do wind up seeing some overfitting, then adding the dropout to a couple of layers ought to help a lot."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_nn_model(input_shape):\n    inp = Input(shape=(input_shape,))\n    x = Dense(256)(inp)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dropout(0.4)(x)\n    x = Dense(1024)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dropout(0.2)(x)\n    x = Dense(1024)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dropout(0.2)(x)\n    x = Dense(512)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dropout(0.4)(x)\n    x = Dense(512)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    #x = Dropout(0.4)(x)\n    x = Dense(256)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dropout(0.4)(x)\n    out1 = Dense(2, activation=\"linear\")(x)#mulliken charge 2\n    out2 = Dense(6, activation=\"linear\")(x)#tensor 6(xx,yy,zz)\n    out3 = Dense(12, activation=\"linear\")(x)#tensor 12(others) \n    x = Dense(128)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dropout(0.2)(x)\n    x = Dense(128)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dense(64)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(alpha=0.05)(x)\n    x = Dropout(0.2)(x)\n    out = Dense(1, activation=\"linear\")(x)#scalar_coupling_constant    \n    model = Model(inputs=inp, outputs=[out,out1,out2,out3])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot Function\nI rely a lot on loss plots to detect when learning has stopped as well as when overfitting begins."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_history(history, label):\n    plt.plot(history.history['loss'])\n    plt.plot(history.history['val_loss'])\n    plt.title('Loss for %s' % label)\n    plt.ylabel('Loss')\n    plt.xlabel('Epoch')\n    _= plt.legend(['Train','Validation'], loc='upper left')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Main Routine\n\nA bunch of stuff happens here.  Pay attention to the callbacks.  I train a different model for each molecule type, which allows for future retraining.  If you have kept your network the same (except for dropout, etc.), and want to retrain for a few more epochs without having to go back to the beginning, then set the retrain flag to False and it will grab the trained models as starting points."},{"metadata":{"trusted":true},"cell_type":"code","source":"from datetime import datetime\n\nmol_types=df_train[\"type\"].unique()\ncv_score=[]\ncv_score_total=0\nepoch_n = 300\nverbose = 0\nbatch_size = 2048\n    \n# Set to True if we want to train from scratch.  False will reuse saved models as a starting point.\nretrain =False\n\n\n# Set up GPU preferences\nconfig = tf.ConfigProto( device_count = {'GPU': 1 , 'CPU': 2} ) \nconfig.gpu_options.allow_growth = True\nconfig.gpu_options.per_process_gpu_memory_fraction = 0.6\nsess = tf.Session(config=config) \nK.set_session(sess)\n\nstart_time=datetime.now()\n\n# Loop through each molecule type\nfor mol_type in mol_types:\n    model_name_rd = ('../keras-neural-net-for-champs/molecule_model_%s.hdf5' % mol_type)\n    model_name_wrt = ('/kaggle/working/molecule_model_%s.hdf5' % mol_type)\n    print('Training %s' % mol_type, 'out of', mol_types, '\\n')\n    \n    df_train_=df_train[df_train[\"type\"]==mol_type]\n    df_test_=df_test[df_test[\"type\"]==mol_type]\n    \n    # Here's our best features.  We think.\n    input_features=[\"x_0\",\"y_0\",\"z_0\",\"x_1\",\"y_1\",\"z_1\",\"c_x\",\"c_y\",\"c_z\",\n                    'x_closest_0','y_closest_0','z_closest_0','x_closest_1','y_closest_1','z_closest_1',\n                    \"distance\",\"distance_center0\",\"distance_center1\", \"distance_c0\",\"distance_c1\",\"distance_f0\",\"distance_f1\",\n                    \"cos_c0_c1\",\"cos_f0_f1\",\"cos_center0_center1\",\"cos_c0\",\"cos_c1\",\"cos_f0\",\"cos_f1\",\"cos_center0\",\"cos_center1\",\n                    \"atom_n\"\n                   ]\n    \n    # Standard Scaler from sklearn does seem to work better here than other Scalers\n    input_data=StandardScaler().fit_transform(pd.concat([df_train_.loc[:,input_features],df_test_.loc[:,input_features]]))\n    \n    target_data=df_train_.loc[:,\"scalar_coupling_constant\"].values\n    target_data_1=df_train_.loc[:,[\"charge_0\",\"charge_1\"]]\n    target_data_2=df_train_.loc[:,[\"XX_0\",\"YY_0\",\"ZZ_0\",\"XX_1\",\"YY_1\",\"ZZ_1\"]]\n    target_data_3=df_train_.loc[:,[\"YX_0\",\"ZX_0\",\"XY_0\",\"ZY_0\",\"XZ_0\",\"YZ_0\",\"YX_1\",\"ZX_1\",\"XY_1\",\"ZY_1\",\"XZ_1\",\"YZ_1\"]]\n    \n    #following parameters should be adjusted to control the loss function\n    #if all parameters are zero, attractors do not work. (-> simple neural network)\n    m1=1\n    m2=4\n    m3=1\n    target_data_1=m1*(StandardScaler().fit_transform(target_data_1))\n    target_data_2=m2*(StandardScaler().fit_transform(target_data_2))\n    target_data_3=m3*(StandardScaler().fit_transform(target_data_3))\n    \n    # Simple split to provide us a validation set to do our CV checks with\n    train_index, cv_index = train_test_split(np.arange(len(df_train_)),random_state=111, test_size=0.1)\n    \n    # Split all our input and targets by train and cv indexes\n    train_input=input_data[train_index]\n    cv_input=input_data[cv_index]\n    train_target=target_data[train_index]\n    cv_target=target_data[cv_index]\n    train_target_1=target_data_1[train_index]\n    cv_target_1=target_data_1[cv_index]\n    train_target_2=target_data_2[train_index]\n    cv_target_2=target_data_2[cv_index]\n    train_target_3=target_data_3[train_index]\n    cv_target_3=target_data_3[cv_index]\n    test_input=input_data[len(df_train_):,:]\n\n    # Build the Neural Net\n    nn_model=create_nn_model(train_input.shape[1])\n    \n    # If retrain==False, then we load a previous saved model as a starting point.\n    if not retrain:\n        nn_model = load_model(model_name_rd)\n        \n    nn_model.compile(loss='mae', optimizer=Adam())#, metrics=[auc])\n    \n    # Callback for Early Stopping... May want to raise the min_delta for small numbers of epochs\n    es = callbacks.EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=8,verbose=1, mode='auto', restore_best_weights=True)\n    # Callback for Reducing the Learning Rate... when the monitor levels out for 'patience' epochs, then the LR is reduced\n    rlr = callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1,patience=7, min_lr=1e-6, mode='auto', verbose=1)\n    # Save the best value of the model for future use\n    sv_mod = callbacks.ModelCheckpoint(model_name_wrt, monitor='val_loss', save_best_only=True, period=1)\n\n    history = nn_model.fit(train_input,[train_target,train_target_1,train_target_2,train_target_3], \n            validation_data=(cv_input,[cv_target,cv_target_1,cv_target_2,cv_target_3]), \n            callbacks=[es, rlr, sv_mod], epochs=epoch_n, batch_size=batch_size, verbose=verbose)\n    \n    cv_predict=nn_model.predict(cv_input)\n    plot_history(history, mol_type)\n    \n    accuracy=np.mean(np.abs(cv_target-cv_predict[0][:,0]))\n    cv_score.append(np.log(accuracy))\n    cv_score_total+=np.log(accuracy)\n    \n    # Predict on the test data set using our trained model\n    test_predict=nn_model.predict(test_input)\n    \n    # for each molecule type we'll grab the predicted values\n    test_prediction[df_test[\"type\"]==mol_type]=test_predict[0][:,0]\n    K.clear_session()\n\ncv_score_total/=len(mol_types)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare results for Submission\n\nThe total CV score matches Kaggle's score pretty closely."},{"metadata":{"trusted":true},"cell_type":"code","source":"def submit(predictions):\n    submit = pd.read_csv('sample_submission.csv')\n    print(len(submit), len(predictions))   \n    submit[\"scalar_coupling_constant\"] = predictions\n    submit.to_csv(\"/kaggle/working/workingsubmission-test.csv\", index=False)\nsubmit(test_prediction)\n\nprint ('Total training time: ', datetime.now() - start_time)\n\ni=0\nfor mol_type in mol_types: \n    print(mol_type,\": cv score is \",cv_score[i])\n    i+=1\nprint(\"total cv score is\",cv_score_total)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras\nprint(keras.__version__)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}