{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n![Molecules](http://media.gettyimages.com/photos/science-or-medical-background-with-molecules-and-atoms-picture-id852106780?s=170667a&w=1007)\n\n**This kernel is based on the kaggle competition [Predicting Molecular Properties](https://www.kaggle.com/c/champs-scalar-coupling/overview). This Kernel consists of Visualzation of all the datasets in the Data segment of the competition for better understanding of properties of Atoms and Molecules.**\n* **Atom** - The atom is the smallest object that retains the properties of an element. Atoms are composed of electrons and a nucleus. The nucleus contains protons and neutrons. \n* **Molecules** - A molecule is a particle made up of two or more atoms that are chemically bonded together; the number of atomic nuclei making up a molecule is a determinate number. \n* **Dipole Moment** - A dipole moment is a measurement of the separation of two opposite electrical charges. Dipole moments are a vector quantity. The magnitude is equal to the charge multiplied by the distance between the charges and the direction is from negative charge to positive charge: μ = q · r where μ is the dipole moment, q is the magnitude of the separated charge, and r is the distance between the charges.\n* **Potential energy** - Potential energy is that energy which an object has because of its position. It is called potential energy because it has the potential to be converted into other forms of energy, such as kinetic energy. \n"},{"metadata":{},"cell_type":"markdown","source":"> ## SPECIAL NOTE:\n* **I have used both matplotlib and plotly for visualization purpose but, I have commented out the plotly visualizations because every time I tried to \"Commit and Run\" it took more than eight hours and failed after that. Eight versions of my kernel failed like that.**\n* **The Plotly visualizations ran fine before commiting, and also they were some awesome 3D-visualizations. Please fork this kernel un-comment out the plotly visualization sections and you would also enjoy some awesome 3D-visualization.**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.plotly as py\nimport plotly.offline as py\n#py.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nfrom mpl_toolkits.mplot3d import Axes3D\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"potential = pd.read_csv('../input/potential_energy.csv')\npotential.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"charge = pd.read_csv('../input/mulliken_charges.csv')\ncharge.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scalar = pd.read_csv('../input/scalar_coupling_contributions.csv')\nscalar.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"structures = pd.read_csv('../input/structures.csv')\nstructures.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"magnetic = pd.read_csv('../input/magnetic_shielding_tensors.csv')\nmagnetic.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dipole = pd.read_csv('../input/dipole_moments.csv')\ndipole.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualizations"},{"metadata":{},"cell_type":"markdown","source":"## Dipole Moment of Molecules"},{"metadata":{"trusted":true},"cell_type":"code","source":"%config InlineBackend.print_figure_kwargs = {'bbox_inches':None}\nfig = plt.figure(figsize=(15,10))\nN = 10000\nax = fig.add_subplot(111, projection='3d')\nax.scatter(dipole['X'].values[0:N],dipole['Y'].values[0:N],dipole['Z'].values[0:N],\n           c=dipole['Z'].values[0:N],s=20,alpha=0.2)\nax.set_xlabel(\"X\", fontsize=18)\nax.set_ylabel(\"Y\", fontsize=18)\nax.set_zlabel(\"Z\", fontsize=18)\nax.set_title(\"Dipole Moment of Molecules\",fontsize=18)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f,ax = plt.subplots(1,3,figsize=(20,10))\nn1 = sns.scatterplot(dipole['X'].values[0:N],dipole['Y'].values[0:N],hue=dipole['Z'].values[0:N],s=50,ax=ax[0])\nn2 = sns.scatterplot(dipole['Y'].values[0:N],dipole['Z'].values[0:N],hue=dipole['Z'].values[0:N],s=50,ax=ax[1])\nn3 = sns.scatterplot(dipole['Z'].values[0:N],dipole['X'].values[0:N],hue=dipole['Z'].values[0:N],s=50,ax=ax[2])\nax[0].set_xlabel(\"X\", fontsize=18)\nax[1].set_xlabel(\"Y\", fontsize=18)\nax[2].set_xlabel(\"Z\", fontsize=18)\nax[0].set_ylabel(\"Y\", fontsize=18)\nax[1].set_ylabel(\"Z\", fontsize=18)\nax[2].set_ylabel(\"X\", fontsize=18)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Implementation with plotly\n#N = 10000\n#trace1 = go.Scatter3d(\n#    x=dipole['X'].values[0:N],\n#    y=dipole['Y'].values[0:N],\n#    z=dipole['Z'].values[0:N],\n#    mode='markers',\n#    marker=dict(\n#        size=5,\n#        color=dipole['Z'],                # set color to an array/list of desired values\n#        colorscale='Jet',   # choose a colorscale\n#        opacity=0.2,\n#        showscale=True,\n#        line = dict(\n#            width = 2,\n#        )\n#    )\n#)\n\n#data = [trace1]\n#layout = go.Layout(\n#    title = 'Dipole Moments',\n#    scene = dict(\n#        xaxis = dict(title=\"X\"),\n#        yaxis = dict(title=\"Y\"),\n#        zaxis = dict(title=\"Z\"),\n#    ),\n#    margin=dict(\n#        l=0,\n#        r=0,\n#        b=0,\n#        t=0\n#    ),\n    \n#)\n#fig = go.Figure(data=data, layout=layout)\n#py.iplot(fig, filename='Dipole_Moments')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Molecular Structure with respect to atomic index"},{"metadata":{"trusted":true},"cell_type":"code","source":"%config InlineBackend.print_figure_kwargs = {'bbox_inches':None}\nfig = plt.figure(figsize=(15,15))\nN = 10000\nax = fig.add_subplot(111, projection='3d')\nax.scatter(structures['y'].values[0:N],structures['x'].values[0:N],structures['z'].values[0:N],\n           c=structures['atom_index'].values[0:N],s=30,alpha=0.2,)\nax.set_xlabel(\"Y\", fontsize=18)\nax.set_ylabel(\"X\", fontsize=18)\nax.set_zlabel(\"Z\", fontsize=18)\nax.set_title(\"Molecule Structures\",fontsize=18)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f,ax = plt.subplots(1,3,figsize=(20,10))\nn1 = sns.scatterplot(structures['x'].values[0:N],structures['y'].values[0:N],hue=structures['atom_index'].values[0:N],palette=\"Blues_r\",linewidth=0,s=20,ax=ax[0])\nn2 = sns.scatterplot(structures['z'].values[0:N],structures['y'].values[0:N],hue=structures['atom_index'].values[0:N],palette=\"Blues_r\",linewidth=0,s=20,ax=ax[1])\nn3 = sns.scatterplot(structures['x'].values[0:N],structures['z'].values[0:N],hue=structures['atom_index'].values[0:N],palette=\"Blues_r\",linewidth=0,s=20,ax=ax[2])\nax[0].set_xlabel(\"X\", fontsize=18)\nax[1].set_xlabel(\"Z\", fontsize=18)\nax[2].set_xlabel(\"X\", fontsize=18)\nax[0].set_ylabel(\"Y\", fontsize=18)\nax[1].set_ylabel(\"Y\", fontsize=18)\nax[2].set_ylabel(\"Z\", fontsize=18)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Implementation with Plotly\n#N = 10000\n#trace2 = go.Scatter3d(\n#    x=structures['x'].values[0:N],\n#    y=structures['y'].values[0:N],\n#    z=structures['z'].values[0:N],\n#    mode='markers',\n#    marker=dict(\n#        size=5,\n#        color=structures['atom_index'],                # set color to an array/list of desired values\n#        colorscale='Jet',   # choose a colorscale\n#        opacity=0.2,\n#        showscale=True,\n#        line = dict(\n#            width = 2,\n#        )      \n#    )\n#)\n\n#data = [trace2]\n#layout = go.Layout(\n#    title = 'Molecule Structures',\n#    scene = dict(\n#        xaxis = dict(title=\"X\"),\n#        yaxis = dict(title=\"Y\"),\n#        zaxis = dict(title=\"Z\"),\n#    ),\n#    margin=dict(\n#        l=0,\n#        r=0,\n#        b=0,\n#        t=0\n#    ),\n    \n#)\n#fig = go.Figure(data=data, layout=layout)\n#py.iplot(fig, filename='Molecule_Structures')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**View it with respect to X-Z plane, to watch a donut shaped  pattern....(While Using Plotly)**"},{"metadata":{},"cell_type":"markdown","source":"## Scalar Coupling of Molecules"},{"metadata":{"trusted":true},"cell_type":"code","source":"f,ax = plt.subplots(figsize=(20,15))\nsns.heatmap(scalar[['fc','sd','pso','dso']].corr(), ax=ax,cmap=\"YlGnBu\")\nplt.title(\"Correlation Matrix\",fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f = plt.figure()\nsns.pairplot(scalar[['fc','sd','pso','dso']])\n#plt.title(\"Pair plot\",fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(20,15))\nsns.set_color_codes(\"bright\")\nsns.barplot(scalar['dso'],scalar['type'],label=\"Diamagnetic spin-orbit\", color=\"b\")\nsns.set_color_codes(\"pastel\")\nsns.barplot(scalar['pso'],scalar['type'],label=\"Paramagnetic spin-orbit\", color=\"b\")\nsns.set_color_codes(\"muted\")\nsns.barplot(scalar['sd'],scalar['type'],label=\"Spin-dipolar\", color=\"b\")\nplt.title(\"Bar plot\",fontsize=20)\nplt.legend(loc = 'lower right',fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Mulliken charge of the atom"},{"metadata":{"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(20,15))\n\nsns.jointplot(charge['atom_index'],charge['mulliken_charge'],kind=\"kde\", color=\"#4CB391\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## magnetic shielding tensors\n**3d-Surface Pot\n**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Implementation with Plotly\n#N = 10000\n#data = [\n#    go.Surface(\n#        z=magnetic.values[:N]\n#    )\n#]\n#layout = go.Layout(\n#    title='magnetic shielding tensors',\n#    autosize=False,\n#    scene=dict(camera=dict(eye=dict(x=1.87, y=0.88, z=-0.64))),\n#    margin=dict(\n#        l=0,\n#        r=0,\n#        b=0,\n#        t=0\n#    )\n#)\n#fig = go.Figure(data=data, layout=layout)\n#py.iplot(fig, filename='elevations-3d-surface')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training Data Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"k = train.type.astype('category').cat.codes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Relation between Scalar Coupling(Y-axis), Spin-dipolar, Paramagnetic spin-orbit, Diamagnetic spin-orbit"},{"metadata":{"trusted":true},"cell_type":"code","source":"N = 10000\nplt.figure(figsize=(15,15))\nplt.scatter(scalar['sd'][:N],train['scalar_coupling_constant'][:N], c='b',edgecolor='black', marker='o',alpha=0.5, label='Spin-dipolar')\nplt.scatter(scalar['pso'][:N],train['scalar_coupling_constant'][:N], c='red',edgecolor='w', marker='o',alpha=0.5, label='Paramagnetic spin-orbit')\nplt.scatter(scalar['dso'][:N],train['scalar_coupling_constant'][:N], c='y',edgecolor='b', marker='o',alpha=0.5, label='Diamagnetic spin-orbit')\nplt.legend(loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Implementation with Plotly\n#N = 10000\n#trace0 = go.Scatter(\n#    y = train['scalar_coupling_constant'][:N],\n#    x = scalar['sd'][:N],\n#    mode = 'markers',\n#    name = 'Spin-dipolar',\n#    marker = dict(\n#        opacity=0.2,\n        \n#        line = dict(\n#            width = 0.5,\n#        )\n#)\n#)\n#trace1 = go.Scatter(\n#    y = train['scalar_coupling_constant'][:N],\n#    x = scalar['pso'][:N],\n#    mode = 'markers',\n#    name = 'Paramagnetic spin-orbit ',\n#    marker = dict(\n#        opacity=0.2,\n        \n#        line = dict(\n#            width = 0.5,\n#        )\n#)\n#)\n#trace2 = go.Scatter(\n#    y = train['scalar_coupling_constant'][:N],\n#    x = scalar['dso'][:N],\n#    mode = 'markers',\n#    name = 'Diamagnetic spin-orbit',\n#    marker = dict(\n#        opacity=0.2,\n        \n#        line = dict(\n#            width = 0.5,\n#        )\n#)\n#)\n#data = [trace1, trace2, trace0]\n\n#py.iplot(data, filename='line-mode')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Implementation"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pre-Process Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#train = pd.merge(train, scalar, how = 'left',\n#                  left_on  = ['molecule_name', 'atom_index_0', 'atom_index_1', 'type'],\n#                  right_on = ['molecule_name', 'atom_index_0', 'atom_index_1', 'type'])\n#train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#test = pd.merge(test, scalar, how = 'left')\n                  #left_on  = ['molecule_name', 'atom_index_0', 'atom_index_1', 'type'],\n                  #right_on = ['molecule_name', 'atom_index_0', 'atom_index_1', 'type'])\n#test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Source of inspiration for the code - [Tod Newman](https://www.kaggle.com/todnewman/keras-neural-net-for-champs), [Andrew Lukyanenko](https://www.kaggle.com/artgor/molecular-properties-eda-and-models)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def map_atom_info(df, atom_idx):\n    df = pd.merge(df, structures, how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    \n    df = df.drop('atom_index', axis=1)\n    df = df.rename(columns={'atom': f'atom_{atom_idx}',\n                            'x': f'x_{atom_idx}',\n                            'y': f'y_{atom_idx}',\n                            'z': f'z_{atom_idx}'})\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_features(df):\n    df['dx']=df['x_1']-df['x_0']\n    df['dy']=df['y_1']-df['y_0']\n    df['dz']=df['z_1']-df['z_0']\n    df['distance']=(df['dx']**2+df['dy']**2+df['dz']**2)**(1/2)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = map_atom_info(train, 0)\ntrain = map_atom_info(train, 1)\n\ntest = map_atom_info(test, 0)\ntest = map_atom_info(test, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_=make_features(train)\ntest_=make_features(test) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_.drop(\"molecule_name\", axis=1, inplace=True)\ntest_.drop(\"molecule_name\", axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_id = test_['id']\ntrain_.drop(\"id\", axis=1, inplace=True)\ntest_.drop(\"id\", axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_type = pd.get_dummies(train_['type'])\ntest_type = pd.get_dummies(test_['type'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_new = pd.concat([train_, train_type], axis=1)\ntrain_new.drop(\"type\", axis=1, inplace=True)\ntest_new = pd.concat([test_, test_type], axis=1)\ntest_new.drop(\"type\", axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_new.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_new['atom_0'] = train_new['atom_0'].astype(\"category\").cat.codes\ntrain_new['atom_1'] = train_new['atom_1'].astype(\"category\").cat.codes\ntest_new['atom_0'] = test_new['atom_0'].astype(\"category\").cat.codes\ntest_new['atom_1'] = test_new['atom_1'].astype(\"category\").cat.codes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.metrics import explained_variance_score,mean_absolute_error,mean_squared_error,r2_score\nfrom sklearn.neighbors import KNeighborsRegressor\nfrom catboost import CatBoostRegressor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y = train_new[\"scalar_coupling_constant\"]\ntrain_new.drop(\"scalar_coupling_constant\", axis=1, inplace=True)\nX = train_new","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_test, y_train, y_test = train_test_split(X, y, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Lets see which regressor works better CatBoost or my all time favourite KNN regressor..."},{"metadata":{},"cell_type":"markdown","source":"### KNN Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_KNN = KNeighborsRegressor(n_neighbors=75, algorithm='auto', n_jobs=-1,p=2)\nmodel_KNN.fit(x_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = model_KNN.predict(x_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Variance_Score(KNN_Regressor)\\t:\"+str(explained_variance_score(y_test,y_pred)))\nprint(\"Mean_Absolute_Error(KNN_Regressor)\\t:\"+str(mean_absolute_error(y_test,y_pred)))\nprint(\"Mean_Squared_Error(KNN_Regressor)\\t:\"+str(mean_squared_error(y_test,y_pred)))\nprint(\"R2-Score(KNN_Regressor)\\t:\"+str(r2_score(y_test,y_pred)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CatBoost Model🐱"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model_cat = CatBoostRegressor(iterations=3000,depth= 13,random_seed = 23,\n                           task_type = \"GPU\")\nmodel_cat.fit(x_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred_cat = model_cat.predict(x_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Variance_Score(cat_Regressor)\\t:\"+str(explained_variance_score(y_test,y_pred_cat)))\nprint(\"Mean_Absolute_Error(cat_Regressor)\\t:\"+str(mean_absolute_error(y_test,y_pred_cat)))\nprint(\"Mean_Squared_Error(cat_Regressor)\\t:\"+str(mean_squared_error(y_test,y_pred_cat)))\nprint(\"R2-Score(cat_Regressor)\\t:\"+str(r2_score(y_test,y_pred_cat)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## SOooooo..................\n## CatBoost Won the match 🎊🎊..."},{"metadata":{},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred_test = model_cat.predict(test_new)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_submission = pd.DataFrame({'id': test_id, 'scalar_coupling_constant': y_pred_test})\nmy_submission.to_csv('SubmissionVictor2.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## I hope this Kernel was helpful to you (ﾉ^_^)ﾉ...\n## Please Upvote this Kernel if you like it....\n![](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBxIQEhUQEBMVEhAQFQ8VEhAQFQ8VFRAVFREWFhUVFxcYHSghGBolGxUVITEhJSorLi4uFx8zODMsNygtLisBCgoKDg0OGxAQGi0mICUtLS0tLS0tLS0vLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLf/AABEIAMYA/gMBEQACEQEDEQH/xAAcAAABBQEBAQAAAAAAAAAAAAAAAQIDBAUGBwj/xABDEAABAwIDBAgEAggDCQEAAAABAAIDBBEFITESQVFhBhMiMnGBkbEHocHRQlIUFSNicoKS8EPh8RYkM1NUc6LS4hf/xAAbAQEAAgMBAQAAAAAAAAAAAAAAAgMBBAUGB//EADURAAICAQMCAwUHBAIDAAAAAAABAgMRBBIxIUEFE1EiMnGBkRQjQmGhsdFSweHwFTMGYnL/2gAMAwEAAhEDEQA/APcUAIAQAgBACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBAJdYygKsgEAIAQAgBACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBACAEAICOWZrRdxAHNQnZGCzJ4Mxi5PCKE2LAdwX5nILQs8QS9xfU2Y6VvllOTEJHb7fwhaktZdLvgvjp4LsQOkcdXE+JKoc5vlssUIrhDFHr6khQSNCUTa7mMJkrKqRujj5m/urY32x4kyDqg+UWYsVeO8A7wyK2YeITXvLJVLSx7Mv0+IMflfZPB2XzW/Vq67OieGa06Zx7FpbJUKgBACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBAISsN4BmVeJ7o8/3jp5cVzr9djpX9Tar0zfWRmPeXG7iSeJXNk5SeZPJuRSisISybTItlnACyYAWTACyYAWTAEssYAWWHEFimrXx82/lP04LZp1U6/wA0U2URkbNNVNkHZ13g6hdam+NqzE0Z1uHJOriAIAQAgBACAEAIAQAgBACAEAIAQAgBACAa94aLk2A1KjKSisvgyk30RiVtaZMhkzhx8VxtRqXa8L3TfqpUOr5KoC10i8HEAXOQ4nRSSZgoy41TNNnVEIPAyR391LZL0MZRLTYhDLlHLG88GPY4+gKbWuUZyi0ogFkAmANkeGi7iGgb3EAepRJsFF2OUoNjUQ3/AO5H91nZL0MZRZp6yKT/AIcjH/wOa72Kw4tdjOUTWUcAVri03BsRvCwm4PdHkSipLDNmgrhJkcnjdx5hdjTalWrD5OfbS4P8i6tspBACAEAIAQAgBACAEBSnrwMmi/PctC7XRi8R6l8KG1lkH6wfwHoVrfb7PRFnkRLEFcDk4WPHctmnWxm8S6MqnS49UXFvlIIAQCEowYeIVfWGw7g+Z4ri6rUebLauDoU1bFl8lULXSLjjMb6aHaMNE3bdmOtsXC/7jR3vE5citmNPTMiDn2RiuwHEKvtS7Rv/AM59h5N0HkFPfCPBHbJkn+wlT+aPwufsnnxM+WypUdEKtmfVh1vyOBUvNizGxnd9E4qhlOG1JJftHZDjdzWWFgTvzv8AJa1jTl0LIp46mzdQJYC6ZGDzrpJgtbPUPJa6Rm0erN+w1m4AbjbVbUJwUSlptkEXQeqOuw3xdf6J58RsYsnQeqbm3YJHBxBTzoszsZJBi2IUJtM1z4x+GW7hbk/UeeXJHCE+Am1ydtgeNRVjNuPItsHxu7zCdPEcCtacHEmnk0QSDcZEaFVpuLyjLSawzcoKvrBn3hqPqu1pr1bHPfuc62pwZbWyVAgBANc4AXOijKSissJZ4KMuIflHmfsubZ4h19hfU2Y6f1IxiDt4HzVa8QsXKRJ6eJbpqsPy0PDj4Leo1ULenDKJ1OJZW0VlXEJLMy3my09ZY4V9O5bTHMjILlw8o38Fd0lzx1sL2GWSpc8smo4HiXL1+WoUt5jabWHVG02x1b8wu9or/Mhh8o591e2RcW8UggMzFqmw6sanvchwXO11+F5a+ZtaavL3MylzUbo2VocC06OBB8CLFSTwMFPD8LhpxaKNree8+ay5uXIUUi0+do1IHmqZ31w6Skiag3whjaph/EPVRjqam8KSM+XL0JdpXZI4E2kyMBtJkYC6DAXTIwI6UDUgeKhK2EPeeDKi3wMFWz8wVa1dL/EjPlS9CQ2cLGzmnjYgq+Mk+qZBoqUWFQwvdJEwMc8Wds6HO+im5t8mFHBeuoMEkExY4OG7UcRwUqrHVPcvmQshvjg6GJ4cA4aHRd6MlJZRzGsPA9SMAgMrFanPYG61/E6LjeIaj2ti7G5p6/xGa6TlfXfwXLczbUQilzt45cLarMJ9hKJO128aq6E3F5RU45WDdhddoPEAr0kHuimc5rDwQYhHtMNtRn6LW1te+p47dSymW2ZhOevOSkdNIru1VLfUsS6ADdE2GjTwiTt24g/ddXw2f3uPVGnqo+xk3F3znjZH7IJOgF1GclFNsyll4Obmk2iXHUlefnJzk5PudSEdqwRFyjkswN2ljJnA+jpXTnIlsYyLhq/kOA5rOm009Z1ziHr3l8PyKr7409Esy/b4m5TYZGwZNHja59Tmu3To9PSsQgjnTusm8ykSy0THCxaD4gFWTpqmsSin8iMZzj1Tf1Mauwkx9qHQaxm5B/h4FcjVeGulOen47x/g3adZue236/yUWSAi4WnCxTjuRuuOHhjrqWTGBLrORgWGN0rthmVu8/c3lzKjXCzUWOuvoly/T/JC2yNMd0uXwjbpMIjZnbadvc/Mn10Xbo8P09PEcv1fVnNs1NtnLwvRFt1I05EC3MBbLrrfRxX0RVmS4b+pmVuCjvRdh3Ad13iN3kuZqPC4+9p/Zfp2Zt1a2S6WdV+qMtjzmCLObk5p3Fc2uxyypLDXK9DfaXK4Y8OVuSOB4KwYNXB5tWHxH1C6egs6OD7cGjqYYe41F0TVEWAc1VP2nO5k+68pdPdJv8zr1xxFFUuK19xdgdFlmsxeCMkWYrkgDU5BbNSc5KK7lM+iydHG2wA4ABepisLBy289RSsmDncVpurdl3XZjlxC8z4hp3TPK4f+4OrprN8cPkzXPXMcjbURWvWVIOJpYNnIOQd7LqeGdb18GaWr/wCs6JekOYUcVks3Z4+wWjrp4go+psaeOZZMN5XIZ0EREqLZNEbwXWYNXkNvwvqfRU2J2ONUeZNL+f0JZUU5Pt1OspIQxoa0WAAA8F6uMI1xUI8I4W5ybk+Web9MPiC5jnNp3iKJhLTNYOdI4a7NwbDXdc6row08IR32mpK2cpbYFDo701rHsfUtk6+CF7GStl2Gm79LZB3mLj0Utmnt6Q6MOVtfWXB6jhtcyphZNH3ZBex1adC08wbjyWjKLhLazZTUllGFicPVym2kgLv5h3vofNeX1lSo1LS4ks/PudjS2eZV15XQr3VW4vGyPsCeAULbNkGzMVlnR4NS9XGAe8c3HiTr9vJej0WnVFEYd+X8WcW+x22OX0+BhdJukkrZm0VGGmodbae/Zswltw0Xy2rZ58tb5dWmiOzzLODTste7ZHk89pfiFUNlLP0kue1xaWSNYWOINiBlbUbiOSvS01j2rkrfnQ6s9W6L482ui27bMjDsyM1sbXBHI/fgtK6l1SwbFdinHJBj8Gy5sg3nZdzB0Pkfdee8Vq2WQuXf2X/Y6mgsynW/iigHLTybmCRpUkyLRapJNlwPA/6rYontmma9scxaOhC7xzQKwDk6g2cRzPuvG2PEmduHVIrl6oci5RHNepRkRaNvBab/ABD4N+pXofDNNhebL5HM1VmXtRsLrmmCAr11MJGFu/dyO5a+qoV1bj9CyqzZNM454Xi5Jnei8iMO7U8BmVGMu3cyzRw+WRjtoMGhHbNtfC66OjttplvUfqzUvjCaxn6GkzEZN+x4AO97rqLX298f7+ZqPTw7ZIquYvzOVgq7rna8tE6oKHQoPWq2bKIXKtsmhaU/tY7/AJj6lpss6dpaurPq/wBiF6+5lg6qZpLHBveLXAHmRkvVrpI4nY8BxTonPU0stW17GsoRJ1kchcHOLWhz7ZWBDdL6nLJbuueZJIo0vRM4/A4i6dhaO6bk8BY39dPNU6VSdiwW3tKDyfRPw1jcKO7tHSSFv8OQ9w5S1jTtIadYh1LHSI9uPj+09LNXlvGWvOqX5S/sdjw/3J/IzlzdxvYI5+6fL3VdkunX1X7mUv2OzgOS9q+p55HhHxNppOuqWkEnrdsjiw9pvyLT5Lo2JvTLaalbUbnk5Loj0ckxKo/RoXsjcGPkL5CbBrC0ZAZk3c358Fzl06m6+D2j4YUr45KkO0ZsRuIzaXtc69jv3+oW/rZJqJqaZPLOo6SH9nzLmW/qH+a834y0tOv/AKR1dD/3fJmLdcXedPA9hU1Mi0WYirosqkjUFeQ0ADMAXJXReuajhI0/s/XLIjiMg02DyO0Pnn7Kl+IXLhJ/VEvs8H6mPUlxJJYcyT2e0Mzfx+S4l2/c3KP06nQr24STKhN9FrN54Ly1QQbb2s4nPw1K3dFR51sYGvqLNkGzr2NAFhkBkAvYpJLCOG3kcsgEBDUVAYM9dwVF18al15Jwg5voc7JTtJJO8k2vzXmZ1Qcm2dSM5JJCtsMgABwCJqPRDq+RdtY3jaAepKYwPDlYpEcFMHNzfyn5EBw97eSTJw4GuWu5FiIn31HeBBHiNFVKTWJR5TyviieE8p8M6nDasSsDhv1HA7wvXafUR1FSsj3/AEfocKyt1TcJHO9JOhLalzpIX9U+TORhBMcp4kDQ+vgulTqtqxJZNWdGXmLwZOFfDXZd+3kYGA3LIAQXfzEC3orJa1JYgiK07b9pnfMjbEwMYA1jAA1o0aAMgtCT5lJmzjsjla6p62QvHdHZbzF8z5leO1mq+0XuxcLovh6/M7mnq8qtRfL6siutfcXCOFxbioy9pYYXR5N/AazaZsHvMsDzG4+i9X4ZqvPpSfvR6P8As/mcbV0+VZns+Ct0o6Kx1tn36uZosJAL3HBw3j2XYo1EqunY0bKVPr3OWo/htK1+0Zo2De+IO2yPlb1Ww9ZDHSJUtPLuzu8Kw2KkiEUQs1tySdXE6ucd5WlOcpyyzZjFRWEY2OVXWPDB3Yzd38W4eX1XlfFtUrbVXHiPPx/wdfQ1OEHOXL/Yo3XO3m20PaVKMyLRNTvu8j8jQfNxIHoGn1W5W/ZyUT5wWS9HMikN21DeSwG2sbxtGvY13eAPujjGXKCbXBNQhsbw/M2vlfitrRuFFm4ru3WR2m/FIHC40XooTU1lHNcWnhj1IwMleGgk7lCc1CLkzMVl4MKaUuJJ1K89da5yyzpQgorCK7nLUlItSIy5UuRNIbtLG8zgcHLKkYwO6wDerFNEXEqxXLpHEEAuAF94a0C/rdW2S6IxBDnFa0pFqGFUuRJDqeodG7aYczq06O+x5q3TayennurfPKfD/h/mQtphasS+vobEOPM/GHMPMEjyIXfq8a08l94nF/X9Uc6WhtXu4ZI/HotxLjwa131AVk/GNJFdG38EzC0V75WPmZNfiL5ez3Gb237TvE7hyXE13ic9Qtq6R9O7+P8ABvUaSNTznLKl1zNxtBdY3ALpuArHlpDmnZcND9CN4VtN86pqcHh/7yQnCM1tkso2KbHbC0jSP3m3c37heio8bra+9jh/l1RzZ6CafsPPx6MsnHotzr8g19/ZbMvF9IvxZ+T/AIK1or/6f1Rn1uMPfkwFg/M7veQ3eK5er8ZlYtlS2r1fP+Dap0MYvdN5f6GY3LRcPcb4t03Ac0qUZGGOpyWyOuOy5jDfcC0uBF/5gt+ufsGvJe0Wdu+hv4LDmmZ2jS5VuRnAbSbjOBzXKakRaJGuV0ZEGi7RVGy7kdfuulo73CWHwzWuryjaC7ZolPFHWZ4kfdaWuliov06zMxXlcCbN9IhcVryZaiJ7rKmUiSRGXk6DzOXyVTm3wT2ruAjJ7xJ5DIfJZUW+WG8cImjaBoAFfBJFcm3ySOFxz3LYxlFfBU275/2OIWjN4eC5deo0lVORNIS6rcjIArG9oYAu5o7G+42oFjJkRMgVMgEyATIC6ypYGEG0eKz5j9TGEIo7jOBbrGQImQOjzIHH5Aan++S2NPDfIrnLCLxy0XQeEUogkjB1GfHQrWnFMsTZEWuGhvydn81V7S4ZPKfIok4i3PUeqKfqHH0JmlXRZBolYVfFkGTNK2YPBVJHQU7rtaeIHsvTweYpnLksNlXFh2L8CPqtPXr7r5l2nftmK5cCR0EQuWtIsRGVS0TQ1QJChSRFkjVfEiyQFXRIMzWyXfIBoH5f0NJ+ZK0NXJKZdXwOutPcWYGySBou4gDiUWX0RkpjF4L7PWNLjo0XJPkrvs1z/Cw2kTfpsf5gPG491CVNkeYswmnwSsladHA+BBVeGjOB6xkAm4AmQITbVZ6gidVMGr2+oUlGT4QIZsThYLueGg5AnaAJ8bK1aa5/hZhND6evik7j2u8Cq51WQ95MyWLqrJkEyYHUj/2rgdzGkebnbXs1dXRY2M17feLxKvkRRGVTIkhhVLJiBYA9qsRhkrVfEgyVpWzAqZ0VMLMaOTfZeorWIpHKlyxKuPaY4b7ZeIzChfDfW4kq3tkmc4SvMSOoiNyokixEZCqaJISyhgyGilxyORolv3RfnoPVZU8+6s/sNnqOAO8+QyCuhF92RbXYqy2DrDIPuf5ha/qPYrT11ePbROt9hLrnZLjzv4m4lIHsha4taQS6xIvbZy8M/kF6jwLTwcXY11ITeDjsEr/0WohqAL9TJG8tFu01rhtDzFwvRroa9kd8XE+oaNtPUxtmi2XxStDmPbYhwKswmcZ5i8Mjm6P07tY2/wBIVUtPVLmK+hNWzXDZWd0Vg3C3gXD2Kol4dpZcwRYtVcvxDP8AZSL8z/65Puq/+K0n9BL7Zd6ijopDvLj4vefqpLwvSr8CMPV3epNH0Ypx+AHxF/dXR0WnjxBfQg9Ra+ZMuRYTC3RjR4ABXxrhHhIqcpPlnj/xwxeJ8kNFDYmAukmLbdlzm7LGHnYuJHMLEn2N/RwaTkzzSkqnwuEkZLXNzy0PI8QqLaY2xcZI3k2j3PDpi+JjjqQvnl0ds3FF5ZCgjBJSkG7uNwDyB+pv8l29PXtrw+5RJ5ZOdoaG/I/dJKS46mE4vkb1w0PZPA/dVOa4fT4ktr7DiFjAyFkSA4BTSMMkaroogyenZtODeJH+a3NNDfYolNstsWzpAvTHLFQHOYlDsPI3HMee5ed1lPl2P0fB0qJ7oFQlaLRsCFVtEhjr7vU/ZQcX2MrAzqxqe0eenosKpcvqZ3vsPLlckQIpJFLJnBQnmu9gGu0T5Bjr+61dS81smlhllchlx558UqQ3jmGmbSfEf/I9V6f/AMftWJQ+ZCw4FekKjZ6PdKaygP8AuszmNJu6J3ajceJYcgeYsVlPBXOqE/eR29F8aaloAmpopOLmOey/kQ5S3Gu9HHszTj+NrPxUj/5ZGH3sm4h9i/Ml/wD2yD/pJf6ovum4fYn/AFDXfGyLdSSeb403D7F/7FOp+Nj/APCox4ySn2DU3klol3ZzON/FDEakFjXtp2HUU4Idbhtk3HiLLDky6Gmrj1wcWTfM5k3JJzJJ1JUS8kpYDI9sY1e5rfU2ULZqEHJ9jKXU92oo9mNreAH3XzqyW6bZeThILqjDK2Gz9ho4AA+IyK7kZFGOhoskUsmMDzY65quUU+jCeBgZbum3I5hVeXjglu9SRvNSSIjgrEjDHXVkURZqYJDcmQ6DIeO/++a7PhtPR2M0tVP8JsrrGmCAp4lSdayw7wzafotXV6fzYdOVwW02bJfkcwX2NjkRkQdxXm5Jp4Z1VhrKE6xQJYEMiwBrpUM4IXzrDZlRKU9SdwJztkCc+HisJSl7pJuMeSSjpHA9ZJ3iLBv5B9ytHUy6bUIvLyyyVoMtMrpLhgqoHxnW12ngRmD6gHyW5oNS6LlP6hrJ4tNE5jix4s5pII4EL30ZqSUl3NdrA1jC42Aus5wZjFyeEXI8PO8+QUd6NqOjl3JP0Ac1jeW/ZCJ9FzIWdxVLTdSVtAOaxvLFpOgjsP4EhN6MPR+hWmpXN5jiFJSTNadE4ECkVHX/AA7wYyy9e4diO4bfe7f8svM8FwfG9Xsr8qPLLILuepLyBaOYFbBdSLKc9E9ji+MFzXG5aNWk6m3BdiiErFhc/ua7sUOeB7ZS07LwWuGrXAghSeYvbJYZYkpLKLLJllMi0SCRCOB4kQYDrFJDBLSxmVwY3U6ngOJWxp6ZXT2oqsmoRyzq6eIMaGjQD+yvSwgoRUV2OTKTk8skUzAIAQGXi2ECbtNOzIN+53I/daGr0Ku9qPSX7mzRqHX0fBytVDLEbPY4cwCWnzGS8/bTbW8SizpwsrmujKwnJ0BPgCVV7T4T+hbiPqWoMPqJe7E4Di/sD/yV9ek1FnEfr0KpX1Q5Zr0XRbfO/a/cZcDzdqfKy6dHhC5tefyRqWa98QRutoowzq2tDW7gAMjx8ea6b09fl+WlhGj5kt25vqc3iFOWOLT5HiOK8drtNKqbizr0W71lFBy5Elg2kIokjhOnfRcyXqYB2x32D8Q+/wDpwXo/CPElH7mzjsyucc9TjcPAtz38Qd916OZvaVRwXVWb4IBjmqSKpLqPCiWJdAQyNesorsSx1K+F4K+rm6uIdnLbeNG8QOar1eshpq90ueyONOKc+nB69heHsp42xMFg0DzXhr75XWOciaLipQJIwtqmGWVyZ0eDUdh1jt/dH1XsfDNJ5cd8uWcnU3bntRbrsOjnFpGg8HaOb4Fb1+lquWJoprunW/ZZztZ0YkbnC8PH5X5O9dD8lxrvCLI9a3n8mb9eui/fWDLmp54+/E8cw0keouFz503V+9Bm1GyuXEkQioPA+hUMy9H9CWEaFDQTTHJpa387wQPK+Z8lt0aO658YXqyi2+uvvlnWYfQNhbZuZPecdXH7cl6KjTwpjticq22Vjyy2rysEAIAQAgEsgCyxgBZZAqAEBVxCiErbHIjuu4H7LU1mkjqIbXz2ZbTa63lHG1sTonFjxYj0I4jiF4fVaadM9s11O3VZGaymQiRaeC0XaBWMYBynSLoe2YmWnPVynUfhf4j66+K7mh8XlUtlnVBZXVM4mshmpzaeNzbfjAJYfPd5r0dN9NyzCWS+Oqa5I2VQOhBV2w2I6lPuSdcsbSfnIQzJsMPUIiNWCbNu5x0a0FxPkEcVFZfQplqkbeEdF6ipIMt4Yt/53f8Ar7rl6vxaqlNV9Wa87Zz5PQcLw2KmYI4mhoHqV5W++y+W6bIcFsvVOAN6xWRgYZvYHhhfaR4sz8LT+Pn4e69P4X4a3iyxdOy9fic3Vale7FnSAL0pzRUAIBEAWWMAFkCoAQAgBACAEAIAQAgBAJdAMc9AUcSp2TN2XjwcNWniCtbVaSvUx2zXzLarZVvMTjMSopIDc9qPc8afzDcvI6zwy3TvLWV6nXp1MLOnf0Kbalc5wNklFQo7DAPcx2T2hw5qdbcGYZk1fRahlzLA0nhdvzaujV4hfDorPqR+RSPQSl3PI5bb/qVs/wDK6j1iYyvRkkXQiibm7teLpD8r2UJ+Kah/jXyQ+RrUeH00AtFE0c7AfIarn3amdnvSb+JJZLRqFq7MksjHVKyoAjE5cQ1oJcdAMyVbXRKyW2KyzEpKKyzpsGwYNs+exdqI9QP4uJ5aeK9PoPB414nd1fZdl8Tl6jWbvZhwdM2Vd00CQOQDkAIAQAgBACAEAIAQAgBACAEAIBpKAie9AVpJEBTmmQyUJ6hYaT6McHPV9Cw5s7B4Duny3eS5eo8Iqs6w6P8AQ3KtZOPSXVGPKZGai44tz/zXFu8Lur7Z+BvQ1Vc+/wBSIYgOK03Q0Xbg/WA4rHkmNwfrDmnkmNwfrDmnkjcH6wHFPJM7gbWF2TQT4K2vR2T91ZMSthHll2lpHP77tkcBmfsF1aPBXzY8fA1LNcvwI6LDmsiFmC19TqT4ldujTV0rEEaFls5+8zXhnV5WXYpUBbjkQE7XIYHhAKgBACAEAIAQAgBACAEAIAQDHIZIJAhgqTBAUZwhkzqhpQGZUMKAzKhhQGVVQE7lXOqE/eSJKclwzMmpnblQ9DQ/wlq1Ni7lV0EnFQ/4+n0M/apkkdO/eVleH0rsPtNnqXqemPD1VsdLTHiKISusfc1aaMq9LBU3k1KdhWQadO0oDSgaUBoQhAXIggLTAhglagFQAgBACAEAIAQAgBACAEAIBCEBG9iAgkiQFWWBDJTmpUBSlokBSlw/kgKcuGckBWfhPJARnCOSAVuEckBPHhfJAW4sO5IC5DQoC7DSIC7FTIC3FCgLLI0MEwagHIAQAgBACAEAIAQAgBACAEAIAQAgEsgGOjQEboEBC6lQETqNAROoeSAYcPHBAN/V3JAAw8cEA8UHJASNouSGSVtIgJm0yAlbChgkDEAoCAVACAEAIAQAgBACAEAIAQAgBACAEAIAQAgBACASyANlAJshAGwEAbAQBsoBbIAsgFQAgBACAEAIAQAgBACAEAID/9k=)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}