{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# import data\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")\nstructures = pd.read_csv(\"../input/structures.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's annoying to print the head of a Dataframe in stdout because it will often truncate the columns, so let's write a function to force it to print all columns."},{"metadata":{"trusted":true},"cell_type":"code","source":"def PrintDataframe(df):\n    print(df.to_string())\n\nPrintDataframe(train.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's find the number of molecule types, and how many times they appear. Interestingly, the value counts appear normally distributed. In retrospect, this information may not be so useful, since the molecule name will not be used a feature. According to the competition data description, \"...no molecule in the training data is found in the test data.\""},{"metadata":{"trusted":true},"cell_type":"code","source":"# analyze molecule types\nmol_name_col = train.molecule_name\nprint(\"number of unique molecules =\",len(mol_name_col.unique()))\nplt.figure(figsize=(3,3))\nplt.ylabel('counts')\nplt.xlabel('number of times a molecule appears')\nplt.title('Train')\nsns.distplot(mol_name_col.value_counts().values,kde=False,bins=range(120))\n# cleanup\ndel mol_name_col","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# analyze molecule types\nmol_name_col = test.molecule_name\nprint(\"number of unique molecules =\",len(mol_name_col.unique()))\nplt.figure(figsize=(3,3))\nplt.ylabel('counts')\nplt.xlabel('number of times a molecule appears')\nplt.title('Test')\nsns.distplot(mol_name_col.value_counts().values,kde=False,bins=range(120))\n# cleanup\ndel mol_name_col","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next consider the coupling types. Notice that they are very unbalanced, but train and test are unbalanced in the same way."},{"metadata":{"trusted":true},"cell_type":"code","source":"# look at couping types\ndef VisualizeCounts(*cols):\n    \"\"\"\n    takes a dataframe column (i.e. a Series)\n    prints the counts and graphs \n    \"\"\"\n    fig, axs = plt.subplots(ncols=len(cols), figsize=(5*len(cols), 4))\n    titles = ['train','test']\n    for i,col in enumerate(cols):\n        print(col.value_counts(),'\\n')\n        sns.countplot(col,ax = axs[i], order = sorted(col.unique()))\n        axs[i].title.set_text(titles[i])\n\nVisualizeCounts(train.type,test.type)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we consider the scalar coupling values, we see that they are basically clustered around 0, with a peak at around 80."},{"metadata":{"trusted":true},"cell_type":"code","source":"# looking at scalar_coupling values\nsns.distplot(train.scalar_coupling_constant)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see how well the coupling type corresponds to the coupling constant.\n\nNote that the coupling type can be easily interpreted. The number prefix tells you how many bonds there are between the two atoms; the last two letters are the two elements involved. For example, 1JHC, means that the coupling is between one hydrogen and one carbon atom, and there is only one bond between them. \n\nSince the coupling strength is inversely proportional to the number of bonds, 1JHN and 1JHC have the strongest coupling. \n\nAll of the coupling constants are positive except for 2JHH, which is negative. \n\nNote that since the maximum number of bonds is three, there are at most two atoms in between the atoms of interest. "},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.FacetGrid(train, col=\"type\", col_order = sorted(train.type.unique()),sharex=False,sharey=False)\ng.map(sns.distplot, \"scalar_coupling_constant\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can extract the number of bonds and the atom types just using the coupling type. Note that the first atom in the coupling type is always H. You can confirm by yourself (using some code further down that extracts the atom type from the structures data) that atom_0 is always H. Therefore, we need only extract atom_type_1."},{"metadata":{"trusted":true},"cell_type":"code","source":"def DecomposeType(df):\n    df['num_bonds'] = df.type.map(lambda s: int(s[0]))\n    df['atom_1_type'] = df.type.map(lambda s: s[-1])\n    df.drop(columns='type',inplace=True)\n\n\nDecomposeType(train)\nDecomposeType(test)\n\nPrintDataframe(train.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see what we can get out of structures"},{"metadata":{"trusted":true},"cell_type":"code","source":"PrintDataframe(structures.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The most obvious feature to extract is the distance between atom 0 and atom 1. I copied the method in this [kernel](https://www.kaggle.com/seriousran/just-speed-up-calculate-distance-from-benchmark) by Chanran Kim"},{"metadata":{"trusted":true},"cell_type":"code","source":"def map_atom_info(df, atom_idx):\n    df = pd.merge(df, structures, how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    \n    df = df.drop(['atom_index','atom'], axis=1)\n    df = df.rename(columns={'x': f'x_{atom_idx}',\n                            'y': f'y_{atom_idx}',\n                            'z': f'z_{atom_idx}'})\n    return df\n\ntrain = map_atom_info(train, 0)\ntrain = map_atom_info(train, 1)\n\ntest = map_atom_info(test, 0)\ntest = map_atom_info(test, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_r0 = train[['x_0', 'y_0', 'z_0']].values\ntrain_r1 = train[['x_1', 'y_1', 'z_1']].values\ntest_r0 = test[['x_0', 'y_0', 'z_0']].values\ntest_r1 = test[['x_1', 'y_1', 'z_1']].values\n\n# calculate through space distance\ntrain['space_dr'] = np.linalg.norm(train_r0 - train_r1, axis=1)\ntest['space_dr'] = np.linalg.norm(test_r0 - test_r1, axis=1)\n\n# drop coordinates\ntrain.drop(columns = ['x_0', 'y_0', 'z_0','x_1', 'y_1', 'z_1'], inplace = True)\ntest.drop(columns = ['x_0', 'y_0', 'z_0','x_1', 'y_1', 'z_1'], inplace = True)\n\nPrintDataframe(train.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It might also be useful to have an idea of what other atoms are in the molecule. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Group by molecule_name and atom, then use size() to count how many of each atom\n# Then use unstack to make the atom types, which groupby made into indices, into columns\n# Finally, because not all molecule contain all five atoms, fill NAs with 0\nstructure_atoms=structures.groupby(['molecule_name','atom'],sort=False).size().unstack('atom').fillna(0)\n# calculate the total number of atoms\nstructure_atoms['total_atoms']=structure_atoms.sum(axis=1)\n# remove index names (optional)\nstructure_atoms.columns = structure_atoms.columns.set_names(None)\nstructure_atoms.index=structure_atoms.index.set_names(None)\n# rename columns\nstructure_atoms = structure_atoms.rename(columns={'C':'num_C','H':'num_H','N':'num_N',\n                                                  'O':'num_O','F':'num_F'})\nplt.figure(figsize=(3,3))\n#plt.hist(structure_atoms.total_atoms,)\nsns.distplot(structure_atoms.total_atoms,kde=False,bins=range(5,30))\nplt.ylabel('counts')\nplt.xlabel('number of atoms')\n\n# extend train and test\ntrain=pd.merge(train, structure_atoms, how = 'left', \n                     left_on  = ['molecule_name'], right_index = True)\n\ntest=pd.merge(test, structure_atoms, how = 'left', \n                     left_on  = ['molecule_name'], right_index = True)\n\nPrintDataframe(train.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"structure_atoms.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can then use this information to confirm that the number of times a molecule appears in either the training set or test set is an indication of the molecule size."},{"metadata":{"trusted":true},"cell_type":"code","source":"# analyze molecule types\nmol_name_col = train.molecule_name\nmol_name_counts=mol_name_col.value_counts()\nmerged=pd.merge(mol_name_counts, structure_atoms, how = 'left', left_index = True, right_index = True)\nplt.figure(figsize=(3,3))\nsns.scatterplot(x=\"total_atoms\",y=\"molecule_name\", data = merged)\nplt.ylabel('number appearances in train')\nplt.xlabel('number of atoms')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# analyze molecule types\nmol_name_col = test.molecule_name\nmol_name_counts=mol_name_col.value_counts()\nmerged=pd.merge(mol_name_counts, structure_atoms, how = 'left', left_index = True, right_index = True)\nplt.figure(figsize=(3,3))\nsns.scatterplot(x=\"total_atoms\",y=\"molecule_name\", data = merged)\nplt.ylabel('number appearances in test')\nplt.xlabel('number of atoms')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}