{"cells":[{"metadata":{},"cell_type":"markdown","source":"# The story\n\nIn [this forum](https://www.kaggle.com/c/champs-scalar-coupling/discussion/104241#latest-605432) Scirpus suggested that 1JHC is not only one class and can be further divided. Lars Bratholm (organizer of the competition) said, that it is true, because carbon atoms can have different hybridization (i.e. they have different number of neighbours to which they are connected with bonds). Giba (Kaggle grandmaster) said that samples can be easily splitted by distance between hydrogen and carbon, setting the threshold to 1.065.\n\nI haven't seen a kernel or a graph showing this, so I made one. I try to split samples by distance between hydrogen and carbon and then to split them by number of neighbours of carbon. I find that splits are the same, except one sample, which is just between two groups and according to different criteria belongs to different groups."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Imports\n\n# Basic imports\nimport numpy as np\nimport pandas as pd\n\n# Graphs\n# %matplotlib widget\n%matplotlib notebook\nimport matplotlib\nimport matplotlib.pyplot as plt\nfrom matplotlib.lines import Line2D\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Seaborn advanced settings\n\nsns.set(style='ticks',          # 'ticks', 'darkgrid'\n        palette='colorblind',   # 'colorblind', 'pastel', 'muted', 'bright'\n        #palette=sns.color_palette('Accent'),   # 'Set1', 'Set2', 'Dark2', 'Accent'\n        rc = {\n           'figure.autolayout': True,\n           'figure.figsize': (14, 8),\n           'legend.frameon': True,\n           'patch.linewidth': 2.0,\n           'lines.markersize': 6,\n           'lines.linewidth': 2.0,\n           'font.size': 20,\n           'legend.fontsize': 20,\n           'axes.labelsize': 16,\n           'axes.titlesize': 22,\n           'axes.grid': True,\n           'grid.color': '0.9',\n           'grid.linestyle': '-',\n           'grid.linewidth': 1.0,\n           'xtick.labelsize': 20,\n           'ytick.labelsize': 20,\n           'xtick.major.size': 8,\n           'ytick.major.size': 8,\n           'xtick.major.pad': 10.0,\n           'ytick.major.pad': 10.0,\n           }\n       )\n\nplt.rcParams['image.cmap'] = 'viridis'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/champsbasic1jhc/1JHC-Train.csv\")\ndf = df[[\"molecule_name\", \"atom_index_0\", \"atom_index_1\", \"dist\", \"N_neighbours_Jatom\", \"scalar_coupling_constant\"]]\ntest = pd.read_csv(\"../input/champsbasic1jhc/1JHC-Test.csv\")\ntest = test[[\"id\", \"molecule_name\", \"atom_index_0\", \"atom_index_1\", \"dist\", \"N_neighbours_Jatom\"]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Histograms\n\nLets first plot histograms of scalar coupling constant, distance and number of neighbours of carbon atom."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(\"Histogram - scalar coupling constant\")\nsns.distplot(df[\"scalar_coupling_constant\"], bins=100, kde=False)\nplt.xlabel(\"Scalar coupling constant\")\nplt.title(\"1JHC\")\nplt.savefig(\"HistogramSCC.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(\"Histogram - Distance\")\nsns.distplot(df[\"dist\"], bins=200, kde=False, rug=True)\nplt.xlabel(\"Distance\")\nplt.title(\"1JHC\")\nplt.xlim(1.06, 1.145)\nplt.savefig(\"HistogramDistances-Train.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(\"Histogram - Neighbours\")\nsns.distplot(df[\"N_neighbours_Jatom\"], bins=4, kde=False)\nplt.xlabel(\"Number of neighbours\")\nplt.title(\"1JHC\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All three graphs show a small group of very different samples:\n\n- Scalar coupling constant > 180\n- Distance < 1.07\n- Number of neighbours == 2\n\nAre they the same samples?"},{"metadata":{},"cell_type":"markdown","source":"# Divide by distance or number of neighbours?"},{"metadata":{},"cell_type":"markdown","source":"## Divide by distance"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"distance_threshold = 1.065\nplt.figure(\"Scatter by distance\")\nplt.plot(df[df[\"dist\"]>distance_threshold][\"dist\"], df[df[\"dist\"]>distance_threshold][\"scalar_coupling_constant\"], \"go\", label=f\"Distance > {distance_threshold}\")\nplt.plot(df[df[\"dist\"]<distance_threshold][\"dist\"], df[df[\"dist\"]<distance_threshold][\"scalar_coupling_constant\"], \"bo\", label=f\"Distance < {distance_threshold}\")\nplt.axvline(x=distance_threshold, linestyle=\"--\", color=\"r\")\nplt.xlabel(\"Distance\")\nplt.ylabel(\"Scalar coupling constant\")\nplt.title(\"1JHC\")\nplt.xlim(1.06, 1.145)\nplt.ylim(64, 208)\nplt.legend()\nplt.savefig(\"ScatterByDistance.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The threshold 1.065 looks good, there is a gap big enough between the rightmost and leftmost samples from both groups."},{"metadata":{},"cell_type":"markdown","source":"## Divide by neighbours"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(\"Violin by neighbours\")\nsns.violinplot(x=\"N_neighbours_Jatom\", y=\"scalar_coupling_constant\", data=df)\nplt.title(\"1JHC\")\nplt.xlabel(\"Number of neighbours\")\nplt.ylabel(\"Scalar coupling constant\")\nplt.savefig(\"ViolinsByNeighbours.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When samples are divided by number of neighbours of a carbon atom, we see very different distributions. Note however, that the case with 5 neighbours is very rare and there are only a few samples."},{"metadata":{},"cell_type":"markdown","source":"## Comparison"},{"metadata":{},"cell_type":"markdown","source":"Lets check whether splits are the same:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"first_condition = df[df[\"dist\"]<distance_threshold].shape == df[df[\"N_neighbours_Jatom\"]==2].shape\nif first_condition:\n    second_condition = np.allclose(df[df[\"dist\"]<distance_threshold][\"scalar_coupling_constant\"], df[df[\"N_neighbours_Jatom\"]==2][\"scalar_coupling_constant\"])\nprint(f\"Comparisons by distance and by number of neighbours are {'' if first_condition else 'not '}same.\")\nif not first_condition:\n    print(f\"First comparison by distance contains {len(df[df['dist']<distance_threshold])} atoms, comparison by neighbours contains {len(df[df['N_neighbours_Jatom']==2])} atoms.\")\nelse:\n    print(f\"All pairs are {'' if second_condition else 'not '}identical\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(\"Scatter with differences\")\nplt.plot(df[(df[\"N_neighbours_Jatom\"]>2) & (df[\"dist\"]>distance_threshold)][\"dist\"], df[(df[\"N_neighbours_Jatom\"]>2) & (df[\"dist\"]>distance_threshold)][\"scalar_coupling_constant\"],\n         \"bo\", label=f\"3+ neighbours, distance > {distance_threshold}\")\nplt.plot(df[(df[\"N_neighbours_Jatom\"]==2) & (df[\"dist\"]<distance_threshold)][\"dist\"], df[(df[\"N_neighbours_Jatom\"]==2) & (df[\"dist\"]<distance_threshold)][\"scalar_coupling_constant\"],\n         \"go\", label=f\"2 neighbours, distance < {distance_threshold}\")\nplt.plot(df[(df[\"N_neighbours_Jatom\"]==2) & (df[\"dist\"]>distance_threshold)][\"dist\"], df[(df[\"N_neighbours_Jatom\"]==2) & (df[\"dist\"]>distance_threshold)][\"scalar_coupling_constant\"],\n         \"ro\", markersize=12, alpha=0.8, label=\"Difference in comparisons\")\nplt.axvline(x=distance_threshold, linestyle=\"--\", color=\"red\")\nplt.xlabel(\"Distance\")\nplt.ylabel(\"Scalar coupling constant\")\nplt.title(\"1JHC\")\nplt.xlim(1.06, 1.145)\nplt.ylim(64, 208)\nplt.legend()\nplt.savefig(\"ScatterWithDifferences.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Splits are the same except one sample, but its scalar coupling constant is closer to the group with higher distance, so it is probably indeed better to divide by distance. In test samples there is no such sample with distance right in between two groups (see histogram below), so it should be safe. The question is if this helps you somehow, because from total of cca. 709K samples the smaller group contains only cca. 9K samples and it can be too few for your training, however, this is up to you.\n\nI save groups divided by distance as output."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(\"Histogram - Distance\")\nsns.distplot(test[\"dist\"], bins=200, kde=False, rug=True)\nplt.xlabel(\"Distance\")\nplt.title(\"1JHC\")\nplt.xlim(1.06, 1.145)\nplt.savefig(\"HistogramDistances-Test.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"df[df[\"dist\"]>distance_threshold].to_csv(\"1JHC-HighDistance-Train.csv\", index=False)\ndf[df[\"dist\"]<distance_threshold].to_csv(\"1JHC-LowDistance-Train.csv\", index=False)\ntest[test[\"dist\"]>distance_threshold].to_csv(\"1JHC-HighDistance-Test.csv\", index=False)\ntest[test[\"dist\"]<distance_threshold].to_csv(\"1JHC-LowDistance-Test.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bonus - Number of neighbours\n\nAs a bonus I made a graph of scalar coupling constant dependent on distance, but samples are divided by number of neighbours of carbon atom."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(\"Scatter by neighbours\")\nplt.plot(df[df[\"N_neighbours_Jatom\"]==2][\"dist\"], df[df[\"N_neighbours_Jatom\"]==2][\"scalar_coupling_constant\"], \"o\", label=f\"2 nbs\", alpha=0.1)\nplt.plot(df[df[\"N_neighbours_Jatom\"]==3][\"dist\"], df[df[\"N_neighbours_Jatom\"]==3][\"scalar_coupling_constant\"], \"o\", label=f\"3 nbs\", alpha=0.025)\nplt.plot(df[df[\"N_neighbours_Jatom\"]==4][\"dist\"], df[df[\"N_neighbours_Jatom\"]==4][\"scalar_coupling_constant\"], \"o\", label=f\"4 nbs\", alpha=0.025)\nplt.plot(df[df[\"N_neighbours_Jatom\"]==5][\"dist\"], df[df[\"N_neighbours_Jatom\"]==5][\"scalar_coupling_constant\"], \"o\", label=f\"5 nbs\", alpha=0.7)\nlegend_elements = [Line2D([0], [0], marker='o', markersize=12, color=sns.color_palette()[0], linestyle='none', label='2 nbs'),\n                   Line2D([0], [0], marker='o', markersize=12, color=sns.color_palette()[1], linestyle='none', label='3 nbs'),\n                   Line2D([0], [0], marker='o', markersize=12, color=sns.color_palette()[2], linestyle='none', label='4 nbs'),\n                   Line2D([0], [0], marker='o', markersize=12, color=sns.color_palette()[3], linestyle='none', label='5 nbs')]\nplt.xlabel(\"Distance\")\nplt.ylabel(\"Scalar coupling constant\")\nplt.title(\"1JHC\")\nplt.xlim(1.06, 1.13)\nplt.ylim(64, 208)\nplt.legend(handles=legend_elements)\nplt.savefig(\"ScatterByNeighbours.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"P. S. There is one sample with very high distance between hydrogen and carbon (over 1.25), but I excluded it from graphs, so they are more readable."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}