{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Predicting Molecular Properties EDA\n\nUpon first inspection of train & test it appears to be a reletively clean data set with no missing values and easy to merge with the stuctures file. The dataset contains data on a number of molecules each consisting of multiple atoms. Each row is an observation of properties between pairs of atoms within the molecule and therefore each individual molecule consists of observations over multiple rows, in order to include each atom pair combination within the molecule. Additional properties on the atoms and molecules are contained in the structures file and Additional Data and this is easily merged with train and test (where applicable - see below). It should be remembered that observations for a whole molecule are grouped across multiple rows. Therefore, if splitting the train set into a train and cross validation (CV) set, the user should not randomly select rows from train to split into a CV, but should randomly select whole *molecules* to split into CV and take *all observations* from those molecules into CV.\n\nWhere this competition data differs to the usual is the Additional Data files are for the train set only, and not provided for the submission data (test). This may be challenging in that it may be necessary to derive, or predict, some of these additional variables for the test data set, with the objective of then using them to predict the target variable in test - like a two stage prediction. This would likely introduce more noise into the test set, so if used this approach should be pursued with caution.\n\n\n# Target Variable: scalar_coupling_constant\n\nStarting with the target variable, load the data sets and analyse the distribution of this variable.\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"library(dplyr)\nlibrary(stringr)\nlibrary(ggplot2)\n\ntrain.df <- read.csv(\"../input/train.csv\", stringsAsFactors = FALSE)\n\nhead(train.df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(train.df, aes(x = scalar_coupling_constant)) + \n        geom_area(stat = \"bin\", \n                     binwidth = 0.5, \n                     colour = \"royalblue\",\n                     fill = \"skyblue\",\n                     linetype = \"solid\") +\n        ggtitle(\"Distribution of Target Variable in Train\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is not a normal gaussian distribution. There appears to be discrete factors affecting this variable. Investigate one of those discrete variables, the 'type' variable to see if that is affecting the scalar coupling constant. Type consists of a single digit, 1, 2 or 3, representing the number of bonds between the two atoms in the observation, followed by three letters: J (which represents \"Join\"or similar) then the chemical symbol for the two atoms in the observation, H for Hydrogen, C for Carbon, N for Nitrogen. For example, 1JHC is a type value: 1 bond between a Hydrogen atom and Carbon atom within a molecule.\n\nPlot the Target variable distribution again, this time distinguished by the Type Variable."},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(train.df, aes(x = scalar_coupling_constant, fill = type)) +\n        geom_area(stat = \"bin\", \n                     binwidth = 0.5, \n                     linetype = \"solid\",\n                     alpha = 0.5) + \n        ggtitle(\"Distribution of Target Variable in Train, by Type\") +\n        scale_fill_brewer(palette = \"Dark2\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Type clearly has an impact on the target variable. Starting with one value of Type, 1JHC, examine this distribution closer."},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(filter(train.df, type == \"1JHC\"), aes(x = scalar_coupling_constant)) +\n        geom_area(stat = \"bin\", \n                     binwidth = 0.5, \n                     colour = \"royalblue\",\n                     fill = \"skyblue\",\n                     linetype = \"solid\") +\n        ggtitle(\"Distribution of Target Variable in Train, Type == 1JHC\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another factor that may affect the target variable is the distance between the two atoms. The x, y, z co-ordinates for each molecule are provided in the structures file. Load & merge that data and calculate the distance between each pair of atoms.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# load structures data \nstructures.df <- read.csv(\"../input/structures.csv\",\n                          stringsAsFactors = FALSE)\nnames(structures.df) <- c(\"molecule_name\",\n                          \"atom_index_0\",\n                          \"atom_0\",\n                          \"ai0_str_x\",\n                          \"ai0_str_y\",\n                          \"ai0_str_z\")\n# merge structures data with first atom in the row (\"atom_index_0\")\ntrain.df <- left_join(train.df,\n                      structures.df,\n                      by = c(\"molecule_name\", \"atom_index_0\"))\n\nnames(structures.df) <- c(\"molecule_name\",\n                          \"atom_index_1\",\n                          \"atom_1\",\n                          \"ai1_str_x\",\n                          \"ai1_str_y\",\n                          \"ai1_str_z\")\n# merge structures data with first atom in the row (\"atom_index_1\")\ntrain.df <- left_join(train.df,\n                      structures.df,\n                      by = c(\"molecule_name\", \"atom_index_1\"))\n\n# calculate distance between each pair of atoms\ntrain.df <- mutate(train.df,\n                 coupling_distance = sqrt((ai1_str_x - ai0_str_x)^2 +\n                                          (ai1_str_y - ai0_str_y)^2 +\n                                          (ai1_str_z - ai0_str_z)^2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now examine the coupling distance distribution, just for Type == 1JHC"},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(filter(train.df, type == \"1JHC\"), aes(x = coupling_distance)) +\n        geom_area(stat = \"bin\", \n                     binwidth = 0.001, \n                     colour = \"royalblue\",\n                     fill = \"skyblue\",\n                     linetype = \"solid\") +\n        ggtitle(\"Distribution of Coupling Distance, Type == 1JHC\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a possible relationship between coupling distance and target variable here, in that they may be *inversely* proportional to each other. Examine this closer."},{"metadata":{"trusted":true},"cell_type":"code","source":"temp.df <- filter(train.df, type == \"1JHC\") \n# divide coupling distance into 3 bins: H, M, L\ntemp.df$coupling_distance_range[temp.df$coupling_distance <= 1.07] <- \"L\"\ntemp.df$coupling_distance_range[temp.df$coupling_distance > 1.07 & temp.df$coupling_distance <= 1.088] <- \"M\"\ntemp.df$coupling_distance_range[temp.df$coupling_distance > 1.088] <- \"H\"\n\n# now plot graph\nggplot(temp.df, aes(x = scalar_coupling_constant,\n                             fill = coupling_distance_range)) + \n        geom_area(stat = \"bin\", \n                           binwidth = 0.05, \n                           linetype = \"solid\",\n                           alpha = 0.5) + \n        labs(title = \"scalar_coupling_constant vs coupling_distance\",\n             subtitle = \"type == 1JHC\") +\n        scale_fill_manual(name = \"coupling_\\ndistance\",\n                          breaks = c(\"H\", \"M\", \"L\"),\n                          labels = c(\"> 1.088\", \"<= 1.088 & > 1.07\", \"<= 1.07\"),\n                          values = c(\"red\", \"green\", \"blue\")) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is clearly a relationship between the coupling distance of the atom pairs, and the target variable. **A Decision Tree algorithm would be ideal to use on this data.**\n\n### Next Type: 2JHC\nNow explore the next Type value: 2JHC. "},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(filter(train.df, type == \"2JHC\"), aes(x = scalar_coupling_constant)) +\n        geom_area(stat = \"bin\", \n                     binwidth = 0.5, \n                     colour = \"royalblue\",\n                     fill = \"skyblue\",\n                     linetype = \"solid\") +\n        ggtitle(\"Distribution of Target Variable in Train, Type == 2JHC\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is closer to a more normal distribution than the previous value of 1JHC.\n\nNow examine the coupling distance distribution for 2JHC:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(filter(train.df, type == \"2JHC\"), aes(x = coupling_distance)) +\n        geom_area(stat = \"bin\", \n                     binwidth = 0.002, \n                     colour = \"royalblue\",\n                     fill = \"skyblue\",\n                     linetype = \"solid\") +\n        ggtitle(\"Distribution of Coupling Distance, Type == 2JHC\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Similar to before, examine to see if there is a relationship between coupling distance and target variable here."},{"metadata":{"trusted":true},"cell_type":"code","source":"temp.df <- filter(train.df, type == \"2JHC\") \n\n# divide coupling distance into 3 bins: H, M, L\n# temp.df$coupling_distance_range <- \"Z\"\ntemp.df$coupling_distance_range[temp.df$coupling_distance <= 2.1] <- \"L\"\ntemp.df$coupling_distance_range[temp.df$coupling_distance > 2.1 & temp.df$coupling_distance <= 2.3] <- \"M\"\ntemp.df$coupling_distance_range[temp.df$coupling_distance > 2.3] <- \"H\"\n\n\n# now plot graph\nggplot(temp.df, aes(x = scalar_coupling_constant,\n                             fill = coupling_distance_range)) + \n        geom_area(stat = \"bin\", \n                           binwidth = 0.5, \n                           linetype = \"solid\",\n                           alpha = 0.5) + \n        labs(title = \"scalar_coupling_constant vs coupling_distance\",\n             subtitle = \"type == 2JHC\") +\n        scale_fill_manual(name = \"coupling_\\ndistance\",\n                          breaks = c(\"H\", \"M\", \"L\"),\n                          labels = c(\"> 2.3\", \"> 2.1 & <= 2.3\", \"<= 2.1\"),\n                          values = c(\"red\", \"green\", \"blue\")) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As this distribution was more gaussian than for the 1JHC type, for 2JHC it appears coupling distance does not have much of a direct effect on the target variable: scalar_coupling_constant."}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}