{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"3.6.3"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# In which section the dataset is mentionned ?\n\nHi everyone ! This notebook is a quick EDA to try to see where the dataset is mentionned in each article.\nAs you probably know, each scientific article is usually structured into something like Abstract, Introduction, and more exotic section names.\nHere I just want to understand where the dataset is more often cited using the label provided in the train dataframe.\n\nIt turns out that there is a lot of section with unual names. Here I focus on the usual ones : abstract, etc. \nOne reason is that, some approach in NLP, like the ULMFiT of Fastai, has a Self-supervised step : the model learn the english structuration of the particular corpus in a unsupervised manner. It is possible to increase the length of the corpus by using some package to got more abstracts, more introductions, more articles, with pubmedR for example.\n\nAs you will see, nonetheless, the \"usual\" sections names does not seem to capture all the positions of the dataset, which can be in a lot of different subsections.","metadata":{}},{"cell_type":"code","source":"# This R environment comes with many helpful analytics packages installed\n# It is defined by the kaggle/rstats Docker image: https://github.com/kaggle/docker-rstats\n# For example, here's a helpful package to load\n\nlibrary(tidyverse) # metapackage of all tidyverse packages\nlibrary(data.table)\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nlist.files(path = \"../input\")\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"051d70d956493feee0c6d64651c6a088724dca2a","_execution_state":"idle","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dealing with json file\n\nFrom a quick glance at the data : each json file contain the differents section of the articles.","metadata":{}},{"cell_type":"code","source":"library(\"rjson\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Opening the first document inside the train csv file : ","metadata":{}},{"cell_type":"code","source":"result <- fromJSON(file = \"../input/coleridgeinitiative-show-us-the-data/train/d0fa7568-7d8e-4db9-870f-f9c6f668c17b.json\")\nlength(result)\nhead(result)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[First tuto on google.](https://www.tutorialspoint.com/r/r_json_files.htm) Unlist on whim.","metadata":{}},{"cell_type":"code","source":"json_data_frame <- as.data.frame(unlist(result), stringsAsFactors = FALSE)\nhead(json_data_frame)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Later this could serve for data analysis (like, in which section are located the words dataset ?).","metadata":{}},{"cell_type":"code","source":"train<-read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\nhead(train)\n#train<-head(train, 200) #quick trial and errors","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dim(train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Function to got the section","metadata":{}},{"cell_type":"code","source":"grep(\"National Education Longitudinal Study\", json_data_frame$`unlist(result)`)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_data_frame$`unlist(result)`[c(2,20,22)]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_data_frame$`unlist(result)`[c(2,20,22)-1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"position<-as.vector(grep(\"National Education Longitudinal Study\", json_data_frame$`unlist(result)`))\nsections_names<-json_data_frame$`unlist(result)`[position-1]\nsections_names","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"table(sections_names)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"canonical_section<-c(\"introduction\", \"result\", \"abstract\", \"material\", \"method\", \"conclusion\", \"data\", \"discussion\", \"supplementary\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_sections_names <- function(train_row){\n    \n    ID<-train_row$Id\n    label<-train_row$dataset_title #The one in the article\n    \n    result <- fromJSON(file = paste0(\"../input/coleridgeinitiative-show-us-the-data/train/\", ID, \".json\"))\n    json_data_frame <- as.data.frame(unlist(result), stringsAsFactors = FALSE)\n    \n        \n    position<-as.vector(grep(label, json_data_frame$`unlist(result)`)) #where the dataset is mentionned\n    sections_names<-json_data_frame$`unlist(result)`[position-1] #title of the section\n    \n    section_article<-data.frame(matrix(ncol=length(canonical_section),nrow=1, dimnames=list(NULL, canonical_section)))\n    section_article[1,]<-0\n    \n    for (section in canonical_section){\n         if (length(grep(section, tolower(sections_names)))>0){\n            section_article[[section]]<-1\n        }\n    }\n    return(section_article)\n}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The only problems with this function is that it does not distinguish between section and subsection inside a section like inside material and methods.","metadata":{}},{"cell_type":"markdown","source":"It looks a bit complicated but a function like the following would return everything including a lot of nonsense that would be need to be cleaned using regex and just explode the memory anyway :","metadata":{}},{"cell_type":"code","source":"grep_sections_names <- function(train_row){\n    \n    ID<-train_row$Id\n    label<-train_row$dataset_title #The one in the article\n    \n    result <- fromJSON(file = paste0(\"../input/coleridgeinitiative-show-us-the-data/train/\", ID, \".json\"))\n    json_data_frame <- as.data.frame(unlist(result), stringsAsFactors = FALSE)\n    \n    position<-as.vector(grep(label, json_data_frame$`unlist(result)`))\n    sections_names<-json_data_frame$`unlist(result)`[position-1]\n    return(sections_names)\n\n}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res<-list()\nfor (article in 1:dim(train)[1]){\n  res[[article]]<-count_sections_names(train[article,])\n}\nres<-rbindlist(res, use.names = TRUE)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"colSums(res)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"table(rowSums(res)) ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotting<-as.data.frame(colSums(res))\nplotting$section<-row.names(plotting)\ncolnames(plotting)<-c(\"ocurrences\", \"section\")\nplotting","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/getting-started/105201\nfig <- function(width, heigth){\n     options(repr.plot.width = width, repr.plot.height = heigth)\n}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig(15, 15)\np <- ggplot(data=plotting, aes(x=reorder(section, ocurrences), y=ocurrences)) +\n    geom_bar(stat=\"identity\", fill=\"steelblue\") +\n    theme_minimal(base_size = 22)\np","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig(15, 15)\n\np + geom_text(aes(label=ocurrences), hjust=1.2, color=\"white\", size=10) + coord_flip()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}