{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.0.5"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This R environment comes with many helpful analytics packages installed\n# It is defined by the kaggle/rstats Docker image: https://github.com/kaggle/docker-rstats\n# For example, here's a helpful package to load\n\nlibrary(tidyverse) # metapackage of all tidyverse packages\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nlist.files(path = \"../input\")\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"051d70d956493feee0c6d64651c6a088724dca2a","_execution_state":"idle","execution":{"iopub.status.busy":"2023-05-15T04:38:47.607743Z","iopub.execute_input":"2023-05-15T04:38:47.609958Z","iopub.status.idle":"2023-05-15T04:38:49.000487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Source\n - https://www.datanovia.com/en/blog/easy-image-processing-in-r-using-the-magick-package\n - https://www.r-bloggers.com/2019/06/extracting-the-data-from-static-images-of-graphs-with-magick/\n - https://themockup.blog/posts/2020-05-22-parsing-json-in-r-with-jsonlite/\n - https://github.com/timelyportfolio/listviewer\n - https://purrr.tidyverse.org/reference/pluck.html\n - https://www.rdocumentation.org/packages/glcm/versions/1.6.5","metadata":{}},{"cell_type":"markdown","source":"

Context

\n 
Millions of students have a learning, physical, or visual disability that prevents a person from reading conventional print. The majority of educational materials for science, technology, engineering, and math (STEM) fields are inaccessible to these students. Technology that makes the written word accessible exists
\n \n*** \n \n

Goal of the Competition

\n 
The goal of this competition is to extract data represented by four types of charts commonly found in STEM textbooks. You will develop an automatic solution trained on a graph dataset.Your task is to predict the data series depicted in the test set figures
\n\n

Dataset Description

\n
This competition's dataset comprises about 65,000 comprehensively-annotated scientific figures of four kinds: bar graphs (both horizontal and vertical), dot plots, line graphs, and scatter plots. While the majority of the figures are synthetic, we also include several thousand figures extracted from professionally-produced sources. Your task is to predict the data series depicted in the test set figures.
\n\n ","metadata":{}},{"cell_type":"code","source":"library(tesseract)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:49.004880Z","iopub.execute_input":"2023-05-15T04:38:49.042364Z","iopub.status.idle":"2023-05-15T04:38:49.349063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eng <- tesseract(\"eng\")\ntext <- tesseract::ocr(\"/kaggle/input/benetech-making-graphs-accessible/train/images/0000ae6cbdb1.jpg\", engine = eng)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:49.365013Z","iopub.execute_input":"2023-05-15T04:38:49.367502Z","iopub.status.idle":"2023-05-15T04:38:50.713371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cat(text)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:50.717916Z","iopub.execute_input":"2023-05-15T04:38:50.721205Z","iopub.status.idle":"2023-05-15T04:38:50.754344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"``` ocr_data() - words from the image with confidence rate and bounding box ```","metadata":{}},{"cell_type":"code","source":"results <- tesseract::ocr_data(\"/kaggle/input/benetech-making-graphs-accessible/train/images/0005413054c9.jpg\", engine = eng)\nresults\n","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:50.758346Z","iopub.execute_input":"2023-05-15T04:38:50.760754Z","iopub.status.idle":"2023-05-15T04:38:56.381842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"library(magick)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:56.384805Z","iopub.execute_input":"2023-05-15T04:38:56.386552Z","iopub.status.idle":"2023-05-15T04:38:56.516308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image <- image_read(\"/kaggle/input/benetech-making-graphs-accessible/train/images/0005e64fdc6e.jpg\")\nprint(image)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:56.520440Z","iopub.execute_input":"2023-05-15T04:38:56.522751Z","iopub.status.idle":"2023-05-15T04:38:56.635811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_info(image)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:56.639749Z","iopub.execute_input":"2023-05-15T04:38:56.641843Z","iopub.status.idle":"2023-05-15T04:38:56.675057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Image border\nimage_border(image_background(image, \"blue\"), \"#0000FF\", \"20x10\")","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:56.678242Z","iopub.execute_input":"2023-05-15T04:38:56.679840Z","iopub.status.idle":"2023-05-15T04:38:56.755460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"```Stack layers on top of each other```","metadata":{}},{"cell_type":"code","source":"image1 <- image_read('/kaggle/input/benetech-making-graphs-accessible/train/images/0000ae6cbdb1.jpg')\nimage2 <- image_read(\"/kaggle/input/benetech-making-graphs-accessible/train/images/0003a50817cf.jpg\")\nimage3 <- image_read(\"/kaggle/input/benetech-making-graphs-accessible/train/images/0005413054c9.jpg\")\nimages <- c(image1,image2,image3)\nimg_stacked <- image_scale(images, \"300x300\")\nimage_info(img_stacked)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:56.758271Z","iopub.execute_input":"2023-05-15T04:38:56.759779Z","iopub.status.idle":"2023-05-15T04:38:56.836856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Combine Images

","metadata":{}},{"cell_type":"code","source":"image_append(image_scale(images, \"x200\"))","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:56.840450Z","iopub.execute_input":"2023-05-15T04:38:56.842708Z","iopub.status.idle":"2023-05-15T04:38:56.920521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Stack images on top of each other:\n\nimage_append(image_scale(images, \"300\"), stack = TRUE) ","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:56.925004Z","iopub.execute_input":"2023-05-15T04:38:56.927649Z","iopub.status.idle":"2023-05-15T04:38:56.997088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Animate Images\nimage_animate(image_scale(images, \"200x200\"), fps = 1, dispose = \"previous\")\n","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:57.000384Z","iopub.execute_input":"2023-05-15T04:38:57.001887Z","iopub.status.idle":"2023-05-15T04:38:57.511047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# library(OpenImageR)\n# setwd(\"/kaggle/input/benetech-making-graphs-accessible/train/images\")\n# Files <- list.files()\n# Results <- list()\n# for(i in seq_along(Files)){\n# Image <- readImage(Files[i])\n# Resized <- resizeImage(Image, width = 200, height = 200)\n# Gray <- rgb_2gray(Resized)\n# Results[[i]] <- Gray\n# }","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:57.513951Z","iopub.execute_input":"2023-05-15T04:38:57.515614Z","iopub.status.idle":"2023-05-15T04:38:57.527637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# library(RJSONIO)\n# library(jsonlite)\n# library(rjson)\n# setwd(\"/kaggle/input/benetech-making-graphs-accessible/train/annotations\")\n# Files <- list.files()\n# Results <- list()\n# for(i in seq_along(Files)){\n# json <- read_json(Files[i])\n# df<-as.data.frame(do.call(\"rbind\", json))\n# }","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:57.531314Z","iopub.execute_input":"2023-05-15T04:38:57.533519Z","iopub.status.idle":"2023-05-15T04:38:57.549162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"```convert JSON files into Data Frames```","metadata":{}},{"cell_type":"code","source":"require(RJSONIO) \njson_file <- fromJSON(\"/kaggle/input/benetech-making-graphs-accessible/train/annotations/0000ae6cbdb1.json\")\njson_file <- lapply(json_file, function(x) {\n x[sapply(x, is.null)] <- NA\n unlist(x)\n})\ndf<-as.data.frame(do.call(\"cbind\", json_file))\n","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:57.552112Z","iopub.execute_input":"2023-05-15T04:38:57.553735Z","iopub.status.idle":"2023-05-15T04:38:57.697197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"library(DT)\ndatatable(head(df))","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:57.708966Z","iopub.execute_input":"2023-05-15T04:38:57.711753Z","iopub.status.idle":"2023-05-15T04:38:58.318464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"library(listviewer)\njsonedit(df)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:58.354183Z","iopub.execute_input":"2023-05-15T04:38:58.357038Z","iopub.status.idle":"2023-05-15T04:38:58.803018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"listviewer::jsonedit(df, height = \"800px\", mode = \"view\")","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:58.839124Z","iopub.execute_input":"2023-05-15T04:38:58.841669Z","iopub.status.idle":"2023-05-15T04:38:59.083143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"library(jsonlite)\nlibrary(httr)\n\n#get data\ndata <- fromJSON(\"/kaggle/input/benetech-making-graphs-accessible/train/annotations/0003a50817cf.json\")\n\n#it's a data frame\nnames(data)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.088572Z","iopub.execute_input":"2023-05-15T04:38:59.111000Z","iopub.status.idle":"2023-05-15T04:38:59.234424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"str(data, max.level = 1)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.238698Z","iopub.execute_input":"2023-05-15T04:38:59.241187Z","iopub.status.idle":"2023-05-15T04:38:59.275561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"str(data, max.level = 2)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.279744Z","iopub.execute_input":"2023-05-15T04:38:59.282267Z","iopub.status.idle":"2023-05-15T04:38:59.330133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data %>% \n purrr::pluck() %>% \n str(max.level = 2)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.334477Z","iopub.execute_input":"2023-05-15T04:38:59.337069Z","iopub.status.idle":"2023-05-15T04:38:59.383758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# library(rlist)\n# files <- list.load('/kaggle/input/benetech-making-graphs-accessible/train/annotations/0000ae6cbdb1.json')\n# data <- as.data.frame(files)\n# data","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-15T04:38:59.388169Z","iopub.execute_input":"2023-05-15T04:38:59.390749Z","iopub.status.idle":"2023-05-15T04:38:59.408783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract the saturation channel\nlibrary(magick)\n\nimage <- image_read(\"/kaggle/input/benetech-making-graphs-accessible/test/images/007a18eb4e09.jpg\")\nim_proc <- image %>%\n image_channel(\"saturation\")\n\nim_proc","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.413337Z","iopub.execute_input":"2023-05-15T04:38:59.415991Z","iopub.status.idle":"2023-05-15T04:38:59.583151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Threshold image at 20%, so that any pixel value that is over 20% saturation gets set to white - this will effectively eliminate all pixels aside from the lines\nim_proc2 <- im_proc %>%\n image_threshold(\"white\", \"20%\")\n\nim_proc2","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.587424Z","iopub.execute_input":"2023-05-15T04:38:59.589974Z","iopub.status.idle":"2023-05-15T04:38:59.702778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Negate the image so that the lines are white and the rest are black\nim_proc3 <- im_proc2 %>%\n image_negate()\n\nim_proc3","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.707145Z","iopub.execute_input":"2023-05-15T04:38:59.709727Z","iopub.status.idle":"2023-05-15T04:38:59.822184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert the pixel values to points suitable for plotting\ndat <- image_data(im_proc)[1,,] %>%\n as.data.frame() %>%\n mutate(Row = 1:nrow(.)) %>%\n select(Row, everything()) %>%\n mutate_all(as.character) %>%\n gather(key = Column, value = value, 2:ncol(.)) %>%\n mutate(Column = as.numeric(gsub(\"V\", \"\", Column)),\n Row = as.numeric(Row),\n value = ifelse(value == \"00\", NA, 1)) %>%\n filter(!is.na(value))","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:38:59.826915Z","iopub.execute_input":"2023-05-15T04:38:59.829555Z","iopub.status.idle":"2023-05-15T04:39:00.783520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"?colors","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-15T04:39:00.788252Z","iopub.execute_input":"2023-05-15T04:39:00.790950Z","iopub.status.idle":"2023-05-15T04:39:01.100350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ggplot(data = dat, aes(x = Row, y = Column, colour = (Column < 300))) +\n geom_point() +\n scale_y_continuous(trans = \"reverse\") +\n scale_colour_manual(values = c(\"red4\", \"grey50\")) +\n theme(legend.position = \"off\")","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:39:01.104900Z","iopub.execute_input":"2023-05-15T04:39:01.107306Z","iopub.status.idle":"2023-05-15T04:39:14.423708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # library(jsonlite)\n\n# annotation <- '/kaggle/input/benetech-making-graphs-accessible/train/annotations'\n# annot_df <- data.frame()\n\n# for (file in list.files(annotation, recursive = TRUE, full.names = TRUE)) {\n# d <- fromJSON(readLines(file))\n# tt <- as.data.frame(d, max.level = 2)\n# tt$file <- tools::file_path_sans_ext(basename(file))\n# annot_df <- rbind(annot_df, tt)\n# }","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:39:14.428930Z","iopub.execute_input":"2023-05-15T04:39:14.431377Z","iopub.status.idle":"2023-05-15T04:39:14.444777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# options(max.print=Inf)\n# head(annot_df, 5)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:39:14.447736Z","iopub.execute_input":"2023-05-15T04:39:14.462804Z","iopub.status.idle":"2023-05-15T04:39:14.475329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"install.packages('glcm')\n","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:39:14.478210Z","iopub.execute_input":"2023-05-15T04:39:14.479804Z","iopub.status.idle":"2023-05-15T04:39:53.685024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculating GLCM Texture\nlibrary(glcm)\nlibrary(raster)\nr <- raster(\"/kaggle/input/benetech-making-graphs-accessible/train/images/0000ae6cbdb1.jpg\")\nplot(r)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:39:53.688787Z","iopub.execute_input":"2023-05-15T04:39:53.690437Z","iopub.status.idle":"2023-05-15T04:40:00.808810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Calculating GLCM textures in one direction\nrglcm <- glcm(r,\n window = c(9,9),\n shift = c(1,1),\n statistics = c(\"mean\", \"variance\", \"homogeneity\", \"contrast\",\n \"dissimilarity\", \"entropy\", \"second_moment\")\n )\nplot(rglcm)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:40:00.811899Z","iopub.execute_input":"2023-05-15T04:40:00.814362Z","iopub.status.idle":"2023-05-15T04:40:06.027552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculation rotation-invariant texture features\nrglcm1 <- glcm(r,\n window = c(9,9),\n shift=list(c(0,1), c(1,1), c(1,0), c(1,-1)),\n statistics = c(\"mean\", \"variance\", \"homogeneity\", \"contrast\",\n \"dissimilarity\", \"entropy\", \"second_moment\")\n )\nplot(rglcm1)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:40:06.032615Z","iopub.execute_input":"2023-05-15T04:40:06.034289Z","iopub.status.idle":"2023-05-15T04:40:24.182841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"?shapeKernel","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:40:24.187703Z","iopub.execute_input":"2023-05-15T04:40:24.190689Z","iopub.status.idle":"2023-05-15T04:40:24.287488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"?erode","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:40:24.292012Z","iopub.execute_input":"2023-05-15T04:40:24.294605Z","iopub.status.idle":"2023-05-15T04:40:24.370867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"install.packages(\"mmand\")","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:40:24.375880Z","iopub.execute_input":"2023-05-15T04:40:24.379157Z","iopub.status.idle":"2023-05-15T04:41:17.050652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Mathematical Morphologies\nlibrary(mmand)\nsk <- shapeKernel(c(9,9), type=\"disc\")\n\n# Convert raster layer into an array wich is used as input for the erode() function.\nrArr <- as.array(r, transpose = TRUE)\nrErode <- erode(rArr, sk)\nrErode <- setValues(r, as.vector(aperm(rErode)))\nplot(rErode)","metadata":{"execution":{"iopub.status.busy":"2023-05-15T04:41:17.053702Z","iopub.execute_input":"2023-05-15T04:41:17.055324Z","iopub.status.idle":"2023-05-15T04:41:17.712225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}