{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# About the Competition\nFor disabled people, understanding graphs in conventional format is quite difficult. To curb this issue, ML is sought after to transform graphs in such a way that understanding them gets easier for such people. In this competition, graph images from different sources are provided alongwith their json annotations using which graphs are to be made understandable for disabled people. The data provided is in json and jpg format representing graphs.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"## Goal of the Competition \nThe competition aims at providing an overview of the graph which includes the type of the chart and the data series represented in the chart by training a Machine Learning model based on the given training data. The first step in the process would be to predict the chart type. The goal of the competition is to derive the data series for a chart directly from the figure itself without the need for manual annotation.\nA thorough understanding of neural networks is necessary to train the model.\n* https://www.analyticsvidhya.com/blog/2022/01/introduction-to-neural-networks/ (for basic understanding of neural networks(an introduction))\n* https://towardsdatascience.com/machine-learning-for-beginners-an-introduction-to-neural-networks-d49f22d238f9 (to understand the complete working of a neural network)","metadata":{}},{"cell_type":"markdown","source":"## Exploratory Data Analysis\n\nTo begin with taking a deeper dive into the graph dataset, we will see what each type of graph class images look like, do they have similarity or not.","metadata":{}},{"cell_type":"markdown","source":"Loading Libaries","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport json   #to work with json files\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg #to display images\nimport glob \nimport os\nimport random\nimport cv2\nimport math\nimport statistics\nfrom collections import Counter","metadata":{"execution":{"iopub.status.busy":"2023-05-03T11:50:28.623923Z","iopub.execute_input":"2023-05-03T11:50:28.624319Z","iopub.status.idle":"2023-05-03T11:50:28.629858Z","shell.execute_reply.started":"2023-05-03T11:50:28.624286Z","shell.execute_reply":"2023-05-03T11:50:28.628856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"directory_path = '/kaggle/input/benetech-making-graphs-accessible/train/annotations'\njson_files = sorted(glob.glob(os.path.join(directory_path, '*.json')))\nimg_dir = '/kaggle/input/benetech-making-graphs-accessible/train/images'\nimg_files = sorted(glob.glob(os.path.join(img_dir, '*.jpg')))","metadata":{"execution":{"iopub.status.busy":"2023-05-03T11:39:16.045666Z","iopub.execute_input":"2023-05-03T11:39:16.046586Z","iopub.status.idle":"2023-05-03T11:39:17.660760Z","shell.execute_reply.started":"2023-05-03T11:39:16.046531Z","shell.execute_reply":"2023-05-03T11:39:17.659623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = []\n\nfor json_file, img_file in zip(json_files, img_files):\n    with open(json_file) as f:\n        json_data = json.load(f)\n    lst = [img_file]\n    for obj in json_data:\n        lst.append(json_data[obj])\n    data.append(lst)\n\ndf = pd.DataFrame(data)\ndf.columns = ['image_loc', 'source', 'chart-type', 'plot-bb', 'text', 'axes', 'visual-elements', 'data-series']\n","metadata":{"execution":{"iopub.status.busy":"2023-05-03T11:39:17.662082Z","iopub.execute_input":"2023-05-03T11:39:17.662500Z","iopub.status.idle":"2023-05-03T11:43:28.424203Z","shell.execute_reply.started":"2023-05-03T11:39:17.662455Z","shell.execute_reply":"2023-05-03T11:43:28.422863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Now, we have a dataframe that stores image location corresponding to the json annotation.","metadata":{}},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-03T11:43:28.426773Z","iopub.execute_input":"2023-05-03T11:43:28.427124Z","iopub.status.idle":"2023-05-03T11:43:28.571754Z","shell.execute_reply.started":"2023-05-03T11:43:28.427093Z","shell.execute_reply":"2023-05-03T11:43:28.570412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['visual-elements'][4]","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:22:16.319416Z","iopub.execute_input":"2023-05-03T14:22:16.320551Z","iopub.status.idle":"2023-05-03T14:22:16.329068Z","shell.execute_reply.started":"2023-05-03T14:22:16.320506Z","shell.execute_reply":"2023-05-03T14:22:16.327815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### EXPLORATORY DATA ANALYSIS","metadata":{}},{"cell_type":"markdown","source":"Let us see if there is an equal amount of distance between every tick on x and y axis.","metadata":{}},{"cell_type":"markdown","source":"##### Let us check for x axis","metadata":{}},{"cell_type":"code","source":"i = 0\nreslt = []\nfalse_data = []\nfor file in df['axes']:\n    st = []\n    ff = json.dumps(file)\n    resp = json.loads(ff)\n    ini = 0\n    for x in resp['x-axis']['ticks']:\n        st = st + [x['tick_pt']['x']-ini]\n        ini = x['tick_pt']['x']\n    if(len(st)>0):\n        st.pop(0)\n    res = all(ele > statistics.median(st)-5 and ele < statistics.median(st)+5 for ele in st)\n    if(res!=True):\n        false_data.append(i)\n    reslt.append(res)\n    i=i+1\nCounter(reslt)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T13:35:21.937186Z","iopub.execute_input":"2023-05-03T13:35:21.937674Z","iopub.status.idle":"2023-05-03T13:35:27.444906Z","shell.execute_reply.started":"2023-05-03T13:35:21.937629Z","shell.execute_reply":"2023-05-03T13:35:27.443319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The observation says that for ticks on the x axis, they are approximately equally spaced for about 95% of data in our dataset. Let us now see the graphs where they are not spaced equally. ","metadata":{}},{"cell_type":"code","source":"false = df.iloc[false_data].reset_index(drop=True)\nfig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 5))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,2):\n    for j in range(0,2):\n        xyz = random.choice(false['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T13:35:27.447165Z","iopub.execute_input":"2023-05-03T13:35:27.447543Z","iopub.status.idle":"2023-05-03T13:35:28.241348Z","shell.execute_reply.started":"2023-05-03T13:35:27.447506Z","shell.execute_reply":"2023-05-03T13:35:28.240131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> On Running this code several times, we observe the false data is of line graphs and vertical bar graphs majorly. Let us now check the category of the labels.","metadata":{}},{"cell_type":"code","source":"print(false['chart-type'].value_counts()) #the observation is quite true\nlst = []\nol = []\nctr=0\nfor i in false['axes']:\n    lst.append(i['x-axis']['values-type'])\n    if(i['x-axis']['values-type'] == 'numerical'):\n        ol.append(ctr)\n    ctr=ctr+1\nCounter(lst)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T13:35:28.243143Z","iopub.execute_input":"2023-05-03T13:35:28.244355Z","iopub.status.idle":"2023-05-03T13:35:28.265781Z","shell.execute_reply.started":"2023-05-03T13:35:28.244302Z","shell.execute_reply":"2023-05-03T13:35:28.264392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Categorical data is not equally spaced. This is no problem since it does not have to be interpolated. For the numerical data, let us see the markings.","metadata":{}},{"cell_type":"code","source":"ol_data = false.iloc[ol].reset_index(drop=True)\nfig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 5))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,2):\n    for j in range(0,2):\n        xyz = random.choice(ol_data['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)\nprint(ol_data['chart-type'].value_counts())\n","metadata":{"execution":{"iopub.status.busy":"2023-05-03T13:35:28.268785Z","iopub.execute_input":"2023-05-03T13:35:28.269420Z","iopub.status.idle":"2023-05-03T13:35:28.998209Z","shell.execute_reply.started":"2023-05-03T13:35:28.269372Z","shell.execute_reply":"2023-05-03T13:35:28.996886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The main issue is with the scatter plots since they require interpolation to obtain the data series.","metadata":{}},{"cell_type":"markdown","source":"##### Now let us check for y axis","metadata":{}},{"cell_type":"code","source":"i = 0\nreslt = []\nfalse_data = []\nfor file in df['axes']:\n    st = []\n    ff = json.dumps(file)\n    resp = json.loads(ff)\n    ini = 0\n    for y in resp['x-axis']['ticks']:\n        st = st + [y['tick_pt']['y']-ini]\n        ini = y['tick_pt']['y']\n    if(len(st)>0):\n        st.pop(0)\n    res = all(ele > statistics.median(st)-5 and ele < statistics.median(st)+5 for ele in st)\n    if(res!=True):\n        false_data.append(i)\n    reslt.append(res)\n    i=i+1\nCounter(reslt)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T13:35:28.999786Z","iopub.execute_input":"2023-05-03T13:35:29.002464Z","iopub.status.idle":"2023-05-03T13:35:34.428286Z","shell.execute_reply.started":"2023-05-03T13:35:29.002388Z","shell.execute_reply":"2023-05-03T13:35:34.426904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The error in tick markings for y axis is negligible. This is good because y axis data has to be interpolated almost all times expect for horizontal bar graphs.","metadata":{}},{"cell_type":"code","source":"false = df.iloc[false_data].reset_index(drop=True)\nfig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 5))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,2):\n    for j in range(0,2):\n        xyz = random.choice(false['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T13:35:34.430308Z","iopub.execute_input":"2023-05-03T13:35:34.431047Z","iopub.status.idle":"2023-05-03T13:35:35.149741Z","shell.execute_reply.started":"2023-05-03T13:35:34.431011Z","shell.execute_reply":"2023-05-03T13:35:35.148519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> So, this data is for horizontal bar graphs as guessed. This indicates no ouliers in the data for y axis ticks.","metadata":{}},{"cell_type":"markdown","source":"## Taking a closer look at each graph type","metadata":{}},{"cell_type":"markdown","source":"1. Vertical Bar Graph","metadata":{}},{"cell_type":"code","source":"bar_graphs = df[df['chart-type']== 'vertical_bar'].reset_index(drop=True)\nprint(len(bar_graphs)) #our dataset contains around 19000 vertical bar graphs\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(20, 10))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,3):\n    for j in range(0,3):\n        xyz = random.choice(bar_graphs['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:24:38.920284Z","iopub.execute_input":"2023-05-03T14:24:38.921452Z","iopub.status.idle":"2023-05-03T14:24:40.991267Z","shell.execute_reply.started":"2023-05-03T14:24:38.921389Z","shell.execute_reply":"2023-05-03T14:24:40.981667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since predicting data series from the plot will require getting a hold on the dimensions of the graph and the relative dimensions of the data points in the graph. We'll start by observing the plot boundary. We'll start to find if the distance of (x0, y0) from the lower left corner of the image has any trend depending on the dimensions of the image.","metadata":{}},{"cell_type":"code","source":"data_f = pd.read_json(bar_graphs['plot-bb'].to_json())\ndata_f = data_f.T\ndata_f['height'].min()\n# plt.scatter(x=data_f['height'], y=data_f['y0'])","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:24:46.980471Z","iopub.execute_input":"2023-05-03T14:24:46.981162Z","iopub.status.idle":"2023-05-03T14:24:53.421201Z","shell.execute_reply.started":"2023-05-03T14:24:46.981124Z","shell.execute_reply":"2023-05-03T14:24:53.420246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"2) Horizontal bar graph","metadata":{}},{"cell_type":"code","source":"hbar_graphs = df[df['chart-type']== 'horizontal_bar'].reset_index(drop=True)\nprint(len(hbar_graphs)) #our dataset contains around 19000 vertical bar graphs\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(20, 10))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,3):\n    for j in range(0,3):\n        xyz = random.choice(hbar_graphs['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:01.148756Z","iopub.execute_input":"2023-05-03T14:27:01.149208Z","iopub.status.idle":"2023-05-03T14:27:02.935215Z","shell.execute_reply.started":"2023-05-03T14:27:01.149166Z","shell.execute_reply":"2023-05-03T14:27:02.934271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"3) Line charts","metadata":{}},{"cell_type":"code","source":"line_graphs = df[df['chart-type']== 'line'].reset_index(drop=True)\nprint(len(line_graphs)) #our dataset contains around 19000 vertical bar graphs\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(20, 10))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,3):\n    for j in range(0,3):\n        xyz = random.choice(line_graphs['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:02.937047Z","iopub.execute_input":"2023-05-03T14:27:02.937455Z","iopub.status.idle":"2023-05-03T14:27:04.740241Z","shell.execute_reply.started":"2023-05-03T14:27:02.937405Z","shell.execute_reply":"2023-05-03T14:27:04.739016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2\nimport numpy as np\n\n# Load the image and the template\nimg = cv2.imread(line_graphs['image_loc'][1])\ntemplate = cv2.imread('/kaggle/input/zero-image-div/zero.png')\n\n# Get the dimensions of the template\nh, w = template.shape[:2]\n\n# Apply template matching\nres = cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED)\n\n# Set a threshold to identify matches\nthreshold = 0.8\nloc = np.where(res >= threshold)\n\n# Draw a rectangle around the matched region\nfor pt in zip(*loc[::-1]):\n    cv2.rectangle(img, pt, (pt[0] + w, pt[1] + h), (0, 0, 255), 2)\n\n# Display the result\nplt.imshow( img)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:04.741870Z","iopub.execute_input":"2023-05-03T14:27:04.742230Z","iopub.status.idle":"2023-05-03T14:27:05.307397Z","shell.execute_reply.started":"2023-05-03T14:27:04.742193Z","shell.execute_reply":"2023-05-03T14:27:05.306172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img = cv2.imread(line_graphs['image_loc'][0])\ngray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)\n\nkernel_size = 5\nblur_gray = cv2.GaussianBlur(gray,(kernel_size, kernel_size),0)\nlow_threshold = 50\nhigh_threshold = 150\nedges = cv2.Canny(blur_gray, low_threshold, high_threshold)\nrho = 1  # distance resolution in pixels of the Hough grid\ntheta = np.pi / 180  # angular resolution in radians of the Hough grid\nthreshold = 15  # minimum number of votes (intersections in Hough grid cell)\nmin_line_length = 50  # minimum number of pixels making up a line\nmax_line_gap = 20  # maximum gap in pixels between connectable line segments\nline_image = np.copy(img) * 0  # creating a blank to draw lines on\n\n# Run Hough on edge detected image\n# Output \"lines\" is an array containing endpoints of detected line segments\nlines = cv2.HoughLinesP(edges, rho, theta, threshold, np.array([]),\n                    min_line_length, max_line_gap)\n\nfor line in lines:\n    for x1,y1,x2,y2 in line:\n        cv2.line(line_image,(x1,y1),(x2,y2),(255,0,0),5)\n# Draw the lines on the  image\nlines_edges = cv2.addWeighted(img, 0.8, line_image, 1, 0)\n# from PIL import Image\n# img = Image.fromarray(lines_edges)\n# img.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:05.310066Z","iopub.execute_input":"2023-05-03T14:27:05.310447Z","iopub.status.idle":"2023-05-03T14:27:05.389213Z","shell.execute_reply.started":"2023-05-03T14:27:05.310396Z","shell.execute_reply":"2023-05-03T14:27:05.388286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nfrom PIL import Image\nfrom scipy.ndimage import generic_filter\nfrom skimage.morphology import medial_axis\n\n# Line ends filter\ndef lineEnds(P):\n    \"\"\"Central pixel and just one other must be set to be a line end\"\"\"\n    return 255 * ((P[4]==255) and np.sum(P)==510)\n\n# Open image and make into Numpy array\nim = Image.open(line_graphs['image_loc'][0]).convert('L')\nim = np.array(im)\n\n# Skeletonize\nskel = (medial_axis(im)*255).astype(np.uint8)\n\n# Find line ends\nresult = generic_filter(skel, lineEnds, (3, 3))\n\n# Save result\nImage.fromarray(result).save('result.png')\n# image = mpimg.imread(\"result.png\")\n# plt.imshow(image)\n# plt.show()\n# cv2.imshow('image', img)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:33.910681Z","iopub.execute_input":"2023-05-03T14:27:33.911965Z","iopub.status.idle":"2023-05-03T14:27:35.064500Z","shell.execute_reply.started":"2023-05-03T14:27:33.911907Z","shell.execute_reply":"2023-05-03T14:27:35.063289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import numpy as np\n\n# img = cv2.imread(line_graphs['image_loc'][0])\n# gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n# edges = cv2.Canny(gray, 50, 150, apertureSize=3)\n# lines = cv2.HoughLinesP(edges, rho=1, theta=np.pi/180, threshold=100, minLineLength=100, maxLineGap=10)\n\n# for line in lines:\n#     rho, theta = line[0]\n#     a = np.cos(theta)\n#     b = np.sin(theta)\n#     x0 = a*rho\n#     y0 = b*rho\n#     x1 = int(x0 + 1000*(-b))\n#     y1 = int(y0 + 1000*(a))\n#     x2 = int(x0 - 1000*(-b))\n#     y2 = int(y0 - 1000*(a))\n#     cv2.line(img, (x1, y1), (x2, y2), (0, 0, 255), 2)\n\n# cv2.imshow('image', img)\n# cv2.waitKey(0)\n# cv2.destroyAllWindows()","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:35.066342Z","iopub.execute_input":"2023-05-03T14:27:35.066814Z","iopub.status.idle":"2023-05-03T14:27:35.071507Z","shell.execute_reply.started":"2023-05-03T14:27:35.066777Z","shell.execute_reply":"2023-05-03T14:27:35.070387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"4) Dot plots","metadata":{}},{"cell_type":"code","source":"dot_graphs = df[df['chart-type']== 'dot'].reset_index(drop=True)\nprint(len(dot_graphs)) #our dataset contains around 19000 vertical bar graphs\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(20, 10))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,3):\n    for j in range(0,3):\n        xyz = random.choice(dot_graphs['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:35.072937Z","iopub.execute_input":"2023-05-03T14:27:35.073365Z","iopub.status.idle":"2023-05-03T14:27:36.928803Z","shell.execute_reply.started":"2023-05-03T14:27:35.073320Z","shell.execute_reply":"2023-05-03T14:27:36.927595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"5) Scatter plots","metadata":{}},{"cell_type":"code","source":"scatter_graphs = df[df['chart-type']== 'scatter'].reset_index(drop=True)\nprint(len(scatter_graphs)) #our dataset contains around 19000 vertical bar graphs\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(20, 10))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,3):\n    for j in range(0,3):\n        xyz = random.choice(scatter_graphs['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:40.262679Z","iopub.execute_input":"2023-05-03T14:27:40.263417Z","iopub.status.idle":"2023-05-03T14:27:41.990038Z","shell.execute_reply.started":"2023-05-03T14:27:40.263372Z","shell.execute_reply":"2023-05-03T14:27:41.989113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Importing libraries","metadata":{}},{"cell_type":"code","source":"with open('/kaggle/input/benetech-making-graphs-accessible/train/annotations/0003a50817cf.json', 'r') as f:\n  data = json.load(f)\nprint(data)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:41.991909Z","iopub.execute_input":"2023-05-03T14:27:41.992536Z","iopub.status.idle":"2023-05-03T14:27:42.023216Z","shell.execute_reply.started":"2023-05-03T14:27:41.992499Z","shell.execute_reply":"2023-05-03T14:27:42.022139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Understanding this data is the key to produce correct answers.","metadata":{}},{"cell_type":"markdown","source":"Let us see the corresponding graph image to this json annotation.","metadata":{}},{"cell_type":"code","source":"img=mpimg.imread('/kaggle/input/benetech-making-graphs-accessible/train/images/0003a50817cf.jpg')\nimgplot = plt.imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:42.024485Z","iopub.execute_input":"2023-05-03T14:27:42.024833Z","iopub.status.idle":"2023-05-03T14:27:42.354872Z","shell.execute_reply.started":"2023-05-03T14:27:42.024798Z","shell.execute_reply":"2023-05-03T14:27:42.353705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let us understand the parts of the json files one by one.","metadata":{}},{"cell_type":"code","source":"data['source']","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:42.357314Z","iopub.execute_input":"2023-05-03T14:27:42.357687Z","iopub.status.idle":"2023-05-03T14:27:42.364378Z","shell.execute_reply.started":"2023-05-03T14:27:42.357650Z","shell.execute_reply":"2023-05-03T14:27:42.362915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The source field tells whether the graph has been generated or extracted.","metadata":{}},{"cell_type":"code","source":"data['chart-type']","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:42.365569Z","iopub.execute_input":"2023-05-03T14:27:42.365873Z","iopub.status.idle":"2023-05-03T14:27:42.376075Z","shell.execute_reply.started":"2023-05-03T14:27:42.365843Z","shell.execute_reply":"2023-05-03T14:27:42.374923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Chart-Type tells the category our graph belongs to - bar/dot/line/scatter.","metadata":{}},{"cell_type":"code","source":"data['plot-bb']","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:42.377474Z","iopub.execute_input":"2023-05-03T14:27:42.377793Z","iopub.status.idle":"2023-05-03T14:27:42.388492Z","shell.execute_reply.started":"2023-05-03T14:27:42.377763Z","shell.execute_reply":"2023-05-03T14:27:42.387548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Gives information about the bounding box of the plot within the figure, given by height, width, x0, and y0.","metadata":{}},{"cell_type":"code","source":"data['text']","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:42.389985Z","iopub.execute_input":"2023-05-03T14:27:42.390286Z","iopub.status.idle":"2023-05-03T14:27:42.408467Z","shell.execute_reply.started":"2023-05-03T14:27:42.390255Z","shell.execute_reply":"2023-05-03T14:27:42.407070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> For every text element in the graph, an ID is assigned alongwith the co-ordinates of the area surrounding the text. The text field has the content to be filled in the area and role tells the use of the label in the plot(whether it is plot title or axis label or tick label).","metadata":{}},{"cell_type":"code","source":"data['axes']","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:44.562340Z","iopub.execute_input":"2023-05-03T14:27:44.562770Z","iopub.status.idle":"2023-05-03T14:27:44.573191Z","shell.execute_reply.started":"2023-05-03T14:27:44.562734Z","shell.execute_reply":"2023-05-03T14:27:44.571755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The axes object contains co-ordinates of the ticks associated with a particular text id. The tick-type tells the visual representation of the tick and value-type tells the data types of the values represented by the tick element.","metadata":{}},{"cell_type":"code","source":"data['visual-elements']","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:44.967367Z","iopub.execute_input":"2023-05-03T14:27:44.967805Z","iopub.status.idle":"2023-05-03T14:27:44.977099Z","shell.execute_reply.started":"2023-05-03T14:27:44.967768Z","shell.execute_reply":"2023-05-03T14:27:44.975825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The height width and co-ordinates of the bars on the graph are stored in this object. This object is significant in calculating the data series of the graph.","metadata":{}},{"cell_type":"code","source":"data['data-series']","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:45.300602Z","iopub.execute_input":"2023-05-03T14:27:45.301451Z","iopub.status.idle":"2023-05-03T14:27:45.311943Z","shell.execute_reply.started":"2023-05-03T14:27:45.301385Z","shell.execute_reply":"2023-05-03T14:27:45.310513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> This object has the x and y coordinates of values depicted in the figure. For the test set images, this is the target to be predicted.","metadata":{}},{"cell_type":"markdown","source":"Now, let us mark these object boundaries on the graph image using json annotation.","metadata":{}},{"cell_type":"code","source":"for text in data['text']:\n    polygon = text[\"polygon\"]\n    coord = dict(polygon.items())\n    point1 = (coord['x3'], coord['y3'])\n    point2 = (coord['x1'], coord['y1'])\n    cv2.rectangle(img, pt1=point1, pt2=point2, color=(255,0,0), thickness=2)\n    plt.imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-03T14:27:46.570289Z","iopub.execute_input":"2023-05-03T14:27:46.571328Z","iopub.status.idle":"2023-05-03T14:27:49.047634Z","shell.execute_reply.started":"2023-05-03T14:27:46.571282Z","shell.execute_reply":"2023-05-03T14:27:49.046406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The boundary is quite defined which gives us the assurance of accuracy of the data. We can do the same for every graph in our dataset.","metadata":{}}]}