{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### INTRODUCTION","metadata":{}},{"cell_type":"markdown","source":"This kaggle competition on graphs asks us to predict the data series and chart type from graph images. This notebook is intended on taking a deep dive in the data provided in the training set. The data given to us has the following specifications-\n1. Graph images - These are png files of graphs which we generally see in textbooks or some reading material. We are given five types of graphs namely Vertical Bar graphs, Horizontal Bar graphs, Line charts, Scatter plots and Dot charts. These are the ones we generally encounter owing to their simplicity and ease of understanding.\n2. Json files - Corresponding to each image, we have a json annotation which gives complete information about every element in the graph.\n\nThe problem at hand is that for people with learning problem or disability, deciphering these graphs is quite difficult because of which resources for their learning decrease considerably. Currently, manual help is taken to convert them into graph understandable text format which is time taking. To sought this issue, we are required to convert these graphs into tabular data to enable their learning using ML.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"### Loading Data","metadata":{}},{"cell_type":"markdown","source":"Okay, so let us start with loading the data - we'll make a list of json file paths and a corresponding list of image file paths. ","metadata":{}},{"cell_type":"code","source":"#importing required libraries\nimport pandas as pd\nimport json   #to work with json files\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg #to display images\nimport glob \nimport os\nimport random\nimport cv2\nimport math\nimport statistics\nfrom collections import Counter","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:12:17.346521Z","iopub.execute_input":"2023-05-10T13:12:17.346917Z","iopub.status.idle":"2023-05-10T13:12:17.353688Z","shell.execute_reply.started":"2023-05-10T13:12:17.346886Z","shell.execute_reply":"2023-05-10T13:12:17.352399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"directory_path = '/kaggle/input/benetech-making-graphs-accessible/train/annotations'\njson_files = sorted(glob.glob(os.path.join(directory_path, '*.json')))\nimg_dir = '/kaggle/input/benetech-making-graphs-accessible/train/images'\nimg_files = sorted(glob.glob(os.path.join(img_dir, '*.jpg')))","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:12:17.356498Z","iopub.execute_input":"2023-05-10T13:12:17.357100Z","iopub.status.idle":"2023-05-10T13:12:17.945327Z","shell.execute_reply.started":"2023-05-10T13:12:17.357056Z","shell.execute_reply":"2023-05-10T13:12:17.944229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we'll convert our json files to csv format and store them in a dataframe. We'll also append the image file path in every row. The data given to us does not have any null values so we should'nt worry.","metadata":{}},{"cell_type":"code","source":"data = []\n\nfor json_file, img_file in zip(json_files, img_files):\n    with open(json_file) as f:\n        json_data = json.load(f)\n    lst = [img_file]\n    for obj in json_data:\n        lst.append(json_data[obj])\n    data.append(lst)\n\ndf = pd.DataFrame(data)\ndf.columns = ['image_loc', 'source', 'chart-type', 'plot-bb', 'text', 'axes', 'visual-elements', 'data-series']\n","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:12:17.946780Z","iopub.execute_input":"2023-05-10T13:12:17.947096Z","iopub.status.idle":"2023-05-10T13:14:05.919769Z","shell.execute_reply.started":"2023-05-10T13:12:17.947066Z","shell.execute_reply":"2023-05-10T13:14:05.918197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we have a dataframe where the first column contains the image location of the graph and the remaining columns have- the source of the graph(generated/extracted), the type as discussed above, the plot boundary, text(it has the coordinates, type of text and text for everything written on the graph), the axes column gives location of x and y ticks corresponding to text ids of tick type, visual elements correspond to coordinates of bars, line or scatter points. The data series columns represents the data given in the graph which we have to predict for the test data as our solution. \n\nIf all of this is not clear to you, no worries as we'll see it in the form of a dataframe and mark these features on a random graph.","metadata":{}},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:14:05.921929Z","iopub.execute_input":"2023-05-10T13:14:05.922483Z","iopub.status.idle":"2023-05-10T13:14:06.098524Z","shell.execute_reply.started":"2023-05-10T13:14:05.922437Z","shell.execute_reply":"2023-05-10T13:14:06.097394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img=mpimg.imread(df['image_loc'][100])\nimgplot = plt.imshow(img)\n\nh,w,x,y = df['plot-bb'][100].values()\ncv2.rectangle(img, pt1=(x,y), pt2=(w+x,h+y), color=(0,0,255), thickness=2)\n\nfor text in df['text'][100]:\n    polygon = text[\"polygon\"]\n    coord = dict(polygon.items())\n    point1 = (coord['x3'], coord['y3'])\n    point2 = (coord['x1'], coord['y1'])\n    cv2.rectangle(img, pt1=point1, pt2=point2, color=(255,0,0), thickness=2)\n\nfor tick in df['axes'][100]['x-axis']['ticks']:\n    cv2.circle(img, (tick['tick_pt']['x'], tick['tick_pt']['y']), radius = 2, color = (255,0, 255), thickness=3)\nfor tick in df['axes'][100]['y-axis']['ticks']:\n    cv2.circle(img, (tick['tick_pt']['x'], tick['tick_pt']['y']), radius = 2, color = (255,0, 255), thickness=3)\nplt.imshow(img)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:14:06.101154Z","iopub.execute_input":"2023-05-10T13:14:06.101961Z","iopub.status.idle":"2023-05-10T13:14:06.452335Z","shell.execute_reply.started":"2023-05-10T13:14:06.101928Z","shell.execute_reply":"2023-05-10T13:14:06.451453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus, we have successfully displayed the 'plot-bb' in blue, 'text' in red and 'axes' ticks in pink. We can show the same for every graph of our dataset. Till now, I hope everyone is clear about the data json file contains. All this data has been manually fed. We can understand how cumbersome it is to annotate everything manually. \n\nNow, we'll use image processing techniques to extract coordinates of objects from the image itself and try to verify it with the json data already provided to us. I'll start by predicting the plot boundaries using Open CV library(used for image processing and computer vision). ","metadata":{}},{"cell_type":"code","source":"def show_plot_bb(i):    \n    flag=0\n    rgb = cv2.imread(df['image_loc'][i])\n    small = cv2.cvtColor(rgb, cv2.COLOR_BGR2GRAY)\n\n    #threshold the image\n    _, bw = cv2.threshold(small, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)\n\n    kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 1))\n    connected = cv2.morphologyEx(bw, cv2.MORPH_CLOSE, kernel)\n\n    # find all the contours\n    contours, hierarchy,=cv2.findContours(connected.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)\n\n    for idx in range(len(contours)):\n        x, y, w, h = cv2.boundingRect(contours[idx])\n        if((x,y)==(df['plot-bb'][i]['x0'], df['plot-bb'][i]['y0'])):\n            cv2.rectangle(rgb, (x, y), (x+w-1, y+h-1), (0, 255, 0), 2)\n            flag = 1\n    plt.imshow(rgb)\n    return flag\n\nf = show_plot_bb(100)","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:14:06.453751Z","iopub.execute_input":"2023-05-10T13:14:06.454146Z","iopub.status.idle":"2023-05-10T13:14:06.754945Z","shell.execute_reply.started":"2023-05-10T13:14:06.454116Z","shell.execute_reply":"2023-05-10T13:14:06.753546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Here I have made a function that plots boundary on the image only if the coordinates match those given in the json file. I have used cv2 and morphological image processing to locate the boundary. Having done this image processing for plot boundaries, we still don't have very satisfactory results. I'll update this once I am able to do something better.","metadata":{}},{"cell_type":"markdown","source":"Now, let us expand our task at hand to axes ticks. Having worked with graphs, we know that x and y ticks are marked at an equal distance between them. Looking at our data, we observe that axes ticks are of two types - Categorical(for eg: {India, US, Brazil...} and Numerical(For eg: {10,20,30...}). For categorical data, equal distance between ticks does not matter much but for numerical data we have to be a bit more careful since they have to be interpolated to reach the desired data series. Therefore, before trying to locate axes ticks, let us check if our data is fault proof of any such thing.\n\nFirst I'll start by reviewing the x axis ticks.","metadata":{}},{"cell_type":"code","source":"i = 0\nreslt = []\nfalse_data = []\nfor file in df['axes']:\n    st = []\n    ff = json.dumps(file)\n    resp = json.loads(ff)\n    ini = 0\n    for x in resp['x-axis']['ticks']:\n        st = st + [x['tick_pt']['x']-ini]\n        ini = x['tick_pt']['x']\n    if(len(st)>0):\n        st.pop(0)\n    res = all(ele > statistics.median(st)-5 and ele < statistics.median(st)+5 for ele in st)\n    if(res!=True):\n        false_data.append(i)\n    reslt.append(res)\n    i=i+1\nCounter(reslt)","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:14:06.756247Z","iopub.execute_input":"2023-05-10T13:14:06.756579Z","iopub.status.idle":"2023-05-10T13:14:15.141865Z","shell.execute_reply.started":"2023-05-10T13:14:06.756552Z","shell.execute_reply":"2023-05-10T13:14:15.140571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The counter that we have as an output is true for those graphs which have an equal distance between ticks and false for the outliers. About 95% of the data is according to our assumption. We'll now check what category do the outliers belong to.","metadata":{}},{"cell_type":"code","source":"false = df.iloc[false_data].reset_index(drop=True)\nfig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 5))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,2):\n    for j in range(0,2):\n        xyz = random.choice(false['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:14:15.143752Z","iopub.execute_input":"2023-05-10T13:14:15.144254Z","iopub.status.idle":"2023-05-10T13:14:15.999350Z","shell.execute_reply.started":"2023-05-10T13:14:15.144213Z","shell.execute_reply":"2023-05-10T13:14:15.998041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay, so after running this code several times, we observe that the false data is of line graphs and vertical bar graphs majorly. Also the labels are categorical for most of the outlier graphs. Let us check if we are actually right...","metadata":{}},{"cell_type":"code","source":"print(false['chart-type'].value_counts()) #the observation is quite true\nlst = []\nol = []\nctr=0\nfor i in false['axes']:\n    lst.append(i['x-axis']['values-type'])\n    if(i['x-axis']['values-type'] == 'numerical'):\n        ol.append(ctr)\n    ctr=ctr+1\nCounter(lst)","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:23:26.302740Z","iopub.execute_input":"2023-05-10T13:23:26.303110Z","iopub.status.idle":"2023-05-10T13:23:26.323318Z","shell.execute_reply.started":"2023-05-10T13:23:26.303082Z","shell.execute_reply":"2023-05-10T13:23:26.322339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are right!!! The outlier graphs are majorly line and bar graphs. The data represented is categorical which is no problem. Our only concern are the 91 graphs that have numerical data and are not equally spaced. ","metadata":{}},{"cell_type":"code","source":"ol_data = false.iloc[ol].reset_index(drop=True)\nfig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 5))\nplt.subplots_adjust(wspace=0.05, hspace=0.25)\n\nfor i in range(0,2):\n    for j in range(0,2):\n        xyz = random.choice(ol_data['image_loc'])\n        img = mpimg.imread(xyz)\n        axes[i,j].imshow(img)\nprint(ol_data['chart-type'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:23:27.397423Z","iopub.execute_input":"2023-05-10T13:23:27.397792Z","iopub.status.idle":"2023-05-10T13:23:28.213235Z","shell.execute_reply.started":"2023-05-10T13:23:27.397763Z","shell.execute_reply":"2023-05-10T13:23:28.212227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Hence, the faulty graphs are scatter and dot plots. This is an abnormality we'll keep in mind while finding the solution for axes ticks.\n\nNow, I'll check for y axis ticks","metadata":{}},{"cell_type":"code","source":"i = 0\nreslt = []\nfalse_data = []\nfor file in df['axes']:\n    st = []\n    ff = json.dumps(file)\n    resp = json.loads(ff)\n    ini = 0\n    for y in resp['x-axis']['ticks']:\n        st = st + [y['tick_pt']['y']-ini]\n        ini = y['tick_pt']['y']\n    if(len(st)>0):\n        st.pop(0)\n    res = all(ele > statistics.median(st)-5 and ele < statistics.median(st)+5 for ele in st)\n    if(res!=True):\n        false_data.append(i)\n    reslt.append(res)\n    i=i+1\nCounter(reslt)","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:32:37.870955Z","iopub.execute_input":"2023-05-10T13:32:37.871377Z","iopub.status.idle":"2023-05-10T13:32:46.207411Z","shell.execute_reply.started":"2023-05-10T13:32:37.871344Z","shell.execute_reply":"2023-05-10T13:32:46.206240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> For the y axis we don't have any outliers as such because the 8 graphs belong to horizontal bar graphs which represent categorical data. Now, we can proceed further and locate axes ticks using image processing. ","metadata":{}},{"cell_type":"markdown","source":"Having detected outliers, which are almost negligible, we'll move on to predicting the co-ordinates of the axes ticks.","metadata":{}},{"cell_type":"code","source":"import cv2\nimport numpy as np\n\n# Read image\nimage = cv2.imread(df['image_loc'][20000])\n\n# Convert image to grayscale\ngray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)\n\n# Use canny edge detection\nedges = cv2.Canny(gray,50,150,apertureSize=3)\n# Apply HoughLinesP method to\n# to directly obtain line end points\nlines_list =[]\nlines = cv2.HoughLinesP(\n\t\t\tedges, # Input edge image\n\t\t\t1, # Distance resolution in pixels\n\t\t\tnp.pi/180, # Angle resolution in radians\n\t\t\tthreshold=100, # Min number of votes for valid line\n\t\t\tminLineLength=5, # Min allowed length of line\n\t\t\tmaxLineGap=10 # Max allowed gap between line for joining them\n\t\t\t)\n\n# Iterate over points\nfor points in lines:\n\t# Extracted points nested in the list\n\tx1,y1,x2,y2=points[0]\n\t# Draw the lines joing the points\n\t# On the original image\n\tcv2.line(image,(x1,y1),(x2,y2),(0,255,0),2)\n\t# Maintain a simples lookup list for points\n\tlines_list.append([(x1,y1),(x2,y2)])\n\t\n# Save the result image\nplt.imshow(image)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-10T13:55:56.707171Z","iopub.execute_input":"2023-05-10T13:55:56.707595Z","iopub.status.idle":"2023-05-10T13:55:57.058004Z","shell.execute_reply.started":"2023-05-10T13:55:56.707566Z","shell.execute_reply":"2023-05-10T13:55:57.055373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}