{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n#
Benetech | Simple OCR Network
\n\n\n### Table of Contents\n\nThe notebook will be divided into separate sections to provide an organized walk-through. The sections are:\n\n
\n
    \n
  1. Import Libraries
    2. \n
  2. Load Image
    3. \n
  3. Load Annotation
    4. \n
  4. Plot all annotation information from image
    5. \n
  5. Object Character Recognition
    6. \n
  6. Predict Bounding Boxes
    7. \n
\n
","metadata":{}},{"cell_type":"markdown","source":"# 1 | Import Libraries [↑](#top) \n***","metadata":{}},{"cell_type":"code","source":"import cv2\nimport easyocr\nimport json\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport numpy as np\n\nfrom pprint import pprint","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-22T20:28:53.919967Z","iopub.execute_input":"2023-03-22T20:28:53.920441Z","iopub.status.idle":"2023-03-22T20:28:53.927236Z","shell.execute_reply.started":"2023-03-22T20:28:53.920400Z","shell.execute_reply":"2023-03-22T20:28:53.925751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2 | Load Image [↑](#top) \n***\n\nYou can load an image by passing its path to `cv2.imread()` function","metadata":{}},{"cell_type":"code","source":"IMAGE_PATH = \"/kaggle/input/benetech-making-graphs-accessible/train/images/0000ae6cbdb1.jpg\"\n\nimage = cv2.imread(IMAGE_PATH)\nplt.imshow(image)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-22T20:10:11.641829Z","iopub.execute_input":"2023-03-22T20:10:11.642726Z","iopub.status.idle":"2023-03-22T20:10:11.913064Z","shell.execute_reply.started":"2023-03-22T20:10:11.642674Z","shell.execute_reply":"2023-03-22T20:10:11.911844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3 | Load Annotation [↑](#top) \n***\n\nWe can load the annotation with the `json` library by passing the annotation's path.\n\nThe annotation JSON has the following keys: `source`, `chart-type`, `plot-bb`, `text`, `axes`, `data-series`\n\nMore info about the data can be found in the competition's [data description](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/data)\n","metadata":{}},{"cell_type":"code","source":"ANNOTATION_PATH = \"/kaggle/input/benetech-making-graphs-accessible/train/annotations/0000ae6cbdb1.json\"\n\nwith open(ANNOTATION_PATH) as f:\n annotation = json.loads(f.read())\n \npprint(annotation)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2023-03-22T20:47:15.048225Z","iopub.execute_input":"2023-03-22T20:47:15.048767Z","iopub.status.idle":"2023-03-22T20:47:15.070050Z","shell.execute_reply.started":"2023-03-22T20:47:15.048727Z","shell.execute_reply":"2023-03-22T20:47:15.068414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4 | Plot all annotation information from image [↑](#top) \n***\n\nWe have information inside the annotations about the major components of the plots, like titles, axis labels, tick labels and their respective bounding boxes.\n\nYou can expand the following cell's code to get a better understanding of the different annotation components.\n\nLet's plot it!","metadata":{}},{"cell_type":"code","source":"color_dict ={\n \"chart_title\": \"r\",\n \"axis_title\": \"y\",\n \"tick_label\": \"c\"\n}\n\n\nfig, ax = plt.subplots()\nplt.rcParams[\"figure.figsize\"] = (20,15)\n\n# === Plot text bounding boxes ===\nfor i in range(len(annotation[\"text\"])):\n role = annotation[\"text\"][i][\"role\"]\n x0 = annotation[\"text\"][i][\"polygon\"][\"x0\"]\n y0 = annotation[\"text\"][i][\"polygon\"][\"y0\"]\n H = annotation[\"text\"][i][\"polygon\"][\"x1\"] - x0\n W = annotation[\"text\"][i][\"polygon\"][\"y2\"] - y0\n rect = patches.Rectangle((x0, y0), H, W,\n linewidth=1,\n edgecolor=color_dict[role], facecolor='none')\n ax.add_patch(rect) \n \n# === Plot axis ticks ===\nfor axis in annotation[\"axes\"].keys():\n for j in range(len(annotation[\"axes\"][axis][\"ticks\"])):\n tick_point = annotation[\"axes\"][axis][\"ticks\"][j]['tick_pt']\n plt.plot(tick_point[\"x\"],tick_point[\"y\"],'wx') \n \n# === Plot axis ticks ===\nfor axis in annotation[\"axes\"].keys():\n for j in range(len(annotation[\"axes\"][axis][\"ticks\"])):\n tick_point = annotation[\"axes\"][axis][\"ticks\"][j]['tick_pt']\n plt.plot(tick_point[\"x\"],tick_point[\"y\"],'wx') \n \n \n \nplot_bb = annotation[\"plot-bb\"]\nrect = patches.Rectangle((plot_bb[\"x0\"], plot_bb[\"y0\"]),\n plot_bb[\"width\"], plot_bb[\"height\"],\n linewidth=1, edgecolor='g', facecolor='none')\nax.add_patch(rect)\n# Display the image\nax.imshow(image)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-22T20:25:49.208343Z","iopub.execute_input":"2023-03-22T20:25:49.208787Z","iopub.status.idle":"2023-03-22T20:25:49.970984Z","shell.execute_reply.started":"2023-03-22T20:25:49.208753Z","shell.execute_reply":"2023-03-22T20:25:49.969739Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5 | Object Character Recognition [↑](#top) \n***\n\nObject Character Recognition (OCR) is a technology that allows a computer to recognize printed or handwritten text characters within an image or document and convert them into editable, searchable, and machine-readable text.\n\nOCR technology works by analyzing the shapes, patterns, and other features of the text characters in an image and then using machine learning algorithms to recognize and convert them into ASCII or Unicode characters. OCR can be used to extract text from a variety of sources, such as scanned documents, photos, or even videos.\n\nOCR is commonly used in many industries, such as finance, healthcare, and government, for tasks such as digitizing paper documents, automating data entry, and improving accessibility for visually impaired individuals. OCR technology has significantly improved over the years and is now capable of recognizing and translating text with high accuracy.","metadata":{}},{"cell_type":"code","source":"reader = easyocr.Reader(['en']) # initialize OCR\nresult = reader.readtext(image) # input image\n\nfor i in range(3):\n print(result[i], \"\\n\")","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2023-03-22T20:53:45.762636Z","iopub.execute_input":"2023-03-22T20:53:45.763080Z","iopub.status.idle":"2023-03-22T20:53:45.771091Z","shell.execute_reply.started":"2023-03-22T20:53:45.763045Z","shell.execute_reply":"2023-03-22T20:53:45.769234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see the OCR network gives us:\n1. **the text** it detects (**what**)\n2. **the bounding box** of the text (**where**)\n3. **the confidence score**, a probability between 0 and 1 of how confident is about the prediction (**how much**).\n\nYou can discard low confidence predictions.","metadata":{}},{"cell_type":"markdown","source":"# 6 | Predict Bounding Boxes [↑](#top) \n***\n\nYou can create a *section* of an image and predict only this portion instead of the whole image.\n\nFor example, you can train or fine-tune an Object Detection Neural Network like yolov5 and predict on the Neural Network's output bounding boxes to get the texts from the images.","metadata":{}},{"cell_type":"code","source":"with open(ANNOTATION_PATH) as f:\n annotation = json.loads(f.read())\n\nfor i in range(2):\n x0 = annotation[\"text\"][i][\"polygon\"][\"x0\"]\n y0 = annotation[\"text\"][i][\"polygon\"][\"y0\"]\n H = annotation[\"text\"][i][\"polygon\"][\"x1\"] - x0\n W = annotation[\"text\"][i][\"polygon\"][\"y2\"] - y0\n mask = image[y0:y0+W, x0:x0+H]\n\nreader = easyocr.Reader(['en']) # this needs to run only once to load the model into memory\nresult = reader.readtext(mask)\nresult","metadata":{"execution":{"iopub.status.busy":"2023-03-22T20:27:37.439528Z","iopub.execute_input":"2023-03-22T20:27:37.440651Z","iopub.status.idle":"2023-03-22T20:27:40.751354Z","shell.execute_reply.started":"2023-03-22T20:27:37.440603Z","shell.execute_reply":"2023-03-22T20:27:40.749929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**I hope you liked it!**","metadata":{}}]}