{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Leveraging a pre-trained Language Model to correct tick label predictions**\n\nThe **[Making Graphs Accessible](https://www.kaggle.com/competitions/benetech-making-graphs-accessible)** challenge is a very interesting puzzle 🧩 in which several pieces must come together to get to the endgame, and as such it allows to tie together machine learning techniques and methods pertaining to multiple modalities! 🖺🖊️🖼️📊📈\n\nIn this notebook I want to share an idea on how one could **improve the predictions for the tick labels** in the graphs, obtained for example by an OCR network, by **using a pretrained Language Model (LM)**, such as BERT. **BERT** (Bidirectional Encoder Representations from Transformers) is a pre-trained language model, based on the Transformer architecture, with context awareness. I am leaving some resources at the end of the notebook for anyone interested in knowing more.\n\n*→→ you can check out [this interesting notebook](https://www.kaggle.com/code/alejopaullier/benetech-simple-ocr-network) for more details on a simple OCR implementation!*.  \n\nLet's dive in! 🌊","metadata":{}},{"cell_type":"markdown","source":"# **Sample image x-axis tick labels**\nFor starters, let's load a sample image from the training set and print out the corresponding annotations. Let's consider for example the x-axis tick labels. The corresponding predictions for the x-axis tick labels will be the target that we'll try to improve with the pre-trained Language Model.","metadata":{}},{"cell_type":"code","source":"import cv2\nimport json\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom pprint import pprint\n\nsample_image = \"/kaggle/input/benetech-making-graphs-accessible/train/images/00c73795f795.jpg\"\nimage = cv2.imread(sample_image)\nsample_annotation = \"/kaggle/input/benetech-making-graphs-accessible/train/annotations/00c73795f795.json\"\n\nwith open(sample_annotation) as f:\n    annotation = json.loads(f.read())\n    \nplt.imshow(image)    \n\nx_axis_annotations = [annotation[\"data-series\"][i]['x'] for i in range(len(annotation[\"data-series\"]))]\nprint(\"\\n\", \"x-axis annotations: \", x_axis_annotations, \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-03-24T19:38:46.222579Z","iopub.execute_input":"2023-03-24T19:38:46.223092Z","iopub.status.idle":"2023-03-24T19:38:46.478180Z","shell.execute_reply.started":"2023-03-24T19:38:46.223050Z","shell.execute_reply":"2023-03-24T19:38:46.476720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Predict the tick labels using an OCR Network**\n\nBy feeding our image to an OCR (**Object Character Recognition**) network we get as an output not only the **text** recognized by the model, but also the various **boxes** bounds and the **confidence scores** associated to the predictions. We could make the life of the OCR easier, by separating the task of recognizing the boxes, using an Object Detection network in front of the OCR, but here I wanted to focus on the postprocesing of the predictions, so I kept this step as simple as possible.  \n","metadata":{}},{"cell_type":"code","source":"import easyocr\nimport pandas as pd\n\nreader = easyocr.Reader(['en']) \nocr_predictions = reader.readtext(image) \n\n# Uncomment the following line to see the whole OCR output\n# pprint(ocr_predictions) \n\n# Here I only extract the text and the confidence\ndf = pd.DataFrame([[ocr_predictions[i][1],ocr_predictions[i][2]]  for i in range(len(ocr_predictions))], \n                    columns=[\"Text predicted\", \"Confidence\"])\ndf.sort_values(by=[\"Confidence\"], ascending=False)\n","metadata":{"execution":{"iopub.status.busy":"2023-03-24T19:38:50.046452Z","iopub.execute_input":"2023-03-24T19:38:50.047003Z","iopub.status.idle":"2023-03-24T19:38:57.458598Z","shell.execute_reply.started":"2023-03-24T19:38:50.046955Z","shell.execute_reply":"2023-03-24T19:38:57.457168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see the accuracy in the tick label predictions is not that great, and the confidence is also rather low for several of them. \n\nSo let's see if we can improve this with the help of a spell checker model! 🚀\n\nI tried **NeuSpell** ([GitHub link](https://github.com/neuspell/neuspell)), which supports the use of multiple LM models, including BERT.","metadata":{}},{"cell_type":"markdown","source":"# **Get the Spell Checker up and running** 📎✔️","metadata":{}},{"cell_type":"markdown","source":"### **Install neuspell**","metadata":{}},{"cell_type":"code","source":"!git lfs install\n!git clone https://huggingface.co/pszemraj/neuspell-subwordbert-probwordnoise","metadata":{"execution":{"iopub.status.busy":"2023-03-24T19:26:37.804372Z","iopub.execute_input":"2023-03-24T19:26:37.805100Z","iopub.status.idle":"2023-03-24T19:28:22.001774Z","shell.execute_reply.started":"2023-03-24T19:26:37.805056Z","shell.execute_reply":"2023-03-24T19:28:21.994884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -U neuspell -q","metadata":{"execution":{"iopub.status.busy":"2023-03-24T19:28:27.614106Z","iopub.execute_input":"2023-03-24T19:28:27.615293Z","iopub.status.idle":"2023-03-24T19:28:51.987193Z","shell.execute_reply.started":"2023-03-24T19:28:27.615190Z","shell.execute_reply":"2023-03-24T19:28:51.985032Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Import and download the pre-trained model**","metadata":{}},{"cell_type":"code","source":"from neuspell import BertChecker\nchecker = BertChecker()\nchecker.from_pretrained(\"./neuspell-subwordbert-probwordnoise/\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-24T19:29:13.689248Z","iopub.execute_input":"2023-03-24T19:29:13.689832Z","iopub.status.idle":"2023-03-24T19:29:43.594165Z","shell.execute_reply.started":"2023-03-24T19:29:13.689788Z","shell.execute_reply":"2023-03-24T19:29:43.592567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Call the spell checker!**\n\nLet's feed the spell checker the list of faulty labels predicted by the OCR, to see if there is any improvement. \n\n\n*NOTE: for the sake of simplicity here I selected the range of OCR predictions related to the x-axis lables manually (11-24) but of course this step would need to be automated*","metadata":{}},{"cell_type":"code","source":"before_spell_check = [ocr_predictions[i][1]  for i in range(11,24)]\nprint(\"Before spell check: \", before_spell_check)\n\n# Call che spell checker\nafter_spell_check = checker.correct_strings([ocr_predictions[i][1] for i in range(11,24)])\nprint(\" After spell check: \", after_spell_check)","metadata":{"execution":{"iopub.status.busy":"2023-03-24T19:41:34.284256Z","iopub.execute_input":"2023-03-24T19:41:34.284720Z","iopub.status.idle":"2023-03-24T19:41:34.751323Z","shell.execute_reply.started":"2023-03-24T19:41:34.284683Z","shell.execute_reply":"2023-03-24T19:41:34.749901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Final result**\n\nIndeed **BERT corrected several things**:\n* Slavenia → Slovenia \n* Atrica → Africa\n* Sti Lanka → Sri Lanka\n* Tatwan → Taiwan\n* Tajkistan → Tajikistan\n* South Kor → South Korea\n\nThere is still plenty of work to be done, for example it could be helpful to perform some clean-up to **remove spurious characters** such as \"~\", \"/\", and \"1\", but overall I thought it was an interesting angle worth sharing.\n\nI hope you found this useful, and any feedback or question you may have is more than welcome! 🧐\n","metadata":{}},{"cell_type":"markdown","source":"### **BERT resources**\n1. The [original BERT paper](https://arxiv.org/pdf/1810.04805.pdf&usg=ALkJrhhzxlCL6yTht2BRmH9atgvKFxHsxQ): \"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding\" by Jacob Devlin et al. \n2. The [BERT GitHub repository](https://github.com/google-research/bert): it contains the code for pre-training and fine-tuning BERT, as well as many pre-trained models and examples of how to use them.\n3. The [BERT Explained series by Jay Alammar](https://jalammar.github.io/illustrated-bert/): if you are not familiar with his work, definitely check out any of the blog posts and videos that he has made on a variety of ML methods and architectures.","metadata":{}}]}