{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

PIX2STRUCT : A GENERAL PURPOSE GENERATIVE MODEL FOR VISUAL LANGUAGE UNDERSTANDING

","metadata":{}},{"cell_type":"markdown","source":"

A BRIEF ON GENERATIVE MODELS

\n
\nThough research into this field has been decades old, the AI world sat up and took notice with the advent of Generative Adversarial Networks(GANs).Then along came, the transformers which allowed for ever larger models to be developed, first focussing on handling text and then moving on to combine other modalities like vision-text, text-image, text-speech and so on.\n\nGenerative models gave the now famous (or infamous depending on how optimistic or pesimmistic one is!) ChatGPT and its variants, generally known as Large Language Models (LLMs). They are nowadays used to generate codes, for language translations, Image captioning, Image generation using text descriptions and for many more applications. New and exiting research is being published on this space at a breakneck speed.\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"
VISUAL LANGUAGE UNDERSTANDING
\n
    \nVisual language understanding (VLU) is the capacity to interpret the relationship between visual content (like images or videos) and its natural language annotations. The basic approach is to employ Computer Vision techniques to extract semantically meaningful information (embeddings) from the image/video and then condition the Natural Language Processing (NLP) techniques on these embeddings, to predict the textual content contained in the data or which is to be inferred from the data.\n    \nVarious objectives like Visual Question Answering (VQA), Image Captioning, Visual Storytelling etc. can be covered under this domain","metadata":{}},{"cell_type":"markdown","source":"
THE MOTIVATION BEHIND PIX2STRUCT
\n
    \nAuthors noted that previous research was usually focussed on how language and vision work separately in tasks like answering questions about images or predicting captions for images.  But in real-life situations, language and vision are closely intertwined, say when we read documents, look at tables, infographics, or use user interfaces, both text and images are consumed together without clear separations between them. Hence they concluded that focus rather should be on understanding how language and vision work together in situations where text and images interact seamlessly. Tackling this requires the ability to recognize text, understand language, and incorporate diverse visual context at the same time.\n    \nAlso previous attempts at decoding visually situated language relied on external and complex tools like OCR engines for document understanding, platform specific metadata for User-Input models etc., which were task specific and could not be scaled and generalized to other tasks. These methods also increased the complexity and computational cost.\n    \nThe development of an OCR-free, end-to-end model for document understanding (OCR-free Document Understanding Transformer (DONUT): https://arxiv.org/abs/2111.15664) was the spark which inspired their work, since DONUT was limited in its scope of understanding only text from the images and leaving out much of the rich knowledge that unsupervised data brought to the table.\n    \nPIX2STRUCT started off from where DONUT drew its line. The idea was to engineer a general purpose model which could be fine-tuned to various task specific domains without much of a fuss.","metadata":{}},{"cell_type":"markdown","source":"
 SO, WHAT IS PIX2STRUCT?
\n
 \nIt is a pretrained model which uses a vision-encoder, text-decoder architecture. The inputs to the model are raw pixel only images, which are then encoded into embeddings using Vision-Transformer(ViT) and the text-decoder is conditioned on these embeddings to generate sequence of tokens. We shall look into how the authors approached the problem to deliver an end-to-end general purpose model for visual language understanding in the following paragraphs:\n\n\nCHOICE OF DATA AND MODEL OBJECTIVE\n\nThe authors chose to pretrain the model on masked screenshots of web images, since the data is readily and abundantly available on the net. They formulated the objective of predicting the HTML parse from the masked images for the model training. They reasoned that HTML structure provides clean and structured information about the layouts, texts and images in the visual data and that masking out certain portion of the images helps the model predict the joint probability of their co-occurance.\n    \nIn order to balance the trade-off between decoder sequence-length and preserving semantics of web page image input, both the screenshots and its HTML ground truth were modified such that it resulted in dense learning of signals for the model during pretraining. This was accomplised by:  \n
  • Only HTML elements or descendants that contribute to the visible content of the webpage are retained
    •    \n
  • In the event an element without visible content has only a single child element, the child is replaced with a grand-child element
    • \n
\n    \n\n
Figure.1: Original web page (left) - Masked input- HTML parse output pairs (right) sampled from original
(Image by original authors)
\n    \n
The authors kept only the text, alt-text and filenames in each node. You can see it in the ground truth HTML parse in the output pair of the above figure. They note that further enrichment with element-tags, urls, titles and styles etc. is possible
\n    \nWHAT DOES THIS OBJECTIVE INTENDS TO ACHIEVE?\n
The authors cleverly crafted this objective to imbibe various facets of Visual Language Understanding into an umbrella model during pre-training, which can then be fine-tuned to further special tasks at hand. 
\n
  • Re-generating visible (unmasked) parts of web-page screenshot subsumes the role of Optical Character Recognition (OCR). Predicting C++ the unmasked portion of image in Figure.1 illustrates this.
    • \n
  • Predicting the masked parts of the image is akin to Masked Language Modelling (MLM). Predicting Python from masked portion of image in Figure.1 illustrates this point
    • \n
  • Recovering the alt-text of the images attends to image-captioning and is a common pretraining strategy for the same. Predicting img alt=C++ illustrates this learning signal.
    •  
\n\n\nARCHITECTURE\n\nPix2Struct uses an Encoder-Decoder architecture. The traditional Vision-Transformer (ViT) acts as the Vision Encoder. But ViT pre-processes the image to fixed resolution/size (usually 224 x 224) before extracting fixed-size patches and messes up the aspect-ratio of the images and also rendering the leverage of model on tasks employing higher resolution images useless. \n\nHerein comes the secret sauce employed by Pix2Struct for its Vision-Encoder:
    \n
  • Rescaling the image flexibly to extract maximal number of fixed size patches and more critically
    • \n
  • Use 2-dimensional absolute positional embeddings  for input patches.
    • \n\n
    Figure.2: Illustration of patch extraction for target sequence length of 36 patches using Pix2Struct (left) & ViT (right).
    (Image by original authors)
    \nAnd the magic this approach delivers is:\n\n
  • Model is able to handle extreme aspect-ratios,
    • \n
  • Making changes to sequence length and resolution is possible on the go.
    • \n","metadata":{}},{"cell_type":"markdown","source":"\n
    TRAINING
    \n
    \n\nCURRICULUM WARM-UP\n\nRather than diving straightforward into training the model on screenshot parsing, authors drew inspiration from curriculum learning where the model is exposed to easy examples of a task and only then on the actual pre-training task. This led to faster and stabler convergence during pre-training. A nice explanation of curriculum learning can be found in this article \n\nFor this \"warm-up\" stage as the authors like to call it, they used images with random and uniformly sampled fonts in Google-Fonts of various sizes ranging from 12pt to 36pt with random colors - and corresponding ground truth\n\n\n
    Figure.3: Image-Ground Truth pairs during Warm-Up
    (Image by original authors)
    \n\nPRE-TRAINING\n\nPre-training continues with the sceenshot parsing objective mentioned above. To achieve reduced sequence length of decoder output, largest portion of ground-truth which fits within a pre-defined sequence length is chosen, and corresponding bounding-box is drawn on the image-pair. Also 50% of text in the images is masked-out randomly.\n\nTwo model variants, Base (262 Million parameters & 768-dimensional hidden state) and Large (1.3 Billion parameters & 1536-dimensional hidden state) were pretrained, with a decoder sequence length of 128 tokens\n\n\n
    Figure.4: Bounding-Box on Image & corresponding Ground Truth pair
    (Image by original authors)
    \n\nFINE TUNING ON DOWNSTREAM DATASETS\n\nFine-tuning on downstream datasets in the sub-domains of Visual Language Understanding was pretty much only a matter of curating the image-text pairs to unequivocally reflect the task at hand. The Datasets used were: ChartQA, AI2D, OCR-VQA, RefExp, Widget Captioning, Screen2Words, TextCaps, DocVQA, InfographicVQA across sub-domains of Illustration Understanding, User-Interface Understanding, Document Understanding and Natural-Image Understanding. \n\nTuning on Image Captioning tasks were easist and most straightforward with image-text pairs. For tasks of Visual Question Answering (VQA) in datasets like ChartQA, DocVQA etc, the question to be answerd was directly rendered on top of the input images, which the model consumes via the Image encoder.\n\nResults on the fine-tuned models with a comparison to State of the Art (Sota) models is provided in the below table by the authors. The Pix2Stuct model outperforms visul only models like GIT2 and DONUT on 8 out of the 9 datasets and Sota models on 6 out of 9.\n\n\n
    Figure.5: Pix2Struct Perfomance compared to other models on 9 benchmark Datasets
    \n\n\nCHALLENGES\n\nThe input image resoluction plays a vital role in the performance of Pix2Struct in Pre-trainig as well as Fine-tuning. It requires a good resolution to perform well, but this same acts as an Achilles Heel for the model as very high resolutions translate to higher sequence length which it may not be able to handle - and this translates to sub-par performance compared to OCR pipelines.\n\nBut it is hoped that developments in Long-range Transformers might enable pixel-only models catch up with OCR based models.","metadata":{}},{"cell_type":"markdown","source":"
    DOWNSTREAM MODELS USING PIX2STRUCT BACKBONE
    \n
    \n\nMATCHA (Math reasoning and Chart derendering pretraining)\n\nMATCHA uses Pix2Struct backbone with the twin objectives of: 
    • Derendering the underlying data-table and and the matplotlib/seaborn code that is used to generate the figures of Chart images during pretraining.
    • Predicting the answer to a math-reasoning question rendered as an image.
    \n\n\nDePlot\n\nDePlot uses Pix2Struct backbone with weights of the MATCHA model to further fine-tuning the model to predict the underlying table of plots in a linear table format and then feeding this prediction to a Large Language Model(LLM) like FlanPaLM and use its One-Shot capabilites to reason over the question asked regarding the plot. \n\nIf you wish to see the capabilities of Deplot, follow this notebook DePlot - Trying Out by Masaki Gotou ","metadata":{}},{"cell_type":"markdown","source":"
    USAGE IN KAGGLE COMPETITIONS
    \n
    \n\nBenetech - Making Graphs Accessible\n\nIn the recently concluded Benetech - Making Graphs Accessible Competition, models using MATCHA and DEPLOT variants of Pix2Struct secured some top positions, beating even object-detection and OCR based models, either on their own (2nd place Solution) or with object-detection model ensembles (1st place Solution, 3rd place Solution).   ","metadata":{}},{"cell_type":"markdown","source":"
    EXAMPLE NOTEBOOKS FOR TRAINING PIX2STRUCT/ VARIANTS ON CUSTOM DATASETS
    \n
    \n\nVarious checkpoints of Pix2Struct and its variants is available on the HuggingFace Hub. Though the training is resource heavy and more so with higher max_patches (2048, 4096), one could use half precision & gradient checkpointing. Below notebooks serve as nice examples of fine-tuning on custom datasets: 
    1. Training Pix2Struct PyTorch + AMP + W&B on Benetech Competition Data by Tanay Mehta.
    2. Fine-tune Pix2Struct using Hugging Face transformers and datasets  by Younes Belkada
    3. Fine_tune_Pix2Struct_on_key_value_pair_dataset_(PyTorch_Lightning) by Niels Rogge
      4. ","metadata":{}},{"cell_type":"markdown","source":"
      LINKS AND CITATIONS
      \n
      \n\n
      1. Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
      2. OCR-free Document Understanding Transformer
      3. How to Improve Your Network Performance by Using Curriculum Learning
      4. DePlot: One-shot visual language reasoning by plot-to-table translation
      5. MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
        6. ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nsubmission=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.loc[0]['value']='Generative AI'\nsubmission.loc[1]['value']='https://www.kaggle.com/code/sandeepmnair/generative-model-for-visual-language-understanding/'\nsubmission.loc[2]['value']='https://www.kaggle.com/code/chinmaypagey/generative-ai-report-by-team-sussy-baka/comments#2345611'\nsubmission.loc[3]['value']='https://www.kaggle.com/code/kkiryll/generative-ai-rise-of-the-transformers/comments#2346804'\nsubmission.loc[4]['value']='https://www.kaggle.com/code/rizvanam/generative-ai-evolution-and-powers/comments#2346806'\nsubmission.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=False)","metadata":{},"execution_count":null,"outputs":[]}]}