{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Table of contents\n1. [Introduction](#intro)\n2. [Transformer](#transformer)\n3. [Vision Transformer](#CV)\n4. [Video Transformer](#vid)\n5. [LLM Transformer](#LLM)\n6. [Graph Transformer](#graph)\n7. [Multimodal: Whole image segmentation](#ssg)\n8. [Limitations and perspectives ](#lim)\n9. [Conclusion](#emojis)\n10. [References](#refs) \n11. [Code](#code) ","metadata":{}},{"cell_type":"markdown","source":"# 1. Introduction [↑](#top)","metadata":{}},{"cell_type":"markdown","source":"The field of artificial intelligence has witnessed remarkable\nadvancements in recent years, revolutionizing various domains and\nreshaping the way we perceive technology. One breakthrough that has\ngarnered significant attention and admiration is the introduction of\ntransformer models. Originally proposed in 2017 by [Vaswani et\nal. 2017](#vaswani2017attention), transformers have quickly become a dominant force in natural language processing (NLP) tasks and\nhave extended their influence to a wide range of applications.\n\n\nHistorically, sequential models, such as recurrent neural networks\n(RNNs) and their variants, were widely used for NLP tasks. However,\nthese models suffered from limitations such as difficulty in\nparallelization and capturing long-range dependencies effectively. This\nmotivated the development of the transformer architecture, which\naddressed these challenges and introduced a new paradigm for modeling\nsequential data. More over, **transformers can be used for both generative (learn the distribution of data) and discriminative (classify, predict or cluster) tasks in artificial intelligence (AI)**.\n\nTransformers and its variants are playing an important role in various area including Natural Language Processing, Computer Vision (CV), Times-series Analysis, Audio Processing, etc. \nThe transformer model introduced a **self-attention mechanism**, which\nallows it to capture dependencies between different positions in a\nsequence more effectively. By attending to relevant information across\nthe entire input sequence, transformers can incorporate global context\nand consider long-term dependencies, enabling them to outperform\ntraditional sequential models in various NLP tasks.\n\nSince their inception, transformers have shown remarkable performance in\nnumerous domains. In addition to NLP, they have been successfully\napplied to computer vision, speech recognition, recommendation systems,\nand many other areas. The inherent ability of transformers to capture\ncontextual information and learn hierarchical representations makes them\nhighly versatile and adaptable to different problem domains.\n\nThis paper aims to explore the applications of transformers across\nvarious domains, highlighting their contributions and advancements in\neach area. By examining the transformation brought about by these\nmodels, we aim to shed light on the potential of transformers and their\nimpact on the future of artificial intelligence. Through a comprehensive\nanalysis of their applications, we can gain insights into the strengths,\nlimitations, and future directions of transformer-based approaches in\ndifferent domains. \n\n
\n \"Overview\"\n

Overview of this note on transformers. In blue, sections that will be dealt with in the following sections.

\n
\n","metadata":{}},{"cell_type":"markdown","source":"# 2. Transformer [↑](#top)","metadata":{}},{"cell_type":"markdown","source":"## Original Transformer\nA transformer is an encoder-decoder model that use the attention mechanism. The origninal model architecture is propose by Google in 2017 [Vaswani et al. 2017](#vaswani2017attention).\n\n
\n \"transformer\n

The transformer model architecture - Image extract from the original publication [Vaswani et al. 2017].

\n
\n\n### Positional Encoding\n\nOne of the key components of the transformer model is the inclusion of\npositional encoding. Traditional sequential models, such as RNNs,\ninherently capture the temporal order of the input sequence, but\ntransformers lack this inherent notion of order. To address this,\ntransformers incorporate positional encoding to provide positional\ninformation to the model.\n\nPositional encoding assigns unique representations to each element in\nthe input sequence based on its position. This encoding is added to the\ninput embeddings and allows the transformer model to distinguish between\ndifferent elements and their relative positions. By incorporating\npositional information, the model can effectively capture the sequential\nnature of the data, enabling it to learn dependencies and patterns\nwithin the sequence.\n\nThere are various approaches to implementing position encoding in\ntransformers. The most common method is to use sine and cosine functions\nof different frequencies to encode the position information. These\nsinusoidal functions are added to the input embeddings, resulting in a\ncombined representation that retains both the semantic information and\nthe positional information.\n\nPosition encoding plays a vital role in the transformer model's ability\nto understand the order and context of the input sequence, contributing\nto its impressive performance in capturing long-range dependencies and\ncontextual relationships between elements.\n\n### Attention Mechanism\n\nThe attention mechanism is a fundamental building block of transformer\nmodels and is responsible for capturing the relationships between\ndifferent positions in the input sequence. It allows the model to focus\non relevant information and weigh the importance of different parts of\nthe sequence when making predictions.\n\nThe attention mechanism operates by calculating attention scores between\neach pair of positions in the sequence. These scores reflect the\nimportance or relevance of one position to another. The scores are\ncomputed using a compatibility function, which compares the\nrepresentations of the positions, and then transformed into attention\nweights through a softmax operation. The attention weights determine the\namount of attention or focus given to each position.\n\nTransformers typically employ a self-attention mechanism, where the\ninput sequence is attended to itself. This means that each position in\nthe sequence can attend to all other positions, capturing both local and\nglobal dependencies. Self-attention enables the model to dynamically\nweigh the importance of different positions based on their relevance to\nthe current position, allowing it to incorporate global context\nefficiently.\n\nThe attention mechanism has been a game-changer in NLP tasks, as it\nprovides a more flexible and powerful way of capturing dependencies\ncompared to traditional fixed-window or fixed-length approaches. By\nattending to relevant information across the sequence, transformers can\ncapture long-range dependencies and effectively model complex linguistic\npatterns, leading to superior performance in various language-related\ntasks.\n\nA lot of good publication such as [Lin 2022](#lin2022survey) ond [Liu 2023](#liu2023survey) give a clear description of the mathematical concept of Attention Mechanism applied on single head or on multi-head. In a few word, it is a specific combinaison of queeries, keys and values that provides a valuable description of the input. The paper [Guo 2022](#guo2022attention) also provide a comprehensive survey of attention mechanisms in computer vision.\n\n### Architecture\nWe can distinguish transformers depending on its architecture[Lin 2022]:\n* encoder-decoder (BART, T5)\n* decoler only (GPT Family)\n* encoler only (BERT, RoBerta)\n\n## Transformer variant\n\nThe survey [Lin 2022](#lin2022survey) provide a conprehensive overview of existing categorise Transformer variants (see image below).\n\n
\n \"Transformer\n

Overview of existing categorise Transformer variants - Image extract from the original publication [Lin 2022].

\n
\n\nThe main concern focusses on the three following parts:\n* Embedding\n* Tokenization\n* Positional embeddings\n\n\n## Initial application domains\n\nThe field of natural language processing (NLP) has greatly benefited\nfrom the advancements of transformer models and their key components\nsuch as attention and self-attention mechanisms. These models have\nsignificantly impacted various NLP tasks, including language\ntranslation, text classification and segmentation, question answering,\ntext summarization, text generation, natural language reasoning, and\nautomated symbolic reasoning.\n\nTransformers have demonstrated remarkable capabilities in language\ntranslation, revolutionizing the way we bridge the gap between different\nlanguages. By leveraging their attention mechanisms and self-attention\ncapabilities, transformers excel at capturing complex linguistic\npatterns and context, enabling accurate and fluent translations.\n\nIn **text classification and segmentation**, transformers have proven to be\nhighly effective. With their ability to understand and process textual\ndata in a hierarchical manner, they can classify documents, categorize\ntexts, and extract meaningful segments with great accuracy and\nefficiency.\n\n**Question answering** systems have also benefited from transformers'\nadvancements. Through their attention mechanisms and self-attention\nmechanisms, transformers can comprehend and analyze large amounts of\ntext, enabling them to provide accurate and relevant answers to user\nqueries.\n\n**Text summarization**, a task aimed at condensing lengthy documents into\nconcise summaries, has also witnessed significant improvements with\ntransformer models. Their ability to capture context, extract salient\ninformation, and generate coherent summaries has propelled the field\nforward.\n\n**Text generation**, including tasks such as language modeling and creative\nwriting, has seen a transformative shift with the introduction of\ntransformers. These models excel at generating coherent and contextually\nappropriate text, making them highly valuable in applications like\nchatbots, dialogue systems, and content generation.\n\n**Natural language reasoning and automated symbolic reasoning** have also\nbenefited from transformer models. By capturing complex linguistic\nrelationships and contextual dependencies, transformers can reason and\nmake inferences, enabling advanced natural language understanding and\nreasoning systems.\n\nMoving beyond natural language processing, transformers have made\nnotable contributions to computer vision tasks as well. By leveraging\ntheir attention mechanisms and self-attention capabilities, transformers\nhave demonstrated superior performance in **image classification,\nrecognition, object detection, segmentation, and generation**. They have\nbeen successfully applied to diverse domains, including natural images,\nmedical images, and biomedical data, significantly advancing the field\nof computer vision.\n\nFurthermore, transformers have extended their capabilities to\nmulti-modal applications, bridging the gap between text, image, video,\nand audio data. By integrating vision and language modalities,\ntransformers enable tasks like visual question answering, visual\ncaptioning, visual common-sense reasoning, and multi-modal generation.\n\nIn the domain of audio and speech processing, transformers have made\nsignificant strides. Their attention mechanisms and self-attention\ncapabilities have enhanced tasks such as audio and speech recognition,\nseparation, classification, and signal processing, improving the\naccuracy and efficiency of these systems.\n\nThe application of transformers is not limited to language and vision\nbut extends to fields like wireless network signal processing and\nmedical signal processing. Transformers have shown promising results in\nanalyzing and processing signals, including wireless signals,\nbiosignals, and medical signals.\n\n## Overview\nWe won't be able to provide as complete state-of-the-art as huggingface has done in this [page](https://huggingface.co/docs/transformers/index). They give more than 200 references on transformers and analyse the supported frameworks. PyTorch is supported for all but three.","metadata":{}},{"cell_type":"markdown","source":"# 3. Vision Transformers [↑](#top)","metadata":{}},{"cell_type":"markdown","source":"## ViT\nAdapting transformers to vision tasks requires building an effective image-to-vector transformation and maintaining image characteristics. The **original Vision Transformer (ViT)** [Dosovitskiy 2020](#dosovitskiy2020) use multi-head self-attention. It is proposed to cut the input image into fixed-size non-overlapping patches which are feed to the encoder. It was the first of a long list and various vision transformers.\n\n
\n \"ViT\n

ViT architecture [Dosovitskiy 2020].

\n
\n\n## STViT\nThe paper titled \"Vision Transformer with Super Token Sampling\", [Huang 2023](#huang2023vision), introduces a method called **STViT** that addresses the issue of redundant information in images. STViT groups similar parts of an image into \"super tokens\" based on superpixels, reducing redundancy and improving efficiency. It utilizes a technique called \"super token attention\" to analyze the important super tokens, focusing on global dependencies and relationships in the image. The paper demonstrates the effectiveness of STViT in various computer vision tasks such as image classification, object detection, instance segmentation, and semantic segmentation.\n\n## Existing surveys\nThe paper [Khan 2022](#khan2022transformers), \"Transformers in Vision: A Survey\" provides an overview of the application of Transformer models in computer vision. It explains the key concepts behind the success of Transformers, covers various vision tasks where Transformers have been applied, and compares different techniques in terms of their architectural designs and experimental value. The survey aims to inspire further research and development in applying Transformer models to solve challenges in computer vision.\n\n
\n \"2022_Khan_Taxonomy.png\"\n

Taxonomy of self-attention mechanism in Computer Vision - Image extract from the original publication [Khan 2022].

\n
\n\nIn \"A survey of visual transformers\" from [Liu 2023](#liu2023survey), the authors propose a taxonomy of visual transformers based on different computer vision tasks such as classification, detection, segmentation, 3D visual recognition, and multi-sensory data stream. They have summarized recent improvements in different tasks:\n* For classification: Using a deep hierarchical Transformer backbone reduces computational complexity and avoids over-smoothing. Early-stage convolution captures low-level features effectively. Convolutional projection and local attention mechanisms improve locality.\n* For detection: Transformer necks benefit from an encoder-decoder structure with less computation. Spatial priors and techniques like sparse attention and scoring network accelerate convergence and reduce computational costs.\n* For segmentation: Encoder-decoder Transformer models unify segmentation tasks. Box-free approaches using learnable mask embeddings achieve state-of-the-art performance. Hybrid models combining box-based Transformers also show better results for instance segmentation.\n* For 3D visual recognition: Local hierarchical Transformers with scoring networks efficiently extract features from point clouds. Transformers handle multi-sensory data in 3D visual recognition, such as multi-view and multi-dimensional data.\n* For visual-linguistic pretraining: Approaches focus on alignments or similarities among different data streams in latent space. Adapting downstream tasks to pre-training schemes enables zero-shot transferring.\n\nMoreover, the relationship between transformers and convolutional neural networks (CNNs) is discussed, and highlight the advantages and limitations of visual transformers. \n\nThis two papers [Khan 2022](#khan2022transformers) and [Liu 2023](#liu2023survey) highlight the advantages of Transformers, such as their ability to capture long-range dependencies, support for parallel processing, and scalability to large networks and datasets. Both papers provide an overview of different vision tasks where Transformers have been successful, including image classification, object detection, segmentation, and 3D visual recognition.\n\n\nAs other Transformers, Vision Transformers can be classified by their architecture: **encoder only, decoder only, encoder-decoder**. The following picture shows the timeline overvier of vision transformers [Huggingface](https://huggingface.co/docs/transformers/model_summary).\n\n
\n \"HuggingFaceCVModels\"\n

Timeline overvier of vision transformers. Original image from [Huggingface] [Update 2.2023]

\n
\n \n\n## Relationship between Transformers, Self-Attention, and CNNs\nTransformers and CNNs have different strengths. Transformers have powerful global modeling capabilities, while CNNs excel at processing low-level features. Transformers can express any convolution operation, but CNNs have stronger inductive bias for low-level semantic processing. [Liu 2023](#liu2023survey).\n\nA discussion on the comparison between Transformers and CNN is provided [kaggle discussion link](https://www.kaggle.com/competitions/petfinder-pawpularity-score/discussion/275278) [Awsaf](#petfinder).\n\nAttention has been shown to effectively handle long-range dependencies and even surpass convolution in achieving state-of-the-art performance on vision tasks. This has prompted investigations into the operational similarities between learned attention layers and convolutional layers. Evidence from the study of [Cordonnier 2019](#cordonnier2019relationship) indicates that attention layers can perform convolution and often exhibit this behavior in practice. Numerical experiments confirm that self-attention layers attend to pixel-grid patterns similarly to convolutional layers, supporting the analysis. Similarly, another study [Raghu 2021](#raghu2021vision) highlights the remarkable differences between Vision Transformer models (ViTs) and convolutional networks (CNNs) in terms of internal representation structure, uniformity, and information aggregation. The crucial roles of self-attention and residual connections in ViTs are identified, along with their impact on spatial localization and preservation of input spatial information. Additionally, the effects of dataset scale and transfer learning are explored, concluding with insights into connections with emerging architectures like the MLP-Mixer.\n\nThe deep learning revolution, with its diverse architectures and integration strategies, has paved the way for sophisticated and intelligent computer vision systems, transforming industries and opening up new possibilities for applications in various domains.","metadata":{}},{"cell_type":"markdown","source":"\n# 4. Video Transformers [↑](#top)\nConcerning video transformers, the paper \"UniFormer\" from [Li 2022](#li2022uniformer) propose to integrate 3D Convolution and Spatiotemporal Self-Attention and so addresses the challenges of learning rich spatiotemporal semantics from high-dimensional videos. Existing approaches, such as 3D convolutional neural networks and vision transformers, have limitations in capturing both local redundancy and global dependency. UniFormer proposes a unified architecture that seamlessly integrates the benefits of 3D convolution and spatiotemporal self-attention in a concise transformer format. It introduces a relation aggregator that learns local and global token affinity in shallow and deep layers, respectively. UniFormer achieves impressive results on popular video benchmarks, including Kinetics-400, Kinetics-600, and Something-Something V1&V2. With only ImageNet-1K pretraining, UniFormer achieves top-1 accuracy of 82.9% on Kinetics-400 and 84.8% on Kinetics-600 while requiring 10 times fewer GFLOPs compared to other state-of-the-art methods. It also sets new state-of-the-art performances of 60.9% and 71.2% top-1 accuracy on Something-Something V1 and V2, respectively.\n\n
\n \"li2022uniformer\"\n

UniFormer architecture - Image extract from the original publication [Li 2022].

\n
\n\n\nThe survay [Selva 2023](#selva2023video) provide an analyse on how to handle video as input, study architecture change to make transformers more efficiency but also how to train it in various regime. The following picture extract from [Selva 2023], show an overview of the input pre-processing step, showing tokenization\nand embedding strategies, as well as positioning.\n\n\n
\n \"2023_Selva_overviewVidTrans.png\"\n

Overview of the input pre-processing step, showing tokenization\nand embedding strategies, as well as positioning - Image extract from the original publication [Selva 2023].

\n
\n \n","metadata":{}},{"cell_type":"markdown","source":"# 5. Transformers in LLM [↑](#top)","metadata":{}},{"cell_type":"markdown","source":"## Advancements in Language Model Editing and Reasoning Abilities\n\nLanguage models have demonstrated impressive capabilities in various\nnatural language processing tasks. However, even the largest pre-trained\nmodels are not immune to errors, limitations, and the potential\nobsolescence of their predictions. To address these challenges,\nresearchers have explored novel approaches to model editing and\nreasoning abilities within large language models. This mini report\npresents the advancements in research related to Language Model Editing\nand Reasoning Abilities, based on three significant papers: \\\"Model\nEditor Networks using Gradient Decomposition (MEND) [Mitchell et al. 2021](#mitchell2021fast),\\\" \\\"Semi-Parametric\nEditing with a Retrieval-Augmented Counterfactual Model (SERAC) [Mitchell et al. 2022](#mitchell2022memory),\\\" and\n\\\"Chain of Thought Prompting for Complex Reasoning in Language Models.\\\"\n\n## Model Editor Networks using Gradient Decomposition (MEND)\n\nThe MEND framework proposed by the authors tackles the challenge of\nenabling post-hoc editing of large pre-trained models while maintaining\ntheir overall functionality. MEND achieves this by employing small\nauxiliary editing networks that make local edits to a model's behavior.\nThe key innovation lies in transforming the gradients obtained from\nstandard fine-tuning, using a low-rank decomposition approach. This\ndecomposition makes the parameterization of the transformation\ntractable, enabling efficient training and application of edits. The\nexperiments with T5, GPT, BERT, and BART models demonstrate that MEND is\neffective in editing models with over 10 billion parameters.\n\n## Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC)\n\nSERAC addresses the limitations of existing model editing approaches by\nintroducing a higher-capacity alternative. It stores edits in explicit\nmemory and leverages this memory to reason and modulate the predictions\nof the base model. By reasoning over the stored edits, SERAC achieves\nbetter expressiveness and accuracy in modulating model behavior. The\nresearchers evaluate SERAC on challenging language model editing\nproblems in question answering, fact-checking, and dialogue generation.\nThe results show that SERAC consistently outperforms existing methods,\ndemonstrating its superior performance in model editing tasks.\n\n## Chain of Thought Prompting for Complex Reasoning in Language Models:\n\nThis paper explores the concept of generating a chain of thought, which\nrepresents a series of intermediate reasoning steps, to enhance the\nreasoning capabilities of large language models. Through a method called\nchain of thought prompting, the researchers provide a few exemplars that\ndemonstrate the desired reasoning process. The experiments conducted on\nthree large language models showcase that chain of thought prompting\nsignificantly improves performance across arithmetic, commonsense, and\nsymbolic reasoning tasks. Notably, even a 540B-parameter language model\nprompted with a few exemplars achieves state-of-the-art accuracy on\nchallenging math word problems.\n\n## Conclusion\n\nThe advancements in research on Language Model Editing and Reasoning\nAbilities presented in the three papers demonstrate the continuous\nefforts to enhance the capabilities of large language models. MEND and\nSERAC provide innovative approaches to enable post-hoc model editing,\noffering tractable and expressive methods to correct errors and update\nmodel behavior. Additionally, the concept of chain of thought prompting\nunlocks complex reasoning abilities within language models, improving\ntheir performance across a wide range of reasoning tasks. These\nadvancements contribute to the development of more reliable and\nadaptable language models, offering potential benefits for various\ndownstream applications.","metadata":{}},{"cell_type":"markdown","source":"# 6. Graph Transformer (GT) [↑](#top)","metadata":{}},{"cell_type":"markdown","source":"Like other methods, graphs are part of Transformer architectures. [Muller et al. 2023](#muller2023attending) The authors propose a toxonomy of graph transformer architectures and give the theoritical background needed to understand deeply this approach. They explored the theoretical properties of graph transformers and how they encode information about the structure and position of graphs. Currently, it's mostly use in 3D molecular property prediction. Experimental studies with increasing difficulty are proposed for edges detection, triangles counting and distinguish circular skip links. There are still several challenges to address both theoretically and practically. On the theoretical side, there is a lack of understanding about the properties of structural and positional encodings and when certain encodings are beneficial. On the practical side, one major limitation of GTs is their quadratic running time wich limit adaptation on real-word tasks where we commonly encountered.\n\n
\n \"ref_muller2023attending\"\n

Figure extract from [Muller et al. 2023].

\n
\n\nIn this context of GTs, common questions remain:\n* How to improve positional embedding?\n* How to improve Attention Matrices?\n\nA first answer it proposed in the overview [Min 2022](#min2022).","metadata":{}},{"cell_type":"markdown","source":"# 7. Multimodal: Whole image segmentation [↑](#top)\n\nIn the field of segmentation, the current state of the art (2023) mainly consists of offereing several types of prompts to calculate a segmentation as close as possible to the idea desired by the user. To achieve this, it is possible to give visual indications such as point, line, box but also to input information in other modalities such as audio or text. The two following papers are **multimodal-prompts and based on a simple Transformer encoder-decoder architexture with an additional encoder for the text**. Both of them combine cross attention and self attention to transfert historical information to the current state. Where SAM produce a segmentation, SEEM also provide labels for a semantic segmentation.\n
\n \"History\"\n

SAM and SEEM comparison- original image from [Zou 2023]

\n
\n\n\n## 7.1 SAM - Segment Anything Model [Kirillov 2023](#kirillov2023segment):\n\n* webpage: https://segment-anything.com\n* github: https://github.com/facebookresearch/segment-anything\n \n
\n \"History\"\n

SAM - Segment Anything Model architecture [Kirillov 2023]

\n
\n\n## 7.2 SEEM - Segment Everything Evrywhere Model [Zou 2023](#zou2023segment):\n\n* webpage: https://huggingface.co/spaces/xdecoder/SEEM\n* github: https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once\n\n
\n \"History\"\n

SEEM - Segment Everything Evrywhere Model architecture [Zou 2023]

\n
\n","metadata":{}},{"cell_type":"markdown","source":"# 8. Limitations and perspectives [↑](#top)\nWhile deep learning architectures like convolutional neural networks (CNNs), encoder-decoder models, and transformers have achieved remarkable success in computer vision, they do come with certain limitations. Here are some common limitations and the related perspective associated with these architectures:\n\n1. **High computational requirements**: Deep learning architectures often require significant computational resources, including powerful hardware and substantial memory capacity especially for complex architectures or large-scale datasets. Training and deploying these models can be computationally intensive, limiting their accessibility and scalability for certain applications. Moreover, fine-tuning or adapting these models to new tasks or domains may also require extensive training or transfer learning techniques. → Some researchers are exploring techniques to optimize and accelerate deep learning computations, such as model compression, quantization, and efficient hardware implementations. Techniques such as transfer learning and pre-training on large-scale datasets have been successful in reducing the need for extensive training. Few-shot and meta-learning approaches are being investigated to enable models to quickly adapt and learn from limited data. Additionally, advancements in cloud computing and distributed systems offer opportunities to alleviate the computational burden. In [Khan 2022](#khan2022), the authors focus mainly on all the same challenges, and a interesting paragraphe highlight high computational requirements of big models.\n\n2. **Large dataset requirements**: Deep learning models typically require large labeled datasets for training. Acquiring and annotating such datasets can be time-consuming and expensive, especially for specialized domains or niche applications. Large-scale dataset also rise ethical and societal questions [Prabhu 2020](#prabhu2020). → Transfer learning and domain adaptation techniques aim to address the limited availability of labeled data by leveraging pre-trained models or utilizing unsupervised or semi-supervised learning approaches. Active learning methods are also being developed to efficiently select and annotate informative samples, reducing the need for extensive labeled datasets.\n\n3. **Lack of interpretability**: Deep learning architectures are often considered black box models, making it challenging to understand the underlying decision-making process. Interpreting and explaining the reasoning behind the predictions made by these models can be difficult, hindering their adoption in domains where interpretability is crucial, such as healthcare. → Interpretability is a growing area of research, and efforts are being made to develop techniques that explain and visualize the decisions made by deep learning models. Methods such as attention mechanisms, saliency maps, and model-agnostic interpretability techniques aim to provide insights into the model's reasoning and improve trust and transparency. A review of taxonomies of XAI (eXplainabilty AI) methods is provided in [Speith 2022](#speith2022).\n\n4. **Limited generalization**: Deep learning models can struggle with generalizing well to unseen data or handling scenarios significantly different from the training data distribution. They may exhibit overfitting when the training data is limited or biased, leading to reduced performance in real-world settings. → Researchers are exploring methods to improve generalization by addressing issues like dataset bias, domain adaptation, and incorporating domain knowledge into deep learning models. Techniques like data augmentation, regularization, and adversarial training aim to enhance the model's ability to generalize to unseen scenarios.The authors of [Bommasani 2021](#bommasani2021) propose a deep analysis of potential impacts of foundation madels.\n\n5. **Limited robustness to adversarial attacks**: Deep learning models can be vulnerable to adversarial attacks, where minor modifications to the input can cause the model to produce incorrect or misleading outputs. Adversarial examples can be crafted to deceive the models, raising concerns about their reliability and security in critical applications. → Adversarial defense strategies are actively being researched to enhance the robustness of deep learning models against adversarial attacks. Techniques like adversarial training, robust optimization, and defensive distillation aim to improve model resilience and security. While the vulnerability of traditional Convolutional Neural Networks (CNNs) to backdoor attacks is well-studied, the specific vulnerabilities of ViTs to such attacks have not been thoroughly explored. In [Zeng 2023](#zeng2023) propose and analyse a new backdoor attack method called TrojViT that is tailored specifically for ViTs.\n\nAddressing these limitations is an ongoing research focus. Efforts are being made to achieve pespectives hightlight and develop techniques that mitigate these challenges and improve the robustness, interpretability, and efficiency of deep learning architectures in AI.","metadata":{}},{"cell_type":"markdown","source":"# 9. Conclusion [↑](#top)\n\nIn conclusion, transformers and more generaly speaking the field of deep learning has made significant progress in addressing various challenges. Researchers are actively working on optimizing computational requirements, overcoming dataset limitations, enhancing interpretability, improving generalization, enabling efficient training, and strengthening model robustness against adversarial attacks. These advancements have the potential to drive the wider adoption and application of deep learning in various domains.\n\nHowever, as deep learning technologies continue to evolve and become more pervasive, it is crucial to consider the ethical implications and societal impact they entail. Explainable AI (XAI) techniques are being developed to provide insights into the decision-making process of deep learning models, ensuring transparency and accountability. This is particularly important to avoid biased or discriminatory outcomes and to build trust between AI systems and users. Additionally, the vulnerability of deep learning models to adversarial attacks raises concerns about the security and integrity of AI systems. Continued research into adversarial defense strategies is necessary to mitigate these risks and ensure the reliability and safety of AI technologies.\n\nOverall, while deep learning holds immense potential, it is imperative to approach its development and deployment with a strong focus on ethics, vulnerability, and the overall impact on society. Striking a balance between technological advancements and societal well-being will be crucial for shaping a future where AI benefits all while minimizing potential risks and ensuring equitable access and usage.\n","metadata":{}},{"cell_type":"markdown","source":"\n# 10. References [↑](#top)\n\n[Lin 2022] Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022). A survey of transformers. *AI Open*. Elsevier\n \n[Vaswani 2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. *Advances in neural information processing systems*, 30.\n\n[Liu 2023] Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J., Tian, J., Zhang, Y., Shi, Z., Fan, J., & He, Z. (2023). A survey of visual transformers. *IEEE Transactions on Neural Networks and Learning Systems*. IEEE.\n\n[Khan 2022] Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022). Transformers in vision: A survey. *ACM computing surveys (CSUR)*, 54(10s), 1-41. ACM New York, NY.\n\n[Huang 2023] Huang, H., Zhou, X., Cao, J., He, R., & Tan, T. (2023). Vision Transformer with Super Token Sampling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 22690-22699).\n\n[Dosovitskiy 2020] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.\n\n[Zheng 2023] Zheng, M., Lou, Q., & Jiang, L. (2023). Trojvit: Trojan insertion in vision transformers. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 4025-4034).\n\n[Selva 2023] Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023). Video transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE.\n \n[Guo 2022] Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022). Attention mechanisms in computer vision: A survey. Computational visual media, 8(3), 331-368. Springer.\n\n[Li 2022] Li, K., Wang, Y., Gao, P., Song, G., Liu, Y., Li, H., & Qiao, Y. (2022). Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv:2201.04676.\n\n[Muller 2023] Müller, L., Galkin, M., Morris, C., & Rampášek, L. (2023). Attending to graph transformers. arXiv preprint [arXiv:2302.04181](https://arxiv.org/pdf/2302.04181.pdf).\n \n[Min 2022] Min, E., Chen, R., Bian, Y., Xu, T., Zhao, K., Huang, W., ... & Rong, Y. (2022). Transformer for graphs: An overview from architecture perspective. arXiv preprint [arXiv:2202.08455](https://arxiv.org/pdf/2202.08455.pdf).\n \n[Mitchell 2021] Mitchell, E., Lin, C., Bosselut, A., Finn, C., & Manning, C. D. (2021). Fast model editing at scale. arXiv preprint [arXiv:2110.11309](https://arxiv.org/pdf/2110.11309.pdf).\n \n[Mitchell 2022] Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., & Finn, C. (2022). Memory-based model editing at scale. In International Conference on Machine Learning (pp. 15817-15831). PMLR.\n\n[Zou 2023] Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Gao, J., & Lee, Y. J. (2023). Segment everything everywhere all at once. arXiv preprint [arXiv:2304.06718](https://arxiv.org/pdf/2304.06718.pdf).\n\n[Kirillov 2023] Kirillov, A. and al. (2023). Segment anything. arXiv preprint [arXiv:2304.02643](https://arxiv.org/pdf/2304.02643.pdf).\n \n[Prabhu 2020] Prabhu, V. U., & Birhane, A. (2020). Large datasets: A pyrrhic win for computer vision?. arXiv preprint [arXiv:2006.16923](https://arxiv.org/pdf/2006.16923.pdf).","metadata":{}},{"cell_type":"markdown","source":"# 11. Code [↑](#top)\nWe propose a world cloud on the title of the article provide in the file 'arxiv_metadata_20230510.json'","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read json file\nimport json\nkey_to_extract = \"title\"\ntitle_list = []\n\n# with open('arxiv.json') as file:\nwith open('/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json') as file:\n num_objects = 0\n for line in file:\n num_objects += 1\n try:\n json_obj = json.loads(line) \n value = json_obj.get(key_to_extract)\n if value is not None:\n title_list.append(value)\n if num_objects < 20:\n print(value)\n except json.JSONDecodeError:\n print(num_objects)\n print(line)\n pass # Ignore lines that are not valid JSON objects\n\nprint(f\"Number of JSON objects: {num_objects}\")\n\n# Concaténer tous les titres en une seule chaîne\ntext = ' '.join(title_list)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\n\nfrom os import path\nfrom wordcloud import WordCloud \n# https://amueller.github.io/word_cloud/auto_examples/index.html","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate a word cloud image\nwordcloud = WordCloud().generate(text)\n\n# Display the generated image:\n# the matplotlib way:\nimport matplotlib.pyplot as plt\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis(\"off\")\n\n# lower max_font_size\nwordcloud = WordCloud(max_font_size=40).generate(text)\nplt.figure()\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis(\"off\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission.loc[0]['value'] = 'Other'\nsubmission.loc[1]['value'] = 'https://www.kaggle.com/bazooo/transformers-from-2020'\nsubmission.loc[2]['value'] = 'https://www.kaggle.com/code/javadbabayian/image-and-video-data-2023-kaggle-ai-report/comments#2338248'\nsubmission.loc[3]['value'] = 'https://www.kaggle.com/code/kkiryll/generative-ai-rise-of-the-transformers/comments#2337980'\nsubmission.loc[4]['value'] = 'https://www.kaggle.com/code/dokster/image-and-video-data-kaggle-ai-report/comments#2337734'\nsubmission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}