{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50947,"databundleVersionId":5741538,"sourceType":"competition"}],"dockerImageVersionId":30513,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\ndf.iloc[0,1] = 'Image and/or video data'\ndf.iloc[1,1] = 'https://www.kaggle.com/code/usharengaraju/semanticsegmentationusingvits'\ndf.iloc[2,1] = 'https://www.kaggle.com/code/kaungmyatkyaw/antique-enthusiasts-image-and-or-video-data/comments#2346316'\ndf.iloc[3,1] = 'https://www.kaggle.com/code/roshinifernando/real-time-object-detection/comments#2346442'\ndf.iloc[4,1] = 'https://www.kaggle.com/code/pandoraalias/unleashing-visual-intelligence-vid-img/comments#2346475'\ndf.to_csv('submission.csv', index=False)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T15:54:52.95Z","iopub.execute_input":"2023-07-16T15:54:52.950418Z","iopub.status.idle":"2023-07-16T15:54:53.014588Z","shell.execute_reply.started":"2023-07-16T15:54:52.950387Z","shell.execute_reply":"2023-07-16T15:54:53.013572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" # Semantic Segmentation using Vision Transformer
","metadata":{}},{"cell_type":"markdown","source":"### Category : Image and/or video data","metadata":{}},{"cell_type":"markdown","source":"**Table of Contents**\n* Introduction\n* Application of Semantic Segmentation across Domains\n* Basic Structure of Semantic Segmentation Models\n* Vision Transformers\n* Evolving Landscape of ViT based Semantic Segmentation Models\n* Comparative Analysis of different ViT based Semantic Segmentation Models\n* Inference of ViT based Semantic Segmentation Models (Code)\n* Foundational Models in Semantic Segmentation using ViTs\n* ViT based Semantic Segmentation Models for Video Data\n* ViT based 3D Semantic Segmentation Models\n* Interpretability of ViT based Semantic Segmentation models\n* Privacy preserving ViT based Semantic Segmentation Models\n* Zero Shot and Few Shot Learning in ViT based Semantic Segmentation Models \n* Self supervised Semantic Segmentation using ViTs \n* Unsupervised Semantic segmentation using ViTs\n* Scalability of Semantic Segmentation Models using ViTs\n* Robustness of Semantic Segmentation Models using ViTs\n* Semantic Segmentation Models using ViTs for Edge/mobile devices\n* Datasets used in Segmentation Models using ViTs\n* Loss Functions used in Segmentation Models using ViTs\n* Evaluation Metrics used in Segmentation Models using ViTs\n* Large Data Requirements\n* Computational Complexity\n* Hardware Requirements\n* Neural Architecture Search for Segmentation using ViTs \n* Challenges and Future Directions \n* Conclusion \n* References\n* Code References\n\n\n","metadata":{}},{"cell_type":"markdown","source":"\n\n# **Introduction**\n\nIn semantic segmentation, each and every pixel of an image is given a class label and this calls for precise pixel-level predictions. And it also seeks to organize the picture or video's pixels into a collection of semantic regions. It has usage in many real-world applications, including robots, automated surveillance, image/video editing, social networking, autonomous driving, etc. It is a fundamental problem in computer vision. Segmentation challenges have been the subject of extensive research, starting with manually created features and traditional machine-learning models. Deep neural networks, Convolution Neural Networks (CNNs), such as Fully Convolutional Networks (FCNs), have gained notable accomplishments for various segmentation tasks over the past 10 years and produced significantly superior outcomes. CNN-based algorithms have more robust generalization capabilities than conventional segmentation methods. CNNs and FCN architecture have become fundamental elements in segmentation research studies because of their excellent performance.\n\nFollowing its enormous success, Transformers became the new benchmark for natural language processing (NLP). This prompted the creation of ViT, which was then applied to computer vision applications like object detection, semantic segmentation, and image classification. A multi-head self-attention (MSA), multi-layer perceptron (MLP), and layer norm (LN) layer make up a standard Transformer encoder. The multi-head self-attention mechanism is the primary engine underlying the ViT which can capture distant relationships with reduced inductive bias. ViT exhibits extraordinary performance when trained on enough data, outperforming the performance of cutting-edge CNNs. \n\nCurrently, both CNN-based and Transformer-based segmentation models are widely used. Simple ViT models cannot be used in segmentation tasks directly because they lack segmentation heads. Transformer-based architectures like SETR[22] and Swin[21] are more widely used for segmentation tasks. The high intra-class variance, context variation, occlusion ambiguities, and low image resolution make dense prediction tasks like semantic segmentation and object recognition more challenging than image classification. In the last few years, the landscape of ViT based architectures has undergone numerous improvements to address these issues, and further advancements are continuously being made to make them effective. The essay focuses especially on semantic segmentation based on Vision Transformers.\n\n\n\n# **Application of Semantic Segmentation across Domains **\n\nSemantic segmentation becomes useful in various domains like autonomous driving, facial recognition, virtual fitting rooms, medical imaging and much more. Segmentation plays a critical role in detecting medical abnormalities that appear in clinical scans, such as CT or MRI scans. Fast and accurate computer vision algorithms allow medical personnel to manage their time. Amazon's Virtual Mirror is the result of semantic segmentation for apparel items. It is a mirror that is both partially reflected and partially transmissive, allowing you to try on garments without ever changing. Autonomous vehicles are one of the most complex applications of computer vision to date. Artificial intelligence companies rely on semantic segmentation to maximize accuracy when it comes to detecting lanes and traffic signs.\n\n\n# **Basic Structure of Semantic Segmentation Models**\n\nThe basic architecture in image segmentation consists of an encoder and a decoder. The encoder extracts features from the image using various filters. The final output, which is often a segmentation mask containing the object's contour, is produced by the decoder. This architecture or a variation of it is present in the majority of architectures. The most basic structure with which it all started was the U-Net.\n\n\n\nFigure 1. [A simple semantic segmentation model architecture](https://www.google.com/imgres?imgurl=https%3A%2F%2Fi0.wp.com%2Fneptune.ai%2Fwp-content%2Fuploads%2F2022%2F10%2FImage-segmentation-architectures.png%3Fssl%3D1&tbnid=AU0Pyq4XgBMhgM&vet=12ahUKEwjrwbLJro6AAxXMnmMGHRrPDJMQMygBegUIARDDAQ..i&imgrefurl=https%3A%2F%2Fneptune.ai%2Fblog%2Fimage-segmentation&docid=l_cm6N2Lf-sSfM&w=624&h=175&q=semantic%20segmentation%20model%20architecture&ved=2ahUKEwjrwbLJro6AAxXMnmMGHRrPDJMQMygBegUIARDDAQ)\n\n\n\n# **Vision Transformers**\n\nThe well-known paper \"Attention is All You Need\" published in 2017 first introduced the universal transformer architecture. Quickly they became the most widely used and popular neural network architecture in the field of Natural Language Processing .\"An image is worth 16x16 words,\" first introduced the Vision Transformer architecture for processing images without the use of convolutions in 2019.\n\nThe main idea is to deconstruct input images into a collection of patches, which, when converted into vectors, are viewed as words in a standard transformer.\nIf in the field of Natural Language Processing the mechanism of attention of the Transformers tried to capture the relationships between different words of the text to be analyzed, in Computer Vision the Vision Transformers instead capture the relationships between different portions of image.\n\n\n\nFigure 2 : [Standard ViT Encoder Structure](https://arxiv.org/pdf/2305.03273.pdf)\n\nA pure transformer model outperformed CNNs in terms of performance and efficiency for image classification in 2021. In the same year, many significant variants of Vision Transformers were released .The major goals of these variations are to be more effective, more precise, or better suited to a certain domain. One of the most pertinent is the Swin Transformer, which, using a multi-stage technique and certain attention mechanism changes, produced state-of-the-art results on a number of object detection datasets, including COCO. With the increasing capabilities of the vision transformers, semantic segmentation began to be seen as a downstream task along with other dense prediction tasks. And with the advent of time, ViTs have become adept in segmentation tasks. Modern cutting-edge techniques all rely on transformer architecture. Most transformer-based methods perform better and have simpler pipelines than CNN methods.\n\n# **Evolving Landscape of ViT based Semantic Segmentation Models**\n\nPrior to the development of ViT and DETR[23], semantic segmentation was frequently handled as a problem of dense pixel classification, as first suggested by FCN. Earlier segmentation models were mostly built using the FCN framework. Better encoder-decoder frameworks, larger kernels, multiscale pooling, multiscale feature fusion, non-local modeling, efficient modeling, and better border delineation are a few of the elements that may be found in these techniques. Post the introduction of transformers, various architectural variants proposed self-attention operators to replace the CNN prediction heads to achieve global context modeling. \n\n\n\nFor complex prediction tasks like picture segmentation that demand simulating complex interactions between pixels, self-attention might be used. Although applying self-attention to big inputs for a dense prediction task like panoptic segmentation is prohibitively expensive, it is excellent at modeling long-range contextual information. Applying self-attention to downsampled inputs or to small areas around each pixel is a simple solution, but even then, it still has quadratic complexity and costs the overall context. Position-sensitive axial attention was introduced to address these problems. It allows models to capture the whole visual context and is compute-efficient. \n\n\nTransformer-based encoder-decoder architectures were proposed after the earlier ideas, which included self-attention in their CNN-based architectures. The Segmentation Transformer (SETR[22]) uses progressive upsampling, multi-level feature aggregation, a ViT encoder, and two decoder designs. SegFormer[24] uses a simple MLP-based decoder with an upsampling operation to obtain the segmentation mask and a hierarchical pyramid ViT as the encoder. To extract image features, the segmenter uses the ViT encoder, while the decoder is a mask transformer module that forecasts segmentation masks using inputs like learnable mask tokens and image-patch tokens. The model architecture of segmentation models based on ViTs has been constantly evolving since then.\n\n# **Comparative Analysis of different ViT based Semantic Segmentation Models**\n\n\n\n\nTable 1 : Shows comparative Analysis of different ViT based Semantic Segmentation Models\n\n\n\nTable 2 : Shows comparative Analysis of different ViT based Semantic Segmentation Models across various parameters\n","metadata":{}},{"cell_type":"markdown","source":"# **Inference of ViT based Semantic Segmentation Models (Code)**\n\n### Code Running Instructions :\n\nEach and every model inference should be run one by one in both Kaggle and colab notebook . Using \"Run all\" command will result in notebook crashing due to low RAM .\n\n#### Colab Notebook : https://colab.research.google.com/drive/1133BZnRsJq45dauz5YHi_yQmSeSvNTNL?usp=sharing\n\n#### Github : https://github.com/ushareng/SemanticSegmentationUsingViTs","metadata":{}},{"cell_type":"code","source":"!pip install -q transformers timm ftfy wandb","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:17:24.784395Z","iopub.execute_input":"2024-03-06T12:17:24.785213Z","iopub.status.idle":"2024-03-06T12:17:36.016168Z","shell.execute_reply.started":"2024-03-06T12:17:24.785179Z","shell.execute_reply":"2024-03-06T12:17:36.015182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import io\nimport requests\nfrom PIL import Image\nimport torch\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom transformers import AutoImageProcessor, DetrForSegmentation\nfrom transformers.image_transforms import rgb_to_id\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:18:00.096328Z","iopub.execute_input":"2024-03-06T12:18:00.097165Z","iopub.status.idle":"2024-03-06T12:18:13.409205Z","shell.execute_reply.started":"2024-03-06T12:18:00.097132Z","shell.execute_reply":"2024-03-06T12:18:13.408244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:18:13.410890Z","iopub.execute_input":"2024-03-06T12:18:13.411256Z","iopub.status.idle":"2024-03-06T12:18:13.887426Z","shell.execute_reply.started":"2024-03-06T12:18:13.411214Z","shell.execute_reply":"2024-03-06T12:18:13.886503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import wandb \nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"api_key\")\nwandb.login(key=secret_value_0)","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:19:14.984971Z","iopub.execute_input":"2024-03-06T12:19:14.985831Z","iopub.status.idle":"2024-03-06T12:19:18.161651Z","shell.execute_reply.started":"2024-03-06T12:19:14.985799Z","shell.execute_reply":"2024-03-06T12:19:18.160630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run = wandb.init(project = 'semantic_seg',\n name='Time Series transformer'\n \n)\nimages = wandb.Image(image, caption=\"input image\")\n\nwandb.log({\"input\": images})","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:20:31.485138Z","iopub.execute_input":"2024-03-06T12:20:31.486383Z","iopub.status.idle":"2024-03-06T12:21:02.903141Z","shell.execute_reply.started":"2024-03-06T12:20:31.486345Z","shell.execute_reply":"2024-03-06T12:21:02.902180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:21:02.937000Z","iopub.execute_input":"2024-03-06T12:21:02.937291Z","iopub.status.idle":"2024-03-06T12:21:24.523236Z","shell.execute_reply.started":"2024-03-06T12:21:02.937263Z","shell.execute_reply":"2024-03-06T12:21:24.522311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_processor = AutoImageProcessor.from_pretrained(\"facebook/detr-resnet-50-panoptic\")\nmodel = DetrForSegmentation.from_pretrained(\"facebook/detr-resnet-50-panoptic\")\n\ninputs = image_processor(images=image, return_tensors=\"pt\")\n\noutputs = model(**inputs)\n\n\nresult = image_processor.post_process_panoptic_segmentation(outputs, target_sizes=[(300, 500)])\n\npanoptic_seg = result[0][\"segmentation\"]\npanoptic_segments_info = result[0][\"segments_info\"]","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:21:24.524989Z","iopub.execute_input":"2024-03-06T12:21:24.525274Z","iopub.status.idle":"2024-03-06T12:21:36.802333Z","shell.execute_reply.started":"2024-03-06T12:21:24.525249Z","shell.execute_reply":"2024-03-06T12:21:36.801480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(panoptic_seg)","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:21:40.492297Z","iopub.execute_input":"2024-03-06T12:21:40.493347Z","iopub.status.idle":"2024-03-06T12:21:40.720409Z","shell.execute_reply.started":"2024-03-06T12:21:40.493312Z","shell.execute_reply":"2024-03-06T12:21:40.719588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import AutoImageProcessor, UperNetForSemanticSegmentation\n\nimage_processor = AutoImageProcessor.from_pretrained(\"openmmlab/upernet-convnext-base\")\nmodel = UperNetForSemanticSegmentation.from_pretrained(\"openmmlab/upernet-convnext-base\")\n\ninputs = image_processor(images=image, return_tensors=\"pt\")\n\noutputs = model(**inputs)\n\nlogits = outputs.logits # shape (batch_size, num_labels, height, width)\nlist(logits.shape)","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:21:43.232386Z","iopub.execute_input":"2024-03-06T12:21:43.232776Z","iopub.status.idle":"2024-03-06T12:22:01.253512Z","shell.execute_reply.started":"2024-03-06T12:21:43.232739Z","shell.execute_reply":"2024-03-06T12:22:01.252499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(np.argmax(logits.detach().numpy()[0],axis=0))","metadata":{"execution":{"iopub.status.busy":"2024-03-06T12:22:01.255452Z","iopub.execute_input":"2024-03-06T12:22:01.255946Z","iopub.status.idle":"2024-03-06T12:22:02.036293Z","shell.execute_reply.started":"2024-03-06T12:22:01.255903Z","shell.execute_reply":"2024-03-06T12:22:02.035414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Foundational Models in Semantic Segmentation using ViTs**\n\nWith new advances in Generative AI, large-scale models called foundation models are developed. The foundation models can be applied to a wide range of downstream applications and are trained on enormous amounts of unannotated data. SAM[7] (Segment Anything Model), a foundation model for image segmentation in computer vision, is constructed on the same premises. It reduces the need for task-specific training, modeling, and annotation for image segmentation. Task, model, and data are the three foundational elements of a SAM. The task is a promptable segmentation task that responds to any prompt with a legitimate segmentation mask.\n\n\n\nFigure 3 : [SAM Architecture](https://pub.towardsai.net/sam-a-image-segmentation-foundation-model-31aea7408c93)\n\n\n\nBy designing the right cues, SAM can handle tasks further down the line. SAM also performs automatic labeling, segmenting an object from a single foreground point, low-level edge detection, mid-level object proposal generation, high-level instance segmentation, and higher-level segmenting of objects from free-form text. In one model, SAM combines interactive and automatic segmentation. By engineering the proper prompt (such as clicks, boxes, or text), the suggested interface enables flexible usage and makes a variety of segmentation tasks possible. SAM can generalize to new sorts of objects and images beyond those detected during training because it was built using a large, high-quality dataset containing more than one billion masks acquired as part of this study. Thus, practitioners no longer need to gather their segmentation data and customize a model, especially for their use case.\n\nNo other picture segmentation software has ever been able to generalize both across tasks and domains like SAM can. BEIT-3 [15], which is pretrained as a Multiway Transformer by doing masked data modeling on images, texts, and image-text pairs, is another all-purpose multimodal foundation model. Some text tokens or image patches are randomly disguised during pretraining. To restore the original tokens—text or visual—from corrupted inputs is the goal of self-supervised learning. The model is versatile in that it can be applied to a variety of tasks regardless of the input or output formats.\n\n# ** ViT based Semantic Segmentation Models for Video Data **\n\nApplications for scene comprehension include human-machine interaction, augmented reality, autonomous vehicles, and image search engines. For these kinds of applications, semantic segmentation makes a greater contribution to complete scene understanding on videos. When a high-resolution video is thought of as a collection of uncorrelated fixed pictures, the objective is often to apply semantic segmentation to individual frames. It is not logical to exclude temporal data from video segmentation and solely concentrate on spatial frame-by-frame features. Even though it is computationally expensive, taking into account the temporal context of a video is crucial for video semantic segmentation since there is a continuous flow across the frames of a video. There has been extensive research done to lower the high computation cost for videos. As a fix, feature warping and reuse have been suggested. Segmentation techniques include selective re-execution of feature extraction layers, optical flow-based feature warping, and fixed-budget keyframe selection rules that have been proposed in recent studies. These methods' primary flaw is that they pay less attention to the video's temporal context. Researchers have established that it makes sense to use an optical flow of video as temporal information to expedite uncertainty estimates in order to meet both spatial and temporal contexts. Some of the Transformer models utilized for video segmentation tasks include VisTR[27], TeViT[26], and SeqFormer[12].\nTo acquire efficient multi-modal representations, certain models, like the VideoBERT [25] model, make use of Transformer networks and the power of self-supervised learning. The prediction of hidden visual and linguistic elements serves as a pretext problem for VideoBERT. This makes it possible to describe high-level semantics and distant temporal dependencies, which are crucial for jobs involving video interpretation. \n\n# ** ViT based 3D Semantic Segmentation Models **\n\nAdopting transformers for semantic level processing mostly concentrates on learning a strong representation, similar to image segmentation and video semantic segmentation. There were parallel efforts that focussed on applying the success in learning image/video representation to the point cloud. Early works directly create U-Net-like segmentation topologies using modified self-attention as backbone networks. To gradually accumulate local features, Point-Transformer, in particular, presents vector self-attention and subtraction relations. In addition, the concurrent work PCT utilizes self-attention operation and improves input embedding with the aid of nearest neighbor searching and farthest point sampling. Long-range context and cross-scale interaction can still only be partially modeled. Swin Transformer's concept is expanded by Stratified-Transformer into the point cloud and 3D inputs that are converted into cubes. For attention input, it suggests a mixed key sampling strategy, and by combining several cube outputs, it increases the effective receptive field.\n\nAdditionally, a number of works concentrate on improving pre-training or condensing the knowledge of 2D pre-trained models. In order to pre-train point cloud transformers, PointBert[28] creates the first Masked Point Modelling (MPM) task. As the input of a typical transformer, it separates a point cloud into a number of small local point patches. Additionally, the reconstruction loss is used to train an additional decoder and pre-train a point cloud tokenizer with a discrete variable autoencoder to encode the semantic contents. Point-MAE randomly masks the input point cloud's irregular point patches at a high ratio. The masked points are then rebuilt using a typical transformer-based autoencoder.\nBy converting the encoder and decoder into pyramid architectures to gradually model spatial geometries and multilevel semantics, Point-M2AE[29] creates a multiscale MIM pretraining. While this is going on, some works adopt the standard image pre-trained transformer by extracting the information from large-scale picture dataset pre-trained models, taking advantage of the same transformer design for point cloud and image.\n\n\n# ** Interpretability of ViT based Semantic Segmentation models **\n\n\nViTs have significant robustness against texture alterations and severe occlusions when compared to CNNs. It is crucial and interesting to understand how Transformer architectures make decisions given their high performance. The fundamental issue is that it is challenging to visualize the relative contribution of input tokens to final predictions since the attention that originates in each layer is complexly inter-mixed in the succeeding levels. Although this is still a concern, some recent studies aim to improve Transformers' interpretability. To assess the precise attention, attention roll-out and attention flow approaches were suggested. \nIt was suggested that relevancy scores be assigned and propagated throughout the Transformer network so that the total relevance remains constant.Their design may accommodate both the favorable and unfavorable attributions that are encountered in the self-attention layer. The ability to offer class-specific visualizations is an additional benefit of the suggested approach. Nevertheless, visualizing and understanding Transformers remains a challenge, and techniques are required to produce spatially precise activation-specific visualizations. Further development in this area may aid in improving comprehension of the Transformer models and recognizing any biases or erroneous behaviors in the decision-making process. Innovative architectures that can aid us in avoiding biases can also be designed with its assistance.\n\n# ** Privacy preserving ViT based Semantic Segmentation Models **\n\nPerceptual encryption-based methods have been verified to be effective in image classification tasks, but other tasks such as semantic segmentation have never been considered under the use of perceptual encryption-based methods because such tasks are required to achieve a pixel-level accuracy. For privacy-preserving semantic segmentation using encrypted pictures and models with the vision transformer (ViT), the segmentation transformer (SETR[22]) was proposed. In some situations, ViT is well recognized to perform better than traditional convolutional neural networks (CNNs). ViT additionally includes an embedding structure. We can avoid block-wise encryption by using embedding, but we can also simply change a secret key in a semantic segmentation work, which is impossible with traditional techniques.\n\n# ** Zero Shot and Few Shot Learning in ViT based Semantic Segmentation Models **\n\nIn the few-shot learning settings, a support set is provided at the inference to adapt to a novel set of categories. Transformer models have been used in this support set to either learn set-to-set mappings or the spatial relationships between samples from the support set and an input query. Compared to an image-level association, the patch-wise spatial self-attention between the query and support set images performs better in terms of absolute performance. In order to connect a particular query image with the sparse examples in the support set, a Cross-Transformer model was created. To accomplish this, the Transformer identifies regions in the query and support set images that are geographically comparable, and the relevant characteristics are then used to derive class decisions for the query. The grid features collected using the query image are used to generate the queries in the Transformer architecture. In addition to a contrastive self-supervision based training mechanism, this strategy results in the best performance on difficult tasks. When making inferences, the transformer model applies the few-shot embeddings learned on the base classes to the few-shot target classes. For the purpose of maintaining discriminative qualities, the embeddings are modified using a contrastive loss function. The visual representation discovered by the models trained on various domains included in the meta-dataset is additionally integrated using a multi-head self-attention based module. The Universal Representation Transformer (URT) layer handles a few shot jobs across a variety of data distributions by dynamically re-weighting the representations from various domain-specific backbones. Models like CyCTR[30] aggregate pixel-wise support features into query features. In particular, CyCTR[30] performs cross-attention between features from different images in a cycle manner, where support image features and query image features are the query inputs of the transformer jointly. Meanwhile, MM-Former adopts a class-agnostic method to decompose the query image into multiple segment proposals. SAM achieves a very good zero-shot performance.\n\n\n\nFigure 4 . [Overview of Embedding Adaptation](https://arxiv.org/pdf/2305.03273.pdf)\n\n# ** Self supervised Semantic Segmentation using ViTs **\n\nThe considerable need for data during the training phase of transformers has made it essential to find alternative methods to train these models, and a central role is now played by self-supervised methods. These techniques make it feasible to train neural networks in a nearly autonomous manner, enabling them to recognize the unique characteristics of a given situation without the need for a substantial training set or carefully chosen labels. The secret to the general adoption of this exciting new architecture may lie in the Vision Transformer's ability to be trained without the need for a sizable vision dataset to be available. Self-attention based Transformer models generally operate in a two-stage training mechanism. First, pre-training is performed on a large-scale dataset in either a supervised or a self-supervised manner. The pre-trained weights are further modified using small to mid-scale datasets for the downstream tasks.\nSelfPatch is a straightforward yet effective visual pretraining job for improving patch-level representations for ViTs. In this work, invariance is imposed against each patch and its neighbors, meaning that each patch views similar neighboring patches as positive samples. As a result, when training ViTs with SelfPatch[19], more semantically significant relationships between patches are learned (without the use of human-annotated labels), which can be advantageous for downstream tasks, particularly those involving dense prediction.\nAnother ground-breaking development in SSL of ViTs is the Multi-Granular Self Supervised Learning(MUGS)[18] framework which explicitly learns Multi-granular visual features. An instance discrimination supervision (IDS), novel local-group discrimination supervision (LGDS), and group discrimination supervision (GDS) are the three complementing granular supervisions. IDS recognizes several instances to learn fine-grained instance-level features. When using LGDS, features from an image's neighbors and other crops are combined into a local-group feature, which is then pulled together for some crops and pushed away for others. Through an additional alignment on local neighbors, it offers IDS complementary instance supervision. It also scatters various local groups individually to improve discriminability. As a result, it aids in the local group learning of high-level fine-grained features. \n\n\n\n Figure 5 .[Self Supervised learning style](https://arxiv.org/pdf/2305.03273.pdf)\n\n# ** Unsupervised Semantic segmentation using ViTs **\n\nUnsupervised segmentation performs segmentation without any labels. It also uses some ideas from the self-supervised techniques. Self-supervised ViT features often contain relevant features of the images. The attention maps and features can be used to describe the semantics of the images. Some approaches also solve unsupervised object discovery by using the key component of the last attention layer for computing the similarities between the different patches. Additionally, many works are implementing sequential pipelines. MaskDistill uses a bottom-up methodology to first identify clusters of pixels that probably belong to the same object. The object masks are then clustered, and the output is used to train an additional model using fictitious ground truths. The offline model's output masks are then chosen based on the object score.\nFew such models are STEGO[31] , SEGCLIP[32] , ACSEG[33]. STEGO is a novel transformer-based architecture for unsupervised semantic segmentation which is capable of jointly discovering and segmenting objects without human supervision. Using a unique contrastive loss, STEGO reduces pretrained unsupervised visual data into semantic groupings. It demonstrates that the correlation patterns between the unsupervised deep network features and actual semantic labels are generally consistent. Another promising framework is the ACSeg technique, or Adaptive Conceptualization for Unsupervised Semantic Segmentation, which explicitly encodes concepts into learnable prototypes. An adaptive concept generator (ACG) then converts each image's prototype to a concept that is instructive. The scene complexity of various images is taken into account, and a modularity loss based on calculating the intensity of pixel pairings belonging to the same concept is proposed to optimise ACG regardless of the concept number. The USS task is finally transformed into an unsupervised classification of the identified ideas. SEGCLIP is a very different model which is based on CLIP for open-vocabulary segmentation in an annotation-free manner. The basic goal is to train on text-image pairs in order to collect patches with learnable centres to semantic regions. The semantic groupings can be dynamically captured by the gathering process and used to produce the final segmentation results. To improve the visual representation, a reconstruction loss on masked patches and a superpixel-based KL loss with pseudo-labels are also suggested.\n\n# ** Scalability of Semantic Segmentation Models using ViTs **\n\nTransformers' simple architecture makes it possible to handle several modalities, such as photos, videos, text, and audio, using the same processing blocks. It also shows great scalability to very large capacity networks and enormous datasets. For instance, the current mixture-of-experts Switch transformer scales up to an astounding 1.6 trillion parameters, substantially outperforming the BERT-large model with 340 million parameters and the GPT-3 model with 175 billion parameters! Transformer networks' scalability and generalization have been greatly aided by the self-supervision-based pre-training stage training. Scaling up compute, model and data together improves representation quality. Smaller models don’t benefit from increasing dataset size/compute resources while large models seem to significantly benefit from even over 1B images.\n\n# **Robustness of Semantic Segmentation Models using ViTs **\n\nDeep neural networks are vulnerable to adversarial attacks. The models can be easily fooled by adding artificial small and imperceptible perturbations to input images. Recent research demonstrates the great robustness that Vision Transformers (ViTs) exhibit against various corruptions. Although the self-attention mechanism is partially responsible for this trait, systematic explanations are still lacking. Self-attention promotes robustness through improved mid-level representations. Fully attentional networks (FANs) also strengthen this capability by incorporating an attentional channel processing design. ViTs demonstrate strong robustness against texture changes and severe occlusions, e.g.ViTs retain up to 60% top-1 accuracy on ImageNet once 80% of the image content is randomly occluded. SegPGD[20] is a very efficient and effective method for adversarial training of segmentation models. In the particular case of segmentation models, generally, single-step adversarial attacks are used for adversarial training but it is not very effective since the single-step attacks are not sufficient to fool the models but on the other hand, the iterative multi-step attacks are very time-consuming. SegPGD particularly solves this problem.\n\nThe foundation model SAM is thoroughly tested using SegPGD and other state-of-the-art adversarial attack algorithms. It was shown that SAM worked very well against the attacks while giving most of the predictions correctly. This can be attributed to its large dataset training and unique and versatile model architecture. SAM being the foundation model is expected to lead the way for the ViT based segmentation models both in terms of accuracy and robustness.\n\n\n\n\nFigure 6 and 7 [SegPGD](https://arxiv.org/pdf/2207.12391.pdf)\n\n# ** Semantic Segmentation Models using ViTs for Edge/mobile devices **\n\nThe majority of transformer-based segmentation techniques have high computational and memory overheads, making them unsuitable for mobile devices. In contrast to earlier real-time segmentation techniques, mobile segmentation techniques must be implemented on mobile devices while taking latency and battery costs into account. Mobile-ViT[34], in particular, unveils the first transformer backbone for portable electronics. Prior to conducting MHSA, it lowers picture patches via MLPs and exhibits higher task-level generalization features. Transformers are used in various studies to develop mobile semantic segmentation. To create the scale-aware semantic feature, TopFormer suggests a token pyramid module that accepts tokens from multiple scales as input. A squeeze-enhanced axial transformer with a universal attention block is what SeaFormer suggests. \n\n# ** Datasets used in Segmentation Models using ViTs **\n\nThe commonly used datasets for image and video segmentation. For image segmentation, the most commonly used datasets are COCO, PASCAL Context, ADE20k and Cityscapes. For video segmentation, the most used datasets are VSPW and Youtube-VIS. \n\n\n\nTable 3 : Various datasets used in Segmentation Models using ViTs\n\n# ** Loss Functions used in Segmentation Models using ViTs **\n\nThe usage of loss functions as well as model architectures has an impact on the accuracy of the models. During training batches, the loss function computes the overall error and uses backpropagation to change the weights. To deal with different domains, many loss functions have been developed, some of which are derivations of already existing loss functions. The imbalances in the dataset are also taken into consideration by these loss functions.\nThe cross-entropy loss and weighted cross-entropy loss, which is performed pixel-wise, is the standard option and most frequently utilized in the context of semantic segmentation.\nDice loss is another often used metric in computer vision tasks to determine how similar two images are. The dice coefficient, which was later evolved as the dice loss function in the segmentation domain, serves as its foundation. Nowadays a mix of dice loss and bce loss is also used.\n\n# ** Evaluation Metrics used in Segmentation Models using ViTs **\n\nThe Jaccard Index, also known as mean intersection over union (mIoU), is a commonly used metric for semantic segmentation since it determines the pixel-wised Union of Interest between the output image and the ground truth masks. The mean IoU for multi-class segmentation is calculated by averaging the IoU for each class separately. This is frequently used to evaluate and compare the performance of dense prediction models. \nAnother popular F-measure based metric is the Dice similarity coefficient (DSC), also known as F1 score or Sørensen-Dice index. It is defined as the harmonic mean between sensitivity and precision. IoU penalizes under and over-segmentation more than DSC. Accuracy or Rand index or pixel accuracy, is one of the most known evaluation metric in statistics. It is defined as the number of correct predictions, consisting of correct positive and negative predictions, compared to the total number of predictions. Despite being popularly known, it is very less used for actual benchmarking due to the usual imbalanced nature of the datasets. One of the lesser-known metrics is the Cohen’s Kappa (Kap). Similar to the AUC value, Kap assesses the degree of agreement brought about by chance. It goes from -1 (worst) to +1 (best), whereas a Kap of 0 denotes a random classifier. It has grown in favor of the field of machine learning thanks to its capacity to be applied to datasets with imbalances. A metric that is very intrinsic to the segmentation use case is the Average Hausdorff Distance. The HD focuses on boundary delineation (contour) to score localization similarity by calculating the distance between two sets of points, such as the ground truth and anticipated segmentation. Exact contour prediction is crucial, particularly for increasingly difficult and detailed segmentation tasks. For this reason, HD-based evaluations have gained popularity, particularly in the segmentation of medical images. \n\n\n\nFigure 8 : [IoU Representation](https://www.google.com/imgres?imgurl=https%3A%2F%2Fmiro.medium.com%2Fv2%2Fresize%3Afit%3A1200%2F0*zsRJ3DbStcZOkTJF.png&tbnid=VZCqRkFH_nRB5M&vet=12ahUKEwiD7vXdq46AAxVCkWMGHXE5BW8QMygAegUIARC1AQ..i&imgrefurl=https%3A%2F%2Fmedium.com%2F%40cyborg.team.nitr%2Fmiou-calculation-4875f918f4cb&docid=1amX9Y86OOD0hM&w=1200&h=555&q=miou%20segmentation%20equation&ved=2ahUKEwiD7vXdq46AAxVCkWMGHXE5BW8QMygAegUIARC1AQ) \n\n# ** Large Data Requirements **\n\nTransformer architectures often require extensive training to understand the underlying modality-specific principles because they do not inherently encode inductive biases to cope with visual data. Longer training timeframes, a major rise in computational demands, and enormous datasets for processing follow as a result. For the ViT model to perform reasonably on the ImageNet benchmark dataset, hundreds of millions of image examples are required. An open research challenge is how to learn a Transformer in a data-efficient way, and recent publications show hopeful progress in that direction. ViTs may be trained from the start on tiny datasets like CIFAR-10 by including feature hierarchies similar to those in CNNs to efficiently capture local picture cues. Self-supervised training techniques can be used to address the sizable data requirement for training ViTs. These techniques make it feasible to train a neural network in a nearly autonomous manner, enabling it to identify the unique characteristics of a particular problem without the need to create a sizable dataset or furnish it with precisely assigned labels.\n\n# ** Computational Complexity **\n\nComputer vision models are similarly hampered by the high computing cost of Transformer models in the NLP domain. Transformers' main self-attention process takes up O(n2) time and memory for n image patches or a quadratic function of the number of patches. This is a significant flaw in current Transformers that prevents their use for the majority of tasks involving high-resolution (HR) images, such as segmentation. The use of pooling/downsampling in self-attention, local window-based attention, axial attention, low-rank projection attention, kernelizable attention, and similarity-clustering based methods are just a few of the many techniques that have been suggested to perform self-attention more \"efficiently.\" But nearly all of these methods either compromise between complexity and accuracy, call for specialized technology or are still inapplicable to very large images. Therefore, it is imperative to provide an effective self-attention mechanism that can be used to process HR pictures on systems with low resources without sacrificing accuracy. The high computing cost can be decreased by introducing inductive biases depending on our comprehension of the visual learning tasks (for example, spatial relationships in the immediate area). \n\n# ** Hardware Requirements **\n\nLarge-scale Transformer networks can have high power and compute requirements, making it difficult for them to be deployed in environments with limited resources, such as internet-of-things (IoT) platforms and edge devices. High computational units are needed for its deployment and training. On mobile devices, the mobile segmentation methods must be implemented while taking latency and battery costs into account. Mobile-ViT, in particular, unveils the first transformer backbone for portable electronics. Prior to conducting MHSA, it lowers picture patches via MLPs and exhibits higher task-level generalization features. To create the scale-aware semantic feature, TopFormer suggests a token pyramid module that accepts tokens from multiple scales as input. \n\n# ** Neural Architecture Search for Segmentation using ViTs **\n\nDesigning precise and effective vision transformers (ViTs) is a crucial yet difficult task. Fast architectural optimization is made possible by supernet-based one-shot neural architecture search (NAS), which has produced convolutional neural network (CNN) results that are state-of-the-art (SOTA). However, using the supernet-based NAS to optimize ViTs directly results in subpar performance, which is worse than training a single ViT. It is discovered that the poor performance is caused by a gradient conflict problem: gradients of various sub-networks clash more violently with those of the supernet in ViTs than CNNs, which causes early training saturation and subpar convergence. A number of methods are provided to address this problem, including a gradient projection approach, a switchable layer scaling architecture, and a streamlined training recipe for data augmentation and regularization. The hybrid ViT model family, known as NASViT[4], surpasses all earlier CNNs and ViTs, including AlphaNet and LeViT, and achieves top-1 accuracy on ImageNet from 200M to 800M FLOPs from 78.2% to 81.8%. NASViTs also outperform earlier backbones on the Cityscape and ADE20K datasets when used to semantic segmentation tasks, obtaining 73.2% and 37.9% mIoU with only 5G FLOPs, respectively.AutoFormer is a one-shot NAS that Chen et al. suggest for vision transformers. A hybrid architecture (CNN and Transformer) is sought after by BossNAS[3]. Another recent study examines how global and local information is traded off in Transformers when used for vision applications. Further investigation of the domain-specific design decisions made utilizing NAS to create more effective and lightweight models akin to CNNs will be helpful.\n\n# ** Challenges and Future Directions **\n\nThe Transformer is invariant with regard to permutations of the input tokens, which presents a challenge because it is unaware of any positional link between them. Since the Transformer must reclaim the 2D positional information of the image patches, or the location in the image where the patches were extracted, ViT is a little trickier than NLP. For each of the N input tokens, ViT employs learnable 1D positional embeddings, which are merely a collection of N learnable vectors of dimension D. These embeddings are not explicitly supervised during training, therefore the ViT is never made aware of the true 2D connection of its input. \nThe length N of the input sequence must be fixed throughout training, though, with this method. This indicates that training needs to be done using input images at a fixed resolution because the patch size also needs to be set (as the linear embedding layer will change otherwise). Although in theory, this should not be the case with CNNs, in practice it is because mini-batch training demands that the input samples within each mini-batch be of the same resolution. ViT may generalize to greater (or smaller) input resolutions during inference, again assuming a fixed patch size, but unlike CNNs, caution must be taken because larger input resolutions result in longer input token sequences. Fortunately, we can utilize a 2D interpolation of the embeddings to generalize to arbitrary input resolutions while maintaining the acquired positional information because positional embeddings that share a 2D positional connection are in fact discovered to be comparable.\n\nSecond, dense predictions, such as the pixel-by-pixel classification in a semantic segmentation job, can only be carried out natively at the patch level since the positional link between the pixels inside each patch is lost. There is a trade-off between expressiveness and throughput in this sense: while small patch sizes, down to 1 x 1 px in the limit, retain more of the underlying spatial relationship of the original image and produce dense latent representations, the throughput suffers and the memory demand of the Transformer becomes unmanageable. Because of this, earlier research, for instance, restricted the attention mechanism to specific pixel neighborhoods or used downsampling to only allow for very low input resolutions. When a patch exceeds the large-patch-size limit, or when a single patch covers the entire image, positional information cannot be used, and the token embedding layer becomes the architecture's bottleneck, severely limiting its capacity.\nSimilar to this, the connection between adjacent pixels that are divided by patch boundaries is lost. Patches join together pixels that might not share contextual commonalities because patching is done in an image-agnostic manner and is therefore somewhat random. As a result, when the input is divided up into patches, crucial components may not be immediately accessible or may need to be reassembled from the corresponding token embeddings. This makes the downstream task more challenging and places more demands on the patch embedding layer. The absence of translational equivariance is related to this as well.\n\nFinally, as was already indicated, the use of small patch sizes and applications of ViT to high-resolution photos and other higher-dimensional image-like data are both prohibited by the typical Transformer architecture, which scales quadratically with the input sequence length. After ViT's first publication, other modifications have been put out to address these problems.\nThere are many open problems in semantic segmentation using ViTs like Joint Multi-Modality Learning, Long Video Unified Segmentation, and Neural Architecture Search for optimal structures. Due to their huge data and computational requirements, it is also important to find out methods to create models which can be trained and deployed on mobile and other edge devices while using small datasets for training. \n\n# ** Conclusion **\n\nViTs are still in the early stages of development, in contrast, to mature convolutional neural networks. However, we saw how strong and competitive they are with their CNN rivals. ViTs are improving, and it is anticipated that in the near future, they will take the place of conventional CNN-based deep learning techniques. \nDespite their success, ViTs might be difficult to work with due to their high computing requirements. Therefore, the ViT design needs to be improved in order to make it lighter and more effective. The community will be motivated by this to create new ViTs-based pathways. We methodically list the major benefits and drawbacks of the current approaches, and we especially go into detail about the crucial directions for future research. \nWe think that ViT and semantic segmentation can be used in a wide range of new study areas to address practical issues.\n\n# ** References: **\n\nBaevski, Alexei, Arun Babu, et al. “Efficient Self-Supervised Learning with Contextualized Target Representations for Vision, Speech and Language.” arXiv.Org, 15 June 2023, arxiv.org/abs/2212.07525.\n\nBaevski, Alexei, Wei-Ning Hsu, et al. “Data2vec: A General Framework for Self-Supervised Learning in Speech, Vision and Language.” arXiv.Org, 25 Oct. 2022, arxiv.org/abs/2202.03555.\n\nDing, Jian, et al. “Decoupling Zero-Shot Semantic Segmentation.” arXiv.Org, 15 Apr. 2022, arxiv.org/abs/2112.07910.\n\nGong, Chengyue, et al. “Nasvit: Neural Architecture Search for Efficient Vision...” OpenReview, 13 Feb. 2023, openreview.net/forum?id=Qaw16njk6L.\n\nGriebel, Matthias, et al. “Deep-Learning in the Bioimaging Wild: Handling Ambiguous Data with Deepflash2.” arXiv.Org, 12 Nov. 2021, arxiv.org/abs/2111.06693.\n\nHe, Kaiming, et al. “Deep Residual Learning for Image Recognition.” arXiv.Org, 10 Dec. 2015, arxiv.org/abs/1512.03385.\n\nKirillov, Alexander, et al. “Segment Anything.” arXiv.Org, 5 Apr. 2023, arxiv.org/abs/2304.02643.\nKiya, Hitoshi, et al. “Privacy-Preserving Semantic Segmentation Using Vision Transformer.” MDPI, 30 Aug. 2022, www.mdpi.com/2313-433X/8/9/233.\n\nLin, Jingbo, et al. “San: Scale-Aware Network for Semantic Segmentation of High-Resolution Aerial Images.” arXiv.Org, 6 July 2019, arxiv.org/abs/1907.03089.\n\nShenwai, Tanushree. “Meta AI Releases Data2vec 2.0: An Efficient Self-Supervised Learning for Machine Learning Tasks.” MarkTechPost, 15 Dec. 2022, www.marktechpost.com/2022/12/14/meta-ai-releases-data2vec-2-0-an-efficient-self-supervised-learning-for-machine-learning-tasks/.\n\nSu, Xiu, et al. “Vitas: Vision Transformer Architecture Search.” arXiv.Org, 30 Nov. 2021, arxiv.org/abs/2106.13700.\n\nUltralytics. “Fastsam (Fast Segment Anything Model).” FastSAM (Fast Segment Anything Model) - Ultralytics YOLOv8 Docs, docs.ultralytics.com/models/fast-sam/. Accessed 15 July 2023.\n\nUltralytics. “Segment.” Ultralytics YOLOv8 Docs, docs.ultralytics.com/tasks/segment/#models. Accessed 15 July 2023.\n\nWang, Huiyu, et al. “Max-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers.” arXiv.Org, 12 July 2021, arxiv.org/abs/2012.00759.\n\nWang, Wenhui, et al. “Image as a Foreign Language: Beit Pretraining for All Vision and Vision-Language Tasks.” arXiv.Org, 31 Aug. 2022, arxiv.org/abs/2208.10442.\n\nZhao, Xu, et al. “Fast Segment Anything.” arXiv.Org, 21 June 2023, arxiv.org/abs/2306.12156.\n\nZhou, Daquan, et al. “Understanding the Robustness in Vision Transformers.” arXiv.Org, 8 Nov. 2022, arxiv.org/abs/2204.12451. \n\n# ** Code References: **\n\nhttps://github.com/MendelXu/SAN/tree/main\n\nhttps://github.com/google-research/deeplab2/blob/main/g3doc/projects/max_deeplab.md\n\nhttps://github.com/IDEA-Research/MaskDINO\n\nhttps://huggingface.co/openmmlab/upernet-swin-base\n\nhttps://huggingface.co/facebook/sam-vit-base\n\nhttps://huggingface.co/facebook/data2vec-vision-base\n\nhttps://huggingface.co/CIDAS/clipseg-rd64-refined\n\nhttps://docs.ultralytics.com/tasks/segment/\n\nhttps://docs.ultralytics.com/models/fast-sam/\n\nhttps://huggingface.co/nicoelbert/deepflash2_testrepo\n\nhttps://github.com/dingjiansw101/ZegFormer\n\nhttps://huggingface.co/shi-labs/oneformer_ade20k_swin_large\n\nhttps://huggingface.co/openmmlab/upernet-convnext-base\n\nhttps://huggingface.co/apple/deeplabv3-mobilevit-small\n\nhttps://huggingface.co/facebook/mask2former-swin-base-coco-panoptic\n\nhttps://huggingface.co/Intel/dpt-large\n\nhttps://huggingface.co/facebook/maskformer-swin-base-ade\n\nhttps://huggingface.co/microsoft/beit-base-finetuned-ade-640-640\n\nhttps://huggingface.co/nvidia/segformer-b5-finetuned-ade-640-640\n\nhttps://huggingface.co/facebook/detr-resnet-50 \n\n","metadata":{}}]}