{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Advances in AI Vision Models in the Last Two Years

\n \n

By: Dmitri Kalinin

\n\n

Category: Computer Vision

\n","metadata":{}},{"cell_type":"markdown","source":"\n## Introduction","metadata":{}},{"cell_type":"markdown","source":"      AI as a field has had a spike in growth the last two years, and the purpose of this article is to highlight the advances in AI experiments with image and video data. At an MIT event, Sam Altman, founder of Open AI, declared that “The era of giant models is over”, as just adding more neurons to a machine learning model is getting to the point of diminishing returns (Knight, 2023). To prove Altman’s point, an important field in AI experimenting with image and video-based data in the last two years is finding ways to extract more information from a limited starting amount of information using machine learning: such as utilizing few- and zero-shot methods to make effective models with little data and annotation. However, one cannot condense the advances of AI in computer vision only to this.\n\n      This essay covers advances in six main topics in computer vision, starting with semantic segmentation. From semantic segmentation, a discussion of the newly-released Segment Anything Model follows. Then, recent advances in vision transformers are discussed, followed by sections on few- and zero- shot learning. After those parts of the article, generalization of AI models is examined, going from there to explore Continual Learning and, lastly, Human-assisted AI. Although this examines the last two years, an emphasis is put on more recent studies, as they are the best indicator for the future direction of research in these areas. For each field, important subsections are pointed out and explored, as well as every studies’ comparison to previous state-of-the-art models in their respective fields. \n","metadata":{}},{"cell_type":"markdown","source":"\n## Semantic Segmentation","metadata":{}},{"cell_type":"markdown","source":"      First off, the task of semantic segmentation, where every pixel of an image is classified as being part of a particular class within the image (example: the canonical problem of identifying a cat), has always been important for a large variety of practical problems such as medical imaging and autonomous driving. As always, there have been innovations in training approaches to improve semantic segmentation accuracy. New loss functions have been developed to improve semantic segmentation performance without increasing computational cost, such as a hardness-weighted segmentation loss (Xiao et al., 2023), and some, such as minimum completion distance, have been combined with new semantic segmentation models to achieve state-of-the-art (SOTA) performance (Konz et al., 2023). A new neuron architecture was developed to handle learning data with reflectional variance for a Point-Cloud model (Cui, Y. et al., 2023), and adversarial approaches for estimation of both positives and negatives have been developed to counter Out of Distribution classes and In-Distribution Miscalculation (Gudovsky et al., 2023). \n\n\n#### Semantic Segmentation in Autonomous Driving\n\n\n      One field that semantic segmentation has been used in more these past two years is for autonomous driving. For this, there are ten challenges for the future research in the field as established by [Barbosa and Osorio, 2023]: dealing with data scarcity, alternative learning procedures (weakly and semi-supervised learning), using synthetic data, unsupervised domain adaptation, dealing with data imbalance, robustness to unseen classes, making a standardized evaluation criteria, efficiency-oriented model development, RGB-depth data integration, and models that use temporal information. For 3D [LiDAR](https://oceanservice.noaa.gov/facts/lidar.html#:~:text=Lidar%20%E2%80%94%20Light%20Detection%20and%20Ranging,the%20surface%20of%20the%20Earth.), new loss functions have been developed to better utilize background information while maintaining the same efficiency (Mei et al., 2023), and the use of pre-trained transformers to train models for autonomous driving have achieved SOTA while drastically reducing training time (Ando et al., 2023). Multimodal approaches using both camera and LiDAR data have been developed and optimized to increase robustness and balance efficiency and accuracy (Li et al., 2023). And methods have been developed to fully utilize spatio-temporal information for unsupervised and source-free domain adaptation in Video Semantic Segmentation (Lo et al., 2023). \n\n\n#### Semantic Segmentation in Medical Imaging\n\n\n      New semantic segmentation approaches have also been utilized in the field of medical imaging. A recent semantic segmentation model based on pre-trained vision representation and segmentation networks so the model can successfully capture global and local features achieved state-of-the-art performance in medical images (Mashayekhi et al., 2023). Similar to autonomous driving, multi-modal methods have been developed and optimized to utilize both uni- and multi-modal information, achieving state-of-the-art performance on medical segmentation tasks such as Brain Tumor segmentation (Marinov et al., 2023).\n\n      [Weakly supervised semantic segmentation](https://paperswithcode.com/task/weakly-supervised-semantic-segmentation) has been becoming particularly popular in medical imaging, as it is an efficient way to deal with the need to label large amounts of image data due to only having image-level labels. For this field, most studies use the BRATS (Menze et al., 2014; Bakas et al., 2018), DECATHLON (Antonelli et al., 2022), and the Breast Cancer Ultrasound (BUSI) dataset (Al-Dhabyani et al., 2020). One such weakly supervised approach combined low and high threshold Class Activation Maps (CAMs) to average out the location of the target, proving to be SOTA (Ostrowski et al., 2023). Another weakly-supervised method for medical imaging revolves around combining a unsupervised segmentation network’s outputted edge map with a response map using GradCAM developed by [Selvaraju et al., 2017] and then refining boundaries using BoundaryFit, improving up to 10% on current medical imaging datasets (Prabakaran et al., 2023).\n\n","metadata":{}},{"cell_type":"markdown","source":"\n## Segment Anything Model (SAM)","metadata":{}},{"cell_type":"markdown","source":"      A new large model made for semantic segmentation deserves its own section: the Segment Anything Model (SAM). Similar to ChatGPT and VGG16, it is a foundational model for the field of semantic segmentation, able to segment any image to a high degree of accuracy for most cases when given prompts for what mask to make. It uses an image encoder and prompt encoder (that considers both sparse prompts of points, boxes, and text and dense prompts of masks) to process image and prompt inputs, then uses a mask decoder (modeled after a transformer block) to decode those encoded features into the masked image, averaging multiple valid masks if the prompt is ambiguous (Kirillov et al., 2023). The model also comes with an automatic segmentation feature, where it automatically finds every point that it thinks has a unique mask and than segments the whole image using those masks. The dataset made for the training of this model, which consists of 11 million high resolution training images with 11 billion total masks (99.1% that were generated completely autonomously with relatively high mask quality compared to all the datasets), has been made publicly available by the authors to help with the creation of more foundational computer vision models (Kirilov et al., 2023). It even shows promise in zero-shot mask transfer to other datasets, yielding higher [mIoU](https://hasty.ai/docs/mp-wiki/metrics/iou-intersection-over-union#:~:text=To%20define%20the%20term%2C%20in,matches%20the%20ground%20truth%20data.) (Mean Intersection over Union) than the next strongest baseline, RITM, on 16 of 23 other datasets, giving it consistently higher mIoU on average (Kirilov et al., 2023). A demonstration of SAM's multiple prompting features is shown in the figure below.\n\n\n\n**Figure 1**. A demonstration of SAM using the interactive demo on an image of a cat. The 3 images utilize 3 different tools of SAM discussed in the above section: point prompting, box prompting, and the automatic segmentation feature.\n\n\n#### Limitations of SAM\n\n\n      With all of the model’s strengths, however, its effectiveness in more niche, difficult tasks has been questionable, particularly its vulnerability against [adversarial attacks](https://en.wikipedia.org/wiki/Adversarial_machine_learning). Detecting transparent glass objects that are seen in a mirror proves difficult for SAM, although performance compared to SOTA models for images with mirrors far away is similar (Han et al., 2023). The model is also very vulnerable to basic adversarial attacks to change its mask prediction (Zhang et al., 2023). Additionally, it was found that SAM is vulnerable to transfer-based attacks, which directly attack one point-prompt-image pair to affect SAM’s performance on a target point-prompt-image pair for mask removal (Zhang et al., 2023). An evaluation of the robustness of SAM in the presence of various types of image corruptions found that except for zoom blur, SAM still performs well on the most corruptions found in real-world data, including various common corruptions, blurring, and natural disturbances. However, under local occlusion ratios of 60% or higher (meaning 60% or more of pixels are turned into Gaussian noise), SAM’s performance drastically declines, and its vulnerability to adversarial attacks significantly increases (Qiao et al., 2023).\n\n\n#### Applications of SAM\n\n\n      Even though SAM has very many limitations, there have already been efforts to use SAM for useful computer vision applications. The model has been used to improve Weakly Supervised Semantic Segmentation models while significantly reducing computational cost (Chen, T., et al., 2023) and create a less complex SOTA model for zero-shot shadow removal in real world images (Zhang, X. et al., 2023). In an effort to decrease the computational cost of SAM, knowledge distillation was utilized to make a more lightweight image encoder that performs on the level of SAM but is 60 times smaller, taking only about 10 ms to segment images (Zhang, Cha. et al., 2023). SAM has been combined with the DINO network to make a SOTA model for image fusion (Xue et al., 2023). Fine-tuning SAM also achieves SOTA for sonar image segmentation tasks (Wang et al., 2023).\n\n      SAM has found many applications in medical imaging. Just the original SAM has shown promise in clinical radiotherapy segmentation tasks, with plenty of room for further development and optimization to make a true SOTA model in this area (Zhang, L. et al., 2023). SAM has been analyzed to see what prompt modes work the best for medical images, finding that box-prompt mode with zero jitters is the best for SAM on medical images (Cheng et al., 2023). Because of SAM’s vulnerability to prompt forms in Interactive Medical Image Segmentation, temporally extended prompt optimization networks were developed to offer proper prompt forms to human experts, increasing stability of SAM predictions (Shen et al., 2023). Ways to build off of or augment SAM’s architecture have already successfully achieved SOTA performance on segmentation tasks, such as adding a complementary CNN encoder to reduce training time by only training the CNN encoder and SAM decoder (Chai et al., 2023) and replacing the decoder with a CNN or ViT prediction head for task-specific promptless medical image segmentation (Hu et al., 2023).\n\n      Another recent direction of applied SAM research is interpretation of 3D image data. In an effort to model Thermal Protection Systems, SAM has been integrated with the Porous Microstructure Analysis Software to make a framework that is SOTA for 3D image scenarios, although more research needs to be done on it’s performance on noisy data and its applications in clinical scenarios (Semeraro et al., 2023). SAM has also been transferred to 3D for promptable medical image segmentation, setting a new SOTA for a large number of tasks in medical image segmentation (Gong, S., et al., 2023). \n\n\n#### Future Uses of SAM\n\n\n      There are many possible directions that SAM can expand to in the future. One logical area is automated driving, which is already extensively pursued by other semantic segmentation approaches. Work must be done to overcome the limitations of SAM discussed above, especially in overcoming adversarial attacks, before SAM can see use in real-world applications. An effort to make models that utilize SAM more explainable has the potential to establish the underlying causes of the vulnerabilities, thus providing a direction for future research. Applying SAM to 3D segmentation is also very promising, especially for clinical/medical imaging data. It is worth noting that SAM has only been out for two months, so the current trends in finding SAM limitations and application of SAM are subject to change. However, it is still necessary to highlight the current progress in this field, as SAM has the potential to accelerate innovation in the field of semantic segmentation applications through both novel techniques and increased researcher interest.\n","metadata":{}},{"cell_type":"markdown","source":"\n## Vision Transformers","metadata":{}},{"cell_type":"markdown","source":"\n\n      Vision transformers (ViTs) have gained more prominence in the last few years; SAM even used some elements of transformer architecture to make their model. ViTs do not use any convolutional layers to process images, instead ViTs decompose the image into multiple smaller sections, projects them linearally, and then applies an encoder block to process the linear inputs, similar to how transformers were intially proposed for natural language processing in ML (Vaswani et al., 2017). A GIF that visualizes ViT architecture can be found [here](https://en.wikipedia.org/wiki/Vision_transformer).\n\n      It was found that Contrastive Learning and Masked Image Modeling, both ways to implement ViTs, learn basically opposite from one another, with Contrastive Learning focusing more on global relationships and Masked Image Modeling focusing more on local features (Park et al., 2023). As such, Contrastive Learning, combined with Denoising Contrastive Regulation, has been utilized to denoise images with low light levels, with the output images having very high Peak Signal to Noise Ratios (Cui, T. and Dong, 2023). \n\n      Based on research that finds that the emergence of sparse activation maps is a commonality amongst all transformer architectures for both LLMs and vision models (Li, Z., et al., 2023) and in effort to capitalize on the possible reduction of computational budget, a Top-k thresholding algorithm was developed to make the activation map even sparser. This increased the efficiency of the model, increased the model’s robustness, made it better for confidence calibration, and decreased sensitivity to noisy training data while maintaining the same accuracy, questioning the entire usefulness of scaling up transformers at this point (Li, Z., et al., 2023). \n\n\n#### Vision Transformers for Medical Imaging\n\n      As for all other computer vision approaches, the use of Vision Transformers for Medical Imaging is extensively studied. Although vision transformers were found to be useful in many parts of the field, there are some areas that a review from early 2023 highlighted that could use improvement: explainability, richer feature representation, video-based analysis, reducing computational complexity, transformers for image registration, utilizing unsupervised learning to increase robustness and generalizability, and the development of a medical software ecosystem (Azad et al., 2023). For explainability of ViTs, it was found that Layerwise relevance propagation based off the TransLRP network (Chefer et al., 2021), makes a more explainable ViT than an Attention Visualization method based off of [Abnar and Zuidema, 2020], and Local interpretable model-agnostic explanations, a method based off the LIME (Ribiero et al., 2016) framework. [Komorowski et al., 2023] made that determination through evaluation of faithfulness correlation, average sensitivity, and effective complexity to make a unified explainability metric. Multi-stage feature-mixing loss aggregation and a multi-scale hierarchical vision transformer built off of MaxViT (Tu et al., 2022) has achieved SOTA performance on Synapse multi-organ and ACDC segmentation tasks (Rahman and Marculescu, 2023). Combining convolution blocks with transformer blocks to capture both local and global features has set a SOTA for generalization benchmarks for medical images while being less computationally expensive and less vulnerable to adversarial attacks (Manzari et al., 2023). Weight inflation has been utilized to transfer pre-trained ViT weights from 2D to 3D for medical image segmentation and then fine-tune them, achieving SOTA performance on many different medical imaging datasets (Zhang, Yu., et al., 2023). \n\n\n#### Knowledge Distillation in Vision Transformers\n\n     Knowledge distillation normally relies on a larger “teacher” model distilling knowledge in a smaller “student” model (Hinton et al., 2015). This is particularly important for ViTs due to the increased computational complexity of ViTs compared to CNNs. For medical image tasks it can help avoid tedious human annotation of large datasets. A study in early 2023 outlined possible future directions for knowledge distillation to improve ViTs: sparse attention map generation to increase efficiency, attention augmentation, explainability, and exploring the effects of scaling up knowledge distillation in ViTs (Habib et al., 2023). First distilling knowledge to a transformer from a CNN and then letting the transformer train by itself to both capture inductive biases and capture global patterns has achieved SOTA performance compared to a version that trains on the full ImageNet dataset and only 1.0% less accurate than the former for a variant that trains on no real data and only uses self-generated data (Chen et al., 2022). Using knowledge distillation through self-supervised learning and self-training, a vision transformer model was able to get gradually better based off a small amount of labeled images and the introduction of more unlabeled data, outperforming other models trained on the same amount of labeled data, showing the importance of unlabeled data for the training of the ViT (Park et al., 2022).","metadata":{}},{"cell_type":"markdown","source":"\n## Advances in Few-shot Tasks ","metadata":{}},{"cell_type":"markdown","source":"      Although knowledge distillation helps reduce the computational budget of ViTs, another way to make ViTs more efficient is to have them train on less data. A method that uses this is few-shot learning, where models learn from a small number of examples of each class. In few-shot image classification, similar to other computer vision fields, new loss functions have been developed for few-shot scenarios and utilize inter and intra-class information to significantly increase the accuracy of the model (Mendez-Ruiz et al., 2023). For few-shot video action classification, an approach that maximally utilizes both intra- and inter-video information by amplifying critical frames and spaces achieves SOTA performance on many datasets (Liu et al., 2023). New loss functions were also developed for few-shot video action classification based on latent space semantic adjacency and domain prototypical similarity, performing on the same level as SOTA models that were trained on more data (Peng et al., 2023).\n\n#### Few-shot Segmentation\n\n\n      As common to all computer vision tasks, few-shot learning has been applied to semantic segmentation to reduce the memory and storage footprint of those models. In few-shot semantic segmentation (FSSS), there are three major strategies: (1) conditional networks, where the model has a conditional branch that generates a parameter set from a support set and a segmentation branch that uses the parameter set and the input image to make a final segmentation mask, (2) prototypical networks, which use a module to compute class representative prototypes to serve as references for pixel-level segmentation, and (3) latent space optimization (Catalano and Matteucci, 2023). With new methods developed to exploit high certainty areas, a model was able to perform on the level of fully supervised methods in thigh muscle (excluding intramuscular fat) segmentation tasks while only using 1% of the data that the fully supervised models use, providing a baseline for future FSSS studies in this area (Chen, S., et al., 2023). A meta learning episodic training strategy that explicitly simulates structural variances and domain shifts in the training process was developed to address unsupervised domain adaptation (UDA) in WSSS. When tested on the MM-WHS 2017 dataset (Zhuang and Shen, 2016), this new approach performs better than various SOTA models for few-shot UDA tasks (Zhao et al., 2023). To properly adapt to variable intra-class differences from the few amount of support set samples, a new method was developed that extracts a general class prototype from a word embedding to supplement instance information. On the Pascal-5 (Hariharan et al., 2014) and COCO-20 (Nguyen and Todorovic, 2019) datasets, the model outperforms previously SOTA benchmarks for FSSS and each component of the model effectively contributes to making that happen (Yang et al., 2023).\n\n#### Few-shot Object Detection\n\n\n      Like FSSS, Few-shot object detection (FSOD) has also been of great interest, as mitigating the need for too many labeled data points has proved useful for applications such as Wise medical (Quellec et al., 2020; Wei, 2021). One study outlined areas that need to be explored further: domain transfer, efficient architectures, robust feature extractors, mixed-supervised learning, unsupervised learning, and data augmentation (Leng et al., 2021). The common benchmarks used to evaluate FSOD models include the Pascal VOC (Everingham et al., 2010) and MS COCO (Lin et al., 2014) datasets. A meta learning approach, where machine learning algorithms learn from each other, was utilized to improve fine tuning of FSOD models, outperforming the competitive fine-tuning-based FSOD models MPSR (Wu et al., 2020) and DeFRCN (Qiao et al., 2021) on the common benchmarks, showing promise for further advances in combining meta learning and fine-tuning approaches for FSOD (Demirel et al., 2023). In the field of data augmentation, a Variational Autoencoder (VAE) architecture was developed to increase crop-related diversity and reliably generate tasks of certain difficulty for incremental training. When the DeFRCN model (Qiao et al., 2021) was fine-tuned on the new VAE-generated features, its performance was significantly improved on both the Pascal VOC (Everingham et al., 2010) and MS-COCO (Lin et al., 2014) benchmarks, by 5.8% on one-shot tasks, 4.6% on two-shot tasks, and 5.8% on three-shot tasks (Xu, Jin., et al., 2023). New training networks were developed to improve generalization on new classes while not losing accuracy on the base classes through increasing inter-class separation and intra-class compactness. This achieves both great performance on prediction of base classes and adaptation efficiency for novel classes on the Pascal VOC (Everingham et al., 2010) and MS-COCO (Lin et al., 2014) benchmarks, as well as the LVIS dataset (Gupta et al., 2019), showing how it can be extended to long-tailed object detection (Ma et al., 2023). To minimize memory, a data-free knowledge distillation (DFKD) approach for generalized FSOD (G-FSOD) was developed to generate instance-level features without accessing the base images by utilizing regions of interest. This method reduces memory requirements by two orders of magnitude (150 MB to 4 MB) while improving performance on both commonly used benchmarks (Guirgus et al., 2023).","metadata":{}},{"cell_type":"markdown","source":"\n## Advances in Zero-Shot Computer Vision Tasks","metadata":{}},{"cell_type":"markdown","source":"      A logical extension of few-shot learning, where the model learns with only a small number of samples from each class, is zero-shot learning, where the model aims to recognize classes that were not seen during training (Larochelle et al., 2008). Especially with the introduction of more sophisticated vision and language models throughout the years and LLMs, pre-trained Vision Language (VL) models, like CLIP (Radford et al., 2021), BLIP (Li, J., et al., 2022), CoCa (Yu et al., 2022), Stable Diffusion (Rombach et al., 2022), and DinoV2 (Amir et al., 2022), are the most widely used for generalized zero-shot learning. In fact, in semantic correspondence tasks, Stable Diffusion has shown to have high-quality spatial information but sometimes has inaccurate semantic matches, while DinoV2 has accurate semantic matches but low-quality spatial information. Thus, a simple fusion of the features of both models for zero-shot semantic correspondence tasks performs better than SOTA semantic correspondence methods, while enabling high-quality object swapping without task-specific fine tuning (Zhang, J., et al., 2023). With the use of LLMs, an approach was developed for label-free parameter efficient fine tuning by substituting labels with a set of texts describing categories of interest, improving an average of 3.8% in the label-free setting, outperforming zero-shot performance of the base VL model, and average gains of 1.3% over few-shot baselines that do 5-shot supervision (Mirza et al., 2023). To make a model better for compositional zero-shot learning (CZSL), the goal of which is to recognize novel object compositions, placing adapters within the model, something that has already shown to improve LLMs (Houlsby et al., 2019), has found use in the CLIP architecture so specific features of the images can be extracted. On the MIT-States (Isola et al., 2015), C-GQA (Naeem et al., 2021), and UT-Zappos datasets (Yu et al., 2017), this approach outperforms existing benchmarks by a significant margin, even double digit percent improvements as measured by AUC in some cases (Zheng et al., 2023). There still is much more to be explored with LLMs and VL models in zero-shot image tasks, and this will be an even bigger topic in the next two years as more people begin to learn how to use GPT-4 and SAM. ","metadata":{}},{"cell_type":"markdown","source":"\n## Generalization of Vision Models ","metadata":{}},{"cell_type":"markdown","source":"      While zero-shot and few-shot methods aim to perform on par with SOTA computer vision models with less data, it makes sense that there have been a plethora of studies to try to generalize computer vision models. With the increased use of foundational models like ChatGPT and SAM, there is an increased demand for the ability for models to become more generalized. However, generalization is a big challenge for these models. One study that evaluated Human-Object Interaction (HOI) detection models on a new dataset made to evaluate a model’s systematic generalization capabilities found that HOI detection models performed much more poorly on the new dataset, highlighting the challenge of generalization. However, this study suggests four ways that systematic generalization performance of HOI models can be improved: increasing diversity of the training data, introducing modular structures in the models, pretraining the encoder and decoder models, and integrating knowledge derived from natural language resources (Takemoto et al., 2023). Another study highlighted how most models cannot generalize when data from one mode (either video, audio, or inertial motion sensor modalities) is missing for egocentric action recognition by evaluating models on a benchmark dataset that evaluates both missing modality generalization and cross-modal zero-shot generalization. However, the authors of the study also developed a strong baseline method for the benchmark that is composed of three stages: unimodal supervised pre-training (so every modality is trained separately first), multi-modal supervised pre-training to form a multi-modal system, and then multi-modal meta training to further improve cross-modal generalization (Gong et al., 2023). \n\n      Even though generalization has its challenges, there have been many efforts to advance domain generalization in the computer vision landscape. For microscopy images, a domain generalization technique called ContriMix was developed. It learns to generate synthetic images by accounting for both inter- and intra- domain information of the images, so that the images it forms have real content and attributes for domain generalization. It outperforms SOTA methods in two histopathology datasets from [Bandi et al., 2018] and achieves SOTA performance in a fluorescence microscopy dataset, showing the effectiveness of the method (Nguyen et al., 2023). Another domain generalization method utilized synthetic image generation for robust face recognition to help resolve privacy issues with the large amount of photos of people’s faces on the internet, as well as image quality for dataset, noisy labels, lack of annotations, and the large differences that exist in the dataset. This achieves state-of-the-art performance compared to facial recognition on the LFW (Huang et al., 2008), AgeDB-30 (Moschoglu et al., 2017), CA-LFW (Zheng et al., 2017), CFP-FP (Sengupta et al., 2016), and CP-LFW (Zheng et al., 2018) benchmarks (Chung et al., 2023). A transformer, consisting of three encoder blocks and three decoder blocks, and residual upsampling network called TransRUPNet was developed for polyp segmentation, particularly to address out-of-distribution generalization. This model outperformed other SOTA models on one in-distribution dataset from [Jha et al., 2020] by at least 1.15% mIoU and three out-of-distribution datasets from [Ali et al., 2023; Lan et al., 2021; Bernal et al, 2015] by at least 3.43% mIoU, significantly improving not just with out-of-distribution data, but with in-distribution data as well (Jha et al., 2023).\n\n      As an effort to develop a method to improve domain generalization, a study utilized Shape Guided Gradient Voting (SGGV). This technique adds extra inputs to provide shape guidance and a gradient voting approach to balance all the gradients being used to update [weights](https://deepai.org/machine-learning-glossary-and-terms/weight-artificial-neural-network) (which overfits the training set) and only having non-conflicting gradients update weights (which are sensitive to outliers). This has achieved SOTA performance on domain generalization datasets such as PACS, VCLS, and OfficeHome, with performance growth from 2.70% to 11.36% compared to a model that has all gradients being used to update weights (Xu, Jia., et al., 2023).","metadata":{}},{"cell_type":"markdown","source":"\n## Continual Learning ","metadata":{}},{"cell_type":"markdown","source":"      Foundational models need massive datasets, of which the internet is the logical source. To account for the continuously changing nature of this “dataset”, a new method known as online continual learning (OCL) is being developed to improve learning efficiency. In OCL, the model is trained on continuously changing datasets online, where a new incoming batch of data is first used as testing data and then turns into new training data. Because of the unique way that Online Continual Learning models are trained, new benchmarks have to be developed to address the distribution shifts, and common practices to increase the accuracy of traditional models, such as increasing [batch size](https://machinelearningmastery.com/difference-between-a-batch-and-an-epoch/) and using the maximum possible [replay buffer size](https://www.tensorflow.org/agents/tutorials/5_replay_buffers_tutorial), actually decrease the accuracy of models trained by OCL (Cai et al., 2021). New metrics have been developed to allow the model to retain and reuse past information to improve its generalization capability and prevent overfitting through changing accuracy measurements (Hammoud et al., 2023). A variant of experience replay called Strong Experience Replay (SER) was developed that utilizes future experiences mimicked on current training data as well as replaying past training data to preserve training knowledge. By reducing the issue of forgetting, this approach outperforms other SOTA approaches considerably on many datasets (Zhuo et al., 2023). A fairness CL approach to analyze the effect of class distribution in CL was developed in conjunction with a new loss function to help with catastrophic forgetting, outperforming SOTA semantic segmentation methods using this new fairness continual learning (Truong et al., 2023). \n\n#### Increasing Practicality of Continual Learning\n\n\n      Predictably, a lot of CL methods are very computationally intensive, which puts them at a disadvantage in practical real-time benchmarking tests. If the stream does not wait for the model to finish training before making the data for the next prediction, longer-training state-of-the-art CL algorithms do not perform as well. This invites the still ongoing research of making CL algorithms more practical so they can be deployed to real-world datasets (Ghunaim et al., 2023). One such way was to use a new CL setup which assumes that the model continuously encounters new classes as the tasks continue and that classes overlap across tasks. That, in conjunction with a new memory management, memory usage, and learning rate scheduling strategy, was able to outperform contemporary CL models by large margins on multiple datasets (Koh et al., 2022). To solve the memory overfitting and model underfitting problems present in CL, a baseline of Repeated Augmented Rehearsal (RAR) was developed to balance the amount of [experience replay](https://paperswithcode.com/method/experience-replay#:~:text=Experience%20Replay%20is%20a%20replay,episodes%20into%20a%20replay%20memory.) utilized in CL regimes, with models trained using this baseline outperforming models trained using normal experience replay (ER) and other SOTA ER methods (Zhang, Ya. et al., 2022). To address memory and storage problems present with CL, the Summarizing Stream Data (SSD) method was developed to integrate information from the original images into more informative samples via knowledge distillation, becoming more representative of the dataset while reducing the amount of images. Not only does this memory-restricted model achieve performance comparable to SOTA models without memory restrictions, it outperforms them on the sequential CIFAR-100 dataset by more than 3% (Gu et al., 2023). With no storage constraints but reducing computational budget, a simple algorithm that uses a kNN classifier and universal pre-trained feature extractors to compactly store and utilize all incoming data was developed so it would never forget past-seen data. This outperformed previously SOTA models that had higher computational budgets (Prabhu et al., 2023).\n\n      Increased practicality is pretty much a necessity for CL algorithms applied to video learning. Although earlier approaches utilized [subsampling](https://en.wikipedia.org/wiki/Sub-sampling_(chemistry)) frames from the video stream and store them into memory, which is extremely computationally expensive (Villa et al., 2022), a new approach has been developed for memory-limited applications that learns from a small number of individual frames that represent a large number of unique videos, outperforming previous state-of-the-art models by up to 21.49% (Alssum et al., 2023).\n\n#### Continual Learning to Mirror Human Learning\n\n\n      As the CL approach is close to how humans learn, multiple suggestions to improve upon CL come from how the human brain learns, such as reconsidering memory introspection, memory prospection and memory transfer (Chang et al., 2021). Thus efforts have been made to make CL learning approaches more similar to biological learning. A bio-inspired approach was utilized that introduces noise at various stages of the Vision Transformer training while utilizing a memory efficient and corruption-robust CL approach. This model outperforms other SOTA CL models for average accuracy across all buffer sizes (with the accuracy gain increasing the smaller the buffer size), using controllable noise to help to prevent overfitting (Jeeveswaran et al., 2023). Incorporating how the brain replays processed neural patterns instead of raw input and prioritizes the replay of recently learned information, a new neuro-inspired CL approach called SHARP was introduced. It can continually update all layers, not just layers that have been subjected to replay, and only needs to replay a few recently seen classes as opposed to all past classes. This method both outperforms existing SOTA CL methods and can perform well with episodes that aren’t as well-structured (Gurbuz et al., 2023).\n","metadata":{}},{"cell_type":"markdown","source":"\n## Human-Assisted AI in Computer Vision","metadata":{}},{"cell_type":"markdown","source":"      Making continual learning approaches more similar to human learning is only part of a bigger field of having AI understand the visual reasoning process at a more human-like level or to use human reasoning to shape the reasoning of AI. A common task for this field was already mentioned in this review: improving explainability of computer vision models. One way of giving AI human visual sensibilities proposes the use of transformation-driven learning: where a model is given the initial and final states and then has to figure out the transformation, as opposed to being trained to answer certain questions only about one state. Testing SOTA visual reasoning models on a dataset constructed with single-step and multi-step transformation (including transformations with different views), it was concluded that they only perform well on single-step data, highlighting the need for advances in AI research in this area (Hong et al., 2023). Utilizing a dataset of human attention gaze data on the CXR-Eye chest X-ray dataset (Karargyris et al., 2020) for human attention guidance (HAG), a classifier became 4.38% more accurate than similar models that didn’t use HAG (Rong et al., 2021). Another study added on human attention guidance to two standard CNNs, ResNet50 (He et al., 2016) and EfficientNetV2 (Tan and Le, 2019), and, when applied on the same CXR-Eye dataset (Karargyris et al., 2020), outperformed other SOTA models without HAG (Zhu et al., 2022). Another study evaluated more generally how the plausibility and faithfulness of the saliency map of a computer vision model compares with the plausibility and faithfulness of a human attention map, finding that human attention maps are better for explaining object detection while normal saliency maps work best for explaining classification tasks well. Thus, Human-Attention-Guided explainable AI (HAG-XAI) models for computer vision were developed to make the model’s saliency maps match human attention maps, sacrificing faithfulness for plausibility for image classification models but gaining both explanation plausibility and faithfulness for object detection models. (Liu et al., 2023). \n\n

Human-in-the-Loop Methods for Computer Vision

\n\n      Another way that human reasoning is incorporated into AI models is through human-in-the-loop methods, which have shown promise in effectively combining human and AI information to improve generalization, explainability, efficiency, and causal understanding in models (Munro, 2020). These methods use humans as the ground truth, making sure that AI models always have something to fall back on as known. A figure visualizing this process can be found below.\n\n\n\n**Figure 2**. A basic visual description of human-in-the-loop machine learning, examining each part of the machine learning process.\n\n      As the field advances, the important trends are increased interactivity, increased automation for most tasks, and the utilization of human-in-the-loop methods for domain-specific tasks (Mosqueira-Rey et al., 2022). A R-CNN network that used human-specified boundaries for instance segmentation significantly outperformed most R-CNNs on the Partial Sketch Object Boundaries (PSOB) dataset and has the potential to be extended to other computer vision tasks (Korkmaz et al., 2023). To improve the interoperability of a model, interactive human annotation was utilized to direct the attention of a classifier to regions specified by users, with an [active learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)) approach to massively reduce the amount of annotations needed for the model to make accurate predictions. The model performed well and had interpretable attention maps on the COCO-2017 and CelebA datasets, with less required user input and more efficiency compared to previous methods (He et al., 2023). A human-in-the-loop method that involves only fast vision checks of sorting segmented images on if they were segmented well or poorly was able to improve the accuracy of simple segmentation threshold algorithm for a cardiac ultrasound segmentation and outperform more complex algorithms without human-in-the-loop methods that train on more data (Zahn and Perrin, 2023). \n\n      Human-in-the-loop methods have also found many applications in interactive robot training. One such study utilized a human-in-the-loop approach and reinforcement learning to increase the efficiency and effectiveness of surgical robot learning in a simulated environment based off SurRoL (Xu, Jia., et al., 2021), with the human-in-the-loop method using human experts outperforming non-human-in-the-loop methods and human-in-the-loop methods not using experts (Long et al., 2023).\n\n      Although human-in-the-loop methods allow for great advances in the accuracy of AI models for complex tasks, the decrease in efficiency due to human annotation efforts and the increase in bias that humans can introduce can make it undesirable over other automated methods, especially when there is no comparable accuracy increase in the task at hand. In a curriculum learning approach, where models are trained from easier to harder samples, instead of using a human-in-the-loop approach to evaluate the difficulty of the task, a new automated Variance of Gradients (VoG) metric was used to evaluate the objective difficulty of tasks in an elbow fracture classification dataset, achieving similar or higher performance than other human-in-the-loop methods on the dataset (Zhou et al., 2023). Of course, human-in-the-loop learning is still a viable strategy for this task, and measures different aspects of difficulty than the VoG method, but sometimes it is not the clear-cut best for every ML task. \n\n      However, although human-in-the-loop methods have some limitations, there are still many future avenues in this field of computer vision that need to be explored. One such field is synthetic data imaging, where it has been found that human perception and labels of synthetic examples generated by mixup, which generates synthetic linear combinations of existing training examples, don’t align with each other. Although the mixup technique has shown to improve model robustness, generalization, and calibration (Zhang et al., 2018), aligning mixup data with human perception, especially with introducing human uncertainty judgements, has the potential to make more safe, reliable, and improved generalization models (Collins et al., 2023).\n","metadata":{}},{"cell_type":"markdown","source":"\n## Discussion and Conclusion ","metadata":{}},{"cell_type":"markdown","source":"      In this article, a review is presented for six main topics in advances in computer vision study in the last two years: Semantic Segmentation (including a deep dive into the new foundational model: SAM), Vision Transformers, Few- and Zero-shot Learning, Generalization of Computer Vision Models, Continual Learning, and AI Assisted by Humans in Computer Vision. For each field, certain subfields are discussed in detail, with recent studies highlighting directions for future research. A lot of future work in these areas will revolve around combining new techniques from each area together, such as using continual learning to make a more explainable model that uses SAM for medical image segmentation, with an aspiration to produce something greater than the sum of its parts. \n\n      A conceptually new place where these advances in computer vision can be applied is the development of more sophisticated image-language models, helping to usher in more powerful foundational models that allow for more ease of use in the AI community and innovation on top of those large models. \n\n      Another thing to look out for is the expanding real-world footprint, from experimental to engineering tasks, especially as certain roadblocks are circumvented for applications like autonomous driving, medical robots, and automated experiments. It is difficult to predict how many new roadblocks will emerge in translating models that outperform previous state of the art into inventions that can make a true impact in the real world, but every roadblock found will expand our understanding of these models and eventually help identify pathways to improvement.\n\n","metadata":{}},{"cell_type":"markdown","source":"\n## Sources","metadata":{}},{"cell_type":"markdown","source":"Abnar, S., and Zuidema, W. H. “Quantifying Attention Flow in Transformers.” In *ACL*, 2020. https://arxiv.org/pdf/2005.00928. \n\nAl-Dhabyani, W., Gomaa, M., Khaled, H., et al. “Dataset of breast ultrasound images.” *Data in brief*, 28:104863, 2020. \n\nAli, S., Jha, D., Ghatwary, N., et al. “A multicentre polyp detection and segmentation dataset for generalisability assessment.” *Scientific Data*, vol. 10, no. 1, p. 75, 2023. https://www.nature.com/articles/s41597-023-01981-y. \n\nAlssum, L., Alcazar, J. L., Ramazanova, M., et al. “Just a Glimpse: Rethinking Temporal Information for Video Continual Learning.” *Arxiv Preprint*, June 28, 2023. https://arxiv.org/pdf/2305.04769.pdf.\n\nAmir, S., Gandelsman, Y., Bagon, S., et al. “Deep ViT Features as Dense Visual Descriptors.” *Arxiv Preprint*, October 15, 2022. https://arxiv.org/pdf/2112.05814.pdf. \n\nAndo, A., Gidaris, S., Bursuc, A., et al. “RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving.” *Arxiv Preprint*, April 25, 2023. https://arxiv.org/pdf/2301.10222.pdf. \n\nAntonelli, M., Reinke, A., Bakas, S., et al. “The medical segmentation decathlon.” *Nature Communications*, 13(1):1–13, 2022. https://www.nature.com/articles/s41467-022-30695-9. \n\nAzad, R., Kazerouni, A., Heidari, M., et al. “Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review.” *Arxiv Preprint*, January 10, 2023. https://arxiv.org/pdf/2301.03505.pdf. \n\nBakas, S., Reyes, M., Jakab, A., et al. “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge.” *Arxiv Preprint*, April 23, 2019. https://arxiv.org/pdf/1811.02629.pdf. \n\nBandi, P., Geessink, O., Manson, Q., et al. “From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge.” *IEEE Transactions on Medical Imaging*, 2018. https://ieeexplore.ieee.org/document/8447230. \n\nBarbosa, F. and Osorio, F. “A Threefold Review on Deep Semantic Segmentation: Efficiency-Oriented, Temporal, and Depth-Aware Design.” *Arxiv Preprint*, March 8, 2023. https://arxiv.org/pdf/2303.04315.pdf. \n\nBernal, J., Sanchez, F. J., Fernandez-Esparrach, G., et al. “WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians.” *Computerized Medical Imaging and Graphics*, vol. 43, pp. 99–111, 2015. https://www.sciencedirect.com/science/article/abs/pii/S0895611115000567. \n\nCai, Z., Sener, O., and Koltun, V. “Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data.” *IEEE/CVF International Conference on Computer Vision (ICCV)*, August 20, 2021. https://openaccess.thecvf.com/content/ICCV2021/papers/Cai_Online_Continual_Learning_With_Natural_Distribution_Shifts_An_Empirical_Study_ICCV_2021_paper.pdf.\n \nCatalano, N., and Matteucci, M. “Few Shot Semantic Segmentation: a review of methodologies and open challenges.” *Arxiv Preprint*, April 12, 2023. https://arxiv.org/pdf/2304.05832.pdf. \n\nChai, S., Jain, R. K., Teng, S., et al. “Ladder Fine-Tuning Approach for SAM integrating complementary network.” *Arxiv Preprint*, June 22, 2023. https://arxiv.org/pdf/2306.12737.pdf.\n \nChang, Y., Li, W., Peng, J., et al. “Reviewing Continual Learning from the perspective of human-level intelligence.” *Arxiv Preprint*, November 23, 2021. https://arxiv.org/pdf/2111.11964.pdf. \n\nChefer, H., Gur, S., and Wolf, L. “Transformer Interpretability Beyond Attention Visualization.” In *CVPR*, 2021. https://arxiv.org/pdf/2012.09838. \n\nChen, S., Tang, Z., Liu, D., et al. “Precise Few-Shot Fat-Free Thigh Muscle Segmentation in T1-Weighted MRI.” In *ISBI*, April 27, 2023. https://arxiv.org/pdf/2304.14053.pdf. \n\nChen, T., Mai, Z., Li, R., et al. “Segment Anything Model (SAM) Enhanced Pseudo-Labels for Weakly Supervised Semantic Segmentation.” *Arxiv Preprint*, May 9, 2023. https://arxiv.org/pdf/2305.05803.pdf.\n\nChen, X., Cao, Q., Zhong, Y., et al. “DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers.” In *CVPR*, April 28, 2022. https://arxiv.org/pdf/2204.12997.pdf. \n\nCheng, D., Qin, Z., Jiang, Z., et al. “SAM on Medical Images: A Comprehensive Study on Three Prompt Modes”. *Arxiv Preprint*, April 28, 2023. https://arxiv.org/pdf/2305.00035.pdf.\n\nChung, C., Chang, P., Chen, Y., et al. “SASMU: boost the performance of generalized recognition model using synthetic face dataset.” *Arxiv Preprint*, June 2, 2023, https://arxiv.org/pdf/2306.01449.pdf. \n\nCollins, K., Bhatt, U., Liu, W., et al. “Human-in-the-loop Mixup”. *In Uncertainty in Artificial Intelligence* (selected for oral presentation), March 7, 2023. https://arxiv.org/pdf/2211.01202.pdf. \n\nCui, T., and Dong, Y. “Contrastive Learning for Low-Light Raw Denoising.” *Arxiv Preprint*, May 5, 2023. https://arxiv.org/pdf/2305.03352.pdf.\n\nCui, Y., Ruan, L., Dong, H-C., et al. “Cloud-RAIN: Point-Cloud Analysis with Reflectional Invariance”. *Arxiv Preprint*, May 13, 2023. https://arxiv.org/pdf/2305.07814.pdf.\n\nDemirel, B., Baran, O. B., Cinbis, R. G., et al. “Meta-tuning loss functions and Data Augmentation for Few-shot Object Detection.” In *CVPR*, April 24, 2023, https://arxiv.org/pdf/2304.12161.pdf. \n\nEveringham, M., Van Gool, L., Williams, C. K. I., et al. “The pascal visual object classes (voc) challenge.” *International Journal of Computer Vision*, 88(2):303–338, 2010. https://link.springer.com/article/10.1007/s11263-009-0275-4. \n\nFang, C., Xu, Y., and Rockmore, D. N. “Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.” In *ICCV*, 2013. https://ieeexplore.ieee.org/document/6751316. \n\nGhunaim, Y., Bibi, A., Alhamoud, K., et al. “Real-Time Evaluation in Online Continual Learning: A New Hope.” In *IEEE CVF Computer Vision and Pattern Recognition (CVPR)*, February 27, 2023. https://arxiv.org/pdf/2302.01047.pdf. \n\nGong, S., Zhong, Y., Ma, W., et al. “3DSAM-adapter: Holistic Adaptation of SAM from 2D to 3D for Promptable Medical Image Segmentation.” *Arxiv Preprint*, June 23, 2023. https://arxiv.org/pdf/2306.13465.pdf. \n\nGong, X., Mohan, S., Dhingra, N., et al. “MMG-Ego4D: Multi-Model-Generalization in Egocentric Action Recognition.” *Arxiv Preprint*, May 12, 2023. https://arxiv.org/pdf/2305.07214.pdf.\n\nGu, J., Wang, K., Jiang, W., et al. “Summarizing Stream Data for Memory-Restricted Online Continual Learning.” *Arxiv Preprint*, May 26, 2023. https://arxiv.org/pdf/2305.16645.pdf.\n \nGudovsky, D., Okuno, T., and Nakata, Y. “Concurrent Miscalculation and Out-of-Distribution Detection for Semantic Segmentation via Energy-Based Normalizing Flow.” *Arxiv Preprint*, May 16, 2023. https://arxiv.org/pdf/2305.09610.pdf.\n\nGuirguis, K., Meier, J., Eskandar, G., et al. “NIFF: Alleviating Forgetting in Generalized Few-shot Object Detection via Neural Instance Feature Forging.”In *CVPR*, March 9, 2023. https://arxiv.org/pdf/2303.04958.pdf. \n\nGupta, A., Dollar, P., and Girshick, R. “Lvis: A dataset for large vocabulary instance segmentation.” In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 5356–5364, 2019. https://arxiv.org/pdf/1908.03195. \n\nGurbuz, M. B., Moorman, J. M., and Dovrolis, C. D. “SHARP: Sparsity and Hidden Activation RePlay for Neuro-Inspired Continual Learning.” *Arxiv Preprint*, May 29, 2023. https://arxiv.org/pdf/2305.18563.pdf. \n\nHabib, G., Saleem, T. J., and Lall, B. “Knowledge Distillation in Vision Transformers: A Critical Review.” *Arxiv Preprint*, February 4, 2023. https://arxiv.org/ftp/arxiv/papers/2302/2302.02108.pdf. \n\nHammoud, H. A. A. K., Prabhu, A., Lim, S-N., et al. “Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right?” *Arxiv Preprint*, May 16, 2023. https://arxiv.org/pdf/2305.09275.pdf. \n\nHan, D., Zhang, C., Qiao, Y., et al. “Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects Cannot be Easily Detected.” *Arxiv Preprint*, April 29, 2023. https://arxiv.org/pdf/2305.00278.pdf.\n\nHariharan, B., and Arbelaez, P., Girshick, R., et al. “Simultaneous detection and segmentation.” In *European Conference on Computer Vision*, pages 297–312. Springer, 2014. https://arxiv.org/pdf/1407.1808. \n\nHe, K., Zhang, X., Ren, S.,,et al. “Deep residual learning for image recognition.” In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016. https://arxiv.org/pdf/1512.03385.pdf. \n\nHe, Y., Yang, X., Chang, C., et al. “Efficient Human-in-the-loop system for Guiding DNNs attention.” *Internal Conferences of Intelligent User Interfaces*, February 6, 2023. https://arxiv.org/pdf/2206.05981.pdf. \n\nHong, Sus., Seo, J., Hong, Sun., et al. “”Large Language Models are Frame-level Directors for Zero-shot Text-to-Video Generation.” *Arxiv Preprint*, June 1, 2023. https://arxiv.org/pdf/2305.14330.pdf. \n\nHong, X., Lan, Y., Pang, L., et al. “Visual Reasoning: from State to Transformation.” *Arxiv Preprint*, May 2, 2023. https://arxiv.org/pdf/2305.01668.pdf.\n\nHoulsby, N., Giurgiu, A., Jastrzebski, S., et al. “Parameter-efficient transfer learning for nlp.” In *International Conference on Machine Learning*, pages 2790–2799. PMLR, 2019. https://arxiv.org/pdf/1902.00751. \n\nHu, X., Xu, X., and Shi, Y. “How to Efficiently Adapt Large Segmentation Model (SAM) to Medical Images.” *Arxiv Preprint*, June 23, 2023. https://arxiv.org/pdf/2306.13731.pdf.\n\nHuang, G. B., Mattar, M., Berg, T., et al. “Labeled faces in the wild: A database forstudying face recognition in unconstrained environments.” In *Workshop on faces in ‘Real-Life’ Images: detection, alignment, and recognition*, 2008.\n\nIsola, P., Lim, J. J., and Adelson, E. H. “Discovering states and transformations in image collections.” In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1383–1391, 2015. https://www.cv-foundation.org/openaccess/content_cvpr_2015/ext/1B_030_ext.pdf. \n\nJeeveswaran, K., Bhat, P., Zonooz, B., et al. “BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning.” In *The 40th International Conference on Machine Learning (ICML)*, May 8, 2023. https://arxiv.org/pdf/2305.04769.pdf.\n\nJha, D., Tomar, N. K., and Bagci, U. “TransRUPNet for Improved Out-of-Distribution Generalization in Polyp Segmentation.” *Arxiv Preprint*, June 3, 2023. https://arxiv.org/pdf/2306.02176.pdf.\n\nJha, D., Smedsrud, P. H., Riegler, M. A., et al. “Kvasir-SEG: a segmented polyp dataset.” In *Proceedings of the International Conference on Multimedia Modeling (MMM)*, 2020, pp. 451–462. https://arxiv.org/pdf/1911.07069.pdf. \n\nKarargyris, A., Kashyap, S., Lourentzou, I., et al. “Eye Gaze Data for Chest X-rays.” *PhysioNet*, September 12, 2020. https://doi.org/10.13026/qfdz-zr67.\n\nKirilov, A., Mintun, E., Ravi, N., et al. “Segment Anything.” *Arxiv Preprint*, April 5, 2023. https://arxiv.org/pdf/2304.02643.pdf.\n\nKnight, W. “OpenAI’s CEO Says the Age of Giant Models Is Already Over.” *Wired*, April 17, 2023. https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/. \n\nKoh, H., Kim, D., Ha, J., et al. “Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Interference.” In *ICLR*, January 28, 2022. https://arxiv.org/pdf/2110.10031.pdf. \n\nKomorowski, P., Baniecki, H., and Biecek, P. “Towards Evaluating Explanations of Vision Transformers for Medical Imaging.” In *CVPR*, April 12, 2023, https://arxiv.org/pdf/2304.06133.pdf. \n\nKonz, N., Dong, H., and Mazurowski, M. A. “Unsupervised anomaly localization in high-resolution breast scans using deep pluralistic image completion.” *Arxiv Preprint*, May 4, 2023. https://arxiv.org/pdf/2305.03098.pdf.\n\nKorkmaz, M., Buyukyazi, T., and Sezgin, M. T. “HAISTA-Net: Human Assisted Instance Segmentation Through Attention.” *Arxiv Preprint*, May 12, 2023. https://arxiv.org/pdf/2305.03105.pdf.\n\nLan, P. N., An, N. S., Hang, D. V., et al. “NeoUNet: towards accurate colon polyp segmentation and neoplasm detection.” *Arxiv Preprint*, July 11, 2021. https://arxiv.org/pdf/2107.05023.pdf. \n\nLarochelle, H., Erhan, D., and Bengio, Y. “Zero-data learning of new tasks.” In *AAAI*, 2008. https://cdn.aaai.org/AAAI/2008/AAAI08-103.pdf. \n\nLeng, J., Chen, T., Gao, X., et al. “A Comparative Review of Recent Few-Shot Object Detection Algorithms.” *Arxiv Preprint*, October 30, 2021. https://arxiv.org/pdf/2111.00201.pdf. \n\nLi, D., Yang, Y., Song, Y., et al. “Deeper, broader and artier domain generalization.” In *Proceedings of the IEEE international conference on computer vision*, pages 5542–5550, 2017. https://arxiv.org/pdf/1710.03077. \n\nLi, J., Dai, H., Han, H., et al. “MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving.” In *IEEE CVPR*, March 15, 2023. https://openaccess.thecvf.com/content/CVPR2023/papers/Li_MSeg3D_Multi-Modal_3D_Semantic_Segmentation_for_Autonomous_Driving_CVPR_2023_paper.pdf. \n\nLi, J., Li, D., Xiong, C., et al. “BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.” In *International Conference on Machine Learning*, pp. 12888-12900. PMLR, 2022. https://proceedings.mlr.press/v162/li22n/li22n.pdf. \n\nLi, Z., You, C., Bhojanapalli, S., et al. “The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers.” In *ICLR* (shortened version), October 2022. https://arxiv.org/pdf/2210.06313.pdf. \n\nLin, T., Maire, M., Belongie, S., et al. “Microsoft coco: Common objects in context.” In *Proc. European Conf. on Computer Vision*, pages 740–755. Springer, 2014. https://arxiv.org/pdf/1405.0312. \n\nLiu, G., Zhang, J., Chan, A. B., et al. “Human Attention-Guided Explainable Artificial Intelligence for Computer Vision Models.” *Arxiv Preprint*, May 5, 2023. https://arxiv.org/pdf/2305.03601.pdf.\n\nLiu, H., Lin, W., Chen, T., et al. “Few-Shot Action Recognition via Intra- and Inter- Video Information Maximization.” *Arxiv Preprint*, May 10, 2023. https://arxiv.org/pdf/2305.06114.pdf. \n\nLo, S., Oza, P., Chennupati, S., et al. “Spatio-Temporal Pixel-Level Contrastive Learning-based Source-free Domain Adaptation for Video Semantic Segmentation.” In *IEEE CVPR*, March 25, 2023. https://openaccess.thecvf.com/content/CVPR2023/papers/Lo_Spatio-Temporal_Pixel-Level_Contrastive_Learning-Based_Source-Free_Domain_Adaptation_for_Video_Semantic_CVPR_2023_paper.pdf. \n\nLong, Y., Wei, W., Huang, T., et al. “Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learning.” *Arxiv Preprint*, June 6, 2023. https://arxiv.org/pdf/2301.00452.pdf. \n\nMa, J., Niu, Y., Xu, J., et al. “DiGeo: Discriminative Geometry-Aware Learning for Generalized Few-Shot Object Detection.” In *CVPR*, March 16, 2023. https://arxiv.org/pdf/2303.09674.pdf. \n\nManzari, O. N., Ahmadabadi, H., Kasiani, H., et al. “MedViT: A Robust Vision Transformer for Generalized Medical Image Classification.” *Computers in Biology and Medicine*, 2023. https://arxiv.org/pdf/2302.09462.pdf. \n\nMashayekhi, M., Majd, S. A., Amiramjadi, A., et al. “Radious: Unveiling the Enigma of Dental Radiology with BEIT Adapter and Mask2Former in Semantic Segmentation.” *Arxiv Preprint*, May 10, 2023. https://arxiv.org/pdf/2305.06236.pdf.\n\nMarinov, Z., Reiss, S., Kersting, D., et al. “Mirror U-Net: Marrying Multimodal Fission with Multi-task Learning for Semantic Segmentation in Medical Imaging.” *Arxiv Preprint*, March 13, 2023. https://arxiv.org/pdf/2303.07126.pdf. \n\nMei, J., Zhou, J., and Hu, Y. “Few-shot 3D LiDAR Semantic Segmentation for Autonomous Driving.” *Arxiv Preprint*, March 3, 2023. https://arxiv.org/pdf/2302.08785.pdf. \n\nMendez-Ruiz, M., Gonzalez-Zapata, J., Reyes-Amezcua, I., et al. “SuSana Distancia is all you need: Enforcing class separability in metric learning via two novel distance-based loss functions for few-shot image classification.” *Arxiv Preprint*, May 18, 2023. https://arxiv.org/pdf/2305.09062.pdf.\n\nMenze, B. H., Jakab, A., Bauer S., et al. “The multimodal brain tumor image segmentation benchmark (BRATS).” *IEEE transactions on medical imaging*, 34(10):1993–2024, 2014.\n\nMirza, M. J., Karlinsky, L., Lin, W., et al. “LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections.” *Arxiv Preprint*, May 29, 2023. https://arxiv.org/pdf/2305.18287.pdf. \n\nMoschoglou, S., Papaioannou, A., Sagonas, C., et al. “Agedb: the first manually collected, in-the-wild age database.” In *Proceedings of the IEEE conference on computer vision and pattern recognition workshops*, pages 51–59, 2017.\n\nMosqueira-Rey, E., Hernandez-Pereira, E., Alonso-Rios, D., et al. “Human-in-the-loop machine learning: a state-of-the-art.” *Artificial Intelligence Review* 56, 3005-3054, 2023. https://doi.org/10.1007/s10462-022-10246-w. \n\nMunro, R. “Human-in-the-loop machine learning.” *Manning Publications*, Shelter Island, 2020.\n\nNaeem, M. F., Xian, Y., Tombari, F., and Akata, Z. “Learning graph embeddings for compositional zero-shot learning.” In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 953–962, 2021. https://arxiv.org/pdf/2102.01987. \n\nNguyen, K., and Todorovic, S. “Feature weighting and boosting for few-shot segmentation.” In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 622–631, 2019. https://arxiv.org/pdf/1909.13140. \n\nNguyen, T. H., Juyal, D., Li, J., et al. “ContriMix: Unsupervised Entanglement of content and attribute for domain generalization in microscopy image analysis.” *Arxiv Preprint*, June 7, 2023. https://arxiv.org/pdf/2306.04527.pdf. \n\nOstrowski, E., Prabakaran, B. S., Shafique, M. “Exploring Weakly Supervised Semantic Segmentation Ensembles for Medical Imaging Systems.” *Arxiv Preprint*, March 16, 2023. https://arxiv.org/pdf/2303.07896.pdf. \n\nPark, N., Kim, W., Heo, B., et al. “What Do Self-Supervised Vision Transformers Learn?” *Arxiv Preprint*, May 1, 2023. https://arxiv.org/pdf/2305.00729.pdf.\n\nPark, S., Kim, G., Oh, Y., et al. “AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation.” *Arxiv Preprint*, February 13, 2022. https://arxiv.org/pdf/2202.06431.pdf. \n\nPeng, K., Wen, D., Schneider, D., et al. “FeatFSDA: Towards Few-shot Domain Adaptation for Video-based Activity Recognition.” *Arxiv Preprint*, May 15, 2023. https://arxiv.org/pdf/2305.08420.pdf.\n\nPrabakaran, B. S., Ostrowski, E., and Shafique, M. “BoundaryCAM: A Boundary-based Refinement Framework for Weakly Supervised Semantic Segmentation of Medical Images.” *Arxiv Preprint*, March 14, 2023. https://arxiv.org/pdf/2303.07853.pdf. \n\nPrabhu, A., Cai, Z., Dokania, P., et al. “Online Continual Learning Without the Storage Constraint.” *Arxiv Preprint*, May 16, 2023. https://arxiv.org/pdf/2305.09253.pdf. \n\nQiao, L., Zhao, Y., Li, Z., et al. “Defrcn: Decoupled faster r-cnn for few-shot object detection.” In *Proc. IEEE Int. Conf. on Computer Vision*, pages 8681–8690, 2021. https://arxiv.org/pdf/2108.09017. \n\nQiao, Y., Zhang, C., Kang, T., et al. “Robustness of SAM: Segment Anything Under Corruptions and Beyond.” *Arxiv Preprint*, June 13, 2023. https://arxiv.org/pdf/2306.07713.pdf.\n\nQuellec, G., Lamard, M., Conze, P.-H., et al. “Automatic detection of rare pathologies in fundus photographs using few-shot learning.” *Medical image analysis* 61 (2020): 101660. https://www.sciencedirect.com/science/article/abs/pii/S1361841520300256. \n\nRadford, A., Kim, J. W., Hallacy, C., et al., “Learning Transferable Visual Models from Natural Language Supervision.” In *Proc. ICML*, 2021. https://arxiv.org/pdf/2103.00020. \n\nRahman, M. M., and Marculescu, R. “Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation.” In *MIDL*, March 29, 2023. https://arxiv.org/pdf/2303.16892.pdf. \n\nRibeiro, M. T., Singh, S., and Guestrin, C.. “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier.” In *KDD*, 2016. https://arxiv.org/pdf/1602.04938. \n\nRombach, R., Blattmann, A., Lorenz, D., et al. “High-Resolution Image Synthesis with Latent Diffusion Models.” *Arxiv Preprint*, April 13, 2022. https://arxiv.org/pdf/2112.10752.pdf. \n\nRong, Y., Xu, W., Akata, Z., et al. “Human Attention in Fine-Grained Classification.” In *British Machine Vision Conference (BMVC)*, November 2, 2021, https://arxiv.org/pdf/2111.01628.pdf. \n\nSelvaraju, R. R., Cogswell, M., Das, A., et al. “Grad-cam: Visual explanations from deep networks via gradient-based localization.” In *Proceedings of the IEEE international conference on computer vision*, pages 618–626, 2017. \n\nSemeraro, F., Quintart, A., Izquierdo, S. F., et al. “TomoSAM: a 3D Slicer Extension using SAM for tomography segmentation.” *Arxiv Preprint*, June 14, 2023. https://arxiv.org/pdf/2306.08609.pdf. \n\nSengupta, S., Chen, J., Castillo, C., et al. “Frontal to profile face verification in the wild.” In *2016 IEEE winter conference on applications of computer vision (WACV)*, pages 1–9. IEEE, 2016.\n\nShen, C., Li, W., Zhang, Y., et al. “Temporally-Extended Prompts Optimization for SAM in Interactive Medical Image Segmentation.” *Arxiv Preprint*, June 15, 2023. https://arxiv.org/pdf/2306.08958.pdf. \n\nTakemoto, K., Yamada, M., Sasaki, T., et al. “HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models.” *Arxiv Preprint*, June 7, 2023. https://arxiv.org/pdf/2305.09948.pdf.\n\nTan, M. and Le, Q. “Efficientnet: Rethinking model scaling for convolutional neural networks.” In *International Conference on Machine Learning*, pages 6105–6114. PMLR, 2019. https://arxiv.org/pdf/1905.11946.pdf. \n\nTaylor, J., Earnshaw, B., Mabey, B., et al. “Rxrx1: An image set for cellular morphological variation across many experimental batches.” In *International Conference on Learning Representations (ICLR)*, 2019.\n\nTruong, T., Nguyen, H., Raj, B., et al. “Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments.” *Arxiv Preprint*, May 25, 2023. https://arxiv.org/pdf/2305.15700.pdf. \n\nTu, Z., Talebi, H., Zhang, H., et al. “Maxvit: Multi-axis vision transformer.” In *ECCV*, 2022. https://arxiv.org/pdf/2204.01697. \n\nVasileiou, C., Smith, J. W., Thiagarjan, S., et al. “Efficient CNN-based Super Resolution Algorithms for mmWave Mobile Radar Imaging.” *Arxiv Preprint*, May 3, 2023. https://arxiv.org/pdf/2305.02092.pdf.\n\nVaswani, A., Shazeer, N., Parmar, N., et al. \"Attention is All you Need.\" *Arxiv Preprint*, December 6, 2017. https://arxiv.org/pdf/1706.03762.pdf. \n\nVenkateswara, H., Eusebio, J., Chakraborty, S., et al. “Deep hashing network for unsupervised domain adaptation.” In *CVPR*, 2017. https://arxiv.org/pdf/1706.07522.pdf. \n\nVilla, A., Alhamoud, K., Escorcia, V., et al. “vClimb: a Novel Video Class Incremental Learning Benchmark.” In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 19035-19044, 2022. https://arxiv.org/pdf/2201.09381.pdf. \n\nWang, L., Ye, X., Zhu, L., et al. “When SAM Meets Sonar Images.” *Arxiv Preprint*, June 25, 2023. https://arxiv.org/pdf/2306.14109.pdf. \n\nWei, M. “Few shot learning for small lesion detection.” *Meta*, vol. 85, no. 22, pp. 0–54, 2021.\n\nWu, J., Liu, S., Huang, D.,et al. “Multiscale positive sample refinement for few-shot object detection.” In *Proc. European Conf. on Computer Vision*, pages 456–472. Springer, 2020. https://arxiv.org/pdf/2007.09384.pdf. \n\nXiao, X., Zhou, D., Hu, J., et al. “Not All Pixels Are Equal: Learning Pixel Hardness for Semantic Segmentation.” *Arxiv Preprint*, May 15, 2023. https://arxiv.org/pdf/2305.08462.pdf.\n\nXu, Jia., Li, B., Lu, B., et al. “Surrol: An open-source reinforcement learning centered and DVRK compatible platform for surgical robot learning.” In *IROS*, IEEE, 2021. https://arxiv.org/pdf/2108.13035. \n\nXu, Jia., Wang, Y., and Chen, X. “Shape-Guided Gradient Voting for Domain Generalization.” *Arxiv Preprint*, June 19, 2023. https://arxiv.org/pdf/2306.10809.pdf. \n\nXu, Jin., Le, H., and Samaras, D. “Generating Features with Increased Crop-related Diversity for Few-shot Object Detection.” In *CVPR*, April 11, 2023. https://arxiv.org/pdf/2304.05096.pdf. \n\nXue, H., Jin, M., Zhang, Cho., et al. “Automatic Image Blending Algorithm Based on SAM and Dino.” *Arxiv Preprint*, June 11, 2023. https://arxiv.org/pdf/2306.05382.pdf. \n\nYang, Y., Chen, Q., Feng, Y., et al. “MIANet: Aggregating Unbiased Instance and General Information for Few-Shot Semantic Segmentation.” In *CVPR*, May 23, 2023. https://arxiv.org/pdf/2305.13864.pdf. \n\nYu, B. X. B., Chang, J., Wang, H., et al. “Visual Tuning.” *Arxiv Preprint*, May 10, 2023. https://arxiv.org/pdf/2305.06061.pdf.\n\nYu, J., Wang, Z., Vasudevan, V., et al. “CoCa: Contrastive Captioners are Image-Text Foundation Models.” In *Transactions on Machine Learning Research*, May 4, 2022. https://arxiv.org/pdf/2205.01917.pdf. \n\nZahn, M., and Perrin, D. P. “Efficient Human-in-loop deep learning model training with iterative refinement and statistical result validation.” *Arxiv Preprint*, April 3, 2023, https://arxiv.org/pdf/2304.00990.pdf. \n\nZhang, Cha., Han, D., Qiao, Y., et al. “Faster Segment Anything: Towards Lightweight SAM for Mobile Applications.” *Arxiv Preprint*, June 5, 2023. https://arxiv.org/pdf/2306.14289.pdf.\n \nZhang, Che., Zhang, Cha., Kang, T., et al. “Attack-SAM: Towards Attacking Segment Anything Model With Adversarial Examples.” *Arxiv Preprint*, May 8, 2023. https://arxiv.org/pdf/2305.00866.pdf.\n\nZhang, H., Cisse, M., Dauphin, Y. N., et al. “mixup: Beyond empirical risk minimization.” In *ICLR*, April 27, 2018. https://arxiv.org/pdf/1710.09412.pdf. \n\nZhang, J., Hermann, C., Hur, J., et al. “A Tale of Two Features: Stable Diffusion Complements DINO for Zero-shot Semantic Correspondence.” *Arxiv Preprint*, May 24, 2023. https://arxiv.org/pdf/2305.15347.pdf. \n\nZhang, L., Liu, Z., Zhang, L,, et al. “Segment Anything Model (SAM) for Radiation Oncology.” *Arxiv Preprint*, June 20, 2023. https://arxiv.org/pdf/2306.11730.pdf.\n\nZhang, X., Gu, C., and Zhu, S. “SAM-helps-Shadow: When Segment Anything Model meet shadow removal.” *Arxiv Preprint*, June 1, 2023. https://arxiv.org/pdf/2306.06113.pdf. \n\nZhang, Ya., Pfahringer, B., Frank, E., et al. “Repeated Augmented Rehearsal; A Simple but Strong Baseline for Online Continual Learning.” *Arxiv Preprint*, September 28, 2022. https://arxiv.org/pdf/2209.13917.pdf. \n\nZhang, Yu., Huang, S., Zhou, Z., et al. “Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation.” In *ML4H*, 2022. https://proceedings.mlr.press/v193/zhang22a/zhang22a.pdf. \n\nZhao, Z., Zhou, F., Zeng, Z., et al. “Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation.” *MICCAI*, September 16, 2022. https://arxiv.org/pdf/2305.06978.pdf. \n\nZheng, T. and Deng, W. “Cross-pose LFW: A database for studying cross-pose face recognition in unconstrained environments.” *Beijing University of Posts and Telecommunications*, Tech. Rep, 5(7), 2018.\n\nZheng, T., Deng, W., and Hu, J. “Cross-age LFW: A database for studying cross-age face recognition in unconstrained environments.” *Arxiv Preprint*, August 28, 2017. https://arxiv.org/pdf/1708.08197.pdf. \n\nZheng, Z., Zhu, H., and Nevatia, R. “CAILA: Concept-Aware Intra-Layer Adapters for Compositional Zero-shot Learning.” *Arxiv Preprint*, May 26, 2023. https://arxiv.org/pdf/2305.16681.pdf. \n\nZhou, Z., Luo, J., Arefan, D., et al. “Human Not in the Loop: Objective Sample Difficulty Measures for Curriculum Learning.” *Arxiv Preprint*, February 25, 2023. https://arxiv.org/pdf/2302.01243.pdf. \n\nZhu, H., Salcudean, S., and Rohling, R. “Gaze-Guided Class Activation Mapping: Leveraging Human Attention in Chest X-rays Classification.” In *Proceedings of the 15th International Symposium on Visual Information Communication and Interaction*, August 2022. https://arxiv.org/pdf/2202.07107.pdf. \n\nZhuang, X., Shen, J. “Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI.” *Medical Image Analysis*, 2016. https://www.sciencedirect.com/science/article/abs/pii/S1361841516000219. \n\nZhuo, T., Cheng, Z., Gao, Z., et al. “Continual Learning with Strong Experience Replay.” Arxiv Preprint, May 23, 2023. https://arxiv.org/pdf/2305.13622.pdf. \nAl-Dhabyani, W., Gomaa, M., Khaled, H., et al. “Dataset of breast ultrasound images.” *Data in brief*, 28:104863, 2020. ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom pathlib import Path\nsubmission_df = pd.DataFrame({\"type\": [\"essay_category\", \"essay_url\", \"feedback1_url\", \"feedback2_url\", \"feedback3_url\"],\n \"value\": [\"Image and/or video data\", \"https://www.kaggle.com/radbear/ai-report-state-of-computer-vision/notebook\", \n \"https://www.kaggle.com/code/rhysie/learnings-from-the-typical-tabular-pipeline/comments#2346078\", \n \"https://www.kaggle.com/code/kkiryll/generative-ai-rise-of-the-transformers/comments#2346107\", \n \"https://www.kaggle.com/code/dgoldberg98/team-rangers/comments#2346135\"]})\nsubmission_df.to_csv('/kaggle/working/submission.csv', index=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T04:09:44.676939Z","iopub.execute_input":"2023-07-16T04:09:44.677593Z","iopub.status.idle":"2023-07-16T04:09:44.722235Z","shell.execute_reply.started":"2023-07-16T04:09:44.677557Z","shell.execute_reply":"2023-07-16T04:09:44.721203Z"},"trusted":true},"execution_count":null,"outputs":[]}]}