{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## **Innovations and Advancements in Computer Vision (2021-2023)**\n**By: Pretham Nandakumar**\n\nCurrently, around five billion images are uploaded every day [[1]](https://photutorial.com/photos-statistics/)! Because of this large amount of data, Computer Vision has now turned into one of the most important fields in AI. Computer Vision is now not just about machines recognizing objects in pictures anymore, now it's about AI truly \"seeing\" and understanding images. With the amount of innovations being made, exciting new use cases have emerged, ranging from healthcare and autonomous vehicles to diagnosing diseases.\n\nBut, at the heart of all these innovations are three groundbreaking advancements: the Vision Transformer [[2]](https://arxiv.org/pdf/2010.11929.pdf), OpenAI's CLIP [[3]](https://arxiv.org/abs/2103.00020), and Diffusion models [[4]](https://arxiv.org/pdf/2006.11239.pdf)[[5]](https://arxiv.org/pdf/2112.10752.pdf). The Vision Transformer has rapidly become the go to architecture for many tasks, OpenAI's CLIP has bridged the gap between vision and language, and Diffusion models have introduced a novel approach to generating high-quality images from text descriptions. These three advancements power many of the most popular models of today.","metadata":{}},{"cell_type":"markdown","source":"## **Table Of Contents**\n\n#### 1. New Innovations And Models\n\n#### 2. Vision Transformer\n\n#### 3. CLIP\n\n#### 4. Diffusion Models\n\n#### 5. Conclsuion","metadata":{}},{"cell_type":"markdown","source":"## **Vocabulary**\n\n**Large Multimodal Model**: A model that can understand multiple modalities like text, images, video, audio, etc.\n\n**Image Encoder**: An image encoder is a model that is designed to convert an image into information that can be interpreted. This is useful for multimodal models because the information outputted by the image encoder is used by different components.\n\n**Text Encoder**: A text encoder is just like an image encoder but is instead used to understand text.\n\n**Large Language Model**: A large language model (LLM) is a model that is trained on text data and is used for text tasks. An example of a LLM is ChatGPT.\n\n**Zero-Shot**: Zero-Shot is a way of testing models on data/tasks that they have never seen before. This is useful to see if a model can generalize to new data.\n\n**Optimizer**: An optimizer is a component in Deep Learning models that help them improve.\n\n**ImageNet**: ImageNet is a dataset and benchmark used to test how good Image Classification models perform.\n\n**Self-Attention**: Before understanding how models like the Vision Transformer work, it is crucial to understand how self-attention works. In NLP, self-attention is used to weigh the importance of different words in a sentence. For example take the sentence: \"I like eating cheese pizza.\" In this sentence, self-attention would understand that the word 'eating' is directly related to 'pizza' and less related to the word 'cheese'. This is important because attention can be used to find the important parts of the data. Self-Attention is the main building block of the Transformer and is what makes the architecture so good.\n\n**Fine-tuning**: Fine-tuning is the process of taking an already trained model and training it further. Fine-tuning is usually used to train further on a specific task.","metadata":{}},{"cell_type":"markdown","source":"## **New Innovations and Models** \n\nThe following section will contain a summary of many of the best innovations that occured over the past two years. The innovations will be sorted by the tasks they do: \n- Large Multimodal Models\n- Image-to-Image\n- Image-to-Text\n- Text-to-Image\n- Video Generation\n- Deepfake Detection\n- Segmentation\n- Image Classification\n- Object Detection.\n\n### **Large Multimodal Models (LMM)**\n\n- **[EMU](https://github.com/baaivision/emu)** [[6]](https://github.com/baaivision/emu): EMU is a new open source Large Multimodal Model (LMM not LLM) that is similar to GPT-4 in the sense that it is capable of performing many multimodal tasks.  EMU combines EVA-CLIP, LLAMA, and Stable Diffusion to carry out tasks like \"image captioning, image/video question answering, and text-to-image generation\" (Sun et al.). One new ability this model unlocks is \" in-context text and image generation, and image blending\" [(Sun et al.)](https://github.com/baaivision/emu). This means that the model can remember from previous conversations and generate what was requested accordingly. \n\n![](https://github.com/baaivision/Emu/raw/main/assets/generalist.png)\n\n[Figure 1: EMU carrying out many multimodal tasks [7]](https://github.com/baaivision/emu)\n\n- **[CM3leon](https://ai.meta.com/blog/generative-ai-text-images-cm3leon/)** [[8]](https://ai.meta.com/blog/generative-ai-text-images-cm3leon/): CM3leon is a new text-to-image and image-to-text generator created by Meta. One huge development by this model is its ability to generate hands! Finally, a model that actually understands hands. The authors report that the model \"achieves state-of-the-art performance for text-to-image generation, despite being trained with five times less compute than previous transformer-based methods\" [(Yu et al.)](https://ai.meta.com/blog/generative-ai-text-images-cm3leon/). CM3leon can perform many tasks like text-to-image, image editing, image captioning, visual question answering, object-to-image, and segmentation-to-image.\n\n**Text-to-image**\n![](https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/358675736_262727669790925_3190473684047147716_n.png?_nc_cat=111&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=k-a4o-mh01MAX8WBmob&_nc_ht=scontent-sea1-1.xx&oh=00_AfCxhA7DSHr4yzEFA1wKMAhhxTf4cWpwxGePyk0slmpGwQ&oe=64B6CEB6)\n\n**Image Editing**\n![](https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/358713848_660084255621228_746245581913776369_n.png?_nc_cat=111&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=iJ8EajBzr00AX8kYPFr&_nc_ht=scontent-sea1-1.xx&oh=00_AfBQoc187D7zDA324BYVRx6oekfinyjzzPk4SVKgxEPcQw&oe=64B7A72E)\n\n**Object-to-Image**\n![](https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/359198671_298094419267090_6070744248933373628_n.png?_nc_cat=105&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=WNYdcDGIdcwAX-npIO2&_nc_ht=scontent-sea1-1.xx&oh=00_AfDujA1ENwLrIj-ZjDISCXI-giUWmlASEF1cKrutCHlCKA&oe=64B80E51)\n\n**Segmentation-to-Image**\n![](https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/358696519_301734465641511_2309463697092042055_n.png?_nc_cat=104&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=J-hcJaD5gv4AX_LobeC&_nc_ht=scontent-sea1-1.xx&oh=00_AfAKFV7cOg9WXQfpDqGYZ-1uRy8CPJZ16cpudJPTFdudDg&oe=64B79F1C)\n[Figure 2: Examples of CM3leon [9]](https://ai.meta.com/blog/generative-ai-text-images-cm3leon/)\n\n- **[Multi-Modal GPT-4](https://openai.com/research/gpt-4)**: Multi-Modal GPT-4 is the best Large Multimodal Model in the world. The model was trained by OpenAI and has quickly proved that it is the most useful LMM. Use for this model has been released to the public via Bing. One stunning example of its capabilities is below.\n\n![](https://pbs.twimg.com/media/F1D9vQWXwAA6sh0?format=jpg&name=large)\n\n[Figure 3: Multi-Modal GPT-4 on Bing Chat [10]](https://www.reddit.com/r/singularity/comments/14zy2kp/mind_blown/?utm_source=share&utm_medium=web2x&context=3)\n\n### **Image-to-Image**\n\n- **[ControlNet](https://github.com/lllyasviel/ControlNet)** [[11]](https://github.com/lllyasviel/ControlNet): ControlNet is a way of controlling Diffusion models. ControlNet can be used in practically anyway. Some examples of tasks ControlNet can do are scribbles to image, canny edge to image, and pose to image. One extremely interesting way ControlNet was used was by making QR code art (Figure 3). This has quickly became one of the most popular image-to-image techniques. The way this model works is by creating a copy of the model, the copied model is then trained to perform the task that was asked for.\n\nExample of scibble to image:\n\n![](https://github.com/lllyasviel/ControlNet/raw/main/github_page/p7.png)\n\n[Figure 4: Scribble to image of a turtle [12]](https://github.com/lllyasviel/ControlNet)\n\n![](https://huggingface.co/monster-labs/control_v1p_sd15_qrcode_monster/resolve/main/images/tree.png)\n\n[Figure 5: QR Code Art by a ControlNet [13]](https://huggingface.co/monster-labs/control_v1p_sd15_qrcode_monster)\n\n- **[Instruct Pix2Pix](https://github.com/timothybrooks/instruct-pix2pix)**: Instruct Pix2Pix is an image editing model that was trained on data generated by GPT-3 and Stable Diffusion. The model can perform impressive edits as seen below.\n\n![](https://i.redd.it/413x5q54jwea1.jpg)\n[Figure 6: Instruct Pix2Pix image editing model [14]](https://github.com/timothybrooks/instruct-pix2pix)\n\n### **Image-to-Text**\n\n- **[BLIP-2](https://huggingface.co/docs/transformers/main/model_doc/blip-2)** [[15]](https://arxiv.org/pdf/2301.12597.pdf): BLIP-2 is a model that combines \"frozen pre-trained image encoders and large language models (LLM)\" [(Li et al.)](https://arxiv.org/pdf/2301.12597.pdf). The model does this by adding a Querying Transformer (Q-Former) which learns to find the most important information outputted by the image encoder and sends the findings to the LLM. The LLM will then output the requested text. BLIP-2 can be used for visual question answering and image captioning.\n\nArchitecture and example image and question:\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg)\n\n[Figure 7: BLIP-2 Architecture and example [16]](https://arxiv.org/pdf/2301.12597.pdf)\n\n### **Text-to-Image**\n\n- **[Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9)** [[17]](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9): Recently, Stability AI released their newest text-to-image generator. This model is called Stable Diffusion Extra Large 0.9. This model is by far the best open source model available. \n\n- **[Dall-E 2](https://openai.com/dall-e-2)** [[18]](https://openai.com/dall-e-2): Dall-E 2 is a OpenAI text-to-image diffusion model. Dall-E 2 is closed source but can be easily accessed with OpenAI's api. Another way of using Dall-E 2 is with Bing. Dall-E 2 was one of the first text-to-image generators that showed the potential of this field and the quality of the images are extremely good but most of the time fail in comparison to Midjourney.\n\n- **[MidJourney](https://www.midjourney.com)** [[19]](https://www.midjourney.com): Midjourney is probably the best text-to-image generator on the market. The images generated are incredibly stunning and feels much more ahead when compared to the rest of these image generators. One drawback is that this model is incredibly slow at generating images. Midjourney also is closed source and can only be accessed through Discord.\n\n### **Video Generation**\n\n- **[Zeroscope](https://huggingface.co/cerspense/zeroscope_v2_576w)** [[20]](https://huggingface.co/cerspense/zeroscope_v2_576w): Zeroscope is a model that can generate videos from text prompts. The resolution of the videos are 1024x576. An example of the model can be found on [YouTube](https://www.youtube.com/watch?v=HO3APT_0UA4).\n\n- **[ControlVideo](https://github.com/YBYBZhang/ControlVideo)** [[21]](https://github.com/YBYBZhang/ControlVideo): ControlVideo is just like ControlNet but for videos. The model extends the output of ControlNet to make it a video.\n\n### **Deepfake Detection**\n\n- **[Intel FakeCatcher](https://www.intel.com/content/www/us/en/newsroom/news/intel-introduces-real-time-deepfake-detector.html#gs.330w9t)** [[22]](https://www.intel.com/content/www/us/en/newsroom/news/intel-introduces-real-time-deepfake-detector.html#gs.330w9t): With the rise of generative ai, detecting the real from the fake has become more important than ever. Intel has developed a deepfake detector that achieves 96% accuracy on videos! The way the model does this is by analyzing \"what makes us human - subtle “blood flow” in the pixels of a video\" [(\"Intel Introduces Real-Time Deepfake Detector\n\")](https://www.intel.com/content/www/us/en/newsroom/news/intel-introduces-real-time-deepfake-detector.html#gs.330w9t). \n\n### **Segmentation**\n\n- **[Segment Anything](https://github.com/facebookresearch/segment-anything)** [[23]](https://github.com/facebookresearch/segment-anything): Segment Anything is a segmentation model developed by Meta. This model has quickly became the most popular segmentation model used. The Github has over 35k stars! \"[Segment Anything] has been trained on a dataset of 11 million images and 1.1 billion masks, and has strong zero-shot performance on a variety of segmentation tasks\" [(Kirillov et al.)](https://github.com/facebookresearch/segment-anything).\n\nExample Image:\n![](https://github.com/facebookresearch/segment-anything/raw/main/assets/masks2.jpg?raw=true)\n\n[Figure 8: Segment Anything example [24]](https://github.com/facebookresearch/segment-anything)\n\n- **[Segment Everything Everywhere All At Once](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once)** [[25]](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once): Segment Everything Everywhere All at Once (SEEM) is a segmentation model that \"allows users to easily segment an image using prompts of different types including visual prompts (points, marks, boxes, scribbles and image segments) and language prompts (text and audio), etc\" [(Zou et al.)](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once).\n\n![](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once/raw/main/assets/teaser_new.png?raw=true)\n\n[Figure 9: Tasks that SEEM can perform [26]](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once)\n\n- **[Segment Anything in High Quality](https://github.com/SysCV/sam-hq)** [[27]](https://github.com/SysCV/sam-hq): Segment Anything in Hiqh Quality (HQ-SAM) is a higher quality version of the normal Segment Anything model. The model achieves state-of-the-art results on Zero Shot Segmentation. The difference of this model when compared to Segment Anything can be seen below.\n\n![](https://github.com/SysCV/sam-hq/raw/main/figs/coco_vis_comp.png)\n\n[Figure 10: HQ-SAM vs SAM [28]](https://github.com/SysCV/sam-hq)\n\n### **Image Classification**\n\n- **[LION Optimizer](https://github.com/lucidrains/lion-pytorch)** [[29]](https://arxiv.org/pdf/2302.06675.pdf): LION is a new optimizer that achieved SOTA results on ImageNet. The new optimizer \"boosts the accuracy of ViT by up to 2% on ImageNet\" [(Chen et al.)](https://arxiv.org/pdf/2302.06675.pdf). It was also reported that \"on diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x\" [(Chen et al.)](https://arxiv.org/pdf/2302.06675.pdf).\n\n- **[EVA 2](https://github.com/baaivision/EVA/tree/master/EVA-02)** [[30]](https://arxiv.org/pdf/2303.11331.pdf): EVA 2 is an open-source model that achieves 90% on ImageNet. The model is only 305 million parameters yet achieves the same performance of models that are above a billion parameters. The creators also report that \"EVA-02-CLIP can reach up to 80.4 zero-shot top-1 on ImageNet-1k\" [(Fang et al.)](https://arxiv.org/pdf/2303.11331.pdf). The model is on [HuggingFace](https://huggingface.co/timm/eva02_large_patch14_448.mim_m38m_ft_in1k) and can be used with the timm package.\n\n### **Object Detection**\n\n- **[YOLOv8](https://github.com/ultralytics/ultralytics)** [[31]](https://github.com/ultralytics/ultralytics): YOLOv8 is an open-source state-of-the-art model used for \"detction, segmentation, pose estimation, tracking, and classification\" [(Jocher et al.)](https://github.com/ultralytics/ultralytics). The model can be easily used with the ultralytics package. Everything you need to know to use this model is on the [Github](https://github.com/ultralytics/ultralytics).\n\n### **Computer Vision Uses**\n\n- **[Be My Eyes](https://www.bemyeyes.com/)**: BeMyEyes is a app that has recently allowed people with visual impairments to recieve assistance from Multi-Modal GPT-4. The app originally connected people to other people but has now started trying out AI assistance.\n\n- **Medical Field**: Computer Vision in the medical field has helped automate many tasks like XRay Analysis, Some Computer Vision can see some patterns that we humans do not recognize, making these AI tools incredible helpful. Computer Vision is also used to conduct CT and MRI analysis. Computer Vision models in CT and MRI scans are used to detect cancer, tumors, internal bleeding, and more. Over the past two years, these techonlogies have improved because of new advancements like the Vision Transformer.\n\n- **Autonomous Vehicles**: Computer Vision is what makes autonomous vehicles possible. Computer Vision is used to detect objects, classify objects, and much more. The leading companies in this field are Tesla, Waymo, and Cruise. The past two years has seen some of the biggest improvements to these types of cars. Tesla has even started a beta that the public can access. Elon Musk has even said that the next version (FSD 12) will not be a beta version. ","metadata":{}},{"cell_type":"markdown","source":"One key similarity between many of the models above is the architecture behind them. Many state-of-the-art models use one of the following building blocks to power their models: The Vision Transformer, CLIP, or the Diffusion Architecture. The following sections will be about how these architectures work. The first paragraph of each topic will be an overview that is in Layman Terms, what follows will be a deeper dive into how each mechanism works. The deep dive will still be explained as simply as possible.","metadata":{}},{"cell_type":"markdown","source":"## **Vision Transformer**\n\n![](https://production-media.paperswithcode.com/methods/Screen_Shot_2021-01-26_at_9.43.31_PM_uI4jjMq.png)\n\n[Figure 11: Vision Transformer Architecture [32]](https://arxiv.org/pdf/2010.11929.pdf)","metadata":{}},{"cell_type":"markdown","source":"The Vision Transformer is an architecture proposed in Google's paper called An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Currently, the Vision Transformer is one of the best performing architectures and has been adopted widely. Many state-of-the-art models in Computer Vision use this architecture. In fact, after scaling Vision Transformers, researchers at Google achieved 90.45% accuracy on ImageNet [[33]](https://arxiv.org/pdf/2106.04560.pdf). Vision Transformers are also used in many multimodal tasks like image generation and visual-question answering. The main trick that the Vision Transformer introduces is patches. Instead of treating each image like normal, the image is split into patches. The reason for this is to reduce the computation necessary to run the model.\n\n### How Vision Transformers Work\nIn the field of Natural Language Processing (NLP), the Transformer architecture has become the standard. However, its use in Computer Vision was severely limited because the Transformer architecture is extremely computationally expensive due to the fact that it would treat each pixel as a token (think of a token as a word). However, Google's paper which introduced the Vision Transformer, treats each image as a sequence of patches as seen above. The difference between using attention across patches versus the full image is that instead of calculating attention across each pixel, you would only calculate attention across each patch and then treat that patch as a token. For example, in a standard 256x256 image, if attention was to be calculated across each pixel, the cost would be 256x256 squared. But when the image is treated as patches, you would divide a 256x256 image into 16 patches. This reduces the amount of computation to a scale achievable by current computers.","metadata":{}},{"cell_type":"markdown","source":"![](https://theaisummer.com/static/aa65d942973255da238052d8cdfa4fcd/7d4ec/the-transformer-block-vit.png)\n\n[Figure 12: Vision Transformer Architetchure [34]](https://arxiv.org/pdf/2010.11929.pdf)\n\nThe architecture of the Vision Transformer is still extremely similar to a standard Transformer Encoder (Figure 12). After the patches are processed through the Transformer Encoder, they are sent to a classifier so that the images can be classified.\n\n### Vision Transformer vs Convolutional Neural Network (CNN)\n![](https://anhreynolds.com/img/cnn.png)\n[Figure 13: Convolutional Layer [35]](https://anhreynolds.com/blogs/cnn.html)\n\nThe main improvement that Vision Transformers make when compared to CNNs (most popular architecture in Computer Vision) is its ability to understand long range relationships. For example in an image of a park, there could be a person looking at a frisbee and a dog running towards a frisbee. If the dog, frisbee, and person are far away from each other, a CNN might not be able to understand this image because of the core design. The basic building block of a CNN is the convolutional layer, which uses filters that slide over the image and compute the dot product between the weights of the filter and the pixels. The end product is a feature map that contains local features because the filter that slides across the image is usually only around 3x3 or 5x5 pixels (Figure 13). On the other hand, Vision Transformers will be able to understand this example of the park because it can find relationships between the patches no matter the distance from each other because of the self-attention mechanism used across all patches.\n\nAnother reason why the Vision Transformers is one of the greatest advancements over the past two years in Computer Vision is because of its interesting use cases. One of the main uses is its involvement in OpenAI's CLIP.\n\n","metadata":{}},{"cell_type":"markdown","source":"## **CLIP**\n\nOpenAI's CLIP (Contrastive Language-Image Pre-training) is a groundbreaking innovation in the field of computer vision. CLIP bridged the gap between vision and language making it able to understand visual concepts and put them into words. Unlike traditional computer vision models that are trained for specific tasks by using large labeled datasets, CLIP is trained on crazy amounts of image-text pairs found from the internet. In total, CLIP was trained on 400 million of these pairs! All this training allowed CLIP to have amazing zero-shot capabilities. Zero-shot learning is where a model is able to perform tasks that it was not trained on. CLIP is able to understand images to a new degree, allowing it to generalize its knowledge to new tasks. This innovative approach to Computer Vision reduces the need for dataset creation which can be expensive and time-consuming.\n\n### Performance\n\nThe creators of CLIP report that it \"matches the performance of the original ResNet-50 on ImageNet zero-shot\" (Radford et al.). CLIP achieves 76.2% on ImageNet zero-shot and recent variants of CLIP have achieved above 81% on ImageNet zero-shot [[36]](https://huggingface.co/papers/2306.15658)! But, unlike other models trained for ImageNet, CLIP performs well on everything and not just one dataset. \n\n### Training CLIP\n![](https://images.openai.com/blob/fbc4f633-9ad4-4dc2-bd94-0b6f1feee22f/overview-a.svg?width=10&height=10&quality=50)\n[Figure 14: How CLIP is trained [37]](https://openai.com/research/clip)\n\nTraining CLIP involves the use of an image encoder and text encoder. There are multiple CLIP variants, but the most popular CLIP variant uses a Vision Transformer as the image encoder and a Transformer model as the text encoder. To train the model, a batch of 32,768 images and their text descriptions are sent to their respective encoder. The encoders then create image embeddings and text embeddings. Embeddings are used to represent the data in a way that the model can understand it. By creating two embeddings, the model is trained to match the corresponding image embedding and text pairs while attempting to stay away from matching the wrong embedding pairs. Or, in other words the model is trained contrastively to match the correct image embedding to the correct text embedding. Figure 14 shows this entire process, I1 and T1 are the image and text embeddings while I1\\*T1 and all those other pairs in the grid are the embeddings being matched together. CLIP is then trained to correctly identify which image embedding matches the text embedding. This process of training both encoders at the same time is called Contrastive pre-training. Now that CLIP has been pre-trained on all these pairs, the model can now perform zero-shot classification. \n\n### Zero-Shot Classification\n![](https://images.openai.com/blob/d9d46e4b-6d6a-4f9e-9345-5c6538b1b8c3/overview-b.svg?width=10&height=10&quality=50)\n[Figure 15: Zero-Shot Classification [38]](https://openai.com/research/clip)\n\nTo conduct zero-shot classification, the model is sent a bunch of text descriptions. For example if CLIP was going to be used on ImageNet, the descriptions would be something along the lines of \"a photo of a [object]\". The ImageNet classes would be put into the object placeholder and all this text would be encoded by the Text Encoder. Then, the image that needs to be classified will be encoded by the image encoder. CLIP then attempts to match the corresponding image embedding to the text embedding. The resulting pair will be the output of the model. This can all be seen in Figure 15.\n\n### Takeaways\n\nCLIP is a powerful multimodal computer vision model that combines a Vision Transformer with an LLM. CLIP's ability to perform zero-shot learning at such a high level is a promising development. CLIP can be used in many tasks, but one of the best uses of CLIP came from DALL-E, Stable Diffusion, and other image generators.","metadata":{}},{"cell_type":"markdown","source":"## **Diffusion Models**\n\nIn the past two years, text-to-image generation has exploded in popularity. By simply providing a text description, these generators can produce breathtaking images making these models incredibly useful. The main architecture text-to-image generators use is Diffusion. The main idea behind these models is as follows:\n\n### Forward Diffusion Process:\n1. Imagine an image \n2. Imagine specks of noise (random pixels) are added throughout the image. This makes the image basically all random pixels and unrecognizable.\n\n![](https://i0.wp.com/stable-diffusion-art.com/wp-content/uploads/2022/12/image-81.png?resize=768%2C336&ssl=1)\n\n[Figure 16: Forward Diffusion Process [39]](https://stable-diffusion-art.com/how-stable-diffusion-work/)\n\n### Reverse Diffusion Process:\n\n3. Now that the image is unrecognizable, a model is used to reverse this process of adding noise to make the image look like the original image. The model reverses noise in a step-by-step manner as seen in Figure 17 where it removes noise little by little.\n\n4. The model that does this is trained on many many images that have noise added to it. The model's job is to reverse this process.\n\n![](https://i0.wp.com/stable-diffusion-art.com/wp-content/uploads/2022/12/image-82.png?resize=768%2C440&ssl=1)\n\n[Figure 17: Reverse Diffusion Process [40]](https://stable-diffusion-art.com/how-stable-diffusion-work/)\n\n### Text-To-Image:\n\n5. Now that a model can perform the reverse diffusion process, we send the model random noise and some text. The model has now been fine-tuned to also take in text. So, with this random noise, it gradually turns the pixels into an actual image that is similar to the text prompt. \n\n### Architecture\n\nThere are two main parts of the Diffusion model, the noise remover and the text understander. In models like Stable Diffusion, the model that is used to remove noise is a U-Net type model. The U-Net has an architecture like a U. The model is trained to reconstruct its input making it great for the reverse diffusion process. \n\n![](https://miro.medium.com/v2/resize:fit:1400/1*VUS2cCaPB45wcHHFp_fQZQ.png)\n\n[Figure 18: U-Net Architecture [41]](https://towardsdatascience.com/u-net-explained-understanding-its-image-segmentation-architecture-56e4842e313a)\n\nThe second model used in Diffusion models is a text encoder. Models like Stable Diffusion and Dall-E 2 use CLIP's text model to create text embeddings. These text embeddings help the U-Net understand what to create. Because the U-Net is only provided random noise, it needs to know what to turn this noise into. The text embedding created by CLIP is provided to the U-Net so that it can turn the noise into the requested image. The entire process can be seen in Figure 18.\n\n![](https://jalammar.github.io/images/stable-diffusion/stable-diffusion-diffusion-process.png)\n\n[Figure 19: Stable Diffusion Architecture [42]](https://jalammar.github.io/illustrated-stable-diffusion/)\n\nThe figure above shows how Stable Diffusion works, one difference in this figure is that the random noise part is labeled as \"Random image information tensor\". Another difference between this figure (Stable Diffusion Architecture) and other Diffusion models is the last part which is labeled as Image Decoder. Other Diffusion models do not have this part. The reason for it is because Stable Diffusion does the reverse diffusion process in the latent space to speed up the process. This means that the inputs to the U-Net are compressed and happen entirely in the latent space. The latent space can be thought of as a low-dimensional representation of the actual data. Because everything happens in the latent space, Stable Diffusion is much faster when compared to standard Diffusion models.\n\nIn summary, random noise and a text prompt is sent to the Diffusion model. The Diffusion model then uses CLIP's text encoder to create text embeddings that are then sent along the random noise. The U-Net takes in the text embeddings and random noise and gradually turns this random noise into the requested image.","metadata":{}},{"cell_type":"markdown","source":"## **Conclusion**\n\nThe past two years in Computer Vision have been extremely eventful. While many advancements were made, three of the main innovations were Vision Transformers, CLIP, and Diffusion Models. These three innovations are not just minor improvements, instead they are reinventing the field of AI. These three advancements have revolutionized what models can 'see' and understand. Because of the way these models work, AI can now truly put images into words! It's safe to say that Computer Vision has seen an unprecedented boost in popularity and with this, more and more game-breaking innovations will continue to be made.","metadata":{}},{"cell_type":"markdown","source":"## **References**","metadata":{}},{"cell_type":"markdown","source":"[1] Matic Broz, 2023. “How Many Photos Are There? (Statistics & Trends in 2023).” Photutorial, 27 June 2023, photutorial.com/photos-statistics/. \n\n[2] Dosovitskiy, Alexey, et al. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” arXiv.Org, 3 June 2021, https://arxiv.org/abs/2010.11929. Accessed 16 July 2023. \n\n[3] Radford, Alec, et al. “Learning Transferable Visual Models from Natural Language Supervision.” arXiv.Org, 26 Feb. 2021, https://arxiv.org/abs/2103.00020. \n\n[4] Ho, Jonathan, et al. “Denoising Diffusion Probabilistic Models.” arXiv.Org, 16 Dec. 2020, https://arxiv.org/abs/2006.11239. Accessed 16 July 2023. \n\n[5] Rombach, Robin, et al. “High-Resolution Image Synthesis with Latent Diffusion Models.” arXiv.Org, 13 Apr. 2022, https://arxiv.org/abs/2112.10752. \n\n[6] Emu Github : https://github.com/baaivision/emu\n\n[7] Figure 2 from : https://github.com/baaivision/emu\n\n[8] Aghajanyan, Armen, et al. “Introducing CM3leon, a More Efficient, State-of-the-Art Generative Model for Text and Images.” Meta AI, 14 July 2023, ai.meta.com/blog/generative-ai-text-images-cm3leon/. \n\n[9] Examples of CM3leon from the blog post by Meta : https://ai.meta.com/blog/generative-ai-text-images-cm3leon/\n\n[10] Multi-Modal GPT-4 on Bing Chat : https://www.reddit.com/r/singularity/comments/14zy2kp/mind_blown/\n\n[11] ControlNet Github : https://github.com/lllyasviel/ControlNet\n\n[12] ControlNet with User Scribble Example : https://github.com/lllyasviel/ControlNet/raw/main/github_page/p7.png\n\n[13] ControlNet QR-Code example : https://huggingface.co/monster-labs/control_v1p_sd15_qrcode_monster\n\n[14] Instruct Pix-2-Pix examples : https://github.com/timothybrooks/instruct-pix2pix\n\n[15] Li, Junnan, et al. “Blip-2: Bootstrapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models.” arXiv.Org, 15 June 2023, https://arxiv.org/abs/2301.12597. Accessed 16 July 2023. \n\n[16] Figure 1 from : https://arxiv.org/pdf/2301.12597.pdf\n\n[17] SDXL 0.9 : https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9\n\n[18] Dall-e 2 : https://openai.com/dall-e-2\n\n[19] Midjourney: https://www.midjourney.com/home\n\n[20] Zeroscope on HuggingFace : https://huggingface.co/cerspense/zeroscope_v2_576w\n\n[21] ControlVideo Github : https://github.com/YBYBZhang/ControlVideo\n\n[22] “Intel Introduces Real-Time Deepfake Detector.” Intel, www.intel.com/content/www/us/en/newsroom/news/intel-introduces-real-time-deepfake-detector.html#gs.330w9t. \n\n[23] Segment Anything Github : https://github.com/facebookresearch/segment-anything\n\n[24] Figure 3 from : https://github.com/facebookresearch/segment-anything\n\n[25] SEEM Github : https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once\n\n[26] Figure 3 from : https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once\n\n[27] HQ-SAM Github : https://github.com/SysCV/sam-hq\n\n[28] Figure 2 from : https://github.com/SysCV/sam-hq\n\n[29] Chen, Xiangning, et al. “Symbolic Discovery of Optimization Algorithms.” arXiv.Org, 8 May 2023, https://arxiv.org/abs/2302.06675. Accessed 16 July 2023.\n\n[30] Fang, Yuxin, et al. “Eva-02: A Visual Representation for Neon Genesis.” arXiv.Org, 22 Mar. 2023, https://arxiv.org/abs/2303.11331.\n\n[31] YOLOv8 Github : https://github.com/ultralytics/ultralytics\n\n[32] Figure 1 from : https://arxiv.org/pdf/2010.11929.pdf\n\n[33] Zhai, Xiaohua, et al. “Scaling Vision Transformers.” arXiv.Org, 20 June 2022, https://arxiv.org/abs/2106.04560.\n\n[34] 2nd Part of Figure 1 from : https://arxiv.org/pdf/2010.11929.pdf\n\n[35] Figure 1 from : https://anhreynolds.com/blogs/cnn.html\n\n[36] CLIPA-V2 : https://huggingface.co/papers/2306.15658\n\n[37] Figure 2 from : https://openai.com/research/clip\n\n[38] Figure 3 from : https://openai.com/research/clip\n\n[39] Figure 4 from How does Stable Diffusion Work: https://stable-diffusion-art.com/how-stable-diffusion-work/\n\n[40] Figure 7 from How does Stable Diffusion Work: https://stable-diffusion-art.com/how-stable-diffusion-work/\n\n[41] Figure 1 from U-Net Explained: Understanding its Image Segmentation Architecture : https://towardsdatascience.com/u-net-explained-understanding-its-image-segmentation-architecture-56e4842e313a\n\n[42] Figure 8 from The Illustrated Stable Diffusion : https://jalammar.github.io/images/stable-diffusion/stable-diffusion-diffusion-process.png\n","metadata":{}},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\ndf.iloc[0,1] = 'Image and/or video data'\ndf.iloc[1,1] = 'https://www.kaggle.com/code/prethamnandakumar/innovations-and-advancements-in-computer-vision'\ndf.iloc[2,1] = 'https://www.kaggle.com/code/soumyaoruganti/image-segmentation-ai-report-2023/comments#2330335'\ndf.iloc[3,1] = 'https://www.kaggle.com/code/omarrajaa/image-data-report/comments#2330348'\ndf.iloc[4,1] = 'https://www.kaggle.com/code/ethanhu8093/meta-analysis-on-kaggle-computer-vision/comments#2330358'\ndf.to_csv('submission.csv', index=False)\ndf.head()","metadata":{},"execution_count":null,"outputs":[]}]}