{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Essay Topic: Advances in Computer Vision from 2021 to 2023","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\nFor the past 2 years, rapid technological advancements have led to a significant increase in the amount of image and video data. This has emerged image and video processing as a valuable area with usage in various industries, like healthcare, security, autonomous driving, and entertainment. Therefore, computer vision (CV) and artificial intelligence (AI) became transformative technology, revolutionizing the way we analyze and understand images and videos. With the advent of deep learning techniques, computer vision has reached new heights, enabling us to extract valuable information and automate tasks. As you can see on the graph below, after AlexNet's publishing in 2012, it is considered to be one of the most influential papers published in computer vision, having spurred many more papers published employing Convolutional Neural Networks (CNNs) to accelerate deep learning nowadays.\n\n
\"Chart\"
\n
\n \n Figure 1: Amount of published papers about CV (from 2013 to 2022)\n\n
","metadata":{}},{"cell_type":"markdown","source":"# Image and Video Data Processing\n\n### **Data Collection**\nEven, if data preprocessing is not a recent advancement in current systems, it remains a crucial part of every SOTA model and solution. Every Kaggle Competition or research starts with data, in our situation it is image and video data, which is primarily used in every competition. Data could be collected differently, like photographs or cameras, as real data; another one is generated, synthetic data. We know that image and video data are the most difficult to collect because it needs to be high-quality and diverse for one label instance, but what if the second option is not applied, so generated data comes to help us? Nowadays, we use both types of data to increase our scores and create robust algorithms. As research paper shows, people get high scores with using these types of data (Roy Voetman et. al., 2023 (Ref. 1))\n\n
\"Chart\"
\n
\n \n Figure 2: Example of Real and Generated Data on MNIST (Ref. 2)\n \n
\n \n### **Data Preprocessing**\n\nData preprocessing plays a significant role in Deep Learning (DL), so we spend a good amount of time in data preprocessing before building the model. Image preprocessing is some amount of operations that do not increase image information content but it decreases noise or distort, therefore decreasing the entropy of the data and usually enhancing image features. Here are some examples of today's preprocessing:\n\n
\n \n \n \n \n \n \n
Gray scale transformation Changes from RGB to Gray colors
Brightness transformationChanges brightness of the image or video
Histogram equalization Contrast enhancement technique
Geometric TransformationElimination of geometric distortion when image is captured
Fourier transformationDecompose image into its sine and cosine components
\n
\n
\n \n Table 1: Preprocessing Methods\n\n
\n\n### **Data Augmentation**\n\nData Augmentation also plays a crucial role in creating robust solutions in today's Kaggle Competitions. In simple words, it creates new images based on the existing data. Image augmentation techniques allow you to artificially increase the size of your training set, thereby providing much more data to your model for training. Image and video data augmentation libraries such as, VidAug, TorchVision Transforms and Albumentations are image and video data augmentation libraries comprise of ways such as: RandomCrop, RandomRotate, HorizontalFlip, VerticalFlip, GaussianBlur,etc and as you can see in the graph below, augmentation brings significant benefits for DL models.\n\n\n
\"Chart\"
\n
\n \n Figure 3: Impact of data augmentation for skin lesion classification (Ref. 3)\n \n
\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Image and Video Data in Different Areas\n\n### **Image Classification**\nThis subfield of AI has significantly contributed to the success of computer vision. CNNs become a core of a lot of state-of-the-art (SOTA) computer vision systems. As CNNs are designed to mimic human brain activities, especially the visual cortex, these networks can learn complex patterns, enabling them to recognize objects, detect features and classify them. CNN architectures such as AlexNet, VGGNet, and ResNet have achieved groundbreaking results and probably surpassed human benchmarks. These advancements as we can see brought to current models such as Visual Transformers (ViT) to achieve around 90% accuracy on image classification task - ImageNet. Applying transformers to visual tasks helped in building an architecture that uses self-attention mechanisms to process images (Dosovitskiy et al., 2020 (Ref. 4)). So, ViT consists of a series of transformer blocks and each consists of layers: multi-head attention and feed-forward layer. As CNN uses pixel arrays, whereas ViT splits the input images into visual tokens or fixed-size patches which are fed to the encoder. So, when it comes to accuracy and efficiency ViT outperforms CNNs, as the attention layer in ViT embeds information across the whole image.\n\n
\n
\"Chart\"
\n
\n \n Figure 4: ViT model's Architecture (Ref. 4)\n \n
\n
\n\nAnother advancement is probably ConvNeXt (Zhuang Liu et al., 2022 (Ref. 5)), which is created by taking ideas from Transformers and ViTs. Basically, it is ResNet-50 model, but trained like ViT, so we change number of blocks, patch the image like Transformers, invert bottlenecks, use GELU instead of ReLU, and get replace BatchNorm with LayerNorm. As a result, we get 87.8% accuracy on ImageNet that outperforms SwinTransformers and ViTs.\n\n
\n
\"Chart\"
\n
\n \n Figure 5: CV Models Timeline (Ref. 6)\n \n
\n
\n \n### **Image Segmentation and Object Detection**\nZero-shot Learning and Image Segmentation from the prompt help to understand text and image relationships. This technology uses advanced algorithms to analyze text descriptions and image data, allowing it to recognize and segment objects without extensive training data. **CLIP-Seg** (Ref. 7) uses a frozen CLIP model as Text-Image embedding and transformer that segments images based on query prompts, so you can classify pixels not like \"plane\", but instead classify as \"wooden plane\" or \"blue plane\".\n\n
\n
\n \"Image\" \n
\n
\n
\n \n Figure 6: CLIP-Seg Architecture Work (Ref. 7)\n\n
\n\n#\n\nAnother advancement happened in 2023 with the release of Segment Anything Model (SAM) (Ref. 8), which uses 11 million images and 1.1 billion segmentation masks. The model consists of three parts:\n\n
\n \n \n \n \n
Image encoder ViT generates image embeddings
Prompt encoderEncodes masks, bounding boxes and texts into embedding vertor
Mask decoder Predicts segmentation masks based on the image and prompt encoders' embeddings
\n
\n
\n \n Table 2: SAM Architecture (Ref. 8)\n\n
\n\nBasically, SAM can be used as enhanced labeler with more improved accuracy, faster annotation and reduced workload, because real-world cases like geospatial imagery or medical imagery show how can such AI assistant achieve endless goals. SAM represented groundbreaking development in CV with adapting to segmentation problems using prompt engineering and NLP.\n \n \n \n### **Visual Question Answering**\nComputer Vision and Natural Language Processing (NLP) can improve performance, generalization, and zero-shot capabilities. So, it helpsto align between visual and language data, because the model could associate a visual context with its corresponding description. In other words, this kind of task is inspired by how human learns information, therefore it emerges better human interpretability and is useful in tasks such as image captioning or visual question answering. **CLIP** (Ref. 9) was released in 2021 and showed us how good AI could be if we connect CV and NLP together. So, while traditional DL image models train feature extractors and linear classifiers, CLIP trains image and text encoders on pairings (image, text). Here text encoder is like a zero-shot linear classifier embedding image captions.\n\n
\n
\n \"Image\" \n
\n
\n
\n \n Figure 7: CLIP Architecture (Ref. 9)\n\n
\n\nAnother area is Video Question Answering, unlike ImageQA it is much more challenging because of the spatio-temporal nature of videos. Therefore, techniques used in ImageQA could lead only to optimal results, and besides other tasks, question answering requires a comprehensive understanding of videos. As Yaoyao Zhong et al. (Ref. 10) described the task, QA consists of inference questions, which explore logical reasoning and factual questions to ask about visual facts. So, basically, we should have Video Encoder and Prompt Encoder to get the answer and there are different techniques that can be used in this area:\n\n
\n
\n \"Image\" \n
\n
\n
\n \n Figure 8: VideoQA Model's Different Architectures (Ref. 10)\n\n
\n\n### **Knowledge Reasoning with Graphs and ViTs**\n\nStandard Graph Neural Networks (GNNs) pass information between connected points across the whole network and how **A Generalization of ViT / MLP-Mixer to Graphs** (Ref. 11) explained it has several problems such as losing important features and referring to far-reaching connections, like recurrent neural networks. So, basically, their paper solves these issues with ViT, and therefore graph splits in patches and is mapped to embedding, information passes through layers, and in the end, there is a linear classifier layer to get predictions. Also in 2023, **A Survey of Knowledge Graph Reasoning on Graph Types** (Ref. 12) paper was released, where Knowledge Graph (KG - is a complex graph with different multi-modal information like text or image) in Knowledge Graph Reasoning (KGR) uncover relations between objects that were not before related. Therefore, as they stated in the abstract KGRs are used in QAs and recommendation systems to deduce new facts from existing ones based on logical rules.\n\n
\n
\n \"Image\" \n
\n
\n
\n \n Figure 9: Categories of Knowledge Graphs (Ref. 12)\n\n
","metadata":{}},{"cell_type":"markdown","source":"# Computer Vision on Conferences\nConference on Computer Vision and Pattern Recognition (CVPR) is conference on computer vision and pattern recognition, which is regarded as one of the most important conferences in its field. This year it was held in Vancouver, Canada and with around 2400 accepted papers, 1700+ were with arXiv versions. From this statistics, just under of 90% of papers had cs.CV listed in primary category (Ref. 13). In 2021, CVPR had different papers, but mainly they were focused on Image, Object, Detection, 3D, Video, Segmentation. However, in 2022, transformers and ViTs topics soared to Moon.\n\n
\n
\"Chart\"
\n
\n \n Figure 10: Top-75 keywords on CVPR 2021 and 2022 (Ref. 14)\n \n \n
\n
\n\n\nIn 2023, CVPR met another rocket, so-called Diffusion models. Diffusion models define a Markov chain of diffusion steps to slowly add random noise to data and then learn to reverse the diffusion process to construct desired data samples from the noise. Former generative models such as, GANs and VAEs also had success, but unlike Diffusion, they are not learned with a fixed procedure and the latent variable. As DALL-E, DALL-E 2, MidJourney and Stable Diffusion appeared, it was stated to be game-changing event for Visual Models. Therefore, it could be predicted that CVPR 2023 would be full of diffusion keywords. Yeah, it happened! If you look to the graph below \"Diffusion\" keyword set a trend for all accepted papers.\n\n
\n
\"Chart\"
\n
\n \n Figure 11: Relative change in keywords on CVPR 2022 and 2023 (Ref. 13)\n \n \n
\n
","metadata":{}},{"cell_type":"markdown","source":"# Image and Video data in Real Life Cases\nNowadays image and video data is a vital component in multiple applications and industries, and like healthcare or autonomous driving. Let's drop down to them how they improve with AI assistance in the past two years.\n\n- Healthcare - medical imaging with MRIs, and CT scans help to diagnose and treat patients with more time efficiency and accuracy. Applications like skin cancer detection or Parkinson's diseases, or even CEREBRA.AI startup - an AI-powered software in Kazakhstan that helps people with early stroke detection for more accurate treatment.\n\n- Self-Driving - real-time video feeds of the surrounding environment, enabling the vehicle's perception system to identify objects, pedestrians, traffic signs, and road conditions. However, currently, it rises several protests as self-driving cars sometimes block streets, park in the appropriate place, or just cannot adequately act in some situations that are understandable to a person. California has issued (Ref. 15) driverless licenses to 7 companies: Apollo, Autox, Cruise, Nuro, Waymo, Weride, and Zoox. The number of drones on the roads of California has increased significantly in the late afternoon, and in 2022 they traveled 25 times more kilometers than in 2021. Now look at the crashes (Ref. 16), in 2023, Cruise reported 28 accidents involving their drones, Waymo - 25, and Zoox - 12. The number of cars without drivers on the roads of California at the end of 2022 can be optimistically estimated: Cruise - 388 cars, Waymo - 688, Zoox - 142. That is, in 2023, Cruise had as many as 7% of cars in an accident. Waymo has 4%, and Zoox has 8%. That's a lot, and it doesn't take into account cases of blocking roads that companies are not required to report. For comparison, the probability of an accident for cars with a driver is <2%. Therefore, it causes light Luddism towards these cars and possible limitations of AI usage, which we will discuss in the next paragraph.\n\n
\n
\"Chart\"
\n
\n \n Figure 12: Self-driving Car Crash Reports in 2022 in California (Ref. 16)\n \n \n
\n
\n","metadata":{}},{"cell_type":"markdown","source":"# Limitations and Future Perspectives\n\n### Limitations\nThese limitations probably the most concerning for past two years, so let's drop down to them.\n\n1. Computational Requirements - CV systems excel at recognizing and localizing objects, they often lack common sense understanding and context. They should require large dataset for training and like current Kaggle Competition - Google Research - Identify Contrails to Reduce Global Warming, uses 451 Gb data of images. Even though, building large-scale models we should require high-definition data, large computational devices, so whole process is time-consuming and not useful for the environment and carbon footprint.\n\n2. Ethical Concerns and Biases - issues related to facial recognition and data privacy need to be addressed to ensure the responsible and unbiased use of computer vision systems. CV algorithms heavily rely on data, and if the data lacks diversity, it could predict unfair results. Addressing dataset bias mitigation requires collecting diverse and representative datasets\n\n### Future Perspectives\n\nComputer Vision still faces limitations, but it does not stop from enhancing and upgrading itself.\n\n1. Enhanced Perception and Understanding - CV models usually struggle with generalization from limited training data, however, systems will strive for improved perception and understanding of visual data. Future research should aim to perform well in uncontrolled environments, as it should with self-driving cars (Ref.15).\n\n2. Multi-modal Adaptation - integration with NLP and other fields has promising prospects. As it enhances combining more tasks and more information we could easily interact with multimedia data. Such applications could enable a comprehensive understanding of VideoQA or image captioning. ","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\n\nIn conclusion, the Computer Vision field has significantly risen and has emerged as a powerful application in current devices and implications. The advancements shown in DL algorithms and other techniques, like SAM, ViTs, and Zero-Shot models have improved the accuracy and efficiency of these programs. Future works in these areas could produce more solutions with the integration of NLP, Graphs, and other methods. \n#\nEven though, we should not forget about limitations that appear, as we need to consider many aspects like computational requirements or ethical concerns, events in 2022, in California (Ref. 15) show that AI usually cannot sometimes work in uncontrolled situations. Therefore, in the future, people should consider creating robust and continually learning AI models and applications.\nOverall, the usage of image and video data changes our daily life, particularly in healthcare or autonomous driving. However, it is difficult to predict what will happen with Computer Vision and Visual data usage, but we can truly impact us, and by understanding how these models should improve in the future we could identify a path for improvement.\n\n","metadata":{}},{"cell_type":"markdown","source":"# References\n\n1. Roy Voetman, Maya Aghaei, Klaas Dijkstra, The Big Data Myth: Using Diffusion Models for Dataset Generation to Train Deep Detection Models, 2022, https://arxiv.org/abs/2306.09762\n2. Code for DC-GAN model creation, https://theailearner.com/2019/09/27/creating-a-deep-convolutional-generative-adversarial-networks-dcgan/\n3. Fabio P, Christina V, Sandra A, Eduardo V. Data augmentation for skin lesion analysis. In: ISIC skin image analysis workshop and challenge, https://arxiv.org/abs/1809.01442\n4. Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2020, https://arxiv.org/abs/2010.11929\n5. Zhuang Liu et al., A ConvNet for the 2020s, 2022, https://arxiv.org/abs/2201.03545\n6. HuggingFace library and their models family, https://huggingface.co/docs/transformers/model_summary\n7. Timo Lüddecke, Alexander Ecke, Image Segmentation Using Text and Image Prompts, 2022, https://arxiv.org/pdf/2112.10003.pdf\n8. Kirillov et al., Segment Anything, 2023, https://arxiv.org/pdf/2304.02643.pdf\n9. Radford et al., Learning Transferable Visual Models From Natural Language Supervision 2021, https://arxiv.org/pdf/2103.00020.pdf\n10. Yaoyao Zhong et al., Video Question Answering: Datasets, Algorithms and Challenges, 2022, https://arxiv.org/abs/2203.01225\n11. Xiaoxin He et al., A GENERALIZATION OF VIT/MLP-MIXER TO GRAPHS, 2023, https://arxiv.org/pdf/2212.13350.pdf\n12. Ke Liang et al., A Survey of Knowledge Graph Reasoning on\nGraph Types: Static, Dynamic, and Multimodal, 2022, https://arxiv.org/pdf/2212.05767.pdf\n13. Blog post about CVPR 2023 statistics and best papers, https://voxel51.com/blog/cvpr-2023-and-the-state-of-computer-vision/\n14. GitHub repository with code to create statistics about CVPR 2021 and 2022, https://github.com/BIGBALLON/CVPR2022-Paper-Statistics/tree/master\n15. Californian Governmental Data about Autonomous Vehicles, https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-testing-permit-holders/\n16. Californian Governmental Data about Autonomous Vehicles Crash Reports, https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/\n","metadata":{}},{"cell_type":"markdown","source":"# Appendix","metadata":{}},{"cell_type":"code","source":"# This code helps to get amount of papers in Computer Vision, it appears as Figure 1 in this essay\n\n# import json\n# import gzip\n# import pandas as pd\n# import matplotlib.pyplot as plt\n\n# df = pd.read_json(\"/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json\", lines=True)\n\n# def increm(cat, name):\n# total = 0\n# final = []\n \n# for i in cat:\n# if i.find(name) != -1 :\n# final.append(cat[total])\n# total += 1\n \n# return final\n\n# new_df = pd.concat([df[df.isin(increm(df['categories'], \"cs.CV\")).any(axis=1)]])\n# comp_data = new_df.astype(str).drop_duplicates().reset_index(drop=True)\n# comp_data['update_date'] = pd.to_datetime(comp_data['update_date'] , format='%Y-%m-%d')\n\n# count = []\n# year = []\n\n# for i in range(2013, 2023):\n# count.append(len(comp_data[comp_data['update_date'].dt.year == i]))\n# year.append(i)\n\n# fig, ax = plt.subplots(figsize=(10, 10))\n\n# plt.plot(year, count, color='red', marker='o')\n# plt.title(\"Amount of papers in CV\")\n# plt.xlabel(\"Year\")\n# plt.ylabel(\"Amount of papers\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\n\ndf.iloc[0, 1] = 'Image and/or video data'\ndf.iloc[1, 1] = 'https://www.kaggle.com/code/dokster/image-and-video-data-kaggle-ai-report'\ndf.iloc[2, 1] = 'https://www.kaggle.com/code/pandoraalias/unleashing-visual-intelligence-vid-img/comments#2343929'\ndf.iloc[3, 1] = 'https://www.kaggle.com/code/radbear/advances-in-ai-vision-models-in-the-last-two-years/comments#2347067'\ndf.iloc[4, 1] = 'https://www.kaggle.com/code/prethamnandakumar/innovations-and-advancements-in-computer-vision/comments#2347085'\n\ndf.to_csv('submission.csv', index=False)\ndf.head()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2023-07-05T10:28:39.094687Z","iopub.status.busy":"2023-07-05T10:28:39.093942Z","iopub.status.idle":"2023-07-05T10:28:39.173648Z","shell.execute_reply":"2023-07-05T10:28:39.172259Z","shell.execute_reply.started":"2023-07-05T10:28:39.094640Z"},"trusted":true},"execution_count":null,"outputs":[]}]}