{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Text to Image Generation Models: The journey so far.\n\n###### Author: Harshada Kushe\nThere are certain portions of the essay that are generated by chatGPT and Bard.\n\n\n# Subject: Text to Image Generation Models: The journey so far.\nSeveral AI/ML models were created overtime that got us closer and closer to the highly advanced text-to-image ai/ml model that we are using today with MidJourney, Dall-E, Stable Diffusion and Google Brain’s Imagen. These models generate human-like visually realistic images from textual descriptions or prompts.\n\n\n---","metadata":{}},{"cell_type":"markdown","source":"## How did it start?","metadata":{}},{"cell_type":"markdown","source":"\n### AlignDRAW\n\nOne of the very first papers published with the text-image generation model was in 2016 by researchers from University of Toronto which was titled – “GENERATING IMAGES FROM CAPTIONS WITH ATTENTION” by Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba & Ruslan Salakhutdinov. They introduced the alignDRAW model which represented a recurrent variational autoencoder (RNN-VAE) with an attention mechanism. The conditioning of this model was imposed by leveraging textual sequences. \n\nThe images generated by alignDRAW lacked photorealism and were blur but had a remarkable ability to generalize object categories which were absent from he training data. For example it could generate an image of a red school bus even though it was not a part of its training data. Since the alignDRAW generated blurry images they added a step where they used GAN(Generative Adversal Network) to create sharp edges. However this was not ideal because the entire model was not trained as a single unit. The alignDRAW method did not have an architecture that handles the complete transformation of data from individual characters to the final pixel-level representation without any breaks or gaps. It required a separate post-processing step to enhance the images and make them sharp.\n\n### GAN: Generative Adversarial Networks\nGAN architectures use two separate neural network models to generate data via an adversarial learning process, which are referred to as generator and discriminator models. In a GAN, the generator and discriminator are two neural networks that compete against each other. The generator tries to create images that are indistinguishable from real images, while the discriminator tries to distinguish between real and fake images.\nSmall Scale Data Regime GAN methods: These are designed for situations where the dataset used for generating images from texts is relatively small. They may not work effectively when dealing with larger datasets. \n\n### Text-conditional GAN\nThis drawback is handled by Text-conditional GAN. It has an end-to-end differential system which is designed to process and transform the data from the beginning character level to end to end pixel level in a differential manner without any gaps in the process. It generated high-quality images that are semantically aligned because of generative adversarial network(GAN). However, the drawbacks is, it faced mode-collapse where the generator learns to generate only a small number of output images, low-image \nquality, semantic misalignment and are more difficult to train.\n\n### StackGAN\nIt uses a two-stage architecture. In the first stage a coarse image is generated from the text and in the second stage a coarse image is refined from the stage. This allowed it to generate more realistic and detailed images than Text-conditional GAN and alignDRAW. It generates better quality of images, less likely to suffer from mode collapse and was more systematically aligned with the text description that was given.\n\n### AttnGAN \nIt generated more sematically aligned images than the other models. It is because attnGAN allows to focus on specific part of text-description while generating images. Also it was less likely to suffer from mode collapse and generated higher quality of images. The biggest drawback it was more expensive than he previous models and it was difficult to train because the attention model can make it more difficult for the generator and the discriminator to learn from each other.\nBigGAN-deep: BigGAN-deep is a deeper version (4x) of BigGAN. It is a variant of BigGAN that uses a deeper generator and discriminator. This allows BigGAN-deep to generate higher-quality images with less data. BigGAN-deep uses a different strategy called \"identity mapping\" to preserve the identity of the image throughout the skip connections. In identity mapping, the output of one layer is simply passed through a 1x1 convolution without any additional parameters. This means that the output of the first layer is exactly the same as the output of the second layer.\n\n### StyleGAN-ADA \nThis is a variant of StyleGAN that uses an adaptive discriminator. The adaptive discriminator is able to learn to focus on different aspects of the image as the generator improves. This allows StyleGAN-ADA to generate more realistic images with less data.\n\n### BigBiGAN\nThis is a bi-directional GAN that uses two generators and two discriminators. The first generator generates images from text descriptions, and the second generator generates text descriptions from images. The two discriminators are used to distinguish between real and fake images. BigBiGAN has been shown to be more effective at generating images from text descriptions with less data than single-generator GANs.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndata = {\n    'Date': ['2019, December', '2020, May 13', '2020, June 9', '2018, November 19', '2017, December 12', '2016, December 12', '2015, November 23'],\n    'Model': ['BigBiGAN', 'StyleGAN-ADA', 'BigGAN-deep', 'BigGAN', 'AttnGAN', 'StackGAN', 'Text-conditional GAN'],\n    'Strengths': [\n        'More effective at generating images from text descriptions with less data than single-generator GANs.',\n        'Uses an adaptive discriminator that can learn to focus on different aspects of the image as the generator improves. This allows StyleGAN-ADA to generate more realistic images with less data.',\n        'Uses a deeper generator and discriminator, which allows it to generate higher-quality images with less data.',\n        'One of the first GAN models to generate high-quality images from text descriptions.',\n        'Uses an attention mechanism that allows the generator to focus on specific parts of the text description when generating the image. This makes it possible for AttnGAN to generate images that are more semantically aligned with the text descriptions they were given.',\n        'Uses a two-stage approach that allows it to generate images that are more realistic and detailed than images generated by previous text-to-image generation models.',\n        'One of the first GAN models to generate images from text descriptions.'\n    ],\n    'Weaknesses': [\n        'More computationally expensive than single-generator GANs.',\n        'Can be more difficult to train than other GAN models.',\n        'Can be more computationally expensive than other GAN models.',\n        'Can be difficult to train and can suffer from mode collapse.',\n        'Can be more computationally expensive than other GAN models.',\n        'Can be more computationally expensive than other GAN models.',\n        'Can be difficult to train and can suffer from mode collapse.'\n    ]\n}\n\ndf = pd.DataFrame(data)\ndf\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T18:27:46.467213Z","iopub.execute_input":"2023-07-04T18:27:46.467665Z","iopub.status.idle":"2023-07-04T18:27:46.485208Z","shell.execute_reply.started":"2023-07-04T18:27:46.467631Z","shell.execute_reply":"2023-07-04T18:27:46.484026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Large scale data Autoregressive methods:\n\n### BigGAN\nIt creates extremely detailed and high resolution images. BigGAN uses a strategy called \"concatenation\" to preserve the identity of the image throughout the skip connections. In concatenation, the output of two layers is simply concatenated together. This means that all of the information from the first layer is preserved in the second layer.\n\n### VQGAN+CLIP\nVQGAN stands for Vector Quantized Generative Adversarial Network, while CLIP stands for Contrastive Image-Language Pretraining. VQGAN+CLIP means interaction between these two networks which are entirely separate models and they work in tadem.\n\n### Parti\nParti is an autoregressive text-to-image model that is similar to DALL-E. It is trained on a dataset of text descriptions and images, and it is able to generate images that are more realistic and detailed than images generated by previous models. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, analogous to machine translation – this allows it to benefit from advances in large language models, especially capabilities that are unlocked by scaling data and model sizes. In this case, the target outputs are sequences of image tokens instead of text tokens in another language. Parti uses the powerful image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens, and takes advantage of its ability to reconstruct such image token sequences as high quality, visually diverse images.\n\n### Midjourney\nMidjourney is a text-to-image generation model that is trained on a dataset of text descriptions and images. It is able to generate images that are more realistic and detailed than images generated by previous models. Midjourney uses a combination of GANs and diffusion models to generate its images. The GANs are used to generate a rough outline of the image, while the diffusion models are used to add detail and creativity. This combination of techniques allows Midjourney to generate images that are both realistic and imaginative.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndata = {\n    'Model': ['BigGAN', 'DALL-E', 'VQGAN+CLIP', 'Parti', 'Midjourney'],\n    'Date Introduced': ['2018', '2021', '2021', '2022', '2022'],\n    'Dataset': ['ImageNet', \"OpenAI's CLIP\", \"OpenAI's CLIP\", 'COCO-Text', 'Midjourney dataset'],\n    'Parameters': ['1.3B', '137B', '10B', '10B', '178B'],\n    'Diffusion Model': ['No', 'No', 'No', 'Yes', 'Yes'],\n    'Resolution': ['128x128', '256x256', '256x256', '1024x1024', '1024x1024'],\n    'FID Score on MS COCO': [30.1, 21.9, 19.9, 17.9, 17.4]\n}\n\ndf = pd.DataFrame(data)\ndf\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T18:18:46.944847Z","iopub.execute_input":"2023-07-04T18:18:46.945902Z","iopub.status.idle":"2023-07-04T18:18:46.966877Z","shell.execute_reply.started":"2023-07-04T18:18:46.945853Z","shell.execute_reply":"2023-07-04T18:18:46.965470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The FID score is a measure of how similar generated images are to real images, and a lower score indicates higher similarity.\nDrawbacks and strengths of these models:\nThe biggest drawback of all these models is that they suffer from high computational cost and has problems of accumulating errors over time leading to a chain of errors.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndata = {\n    'Model': ['BigGAN', 'DALL-E', 'VQGAN+CLIP', 'Parti', 'Midjourney'],\n    'Strengths': [\n        'High resolution images, good at generating images of objects and scenes',\n        'High resolution images, good at generating images of creative concepts',\n        'High resolution images, good at generating images that are both realistic and creative',\n        'High resolution images, good at generating images that are both realistic and creative',\n        'High resolution images, good at generating images that are both realistic and creative'\n    ],\n    'Drawbacks': [\n        'Low FID score, can be blurry and unrealistic',\n        'Can be repetitive, can generate images that are not visually appealing',\n        'Can be slow, can be difficult to generate images with specific details',\n        'Can be expensive, can be difficult to generate images with specific details',\n        'Can be difficult to get access to, can be expensive'\n    ]\n}\n\ndf = pd.DataFrame(data)\ndf\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T18:23:27.242096Z","iopub.execute_input":"2023-07-04T18:23:27.242507Z","iopub.status.idle":"2023-07-04T18:23:27.258429Z","shell.execute_reply.started":"2023-07-04T18:23:27.242477Z","shell.execute_reply":"2023-07-04T18:23:27.257045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Diffusion Models:\n\nDiffusion models are trained in a similar fashion to a wake sleep algorithm. The wake sleep algorithm introduced a way to train two types of models namely inference models and generative models by pitting them against each other (Hinton, 1995; Dayan et al.,1995). This approach remained unexplored for nearly two decades but there has been a surge in its development in the recent times. Researchers have come up with variational learning and inference algorithms that directly train a flexible generative model, which generates data, and an inference model, which tries to understand and infer hidden patterns or variables in the data. These models are trained together so that their performance becomes better with respect to each other. \nThey solve the problems of error accumulation. Error accumulation is a problem that occurs when an autoregressive model generates an image one pixel at a time. As the model generates more pixels, the errors in the earlier pixels can accumulate and make the final image blurry or unrealistic. Diffusion models solve these problems by gradually adding noise to an image until it reaches a desired level of realism. The model then learns to remove the noise and generate images that are both realistic and creative. This process helps to prevent error accumulation and makes the generation process more efficient.\nDiffusion models can have a higher computational cost than autoregressive models, especially for high-resolution images. This is because diffusion models need to generate the entire image at once, while autoregressive models can generate the image one pixel at a time. However, diffusion models can be more efficient in terms of memory usage, as they do not need to store the entire image in memory at once.\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndata = {\n    'Model Type': ['Diffusion Model', 'Autoregressive Model'],\n    'FID Score': ['Lower', 'Higher'],\n    'Computation Cost': ['More computationally expensive', 'Less computationally expensive'],\n    'Memory Usage': ['Less memory intensive', 'More memory intensive'],\n    'Error Accumulation': ['Less likely', 'More likely'],\n    'Efficiency': ['Less efficient', 'More efficient']\n}\n\ndf = pd.DataFrame(data)\ndf\n\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T18:44:43.904412Z","iopub.execute_input":"2023-07-04T18:44:43.904979Z","iopub.status.idle":"2023-07-04T18:44:43.928884Z","shell.execute_reply.started":"2023-07-04T18:44:43.904924Z","shell.execute_reply":"2023-07-04T18:44:43.927061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Markov chain which is used as a tool to evaluate a predefined model. In context of diffusion, we take a different approach, instead of using Markov chain for approximation, the Markov chain itself is considered as a model. More concisely the probabilistic model we want to obtain is defined as the outcome of the Markov chain. Therefore, we can derive analytical expressions for the probabilities associated with each step of diffusion process and understand the behaviour of the chain more precisely. \nDenoising Diffusion Probabilistic models (DDPM): DDPM is an improved variant of DPM, specifically designed to generate high quality samples and denoising. DDPM model was introduced in 2020. The advent of DDPM can be attributed to previous attempts (SGM) Score based generative models and (DPM) Diffusion Probabilistic Models that we discussed earlier.\n\n### Diffusion Probabilistic Model (DPM)\nThe diffusion chain maps data from a complex distribution to a simpler one by estimating the diffusion of Markov diffusion chain. First it uses a forward process called as inference process to transform complex data into simpler data and learns mapping by reversing the diffusion process. DDPM uses advanced techniques in a similar way to DPM.\n\n### Score based generative Models (SGM)\nSGM is a technique that adds different levels of random noise to the data to generate samples. By using the gradient of the logarithm of the probability density as a score function, SGM generates samples by gradually reducing the amount of noise. The model is trained by estimating the score functions for noisy data distribution.\n\nBoth SGM and DDPM share the same objective during training and optimize their models in a similar way though they have different motivations. To add to that, researchers have found a new version of SGM (32) that is specifically designed for generating high resolution and detailed images.\nDDPM for image synthesis: They use a step-by-step process to generate images from noisy data. During training the models learn the process by going in reverse that is by using the Markov chain method. That is, they start with normal images and introduce noise at every step. The goal is to optimize the model by estimating the probability of the noise that has o be added at every step to the data. To summarize DDPM’s are models that generate high resolution images by reducing noise. \n\n## The impact of labels on Image synthesis in GAN’s and Diffusion models:\n\nConditional GAN’s introduced the concept of using class labels as an additional input layer to the model as guidance. Class labels, conditional inputs and user prompts are considered a form of guidance which supplement the learning process of these models. Other techniques such as class-conditional normalization, use of auxiliary classifier loss have also been explored to benefit as labels in GAN’s.\nSince this method was successful in GAN’s, the diffusion models have also incorporated this method. This is achieved by the development of two types of models: the conditional diffusion model and the guided diffusion model.\n\nThe conditional diffusion model takes additional information like class labels as input during the learning process to generate images. In guided diffusion model, the training process involves the involvement of class-induced gradients which guide the sampling process during image synthesis leading to improved image results.\n\nIn the classifier free guidance, a single model is jointly trained with both an unconditional score estimator and a conditional score estimator where the class label is used as a condition. This approach removes the need for a separate classifier and makes the diffusion more efficient.\n\n## Best of Text to Image Diffusion Models:    \nIt can be categorized on where the diffusion process occurs either in the pixel space or the latent space.\n\n## Pixel Space\n\nThe generation of images occurs directly at individual pixel. It evolves by iteratively updating the pixel values based on the surrounding text and the desired distribution of the target image. The model learns the transformation rules through training on every image dataset which generates realistic images. Examples of these methods include Glide and Imagen.\n\n### Glide\n\nIt is a text-to-image(T2I) framework that generated images on given text descriptions. In GLIDE, the text encoder is trained alongside the diffusion prior using paired image-text data. This means that the text encoder learns to encode text descriptions into meaningful representations that can be used for image generation. It uses a technique called as guided diffusion where the original class label is replaced with text allowing the generated images to be conditioned on textual prompt. GLIDE incorporates a text encoder, which is a type of neural network called a transformer. This text encoder consists of 24 residual blocks and has a width of 2048, meaning it has a large number of parameters to capture intricate details in the text. Experimental results demonstrate that GLIDE outperforms DALL-E, another popular T2I model, in terms of both Fréchet Inception Distance (FID) and human evaluation scores. \n \n### Imagen\n\nJust like GLIDE Imagen text-to-image framework which uses classifier-free guidance but the key difference is terms of choice of encoder. Imagen takes a different approach than GLIDE by using a pre-trained large language model as text-encoder. The pre-trained model has all its weights fixed and is already trained on a large amount of data which eases its computational burden. \n\nThe text encoder in Imagen can be pre-trained on either image-text data, such as CLIP, or on text-only corpora, such as BERT, GPT, and T5. Text-only corpora are much larger than paired image-text data, providing a wider range of textual contexts for the language model. For example, the text-only corpus used in BERT is about 20GB, while the one used in T5 is around 800GB. Imagen demonstrates that using larger language models, like different variants of T5, as the text encoder improves the quality and alignment between generated images and text descriptions more effectively than increasing the size of the diffusion model itself.\n\n### SR3\n\nSR3 is a super-resolution diffusion model that takes as input a low-resolution image and builds a corresponding high resolution image from pure noise. The model is trained on an image corruption process in which noise is progressively added to a high-resolution image until only pure noise remains. It then learns to reverse this process, beginning from pure noise and progressively removing noise to reach a target distribution through the guidance of the input low-resolution image. \nGoogle uses SR3 for class conditional image generation which produces high resolution detailed images. Since ImageNet was a high entropy dataset, they built a cascade of multiple diffusion models. CDM is a class-conditional diffusion model trained on ImageNet data to generate high-resolution natural images. Since ImageNet is a difficult, high-entropy dataset, we built CDM as a cascade of multiple diffusion models. This cascade approach involves chaining together multiple generative models over several spatial resolutions: one diffusion model that generates data at a low resolution, followed by a sequence of SR3 super-resolution diffusion models that gradually increase the resolution of the generated image to the highest resolution. It is well known that cascading improves quality and training speed for high resolution data, as shown by previous studies (for example in autoregressive models and VQ-VAE-2) and in concurrent work for diffusion models.\n\n### VQGAN+CLIP\n\nThis model uses a combination of a vector quantization (VQ) model and a text-conditioned diffusion model (CLIP) to generate images. The VQ model learns to represent images as a sequence of discrete tokens, and the CLIP model learns to map text descriptions to these token sequences. This allows the model to generate images that are both realistic and relevant to the given text description.\n\n### BigGAN\n\nThis model is a diffusion model that uses a very large number of parameters (1.3 billion). This allows it to generate images that are very high-quality and realistic. However, BigGAN is also very computationally expensive to train and use.\nStyleGAN\nThis model is a diffusion model that uses a technique called progressive growing to generate images of increasing resolution. This allows StyleGAN to generate images that are both high-quality and realistic, while also being computationally efficient.\n\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndata = {\n    'Model': ['VQGAN+CLIP', 'BigGAN', 'StyleGAN', 'SR3', 'Imagen', 'Glide'],\n    'FID': [20.9, 14.6, 13.4, 12.8, 12.7, 12.6],\n    'Classification Accuracy Score': ['93.50%', '95%', '94%', '95%', '95.50%', '95%'],\n    'Computation Cost': ['High', 'Very high', 'High', 'Very high', 'Extremely high', 'Very high'],\n    'Memory Usage': ['High', 'Very high', 'High', 'Very high', 'Extremely high', 'Very high']\n}\n\ndf = pd.DataFrame(data)\ndf\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T19:39:18.358578Z","iopub.execute_input":"2023-07-04T19:39:18.359035Z","iopub.status.idle":"2023-07-04T19:39:18.378943Z","shell.execute_reply.started":"2023-07-04T19:39:18.359002Z","shell.execute_reply":"2023-07-04T19:39:18.377990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, all of these models have achieved very high FID scores, which indicates that they are able to generate images that are very realistic and indistinguishable from real images. They also have high classification accuracy scores, which indicates that they are able to generate images that are relevant to the given text prompt. In terms of computation cost and memory usage, all of these models are very demanding. They require a lot of computational resources to train and generate images. However, Imagen is the most demanding model, followed by SR3, VQGAN+CLIP, BigGAN, and StyleGAN.\n\n## Frameworks in Latent Space\n\nLike the pixel-space method uses a classifier-free guidance, VQ-diffusion has been used, which uses a mask-then-replace diffusion strategy. The Stable Diffusion model reverses a forward diffusion process that adds noise to the latent space. They also introduce cross-attention as a way to condition the model on different signals, like text. The experimental results show that using diffusion modeling on the latent space is better than using it on the pixel space in terms of reducing complexity and preserving details. Stable Diffusion is a milestone work in the field of latent space. It is a framework that trains diffusion models on the latent space, which is a scaled-up version of the Latent Diffusion Model (LDM). In this framework, they use a technique called VQ-GAN to represent the latent space in the first stage. VQ-GAN improves upon VQ-VAE by adding an adversarial objective to make the synthesized images look more natural.\n\nDALL-E2 is a text-to-image model that operates in the latent space using multimodal contrastive models. In this approach, the model matches image embeddings and text encodings in the same representation space. CLIP is a key component used in DALL-E2, which learns these multimodal representations and has been widely used in text-to-image models. It is also known as unCLIP, uses the CLIP text encoder and inverts the CLIP image encoder with a diffusion model to generate images from the CLIP latent space. This combination of encoder and decoder is similar to the structure of a VAE, although the inverting decoder is non-deterministic. The main task is to train a prior that bridges the gap between the CLIP text and image latent space, called the text-image latent prior. It also finds that this prior can be learned using either an autoregressive method or a diffusion model, but the diffusion prior achieves better performance. Experimental results show that removing this text-image latent prior leads to a significant drop in performance, emphasizing the importance of learning this prior. Inspired by DALL-E2, clip2latent proposes training a diffusion model that bridges the gap between CLIP embeddings and a pretrained generative model, such as StyleGAN.\n\nIn clip2latent, the diffusion model generates the latent space of StyleGAN from the CLIP image embeddings. During inference, the latent space of StyleGAN is generated directly from text embeddings, enabling language-free training for text-to-image diffusion.\n\nThough StyleGAN operates in the pixel space, not in the latent space. In the context of clip2latent, the diffusion model is trained to generate the latent space of StyleGAN from the CLIP image embeddings. However, during inference, the latent space of StyleGAN is generated directly from text embeddings as if they were image embeddings. This enables language-free training for text-to-image diffusion, even though StyleGAN itself operates in the pixel space.\n\n## Applications in the text-to-image space\n\n### Paintings\n\nAI based painting have taken the internet by storm in the recent times.\nIt started with GAN based painting techniques but the problems that they faced was unstable training and model collapse. The recent diffusion models have pioneered the way in creating detailed high-definition images with improved techniques and themes.\n\nThere are several models that are used to enhance the framework of these differential models. One such model is Multimodal guided artwork diffusion (MGAD). MGAD uses the generative process of a diffusion model with both image and text input labels called as multimodal guidance which creates impressive digital artworks.\n\nAnother approach is DiffStyler which maintains global content of the input image. It uses two diffusion models one wherein which it allows for explicit content and the other with abstract aesthetics. By using two separate models it can simultaneously learn and generate both the specific content of the image and the overall artistic style of the image. The performance is evaluated both manually and by using quantitative metrics. The ability to control the noise during the process allows the balance to be made between aesthetics and the explicit content which allows control over the artwork.\n\nTo enhance the creativity of a diffusion, model another study proposes, first to extend textual conditions and retrain the model using wikiart dataset. One can create artwork in the style of famous artists. The aesthetics style is customised by using a set of images provided by the use. To add to that generated images are extended to Scalable Vector Graphics which can be used as digital icons and artistic purposes.\n\n### eDiff-I\n\nThis model uses an ensemble of expert denoisers, each specialized for a particular stage of the diffusion process. This helps to improve the overall quality of the generated images.\n\n### VQGAN+CLIP\n\nThis model combines a diffusion model with a text-based encoder called CLIP. This allows the model to generate images that are more closely aligned with the user's desired output.\n\n### BigGAN\n\nThis model is a large language model that has been trained on a massive dataset of text and code. It can be used to generate text descriptions of images, which can then be used to guide the diffusion model in generating images that match the user's desired output.\n\n### StyleGAN \n\nThis model is a generative adversarial network (GAN) that has been trained on a large dataset of images. It can be used to generate images that are like the images in the training dataset.\n\n### VQVAE\n\nThis model is a variational autoencoder (VAE) that has been trained on a large dataset of images. It can be used to generate images that are like the images in the training dataset, but with different random noise added to each image.\n\n## Video generation and story visualization\n\nText-to-video refers to a technology that can generate videos based on text input. Videos are made up of a series of images, so this application combines text-to-image and video generation techniques. There are two main approaches in text-to-video generation.\n\n### Make-A-Video\n\nIt adapts a pre-trained text-to-image model to generate videos. It enhances the visual quality by incorporating temporal information, using models for spatial super-resolution and frame interpolation. Make-A-Video achieves this without requiring paired text-video data, thanks to the use of pre-trained text-to-image models and unsupervised learning on video data.\nThe second approach, called Video Imagen, extends an existing video generation method to include text-to-video capabilities. Video Imagen utilizes cascaded video diffusion models and draws insights from recent findings in text-to-image generation and video diffusion models. For example, it incorporates frozen encoder text conditioning and v-prediction parameterization.\n\n### Text-to-story generation\n\nWhich aims to generate videos that match text-based stories. Make-A-Story and AR-LDM are two examples of systems that tackle this task. Make-A-Story proposes an autoregressive diffusion-based framework with a visual memory module to capture actor and background context across frames. It also introduces sentence-conditioned soft attention for visio-lingual co-reference resolution to maintain consistency between scenes. AR-LDM, on the other hand, focuses on stable diffusion for text-to-story generation.\n\n### Text2Video-Zero \n\nIt is a text-to-video diffusion model that does not require any paired text-video data to train. It is trained on a dataset of text descriptions and videos, and it learns to generate videos that match the text descriptions without having seen any examples of paired text-video data.\nRunway Gen is a text-to-video diffusion model that is trained on a dataset of text descriptions and videos. It also includes a text-to-pose model that can be used to generate the poses of the actors in the video. This allows the model to generate videos that are more realistic and engaging.\nNuwa-XL is a text-to-video diffusion model that is trained on a dataset of text descriptions and videos. It also includes a text-to-motion model that can be used to generate the motion of the actors in the video. This allows the model to generate videos that are more realistic and dynamic.\n\n## Image Editing\n\nGenerating 3D objects is more complex than generating 2D images. DeepFusion was the first work to successfully use diffusion models for 3D object synthesis. It was inspired by Dream Fields, which used 2D image-text models for 3D synthesis. DeepFusion trained a randomly initialized NeRF model with the knowledge distilled from a pre-trained 2D diffusion model called Imagen.\n\n### Magic3D\n\nHowever, according to Magic3D, DeepFusion had some limitations. The low-resolution image supervision and slow optimization process of NeRF led to low-quality generation and long processing times. To address this, Magic3D proposed a coarse-to-fine optimization approach. It used a coarse representation as an initialization and then optimized mesh representations with high-resolution diffusion priors. Magic3D also accelerated the generation process using a sparse 3D hash grid structure.\n\n### 3DDesigner\n\n3DDesigner focused on the consistency of 3D object generation, specifically the cross-view correspondence. It used low-resolution results from the NeRF-based condition module as a prior and combined it with a two-stream asynchronous diffusion module to enhance consistency. This approach aimed to achieve consistent results from different viewing angles, even up to 360 degrees.\n\n### 3D-GAN\n\n3D-GAN model is a generative adversarial network (GAN) that can be used to generate 3D models from a single image. It is trained on a dataset of 2D images and their corresponding 3D models, and it learns to generate 3D models that are like the images in the dataset.\n\n### Pix3D\n\nPix3D is a 3D image translation model that can be used to translate 2D images to 3D models. It is trained on a dataset of 2D images and their corresponding 3D models, and it learns to generate 3D models that are similar to the images in the dataset.\n\n### StyleGAN3\nStyleGAN3 is a generative adversarial network (GAN) that can be used to generate high-quality images. It is trained on a dataset of images, and it learns to generate images that are similar to the images in the dataset. StyleGAN3 can also be used to generate 3D models by converting the generated images to 3D.\nDeepView3D is a deep learning model that can be used to extract 3D information from a single image. It is trained on a dataset of images and their corresponding 3D models, and it learns to identify and extract 3D features from images.\n\n### 3D-VGAN\n3D-VGAN is a variational autoencoder (VAE) that can be used to generate 3D models from a latent space. It is trained on a dataset of 3D models, and it learns to generate 3D models that are similar to the models in the dataset.\n\n## Text to image Editing:\n\nText-guided image editing is a technique that allows you to modify images based on textual descriptions. Before diffusion models became popular, this task was mainly done using GAN inversion methods combined with CLIP. However, GANs had limitations in accurately inverting images, which sometimes resulted in unintended changes to the image content.\n\n### DiffusionCLIP\n\nDiffusionCLIP was a pioneering work that introduced diffusion models to address this problem. It used a pre-trained diffusion model to convert the input image into a latent space and then fine-tuned the model in reverse using two loss terms: a local directional CLIP loss and an identity loss. The CLIP loss guided the edited image to align with the text, while the identity loss prevented unwanted changes. DiffusionCLIP showed superior performance in both in-domain and out-of-domain manipulation.\n\n### LDEdit\n\nAnother approach called LDEdit proposed combining diffusion models with deterministic forward diffusion in the latent space. This approach performed well in various image editing tasks and provided a generalized framework without the need for fine-tuning.\n\n### Prompt-to-Prompt\n\nTo solve the issue of different outputs with slight modifications in the text prompt, Prompt-to-Prompt used a cross-attention map during the diffusion process to represent the relationship between image pixels and words in the text prompt. In the task of image-to-image translation, manipulating spatial features and self-attention inside the model was found to control the translation process.\nThere were also methods that focused on unsupervised image translation and disentangling style and content representation. CycleDiffusion unified generative models and showed that diffusion models can be guided similarly to GANs.\n\n### Direct Inversion\n\nDirect Inversion used a two-step procedure of encoding the image into noise and generating the edited image with inverted noises. This method did not require optimization or model fine-tuning. DDIM was widely used for its near-perfect inversion, but it could lead to incorrect image reconstruction due to linearization assumptions. EDICT proposed maintaining two coupled noise vectors to achieve higher reconstruction quality, although it required more computation time.\n\nIn the case of image editing with masks, the challenge was to ensure a seamless coherence between the masked region and the background. Blended diffusion addressed this challenge by using pretrained CLIP and incorporating two loss terms: one for aligning the masked image with the text caption and another for preserving the unmasked region. It also employed a progressive blending technique to ensure a smooth transition between the edited region and the rest of the image.\n\nText-guided image editing allows you to modify images based on text descriptions. Diffusion models have been used to improve this process by accurately aligning the edited image with the text and minimizing unintended changes. Different techniques have been developed to address specific challenges, such as maintaining coherence between masked regions and the background. These methods have shown promising results in various image editing tasks.\n\n### Imagic\n\nImagic allows you to perform text-based edits on a single image by optimizing the text representation and fine-tuning diffusion models. There are also other interesting tasks, like semantic image composition and semantic mixing, which involve blending concepts or transforming images based on human instructions. These approaches leverage the power of diffusion models and have shown promising results in generating realistic and faithful images.\n\n## Challenges:\n\n### Dataset Bias: \n\nLarge models like the ones mentioned are trained using text and image data that are collected from various sources. However, this data collection process can introduce biases, such as biases related to race and gender. Additionally, these models mostly use English as the default language for input text, which can disadvantage those who don't understand English.\n\nTo address these issues, it is important to have a more diverse and balanced dataset for training the models. This means including a wider range of text and image samples from different backgrounds and cultures. By having a more representative dataset, the biases introduced during training can be minimized.\n\nFurthermore, new methods and approaches can be developed to reduce the influence of dataset bias on the models. This involves exploring techniques that ensure fair and unbiased representation across different races, genders, and languages. By striving for diversity and inclusivity in the training data and adopting innovative methods, the models can become more equitable and avoid favoring specific groups or languages.\n\n### Computation\n\nThe success of deep learning models, particularly in the text-to-image domain, relies heavily on having access to a large, labelled dataset. Training these models is resource-intensive, making it feasible primarily for large companies, and the resulting models are often too large to be deployed on efficiency-focused devices like edge devices.\n\n### Evaluation\n\nEvaluating text-to-image models is challenging due to limitations in existing automatic evaluation metrics, subjective nature of human evaluation, and potential biases and limitations in the design of evaluation prompts. More reliable and diverse evaluation criteria are needed to overcome these challenges and obtain a comprehensive understanding of the models' performance.\n\n## Conclusion\n\nThis essay has tried to get all the major technological advancements that have happened so far to the very detail. Since different evaluation methods and benchmarks are used, to generalise it, is very difficult. Even while writing this paper I have asked questions to ChatGPT and Bard for latest sources to understand terms and get more information. The exciting times that we see ahead are truly incredible and I am looking forward to enriching the research with more latest research. \n\n\n\n#### References:\n\nText-to-image Diffusion Models in Generative AI: A Survey: Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, In So Kweon\n\nB. Goertzel and C. Pennachin, Artificial general intelligence. Springer, 2007, vol. 2. \nV. C. M¨ uller and N. Bostrom, “Future progress in artificial intelligence: A survey of expert opinion,” in Fundamental issues of artificial intelligence. Springer, 2016, pp. 555–572.\n \nJ. Clune, “Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence,” arXiv. preprint arXiv:1905.10985, 2019. \n \nR. Fjelland, “Why general artificial intelligence will not be realized,” Humanities and Social Sciences Communications, vol. 7, no. 1, pp. 1–9, 2020.\n\nY. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, 2015.\n\nE. Mansimov, E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Generating images from captions with attention,” ICLR, 2016.\n \nS. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” in International conference on machine learning.\n\nPMLR, 2016, pp. 1060–1069. H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 5907–5915. \n\nMasataka Seo , Rihito Tominaga Image Generation from Text Using StackGAN with Improved Conditional Consistency Regularization 26 December 2022\n\nhttps://www.researchgate.net/figure/The-architecture-of-the-proposed-AttnGAN-Each-attention-model-automatically-retrieves_fig1_321374490\n\nhttps://paperswithcode.com/method/biggan-deep\n\nhttps://www.researchgate.net/figure/Network-architecture-of-StyleGAN2-and-StyleGAN2-ADA-a-the-generator-and-b-the_fig2_351004600\n\nhttps://paperswithcode.com/method/bigbigan\n\nhttps://ljvmiranda921.github.io/notebook/2021/08/08/clip-vqgan/\n\nhttps://sites.research.google/parti/\n\nhttps://ai.googleblog.com/2022/05/vector-quantized-image-modeling-with.html\n\nhttps://iterative-refinement.github.io/\n\nhttps://cascaded-diffusion.github.io/\n\nJacob Menick, Nal Kalchbrenner “Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling” Dec 2018\n\nAli Razavi, Aaron van den Oord, Oriol Vinyals “Generating Diverse High-Fidelity Images with VQ-VAE-2” 2 Jun 2019\n\nAlex Nichol, Prafulla Dhariwal “Improved Denoising Diffusion Probabilistic Models”18 Feb 2021\n\nAlex Nichol, Prafulla Dhariwal “Diffusion Models Beat GANs on Image Synthesis” 1 Jun 2021\n\nhttp://3dgan.csail.mit.edu/\n\nhttps://developer.nvidia.com/blog/enhancing-ai-transparency-and-ethical-considerations-with-model-card/\n\nhttps://www.intertek.com/asset-integrity-management/deepview3d-condition-based-maintenance/\n\nhttps://www.all3dfree.net/3d-model-vehicle-collection-delivery-van.html","metadata":{}}]}