\n[$\\mathit{1.\\ Introduction}$](#2)
\n[$\\mathit{2.\\ The\\ history\\ of\\ Generative\\ AI}$](#3)
\n[$\\mathit{3.\\ Advancements\\ in\\ methods}$](#4)
\n[$\\mathit{4.\\ Advancements\\ in\\ models}$](#5)
\n[$\\mathit{5.\\ Relative\\ notebooks\\ in\\ Kaggle}$](#6)
\n[$\\mathit{6.\\ Trend\\ of\\ development}$](#7)
\n[$\\mathit{7.\\ Advancements\\ in\\ Applications}$](#8)
\n[$\\mathit{8.\\ Meet\\ the\\ challenges\\ \\&\\ problems}$](#9)
\n[$\\mathit{9.\\ Conclusion}$](#10)
\n[$\\mathit{Reference}$](#10)
","metadata":{}},{"cell_type":"markdown","source":"\n\n\n\n\n\n
Abstract
\n\nThe history of Generative AI can be traced back to a very long time ago. After entering the era of Deep Learning, the development of Generative AI has accelerated, especially in the recent 2 years, inventing many models that perform astonishingly well in generating new content. Recently, some large generative models have drawn much public attention for its impressive ability of generating new text and images. The AI Generated Content (AIGC) has almost become a symbol of AI, since its impressive performance can easily be visualized and interpreted. However, the rapid development of Generative AI also brings new challenges to us in various fields. We must confront these problems and try to solve them, if we seek further development.\nIn this essay, I will present a comprehensive and analytical introduction of Generative AI, including the history of Generative AI, advancements in recent years, the applications, etc. Moreover, Some existing challenges will also be discussed. I sincerely hope my essay can help people who want to understand Generative AI and bring them inspiration.
\n \n\n","metadata":{}},{"cell_type":"markdown","source":"\n\n\n\n\n\n$\\scriptsize\\bf{1.\\ Introduction}$
\n\n Differing from other AI tasks, such as classification, the fundamental task of Generative AI is to generate new content. The core component in Generative AI is the Neural Network Model. Having access to larger computational resources and dataset, model such as ChaptGPT[1], DALL·E 2[2] is now capable to create new content that's indistinguishable from those made by human author, according to the instruction given by human, expanding the application of AI in many fields. For instance, DALL·E 2 is a state-of-art model developed by OpenAI[3]. It has performed impressively in generating high-quality images based on text instruction, as shown in Figure 1. People may wonder why the evolution of Generative AI is so fast. There are many factors boosting the development of Generative AI. For example, with the increase of available computational resources and the size of dataset, many ideas are able to be verified and come to implementation thereby evolving the Generative AI. It's totally necessary to discover the factors behind such rapid development, so that we can find out the key to keep Generative AI or AI thriving. And that's what this essay is about.
\n This essay is organized as follows: The introduction is section 1. In section 2 we will discuss the overview of the history of Generative AI from different views. In section 3 we will discuss the advancements in methods in recent years, including generative methods and training methods. In section 4, we will help readers cultivate a comprehensive understanding of methods by analyzing some models. In section 5 some excellent notebooks in Kaggle will be presented. Section 6 provides a discussion about the trend of the development of Generative AI. Applications will be discussed in section 7. We will also discuss some challenges Generative AI is facing in section 8.
![](\"https://pic.imgdb.cn/item/64a65ac41ddac507cc8d22c1.png\")
\n Figure 1: 3 images created by DALL·E 2, given 3 particular text instructions. The left part are the instructions and the right part are the corresponding generated images.
$\\scriptsize\\bf{2.\\ The\\ history\\ of\\ Generative\\ AI}$
\n![](\"https://pic.imgdb.cn/item/647d7cdd1ddac507cc754309.png\")
\n Figure 2: The timeline of the appearance of some crucial models. These models are classified into 3 categories, i.e. NLP, CV and Visual-Language, with colors of green, blue, yellow respectively.
\n
Since the flourishng and rapid development of Generative AI began with Deep Learning getting more popular, the introduction will mainly focus on achievements after 2010. You may notice that many powerful models were proposed in the past 3 years, as shown in Figure 2.
\n As mentioned above, the fundamental task of Generative AI is to generate content. From this perspective, Generative AI has a very long history that can be traced back to the 1950s, when Hidden Markov Models (HMMs)[4] and Gaussian Mixture Models (GMMs)[5] were first proposed. Basically, these are both probability models that can learn a distribution from a known dataset. We can generate new content by sampling from the learned distribution.
\n The HMMs is widely used in Natural Language Processing (NLP). As a matter of fact, a language model is a probability model that uses some words to predict the most possible successive words. We can regard \"predict\" as a kind of generation. Generative AI and NLP are inextricably linked, as \"generate\" is a significant part of NLP (one of the basic tasks of training a NLP model is to use some words in a sentence to predict/generate other words). In that case, one may assert that Generative AI actually originated from NLP. Recurrent Neural Networks (RNNs)[6] is adopted to handle the dependency in long distance. After years, a revolutionary structure was proposed, the Transformer[7]. The successive state-of-art NLP models also adopted this structure, such as BERT[8] and GPT[9]. Their astonishing performance is not only in text generation, but also in other tasks like Q&A and translation.
We may notice that some state-of-art Large Language Models (LLMs) are deemed as a primitive version of Artificial General Intelligence (AGI). The data LLMs handle with is the natural language, which contains more semantic information than image. In that case, the way that LLMs answer complicated question could be an evidence for judging whether the model can \"think\" the way humans do.
\n Things are quite different when it comes to Computer Vision (CV). The Generative Adversarial Network (GAN)[10] is undoubtedly a milestone in the generative computer vision area. Many of its variants were also proposed and achieved a remarkable result. Other methods such as Variational Autoencoders (VAEs)[11] and Diffussion Model (DDPM)[12] were also proposed. All of these have become an important part of the generative task in CV, as shown in Figure 2. Moreover, a training strategy called Masked Image Modeling (MIM)[13] is also proposed for pre-training. Its basic principle is to mask a part of an image, and train the model to generate the masked part as close to the original image as it can. Such a way of pre-training makes the model perform well in downstream tasks after being fine-tuned, proving that the model indeed has learned how to extract useful features while generating content.
![](\"https://pic.imgdb.cn/item/647de9251ddac507cc438d4f.png\")
\n Figure 3: Handwriting numbers generated by GAN and VAE.
There are significant distinctions between the generative model in CV and NLP, due to the difference between image and text data. People will naturally start thinking if they can take the complementary advantages from both CV and NLP model so that they can promote their performance. A milestone of combining two fields is the Vision Transformer[14], which applies the Transformer structure in CV task and achieves a impressive result. In fact, combining CV and NLP is a fraction of a broader field called Multimodal Learning[15]. If we want a model to generate a certain content according to our demand, a very intuitive way is to give the model a textual instruction as an input. That's exactly what the researchers have done. Conditional Generative Adversarial Nets (CGAN)[16] is an extension of the original GAN. Both the generator and the discriminator add additional information as the condition. it can be any type of additional information, such as category information or data from other modals. There are some other text-to-image GAN models such as StackGAN[17], which can generate image in multiple stages. Apart from GAN, models in different structures also perform well, such as VisualBERT[18]. A milestone in text-image model is Contrastive Language-Image Pre-training (CLIP)[19]. It uses text as a supervisory signal to train a transferable vision model. After the training is completed, it was applied to the zero-shot classification task. At the same time, a lot of experiments were conducted to prove that CLIP has good performance in terms of representation learning, robustness, and cognitive learning ability. DALL·E 2[20] combines DDPM and CLIP. It can generate original images according to the given concept, characteristics and style. In addition, DALL·E 2 can also modify existing pictures according to the description, such as removing or adding an object, and taking shadows, reflections, and textures into account. Also, even if no language description is given, DALL·E 2 still can generate a series of new pictures with similar styles based on existing pictures. Its mighty power has astonished everyone.
\n Beyond the Visual-Language model, there are still many cross-modal models that are proposed in recent years. We will discuss them in another section.
\n Now we have roughly reviewed the history of Generative AI. If you are interested in any specific model mentioned above, you may read the original papers for more information. However, we still need to dive deeper to unveil the advancement hiding behind these successful models.
$\\scriptsize\\bf{3.\\ Advancements\\ in\\ methods}$
\n If we analyze some generative models, we will find that the feature-extraction part in these models doesn't differ very much from each other. Some of the feature-extraction blocks such as Transformer have been proved to be very effective. What really makes a generative model to perform so well should be the advancement in method. \n If you want to interpret Generative Model mathematically, you can start with the Probabilistic Distribution. In that point of view, all data follows an unknown probabilistic distribution and training a model is actually a process of approximating the distribution of real data. Once the model has learned a distribution that's close to the real distribution, we can sample from this distribution to create new content. In statistics, learning a distribution from a given data is also known as Maximum Likelihood Estimation. Generally speaking, learning a distribution is what all various generative AI models have in common, no matter how distinctive the downstream tasks of these models are. To better demonstrate these advancements, I will introduce some significant methods, starting from VAE to DM. \n
Some people may regard GAN or VAE as a common building block in a generative model. Their structure is usually quite simple, components such as decoder or encoder can be easily implemented. However, if we want to interpret generative models comprehensively, we need to broaden our horizons, penetrating the surface to reach the mathematical theory behind these \"building blocks\", i.e. the method.
These methods refer to the mathematical method behind generative models or the specifically designed training skills that make the model easier to converge or less possible to overfit, and to perform better. Personally, I believe that the advancement in methods is more prominent and worth discussing than other things like using larger dataset or larger models. It's relatively trivial using a larger model to get a better performance. Advancements in method, on the other hand, are more exquisite. I think discussing them could bring some inspiration. Notice that some methods such as GAN were actually proposed many years ago, but they have been adopted in some models proposed recently. Hence, discussing no-recently-proposed methods is necessary. To make it easier to read, rigorous mathematical formulas and inference parts will not be included in this section, only some simple notations will be presented. However, some very basic knowledge about probability and statistics are needed. 3.1 Advancements in generative method
\n$$\\large\\boldsymbol{VAE}$$
\n
![](\"https://pic.imgdb.cn/item/648561961ddac507cc4df182.png\")
\n Figure 4: Structure of VAE.
Variational Autoencoder (VAE), shown in Figure 4, is a milestone among generative models. It consists of 2 components, a encoder and a decoder, which are both multi-layer neural networks. Given some samples $X$, which follow a distribution $p(X)$. Instead of learning $p(X)$ directly, VAE introduces a latent variable $Z$ to learn the distribution by the formula $p(X)=\\sum\\limits_{z} p(X|Z)p(Z)$. $p(X|Z)$ is called Posterior Distribution and $p(Z)$ is called Prior Distribution. However, this is very resource-consuming, since we need to sample $Z$ many times to estimate $p(X)$. Consequently, another distribution $p(Z|X)$ is introduced, from which $Z$ is sampled, making it easier to estimate $p(X)$. \n
The encoder outputs the mean and variance of $p(Z|X)$, which will be used to compute the loss, The closer $p(Z|X)$ to a standard normal distribution $N(0, I)$ is, the smaller loss will be. The decoder plays a role of mapping the $Z$ from standard normal distribution to $p(X)$. Given a latent variable $Z$, a trained generator will output an image.$$\\large\\boldsymbol{GAN}$$
\n
![](\"https://pic.imgdb.cn/item/648899b71ddac507cc22d936.png\")
\n Figure 5: General structure of VAE.
Generative Adversarial Network (GAN) composes of 2 parts, a discriminator $D$ and a generator $G$. The main task is still to learn $p(X)$. The generator $G$ takes $Z$ sampled from a prior distribution as input. It needs to learn a distribution $p(X; \\theta)$ that will approach to the real data distribution $p(X)$ during training. The discriminator $D$ needs to distinguish genuine samples from generative samples. During the training the discriminator will try to distinguish the real samples and generative samples more and more accurately, yet the generator will generate samples that are more and more harder for the discriminator to distinguish, that is exactly where the \"adversarial\" part is. Finally, the generated image $G(Z)$ and real image $X$ will be so similar to each other that the discriminator $D$ can't tell which one is authentic. \n Autoregressive mosel (AR) is relatively easy to understand. It has been widely applied in language model. It uses the samples generated previously to generate new samples, i.e. $p(X) = \\prod_{t=0}^N p(X_{t}|X_{t-1}, X_{t-2},...,X_0)$ AR is more suitable for sequential data, that's what makes it so successful in the language model. \n
Naturally, GAN and VAE have something in common. They both learn a transformation from $p(Z)$ to $p(X)$, whereas GAN didn't explicitly learn $p(Z)$.$$\\large\\boldsymbol{AR}$$
\n$$\\large\\boldsymbol{DM}$$
\n
![](\"https://pic.imgdb.cn/item/648e92f31ddac507cc523ac6.png\")
\n Figure 6: General process of DM.
Since VAE, GAN and AR were proposed, the advancements in method mainly consisted of variants or mixtures of them, until the Diffusion Model (DM) was proposed. DM is one of the most significant advancements in method in recent years. DM is quite similar to VAE, as they both need to confine the distribution of latent variable $p(Z)$ in $N(0, I)$. However, the DM uses Markov Chain in this process. \n Compared with traditional techniques, recent advancements in training techniques have evidently improved the performance of models. When training a generative model, not only do we need to tackle the traditional problems such as overfitting, we also need to think how to make the model generate what we need as much as possible. Consequently, I will introduce some crucial techniques in the next paragraphs. \n Reinforcement Learning in Human Feedback (RLHF) is widely used in Large Language Model (LLM). RLHF includes three steps: pre-training, reward learning, and fine-tuning with reinforcement learning. First, we need to pretrained a language model on a large datasets. In the second step we train a reward model to encode the diversified and complex human preference. Specifically, given the same prompt x, different generated answers $\\{y_1, y_2,...,y_n\\}$ are evaluated by humans in a pairwise manner. The pairwise comparison relationships are later transferred to pointwise reward scalars, $\\{r_1, r_2,...,r_n\\}$, using an algorithm such as ELO[21]. In the final step, the language model is fine-tuned to maximize the learned reward function using reinforcement learning. RLHF has been applied to fine-tune models in various applications such as ChatGPT. The detailed steps are described below. \n 1) Pretrained a language model \n
In the forward process of DM, given a input $X_0$, at moment $t-1$, a noise $Z$ that follows $N(0, I)$ is added to $X_{t-1}$ by weight. Consequently, we acquire the $X_{t}$ by $X_{t} = \\sqrt{a_{t}}X_{t-1} + \\sqrt{1 - a_{t}}X_{t-1}$, $a_{t}$ is the weight, which will decrease during iteration. Finally, if we iterate it enough times, the $X_{t}$ will be very close to $N(0, I)$, just like $Z$ in VAE.
The next step, backward process, is to restore the image $X_{0}$ from $X_{t}$. In forward process, the distribution of $X$ at any moment depends on the previous one, which is $q(X_{t}|X_{t-1})$. We can use that to compute $p(X_{t-1}|X_{t})$ via The Bayes Formula, i.e. $p(X_{t-1}|X_{t}) = q(X_{t}|X_{t-1}) \\frac{q(X_{t-1}) }{q(X_{t})}$. After training the DM, we can sample a noise from $N(0, I)$, then generate an image through iteration.
All of these generative methods share the same task: learning a distribution and making it closer to the real data distribution. 3.2 Advancements in training method
\n$$\\large\\boldsymbol{Reinforcement\\ Learning\\ in\\ Human\\ Feedback}$$
\n
![](\"https://pic.imgdb.cn/item/64a444e01ddac507ccf330bc.png\")
\n Figure 7: Process of pretraining a language model. Source: https://huggingface.co/blog/rlhf
First, we train a language model using the typical pre-trained task. Here, this language model can be fine-tuned with additional text or conditions, for example, OpenAI fine-tuned the \"preferable\" artificially generated text, while Anthropic distilled on context clues according to the \"useful, honest and harmless\" standard out of the original language model. Expensive augmentation data may be used here, but it is not a necessary step for RLHF. Since RLHF is still an unexplored field, there is no clear answer to \"which model\" is suitable as a starting point for RLHF. \n 2) Training reward model \n
![](\"https://pic.imgdb.cn/item/64a446361ddac507ccf5bef8.png\")
\n Figure 8: Process of training a reward model. Source: https://huggingface.co/blog/rlhf
The reward model (RM) can be another fine-tuned language model (LM), or an LM trained from scratch on preference data. For example, Anthropic proposes a special pre-training method, which uses Preference Model Pretraining (PMP) to replace the fine-tuning process after general pre-training. \n 3) Fine-tune via reinforcement learning \n
With respect to the training text, the RM's hint-to-generate pairs are sampled from a predefined dataset, and the initial LM is used to generate text for these hints. Anthropic's data was primarily generated via a chat tool on Amazon Mechanical Turk and made available on the Hub, while OpenAI used user-submitted prompts to the GPT API.
Regarding the value of training rewards, it is necessary to manually rank the answers generated by LM. At first we might think that RM should be trained directly on text annotation scores, but these scores are uncalibrated and full of noise due to the different values of the annotators. Ranking allows you to compare the output of multiple models and build better canonical datasets.
For specific ranking methods, a successful approach is to compare the output of different LMs given the same prompt, and then use the Elo system to build a complete ranking. These different ranking results will be normalized to a scalar reward value for training.
![](\"https://pic.imgdb.cn/item/64a44d661ddac507cc05233f.png\")
\n Figure 9: Process of fine-tunning with RL. Source: https://huggingface.co/blog/rlhf
First, let's make some definitions. Policy is an LM that takes a prompt and returns a sequence of texts (or a probability distribution of texts). The action space of the policy is all tokens corresponding to the vocabulary of the LM, and the observation space is the possible sequence of input tokens. The reward function is a combination of a preference model and a policy shift constraint. \n
The specific calculation of the reward function determined by the PPO algorithm is as follows: input the prompt $x$ into the initial LM and the current fine-tuned LM to obtain the output text $y_{1}, y_{2}$ respectively, and pass the text from the current policy to the RM to obtain a scalar reward $r_{\\theta}$. Comparing the generated text of the two models to calculate the penalty term of the difference, designed in several papers as the scaling of the Kullback–Leibler (KL) divergence between output word distribution sequences, namely $r=r_{\\theta}-\\lambda r_{KL}$. This term is used to penalize the RL policy for generating large deviations from the initial model in each training batch to ensure that the model outputs reasonably coherent text.
Finally, according to the PPO algorithm, we optimize according to the reward index of the current batch of data (from the on-policy characteristics of the PPO algorithm). The PPO algorithm is a Trust Region Optimization (TRO) algorithm that uses gradient constraints to ensure that the update step does not destabilize the learning process.$$\\large\\boldsymbol{Contrastive\\ Learning\\ in\\ Multimodal\\ Model}$$
\n
![](\"https://pic.imgdb.cn/item/64a455081ddac507cc137a2f.png\")
\n Figure 10: Contrastive learning in CLIP. Source: https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-mini--Vmlldzo4NjIxODA
In multimodal learning, samples from different modals will usually be mapped into a laten space where each of them can be represented by a laten variable. These latent variables should be as close as possible to their corresponding variables during training, meanwhile they should get away from those variables that doesn't correspond with them. This kind of training method is called Contrastive Learning. It has been extensively adopted in Multimodal Generative Learning. \n\n","metadata":{}},{"cell_type":"markdown","source":"\n After discussing all kinds of methods in the previous section, let's see how they are implemented in some recently proposed models. In this section I will demonstrate the advancements in generative models in recent years, and I will briefly analyze some of the representative models, since models are also an important part in the advancements in Generative AI. \n There are many generative models proposed in the past few years. These models were developed by different companies, and were designed for different tasks. Moreover, some of them are cross-modal. Due to simplicity I won't introduce each of them, but I will demonstrate a timeline of the birth of Large Language Models instead. \n
CLIP is very good example of using contrastive learning. Suppose a batch in the dataset contains $N$ text-image pairs. The $N$ texts are first encoded by the Text Encoder. Assuming that the Text Encoder encodes each text into a one-dimensional vector with a length of $d_{t}$, then the text data of this batch is output by the Text Encoder as $[T_{1}, T_{2}, ..., T_{N}]$, the dimension is $(N, d_{t})$ ; Similarly, the $N$ images are encoded by the Image Encoder first, assuming that the Image Encoder encodes each text into a one-dimensional vector whose length is $d_{i}$, then The image data of this batch is output by Image Encoder as, $[I_{1}, I_{2}, ..., I_{N}]$, the dimension is $(N, d_{i})$.
In $[T_{1}, T_{2}, ..., T_{N}]$ and $[I_{1}, I_{2}, ..., I_{N}]$, the text and the images are in one-to-one correspondence. For example, $T_{1}$ corresponds to $I_{1}$, $T_{2}$ corresponds to $I_{2}$, etc. We call the $N$ corresponding text-image pairs as positive sample, and we mark the text-image pairs that does not originally correspond as negative sample, for example, $T_{1}$ does not correspond to $T_{2}$. In this way, we have $N$ positive samples and $N^2 - N$ negative samples. Positive and negative samples can be used as positive and negative labels to train Text Encoder and Image Encoder.
We calculate the cosine similarity between $I_{i}$ and $T_{j}$, and use it to measure the corresponding relationship between the corresponding text and the image. The larger the cosine similarity, the stronger the corresponding relationship between $I_{i}$ and $T_{j}$, and vice versa. Now, the task of training becomes very clear: Training the Text Encoder and Image Encoder, maximize the cosine similarity of $N$ positive samples, and minimize the cosine similarity of $N^2 - N$ negative samples.
So far we have reviewed the advancements in methods in recent years. There are still other methods that are relatively less used in models proposed in recent years, so I didn't introduce them. You can find one or more kinds of methods above in almost all generative models. As a matter of fact, some methods are not especially designed for generative models, or they are only suitable in one modal of data. For instance, RLHF can hardly be transferred into training a vision model. However, due to the strong trend of multimodal learning in Generative AI, many methods that seem unlikely for training generative models could be potentially used.$\\scriptsize\\bf{4.\\ Advancements\\ in\\ models}$
\n$$\\large\\boldsymbol{Generative\\ models\\ proposed\\ in\\ recent\\ years}$$
\n
![](\"https://pic.imgdb.cn/item/64b029a31ddac507ccd85be0.png\")
\n Figure 11: Timeline of the development of Large Language Model. Source: https://arxiv.org/abs/2303.18223
$$\\large\\boldsymbol{DALL·E\\ 2}$$
\n![](\"https://pic.imgdb.cn/item/64a59b441ddac507cc64d463.png\")
\n Figure 11: General structure of DALL·E 2. Source: https://arxiv.org/abs/2204.06125
DALL·E 2 is a state-of-art text-to-image model, whose impressive performance has been demonstrated in the introduction section. DALL·E 2 mainly consists of three parts: the CLIP, the prior and the image decoder. These components are trained separately and stacked together to build DALL·E 2. \n
1) CLIP
The details about CLIP have been discussed in the last section. CLIP in DALL·E 2 has nothing different from that. The main task of CLIP is to encode text and image into a feature space. Both text encoder and image encoder will be used in training other parts.
![](\"https://pic.imgdb.cn/item/64a654731ddac507cc801652.png\")
\n Figure 12: Training process of the prior. Source: https://arxiv.org/abs/2204.06125
2) Prior \n
The prior part is actually a diffusion model that transforms the embedding vector of text into a latent vector, which will be used as the input of the decoder. The embedding vector of text is the output of the text encoder in CLIP. During training, the parameters of the prior will be updated so the output of the prior will be closer to the output of the image encoder in CLIP.
3) Image decoder
The decoder is another diffusion model that uses the output of the prior as a condition and generates images we need.
After training the three components, we are able to construct a complete DALL·E 2. We also need to discard the image encoder in CLIP and keep the text encoder in CLIP, the trained prior and the image decoder. In this way, the process of generating a new image is very clear: the text is encoded by the text encoder, then the embedding of text is converted into the embedding of image by the prior, and finally the image is decoded by the decoder.$$\\large\\boldsymbol{InstructGPT}$$
\n
![](\"https://pic.imgdb.cn/item/64a6e3c01ddac507ccef737e.png\")
\n Figure 13: Model structure of GPT series, wherein Trm stands for Transformer block. Source: https://167.114.98.70/model-structure-of-bert-and-gpt-download-scientific-diagram/
GTP-1, GPT-2, GPT-3, InstructGPT[22] share the same core structure, the transformer, as shown in Figure 13. The difference between each of them is the number of layers of the model and the hyperparameters. \n
As you can see, there are hardly advancements in model structure. The reason why InstructGPT is so successful is that the RLHF, which we've discussed before, is adopted in the training of InstructGPT. Moreover, RLHF is also adopted in the training process of ChatGPT.
There are still some problems. Indication is the only clue for the model to generate output, and the model relies heavily on it. How to improve the generalization ability of the model to the instruction and the error correction ability of the error indication is a very important task to improve the model experience. This not only allows the model to have a wider range of application scenarios, but also makes the model more \"intelligent\". Besides, we may need to design a more reasonable way of using human feedback, or a more cutting-edge model structure. Many problems of InstrcutGPT can be solved by providing more labeled data, but this will lead to more serious performance degradation of general NLP tasks.$$\\large\\boldsymbol{StyleGAN}$$
\n
![](\"https://pic.imgdb.cn/item/64aae2f51ddac507cc39e944.png\")
\n Figure 14: Structure of StyleGAN. Source: https://arxiv.org/abs/1812.04948
StyleGAN[23] consists of two main parts: mapping network and synthesis network. In general, the structure of StyleGAN is in coherence with the GAN frame. \n\n\n","metadata":{}},{"cell_type":"markdown","source":"\n Kaggle, as the world’s largest data science community, has many users who have a strong interest in AI and data science. Search on kaggle with \"generative\" as the keyword, you will find a lot of works related to Generative AI. I will present some excellent notebooks with codes related to this essay in the table below. The code provided by these notebooks may help you understand generative models in a more practical way, and create your own model. \n Beside focusing on specific advancements, it is also vital to focus on the trend of development which I will analyze from different angles in this section. Knowing the trend of the development of Generative AI can help us understand the progress made in recent years from a more general perspective. \n A model with more parameters generally performs better than thoes with less parameters, in most circumstances. It may explain the increase number of parameters in models proposed recently. Larger model requires more computational resources. This paper[25] demonstrates the increase of parameters and training speeds of some large models. \n
1)Mapping network
Unlike the previous generator that directly uses the latent code in the $Z$ space as the starting point for generating images, StyleGAN uses a Mapping Network composed of 8 FC layers to map the latent vector in the $Z$ space to a new space, which is called $W$ space (both $Z$ and $W$ are 512-dimensional vector spaces). Moreover, the latent code in the $W$ space is not used as the feature map of the generated image, but is used to control the backbone to generate the network feature map, thereby indirectly controlling the characteristics of the output image.
2)Synthesis network
Next comes the actual generative network. The network is divided into different blocks, and these blocks are some upsampling and convolution kernels or AdaIN[24] operations. Notice that in StyleGAN, hidden variables are not used as the initial input of the generation network, the initial input is a learnable constant matrix, and hidden variables are input to each AdaIN.
The generator network has two inputs in the forward process, one is the learnable parameter initial constant, and the other is the output of the mapping network. The initial constant and $w$ will be input into the generator block, then the final output feature map is used as input to enter the next generator block, and at the same time, it will be input to the ToRGB layer to output an RGB image and accumulate it on the historical accumulated RGB image. After the feature map enters the generator, it is fused together through a custom convolution layer and a demodulated style vector. The style vector is converted from the vector of the $W$ space by the custom fully connected layer. The fused feature map plus bias and noise output. The final accumulated RGB image is output as a generated image.
As for the discriminator, it first converts the image into a feature matrix with a structure similar to the mapping network, then reduces the feature dimension by downsampling n blocks, then finds the standard deviation in a mini-batch, and finally passes a custom convolution layer and fully connected layer to output the final image classification result.
The original paper also discovered the problem of disentanglement. Being able to control features of the generated image is another amazing part of this article. For simplicity I won't discuss that in this essay. If you are interested in it, you may review the original paper.$\\scriptsize\\bf{5.\\ Relative\\ notebooks\\ in\\ Kaggle}$
\n\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"\n\n\n\n\n\n\n \nName of Notebook \n Author \n Link \n\n \nVariantional Autoencoders(VAE) \n fazilbtopal \n https://www.kaggle.com/code/fazilbtopal/variantional-autoencoders-vae \n\n \nGAN Introduction \n jesucristo \n https://www.kaggle.com/code/jesucristo/gan-introduction \n\n \nDeep Learning For NLP: Zero To Transformers & BERT \n tanulsingh077 \n https://www.kaggle.com/code/tanulsingh077/deep-learning-for-nlp-zero-to-transformers-bert \n\n \nStable-Diffusion 🎨 | Custom-pipeline \n danushkumarv \n https://www.kaggle.com/code/danushkumarv/stable-diffusion-custom-pipeline \n\n \n[PART 1] TF/Keras - Train Custom CLIP-like on COCO \n dschettler8845 \n https://www.kaggle.com/code/dschettler8845/part-1-tf-keras-train-custom-clip-like-on-coco \n\n \nCreate your own ChatGPT \n amirmotefaker \n https://www.kaggle.com/code/amirmotefaker/create-your-own-chatgpt \n\n \nAnime with StyleGAN \n basu369victor \n https://www.kaggle.com/code/basu369victor/anime-with-stylegan \n\n \nKeras VQ-VAE for image generation \n ameroyer \n https://www.kaggle.com/code/ameroyer/keras-vq-vae-for-image-generation \n$\\scriptsize\\bf{6.\\ Trend\\ of\\ development}$
\n
In this section I will discuss the trend of development from 3 perspectives, size of model, size of dataset and multimodal learning, which I believe can comprehensively depict the development of Generative AI in recent years. More parameters and more resource-consuming
\n
![](\"https://pic.imgdb.cn/item/649709711ddac507cc0bcc63.png\")
\n Figure 15: Statistics of model size and training speed across different models and computing devices. Source: https://arxiv.org/abs/2303.04226
Another thing substantiating that it's very resource-consuming is the monetary cost of training. We will discuss the details about it later. \n If the number of parameters increase, it is wise to use larger dataset for training, which can alleviate overfit. These large datasets usually hundreds of milions of samples, with size of hundreds of GBs, even more than 1 TB. Table below shows the size of some datasets that used for training large models. \n Larger datasets
\n
![](\"https://pic.imgdb.cn/item/649866dc1ddac507cce2a6a2.png\")
\n Table 1: Size of different datasets used for training large model.
Notice that some datasets are not completely public, so the precise size of these dataset will not be presented in Table 1, yet the estimated size will be presented instead. \n As I have discussed above, a powerful generative model should not only generate high-quality content, it also needs to generate content according to our demands. To achieve that, one may need to use some \"prior knowledge\" to guide the model to generate content that is coherent with one's demand. Under most circumstances, the \"prior knowledge\" and the target content is usally in different modal. Consequently, it's very natural to develop a generative multimodal model. \n Multimodal learning
\n
There are already many multimodal generative models that were proposed in recent years. According to this paper[26], these models can be roughly categorized into 9 domains, as shown in Figure 11.
![](\"https://pic.imgdb.cn/item/64999b551ddac507cc85151b.png\")
\n Figure 16: A taxonomy of the most popular generative AI models that have recently appeared classified according to their input and generated formats. Source: https://arxiv.org/abs/2301.04655
We may notice that many cross-modal models use text input as \"prior knowledge\". This process resembles to real human behaviour. For instance, suppose we want an artist to draw a picture, we first need to tell the artist what picture do we expect. Telling your demands to the artist is like feeding the model with a \"prior knowledge\", text is undoutedly the most efficient way among all. \n\n","metadata":{}},{"cell_type":"markdown","source":"\n It's not hard to realize that Generative AI could have a wide application in many fields. At present, the global capital market is focusing on the field of generative AI, and many star projects have emerged, including OpenAI, Cohere[27], Runway[28], etc. At present, the valuation of OpenAI has reached 29 billion U.S. dollars, and companies such as ADEPT[29], Character.ai[30], Glean[31] and other companies with the lowest valuation have also reached 10 billion U.S. dollars. \n
Based on these trends, one may conceive that the future state-of-art model will be a model with a large number of parameters, handling multimodal data. However, these are the trends of development in the past few years, it may not be very accurate to describe the future. There could be some revolutionary breakthroughs in the future that totally change the Generative AI. Intuitively, I don't believe that increasing the size of the model could be an effective way to get a more powerful model, if we are also seeking a chance of applications. For example, How can one implement a very powerful model with 100 billion parameters on an small embedded system? Very large model undoubtedly has limitations in application.
The dependency of a large dataset could also lead to limitations. Now we use a large dataset that contains millions of text-image pairs to train a powerful model. Suppose we need to develop a multimodal model which can deal with medical images. Can we find or build such a large dataset for it? As we all know, due to many factors such as the privacy of patients, collecting medical images is very hard, not to mention collecting millions of them. Many experiments have demonstrated generative models trained on small dataset can't perform as well as they were trained on large dataset. If we really want to use generative models in the medical domain, getting rid of the dependency of a large dataset, or using dataset more effectively will be a very crucial problem.$\\scriptsize\\bf{7.\\ Advancements\\ in\\ applications}$
\n
![](\"https://pic.imgdb.cn/item/64a3b0d81ddac507ccb47170.png\")
\n Figure 17: 11 most highly valued generative AI companies. Source: https://www.cbinsights.com/research/generative-ai-funding-top-startups-investors/
The application fields of generative AI continue to expand in dialogue, programming, drawing, video production and others. In addition to OpenAI, where the GPT series were originated, Replit, which focuses on the collaborative programming track, Stability.ai in the field of drawing generation, and Runway in the field of video generation, have all been implemented in various scenarios. The products of these companies have a common feature that it can recognize natural language and output the results of the corresponding modality. \n
Instead of merely talking about the general concept of any applications, I will present some examples of applications in a real scenario. I believe that will help readers better conceive how Generative AI could potentially change the world. Generative AI in Autonomous Driving
\n
![](\"https://pic.imgdb.cn/item/64ad84921ddac507ccadfe06.gif\")
\n Figure 18: Video generated by GAIA-1 shows a realistic driving sequence. Source: https://youtu.be/cwJ_TV9Daqo
At the CVPR 2023 autonomous drving workshop, both Tesla and Wayve mentioned their latest exploration direction in using large generative models, that is, using large generative models to generate continuous video scenes related to autonomous driving, which Wayve named GAIA-1. Tesla, on the other hand, named its attempt World Model[32]. Obviously, large language model has achieved great success and detonated the AI industry, the field of computer vision is looking forward to the moment of similar breakthroughs. Technology companies at the forefront of industry technology have given their answer to this question, which is the generative World Model. \n LLM has shown potential abilities to perform natural language processing tasks in different domains, stimulating great interest in finance. Accessing high-quality financial data is the first challenge for LLM for finance. \n
In general, World Model relies on a large amount of live video data, using a generative model to generate scenes and compares them with real data, so as to build the loss function for training, so that the model can be trained without relying on label information. The most exquisite part of it is that in order to successfully predict the scene in the future, you must have a deep understanding of the semantic information of the scene at the present moment and the laws of world evolution. By predicting plausible future scenarios as accurately as possible, the model must learn a lot of complex information. In fact, it is precisely because human beings have been dealing with such actual scenes continuously since birth that they have such a reasonable ability to deduce the development of the future world, and this ability is actually the cornerstone of intelligence. Therefore, by requiring the model to continuously predict the next An image of what is happening next step, and comparing it to what actually happened, may be a great progress in autonomous driving, even could be the begining of AGI. Generative AI in Finance
\n
![](\"https://pic.imgdb.cn/item/64ad8cf21ddac507ccc8c3bf.png\")
\n Figure 19: The framework of FinGPT. Source: https://arxiv.org/abs/2306.06031
FinGPT[33] is a LLM specifically designed for finance. FinGPT takes a data-centric approach and provides researchers and practitioners with accessible and transparent resources to develop their FinLLMs. Through a collaborative effort within the open-source AI4Finance community, FinGPT aims to stimulate innovation, democratize FinLLM, and unlock new opportunities for open finance. \n
FinGPT may find widespread application in financial services, helping professionals and individuals make informed financial decisions. Possible applications include:
1. Robo-advisors: Provide personalized financial advice, reducing the need for regular face-to-face consultations.
2. Quantitative Trading: Generate trading signals to make informed trading decisions.
3. Portfolio Optimization: Use numerous economic indicators and investor profiles to construct optimal portfolios.
4. Financial Sentiment Analysis: Assess sentiment on different financial platforms for in-depth investment guidance.
5. Risk Management: Develop effective risk strategies by analyzing various risk factors.
6. Financial Fraud Detection: Identify possible fraudulent transaction patterns to enhance financial security.
7. Credit Scoring: Predict credit worthiness based on financial data to aid in lending decisions.
8. Bankruptcy Prediction: Predict the possible bankruptcy or bankruptcy of a company based on financial and market data.
9. Mergers and Acquisitions (M&A) Forecast: Help investors predict market movements by analyzing financial data and company files
to predict possible mergers and acquisitions.
10. Low-code development: Facilitates software creation with a user-friendly interface that reduces reliance on traditional
programming.
Overall, the integration of LLMs with the financial industry presents unique complexities and vast opportunities. In the face of high timeliness in financial data, dynamic financial environment and other challenges, we need effective solutions, and FinGPT is a good example. Generative AI in Law
\n
![](\"https://pic.imgdb.cn/item/64aed3461ddac507ccc1cfde.png\")
\n Figure 20: The framework of ChatLaw. Source: https://arxiv.org/abs/2306.16092
ChatLaw[34] is released by a research team of Peking University and is committed to providing inclusive legal services. On the one hand, there is currently a shortage of practicing lawyers in China. On the other hand, ordinary people have limited understanding of legal knowledge and provisions, and cannot use legal weapons to protect themselves. The recent rise of big language models just provides an excellent opportunity for ordinary people to consult legally-related issues in a conversational way. \n\n","metadata":{}},{"cell_type":"markdown","source":"\n So far we have discussed the history of Generative AI, the advancements in methods and models, the trend of development, we can see that so many achievements have been made in recent years. Nevertheless, nothing can go well all the time without encountering an obstacle, that works for Generative AI too. There are still many problems that could hinder, even damage the development of Generative AI. Thus, we need to confront them. Moreover, taking up these challenges could even boost the development of Generative AI, which is another reason why we need to meet them. In the paragraph below some questions and challenges will be presented, as they are very crucial and worth discussing. \n Large model usually has a large number of parameters, and is usually trained on a large dataset, which indicates that it has a very high requirement of hardware and energy to train a large model. Training model is a very crucial part of Generative AI. We have roughly discussed this issue in the previous paragraph, and we are about to discuss it more specifically. \n
ChatLaw's training data is mainly composed of forums, news, laws, judicial interpretations, legal consultation, law examination questions, and judgment documents, and then the dialogue data is constructed after cleaning and data enhancement. At the same time, by cooperating with Peking University School of Transnational Law and well-known legal agency in the industry, the research team of ChatLaw can ensure that the dataset can be updated in a time to ensure the professionalism and reliability.
According to the demo, ChatLaw supports users to upload legal materials such as files and recordings, helps them summarize and analyze, and generates visual maps, charts, etc. In addition, ChatLaw can generate legal advice, legal documents based on facts.
Examples demonstrated above indicate wild application of Generative AI. We can also see that Generative AI is not completely reliable in some sophisticated scenario, a human supervisor could be needed. Though, at its current state, Generative AI is certainly capable of helping people in trivial works.$\\scriptsize\\bf{8.\\ Meet\\ the\\ challenges\\ \\&\\ problems}$
\n 1. Training model could be unaffordable.
\n
![](\"https://pic.imgdb.cn/item/64a3b1d01ddac507ccb6422c.png\")
\n Table 2: Cost of training of different large models
As the data demonstrated in Table 2, training a large model usually costs millions of dollars. It may be a drop in the ocean for tech giants, but it's most part of the ocean for some small companies, faculties, institutions. Most of the companies will usually get funds from investors to help them develop models that are designed for specific tasks. The investors expect the model to have some potential commercial values, so they can benefit from it. That may give account for why large generative models thrived in recent years, even though the cost of training keeps increasing. \n Applications of Generative AI in various fields, such as finance, painting, dialogue, programming, etc, have increased tremendously. Office softwares such as Microsoft office, Photoshop uses Generative AI as a plug-in to help their users work more efficiently. However, the application of Generative AI is still limited, due to public worries about the potential risks of using Generative AI and the incompetence of Generative AI in many fields. \n There are many concerns about Generative AI. Due to the overwhelming capability of large generative models, it has become harder, sometimes even impossible to distinguish generated contents from real ones. People are astounded, wondering if a powerful generative AI could be an intelligent being like the Terminator, and annihilate human civilization. Some people suspect that such a powerful model could be used for criminal purposes, like fraud or even worse, terrorism. \n Artificial General Intelligence (AGI) seems to be the ultimate purpose of the development of AI. Since large language models (LLMs) such as ChatGPT were proposed, their strong ability of reacting to complicated instruction has made people wonder if they are qualified to be called as AGI. \n
However, commercial activities normally aim for profit. In that case, cutting down the training cost is certainly profitable, and it also means more money could be used for other parts of research. In other words, when the training cost increases higher than a certain level, profit will reduce rapidly. If the generative model fails to satisfy the expectations of investors, bringing them with more profit, capitals will soon be transferred somewhere else, hence the research about Generative AI will consequently be harder to proceed. Under such circumstances, I suspect most of the non-profit institutions can afford the cost of training. as training is an important part of research. If the cost of training keeps increasing, it will soon be unaffordable or non-profitable, eventually the development of generative AI will be suspended. Moreover, models relied on time-sensitive data, such as financial data, usually need to be retrained or fine-tuned frequently. If training models are unaffordable, Generative AI could never be used in certain fields.
Fortunately, some effective methods that tackles this problem have been proposed, such as Colossal-AI [35], Deepspeed[36], etc. Moreover, training on cloud platform can also reduce the cost. These methods can relatively reduce the cost of training, yet it will still be very expensive to train an extremly large model. Now the development of hardwares, the integrated circuit, has closed to the physical limitation. We should start looking for a new way to make the model more powerful instead of merely increasing the size of the model or the dataset. 2. Furthur applications
\n
As people are worrying about the potential risks of using it, about which we will discuss in the next paragraph. Many countries are discreet about the usage of Generative AI. For instance, ChatGPT is not officially allowed to be used in many countries, such as China, etc, wherein billions of people could have worked more efficiently and easily via ChatGPT and other models. Generative AI could certainly increase productivity, thus creating a better world.
Due to its inexplainability and uncontrollability in the generation of new content, the generative model is incompetent in some tasks, especially in fields where a tiny mistake could lead to catastrophic consequences, such as the medical field.
Nonetheless, current Generative AI is capable of helping many people by reducing unnecessary and trivial labor. We should not only focus on how to make progress in the model, but also need to expand the applications of Generative AI and build it into social activities so that more people can benefit from it. 3. Security & ethical problems
\n
Such a hypothesis is not completely ludicrous. There is already much evidence that confirms people's worry. For instance, in this news[37], Mr.Guo was defrauded of 4.3 million RMB within 10 minutes during a video chat with a \"friend\", who is actually pretented by a criminal with AI Face Swap. Society's reaction to new techs is always not in time. Currently, it's hard to technically prevent such fraud from happening. Besides, it could take legislators years to fill up the legal loopholes, and to constrain the usage of Generative AI.
When using generative models, especially LLMs, the hidden danger of data leakage is becoming more and more prominent. According to a survey by cybersecurity company Cyberhaven, at least 4% of employees enter corporate sensitive data into ChatGPT, and sensitive data accounts for 11% of the input content. It seems that the proportion is not high, but Cyberhaven's statistics show that 0.9% of employees caused 80% of the data outflow incidents, and the user base of hundreds of millions means that 11% of the input content is still a very large figure. The data outflow problem caused by ChatGPT Can not be neglected.
Another issue is that ChatGPT may have gender, race, cultural and other biases, in some cases it will produce wrong answers or language bias. A considerable part of its training data comes from articles, reports and reviews from the internet. Once plenty of people have presented enough prejudice against something in their articles, the language model will learn that prejudice and regard it as the mainstream view.
As for the problem that if Generative AI could end human civilization, it's quite unlikely for now. Though a large language model can answer questions logically, its ability in causal inference is still questionable. Many experiments have been conducted to discover its ability in causal inference, in this paper[38], experiments show that pure causal inference is a very challenging task for all LLMs. Among them, BART MNLI[39] has the highest F1 value of 33.38%, even higher than GPT-4 (29.08%). It is worth noting that many models perform worse than random guessing, meaning they fail entirely on purely causal inference tasks. Because of its disability in causal inference, it's incompetent in some sophisticated tasks. Anyway, current models don't have the ability to destroy humans. 4. Beginning of AGI?
\n
In fact, there is a dispute about this topic. On March 22, 2023, MSR published an article[40] on arXiv, which comprehensively evaluated GPT-4. The core point of the paper is that \"Given the breadth and depth of GPT-4's capabilities, we believe it should reasonably be considered an early version of an artificial general intelligence (AGI) system.
However, many people are skeptical about whether large language models can achieve general artificial intelligence. Among them, Turing Award winner Yann LeCun clearly expressed his opposition. He believes that ChatGPT is not a strong AI, nor is it AGI, it's just a LLM. Because its model scale is very large (more than 175 billion parameters), it can handle massive Natural language text. It also has strong abilities of language generation, understanding and reasoning, which make it look like AGI, but it is actually an LLM.
It's hard to beileve that we have entered, or to be more specific, completely entered an AGI era. Though LLMs are very powerful, there are still many things that remain mysterious. The extraordinary abilities of LLMs mostly attribute to larger dataset, larger model scale, and exquisite training skills. Yet the occurrence of \"Emergence\" indicates we don't totally understand what could make a model stronger, and why it could make a model stronger. Hypothetically, Even though we have developed an AGI model, we can't explain it at current state, we can't even find out if it's really a AGI model. Using an unexplainable model means potential risks.
![](\"https://pic.imgdb.cn/item/64b170f61ddac507cce2ec43.png\")
\n Figure 21: A generated picture with the prompt of \"draw me a picture that can demonstrate the advancements in generative ai in recent years\". Presumably, the model wanted to generate a picture in a knowledge graph form, yet the random circles and meaningless symbols in the actual generated picture indicate that the model still has difficulties in understanding complicated instructions.
\n Potential risk of unemployment is another reason why some people resist Generative AI. Generative AI can do pretty well in some works, sometimes even better than humans. Compared with human resource, under the same circumstances, using Generative AI could be relatively economic. Therefore, people start worrying about whether they will be unemployed one day, as their jobs could be taken by AI. In the short term, a few jobs may be taken by AI, bringing unemployment to some people. But in the long term, AI will work with humans, instead of taking away their jobs. \n
Nonetheless, we shouldn't be too upset about the fact that we don't know when the AGI era will come. As I mentioned before, expanding the applications of Generative AI can benefit people more. But I didn't mean that we should stop the research on AGI, it should be carried on. During the process of developing AGI, we could discover more impetus to the development of AI. 5. Lead to uneployment?
\n
Things like new inventions replacing humans happened before. In the first industrial revolution, steam engines replaced human resources in many factories, tremendously improving social production efficiency. It's a great advancement in human civilization. However, where were those people who were replaced by steam engines? Did they remain unemployed for a very long time? The answer is no. Instead of doing what they used to do in the factory, they started working in the factory wherein the components of the steam engine were produced. Besides, other workers started working for the maintenance of the steam engine, or working in the boiler room. New inventions may take away jobs, but it also provides jobs.
China, as a large and important market, Generative AI has started to bring new job opportunities to it. According to some reports, the demand for talents of AIGC shows a more obvious upward trend than that of AI. According to the data of Liepin Big Data Research Institute, from the first quarter of 2020 to the first quarter of 2021, the growth of the job opportunities in AIGC and AI is comparable. Since then, AIGC has continued to grow in general, and the number of new jobs in the first quarter of 2023 is 5.63 times more than that in the first quarter of 2020.
![](\"https://pic.imgdb.cn/item/64b0bf6f1ddac507cc0dd62a.png\")
\n Figure 22: Comparison of the demand for talents in AI and AIGC (new job). Source: https://www.aigc.cn/12703.html
We are at the primitive stage of applying AI, which means it will take society years or even longer to integrate Generative AI in it. On the other hand, the development of Generative AI can also promote society, thus creating new job opportunities for people. This phenomenon can be rigorously explained by the Marxist Theory[41]. In general, it takes time. \n
![](\"https://pic.imgdb.cn/item/64b0caea1ddac507cc35c1e1.webp\")
\n Figure 23: The link between superstructure and base, according to the Marxist Theory. Source: https://www.thoughtco.com/definition-of-base-and-superstructure-3026372
$\\scriptsize\\bf{9.\\ Conclusion}$
\n This essay provides a comprehensive analysis of Generative AI. We have discussed the history of Generative AI, the crucial advancements in methods, the trend of the development of Generative AI, and the application of Generative AI. Furthermore, we also discussed some challenges that Generative AI confronts. I sincerely hope this essay could bring you some inspiration. \n \n\n","metadata":{}},{"cell_type":"markdown","source":"\n \n[1] https://openai.com/blog/chatgpt \n \n\n","metadata":{}},{"cell_type":"markdown","source":"\n The codes for submission are copied from this amazing notebook. \n \n\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nsubmission_df = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-02T07:19:37.051002Z","iopub.execute_input":"2023-07-02T07:19:37.051412Z","iopub.status.idle":"2023-07-02T07:19:37.120076Z","shell.execute_reply.started":"2023-07-02T07:19:37.051379Z","shell.execute_reply":"2023-07-02T07:19:37.119016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.loc[0]['value']='Generative AI'\nsubmission_df.loc[1]['value']='https://www.kaggle.com/code/jay2333/understand-generate-and-transform-the-world'\nsubmission_df.loc[2]['value']='https://www.kaggle.com/code/adwivedi/kic-and-ethical-considerations-in-generative-ai/comments#2326534'\nsubmission_df.loc[3]['value']='https://www.kaggle.com/code/harbhajansingh21/kaggle-ai-report-for-generativeai/comments#2326553'\nsubmission_df.loc[4]['value']='https://www.kaggle.com/code/keerthanasrija/generative-ai-kaggle-2023-report/comments#2326559'\nsubmission_df.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the submission to a CSV file\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]}]}
Why is Generative Ai so popular in recent years? The concept of the World Model may explain that. World Model is trained to generate various and complex content, it also needs to handle multimodal data. Some scientists believe that during this process, the World Model will gradually understand the complex connections between objects and acquire the ability of inference, finally understanding the world. This process is similar to the growth of human babies, as they also develop their intelligence via interactions with the world. Hence, the World Model is deemed as the key to AGI.
In general, compared with models trained for a particular downstream task, models trained for generation are more powerful and able to \"understand\" objects and the world, and perform better in some downstream tasks. Moreover, Generative AI also has huge commercial value. That's the reason that makes it so popular these days. Furthermore, the requirement of multimodal learning makes the connection between different fields such as CV and NLP closer. It also created a large number of new research directions, stimulating scientists' interest in exploring the unknown.
Looking at all the history of human civilization from its birth to the present, the development of technology in the past three hundred years is faster than any other period of time. Technology liberates humans from arduous labor. In the first industrial revolution, the appearance of the steam engine greatly improved productivity, a machine can work as 100 human workers. In the second industrial revolution, the usage of more efficient energy such as gasoline and electrical energy further improved productivity, and also brought more convenience in people's daily life. The third industrial revolution is mainly about information technology. which makes the transmission of information more efficient. In general, the purpose of the Industrial Revolution has always been to liberate humans from arduous labor.
As for the fourth industrial revolution, it seems that different countries have different definitions about it. Nevertheless, when people in the future start talking about the fourth industrial revolution, AI is always an indispensable topic. AI will be a very important part in the fourth industrial revolution. Compared with steam engines and internal combustion engines that frees people from mechanical labor, AI can liberate humans from a higher-level labor, labor that needs creativity, emotions, etc. A steam engine can never tell you how to write a fiction, but an AI model like GPT-4 could even teach you how to finish your math homework, wouldn't it be amazing?
As a very important part of AI, Generative AI will undoubtedly improve our life and create a better world. Though there are still many problems, we are now in a very early stage of using AI. Realizing problems and risks early before any mistakes are made means we will have more time to tackle problems, and will be more discreet about using AI.$\\scriptsize\\bf{Reference}$
\n
\n[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. arxiv:2204.06125, 2022.
\n[3] https://openai.com/
\n[4] L. Rabiner, B. Juang. An introduction to hidden Markov models. DOI:10.1109/MASSP.1986.1165342.
[5] Jesús Zambrano. Gaussian Mixture Model - method and application. DOI:10.13140/RG.2.2.32667.77602.
\n[6] Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals. Recurrent Neural Network Regularization. arxiv:1409.2329, 2014.
\n[7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Attention is all you need. arxiv:1706.03762, 2017.
\n[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arxiv:1810.04805, 2018.
\n[9] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever. Improving Language Understanding\nby Generative Pre-Training. 2022.\n
\n[10] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio. Generative Adversarial Network. arxiv:1406.2661, 2014.
\n[11] Diederik P Kingma, Max Welling. Auto-Encoding Variational Bayes. arxiv:1312.6114, 2013.
\n[12] Jonathan Ho, Ajay Jain, Pieter Abbeel. Denoising Diffusion Probabilistic Models. arxiv:2006.11239, 2020.
\n[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. arxiv:2111.06377 2021.
\n[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arxiv:2010.11929, 2020.
\n[15] Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schneider, Rickmer Schulte, Karol Urbanczyk, Jann Goschenhofer, Christian Heumann, Rasmus Hvingelby, Daniel Schalk, Matthias Aßenmacher. Multimodal Deep Learning. arxiv:2301.04856, 2023.
\n[16] Mehdi Mirza, Simon Osindero. Conditional Generative Adversarial Nets. arxiv:1411.1784, 2014.
\n[17] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas. StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. arxiv:1612.03242, 2016.
\n[18] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang. VisualBERT: A Simple and Performant Baseline for Vision and Language. arxiv:1908.03557, 2019.
\n[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arxiv:2103.00020, 2021.
\n[20] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. arxiv:2204.06125, 2022.
\n[21] Daniel Gomes de Pinho Zanco, Leszek Szczecinski, Eduardo Vinicius Kuhn, Rui Seara. A comprehensive analysis of the Elo rating algorithm: Stochastic model, convergence characteristics, design guidelines, and experimental results. arxiv:2212.12015, 2022.
\n[22] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe. Training language models to follow instructions with human feedback. arxiv:2203.02155, 2022.
\n[23] Tero Karras, Samuli Laine, Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. arxiv:1812.04948, 2019.
\n[24] Xun Huang, Serge Belongie. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. arxiv:1703.06868, 2017.
\n[25] Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu, Lichao Sun. A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT. arxiv:2303.04226, 2023.
\n[26] Roberto Gozalo-Brizuela, Eduardo C. Garrido-Merchan. ChatGPT is not all you need. A State of the Art Review of large Generative AI models. arxiv:2301.04655, 2023.
\n[27]https://cohere.com/
\n[28]https://runwayml.com/
\n[29]https://www.adept.ai/
\n[30]https://beta.character.ai/
\n[31]https://www.glean.com/
\n[32]https://worldmodels.github.io/
\n[33] Hongyang Yang, Xiao-Yang Liu, Christina Dan Wang. FinGPT: Open-Source Financial Large Language Models. arxiv:2306.06031, 2023.
\n[34] Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, Li Yuan. ChatLaw: Open-Source Legal Large Language Model with Integrated External Knowledge Bases. arxiv:2306.16092, 2023.
\n[35]https://github.com/hpcaitech/ColossalAI
\n[36]https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/
\n[37]https://www.thepaper.cn/newsDetail_forward_23193760
\n[38] Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, Bernhard Schölkopf. Can Large Language Models Infer Causation from Correlation? arxiv:2306.05836, 2023.
\n[39] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arxiv:1910.13461, 2019.
\n[40] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, Yi Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arxiv:2303.12712, 2023.
\n[41]https://www.thoughtco.com/definition-of-base-and-superstructure-3026372Code for submission
\n