![](\"https://i.imgur.com/WJeMnOm.png\")
Image credit Appen
","metadata":{}},{"cell_type":"markdown","source":"\n\n1. Introduction
¶\n![](\"https://i.imgur.com/g9EAjKF.png\")
Image credit Brett Jordan
","metadata":{}},{"cell_type":"markdown","source":" This report highlights the most significant events in the field of generative AI between 2021 and 2023. Generative AI has seen remarkable advancements during this period, with breakthroughs in image synthesis, audio and language models. The following sections discuss notable developments and achievements in the field.","metadata":{}},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"2. Background
¶","metadata":{}},{"cell_type":"markdown","source":" Before we dive into the advancements in Generative AI over the last two years, it is important to understand the significance of the Generative Pre-trained Transformer (GPT). GPT is a family of large-scale language models developed by OpenAI. The underlying architecture of GPT was based on the [Transformer](https://arxiv.org/abs/1706.03762) which is the classic encoder decoder architecture. The encoder processes the input sequence, while the decoder generates an output sequence based on the encoded information. The paper [Generating Wikipedia by Summarizing Long Sequences](https://arxiv.org/pdf/1801.10198.pdf) proposed another arrangement of the transformer block that is capable of doing language modeling. This model threw away the transformer encoder and only used a stack of decoders. The GPT models also use a similar architecture which consists of a stack of many decoders. The GPT models are autoregressive in nature, which means that the model predicts the next value in a sequence based on the previous values in the sequence. ","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/CNP0S4i.png\")
Image credit Jay Alammar
","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/fhXCY1J.png\")
Image credit tooltester.com
","metadata":{}},{"cell_type":"markdown","source":"3. Year 2021
¶","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/yMr3cPk.png\")
Example prompt \"An astronaut riding a horse in photorealistic style\" using DALL-E
\nImage credit OpenAI
","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"Example of function auto code completion in Github Copilot
\n Animation credit kingslayerr 4. Year 2022 Image credit Google DALL-E 2 Architecture Image credit DALL-E-2 paper DALL-E (left) vs DALL-E 2 (right) for the same prompt Photo credit OpenAI Prompt: Teenager, female, seventeen years old, pop culture, modern clothing. Photorealistic, Natural lighting, Pastel \n Image credit John Severinson Image credit Jay Alammar Prompt: kneeling cat knight, portrait, finely detailed armor, intricate design, silver, silk, cinematic lighting, 4k Image credit Valère Image credit Meta AI Prompt: A teddy bear painting a portrait Image credit Meta AI Example hallucination by Galactica Image credit Michael Black Example conversation with ChatGPT 5. Year 2023 A timeline of LLMs including open ource ones (highlighted yellow)\n Image credit A Survey of LLMs paper Example conversation with Bard for the same prompt used for ChatGPT above Prompt: Aerial drone footage of a mountain range\n Image credit Runway Image credit QLoRA paper Image credit OpenAI 6. Conclusion Image credit Tama66 7. Looking Forward Image credit Greg Rakozy 8. References 9. Citations📌 2021 marked a year in Generative AI where:\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":" 4.1 Pathways Language Model (PaLM) ¶
","metadata":{}},{"cell_type":"markdown","source":" [PaLM](https://arxiv.org/pdf/2204.02311v5.pdf) is a 540B parameter (3x the size of GPT3) large language model (LLM) developed by Google AI. It was designed to be more efficient and scalable than previous LLM architectures, while still achieving state-of-the-art performance on a variety of tasks. PaLM uses Pathways which uses a hierarchical structure of pathways, that allows the model to learn different levels of abstraction about the data. This architecture allowed for scaling the training process to thousands of TPU chips, which is a significant increase compared to other language models. This scaling was achieved using data parallelism, where the data is divided and processed across multiple TPU chips, and model parallelism, where the model is split and processed on a single TPU chip. This architecture enabled the training of a very large model with a large dataset of 780 billion tokens comprising of filtered webpages, books, Wikipedia, news articles, source code and social media conversations.\n
PaLM showed significant improvements over previous language models on a variety of tasks, including text generation, question answering, and machine translation. For example, PaLM was able to generate text that was more factual and creative than text generated by previous language models. It was also able to answer questions more accurately and to translate languages more fluently.\n\n","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/s3phECI.png\")
4.2 DALL-E 2¶
","metadata":{}},{"cell_type":"markdown","source":" In April 2022, Open AI released the succesor to DALL-E, the [DALL-E 2](https://arxiv.org/pdf/2204.06125v1.pdf). While DALL-E used a [CLIP](https://arxiv.org/abs/2103.00020) model for re-ranking, DALL-E 2 uses CLIP embeddings directly, and decodes images via a diffusion model instead of a decoder-only transformer. A diffusion model is a type of generative model that works by gradually adding noise to an image until it matches the text description. This allows DALL-E 2 to generate more realistic images, as it is not limited by the need to generate the image one pixel at a time.","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/nxaIUES.png\")
The combination of using CLIP embeddings, a diffusion prior and a diffusion decoder enabled DALL-E 2 to generate more realistic and accurate images vs DALL-E. Below, is an example of the difference in quality for the same prompt. \"a painting of a fox sitting in a field at sunrise in the style of Claude Monet\"\n\n ","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/JK9Iam5.png\")
4.3 Midjourney¶
","metadata":{}},{"cell_type":"markdown","source":" Midjourney, an image generation model developed by an independent research lab called Midjourney Inc., was released to the public in July 2022. The model's architecture has not been revealed, but its images were generally considered to be of high quality and able to generate a wide variety of different styles and genres. Midjourney was also released in an open beta via Discord, which allowed anyone to sign up and start using the model. This made Midjourney one of the most accessible LLMs for generating images. \n
The following images show how the quality of images generated by Midjourney has improved over time and across different versions of the model. The images were all generated with the same prompt, but the versions of Midjourney used to generate them ranged from version 1, released in 2022, to version 5, released in March 2023. As you can see, the images generated by newer versions of Midjourney are generally more detailed, realistic, and creative than those generated by older versions.","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/gLdrSCY.png\")
4.4 Stable Diffusion¶
","metadata":{}},{"cell_type":"markdown","source":" In August 2022, Stable Diffusion a latent diffusion model, capable of generating images from text descriptions was released. This model was particularly signficant because the code and model weights were released publically, making it one of the earliest open source LLMs that generated images from text. It was developed by researchers from the CompVis Group at Ludwig Maximilian University of Munich and Runway with a compute donation by Stability AI and training data from non-profit organizations. It is a latent diffusion model, which means that it uses a latent space to represent the images it generates and it starts with a noisy image and gradually reduces the noise to generate a more realistic image. This allows it to generate images that are more diverse and creative than traditional diffusion models.","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/TpKNMFV.png\")
\n
","metadata":{}},{"cell_type":"markdown","source":" The picture below shows a sample prompt and the image generated by Stable Diffusion. The fine-grained details of the intricate armor design, result in a stunning picture of a cat in armor. ","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/PgoYDOk.png\")
4.5 Meta announces Make-A-Video¶
","metadata":{}},{"cell_type":"markdown","source":" In September 2022, Meta (Facebook) announced [Make-A-Video](https://arxiv.org/pdf/2209.14792.pdf), a LLM that generates videos from text. The model extends a diffusion-based text-to-image (T2I) model to text-to-video (T2V), through a spatiotemporally factorized diffusion model. Make-A-Video consists of three main components, (1) A base T2I model trained on text-image pairs (2) Spatiotemporal convolution and attention layers that extend the networks’ building blocks to the temporal dimension and (3) Spatiotemporal networks that consist of both spatiotemporal layers, as well as a frame interpolation network for high frame rate generation.","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/qD1uM2T.png\")
![](\"https://makeavideo.studio/assets/A_teddy_bear_painting_a_portrait.webp\")
4.6 Meta releases Galactica¶
","metadata":{}},{"cell_type":"markdown","source":" In November 2022, Meta (Facebook) released Galactica, a LLM that can store, combine and reason about scientific knowledge according to their submitted [paper](https://galactica.org/static/paper.pdf). The models ranged from 125 Million to 120 Billion parameter. \n
In collaboration with Papers with Code, Meta AI open-sourced Galactica and launched a website that allowed visitors to interact with the model. However, three days after Galactica’s release, Meta had to shut down the online demo following a deluge of criticism by scientists and tech media about the model’s incorrect and biased output. A fundamental problem with Galactica is that it is not able to distinguish truth from falsehood, a basic requirement for a language model designed to generate scientific text. The explanations it generated sounded authoritative, but they were often subtly incorrect, biased, or just plain wrong.\n
The example below shared on [Twitter](https://twitter.com/Michael_J_Black/status/1593133722316189696?s=20) by Michael Black (Director, Max Planck Institute), shows Galactica citing fictional sources for the query \"Accurate estimation of body shape under clothing from an image\". It produces an abstract that is plausible but refers to Alldieck et al. \"Accurate Estimation of Body Shape Under Clothing from a Single Image\" which does not exist. ","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/NhiB4G9.png\")
The release of Galactica was a cautionary tale about the dangers of releasing a large language model which everyone could access and that was specifically marketed as a model that could reason about scientific knowledge. Galactica’s output can feel real but it was not grounded in real facts. However, the incident also highlights the importance of carefully evaluating the output of LLMs before using them for important tasks.","metadata":{}},{"cell_type":"markdown","source":" 4.7 ChatGPT - A Tipping Point for Generative AI¶
","metadata":{}},{"cell_type":"markdown","source":" In November 2022, OpenAI released ChatGPT, a generative AI based chatbot that gained worldwide popularity and took the AI world by storm! OpenAI made ChatGPT accessible to everyone for free, and it gained 1 million users in just 5 days. In contrast, Netflix took 3.5 years to get the same amount of users. ChatGPT hit a 100 million monthly active users in January 2023, making it the fastest-growing application in history. \n
ChatGPT was a versatile language model that could generate chat conversations that were often indistinguishable from human-generated conversations. It could also be used for a variety of other tasks, such as answering questions, generating creative content, providing explanations, offering suggestions, writing poems or code, and more.\n
The release of ChatGPT was a pivotal moment in the field of artificial intelligence. It showed that language models were capable of achieving a level of sophistication that was previously thought to be impossible. This led to a flurry of activity in the AI community, as other companies rushed to develop their own language models and tools and it is likely that we will see even more impressive language models in the years to come.
\n
ChatGPT was based on the GPT3 architecture, so it is estimated to have around 175 billion parameters. It was fine tuned on a chat specific task where the model was trained on a smaller curated dataset compared to the pretraining data. Human AI trainers and annotators interact with the model and provide rankings for different possible model-generated responses. The model then learns from this feedback and adjusts its responses accordingly. Fine-tuning allows the model to specialize in particular domains or tasks while incorporating human preferences and guidelines. For example, ChatGPT may have been fine-tuned using conversational data, customer support dialogues, or other relevant datasets to make it more suitable for interactive and conversational applications.","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/S4PPgVM.png\")
📌 2022 was a very significant year for Generative AI where:\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":" 5.1 Meta Releases LLaMA - Open Source LLMs Explode! ¶
","metadata":{}},{"cell_type":"markdown","source":" In February 2023, Meta released a new large language model (LLM) called LLaMA. LLaMA is a 65-billion parameter model that was trained on a massive dataset of text and code. \n
The LLaMA release was significant for a number of reasons. It was one of the largest LLMs that was released to the public. This means that it has the potential to be used for more complex and challenging tasks than previous LLMs. LLaMA was also open source but the model weights were initially distrubuted to a handful of researchers with the intent of only using the model for research purposes. The model weights landed up being leaked online which means that anyone could download and use it. This event accelerated the development of many LLMS and sparked a series of impressive open source alternatives to ChatGPT. ","metadata":{}},{"cell_type":"markdown","source":"\n![](\"https://i.imgur.com/PpToJYE.png\")
The above chart mentioned in the paper [A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf), shows the explosion in the number of open source LLMs after the release of LLaMa in January 2023. Huggingface also maintains an open LLM leaderboard [here](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). \n
The Technology Innovation Institute (TII) in Abu Dhabi released Falcon, a notable example of an open source LLM under the Apache 2.0 license. It was the first truly open model with capabilities rivaling many current closed-source models. There were two versions of Falcon: a smaller 7 billion parameter version and a larger 40 billion parameter version. Falcon 40-B requires only 90GB of GPU memory while Falcon-7B only demands around 15GB. The key ingredient for the high quality of the Falcon models is their training data, which is predominantly (more than 80%) based on RefinedWeb, a novel massive web dataset based on CommonCrawl. TII focused on scaling and improving the quality of web data instead of gathering scattered curated sources. They leveraged large-scale deduplication and strict filtering to match the quality of other corpora. The Falcon models still include some curated sources in their training (eg. Reddit), but to a much lesser extent compared to GPT3 or PaLM. The Falcon 40-B currently leads the Hugginface LLM leaderboard. \n ","metadata":{}},{"cell_type":"markdown","source":" 5.2 Google Releases Bard ¶
","metadata":{}},{"cell_type":"markdown","source":" Google made a significant move in March 2023 by introducing Bard, a chatbot built on a LLM framework. This release aimed to keep up with the immense hype and popularity surrounding OpenAI's ChatGPT. It was a fine tuned model based on PaLM 2 which had around 340 billion parameters. Bard was trained on a dataset of text and code that was 10 times larger than the dataset that ChatGPT was trained on. Bard also has access to the internet, which means that it could continuously learn and update its knowledge base. This made Bard more up-to-date and accurate than ChatGPT, which was limited to the information that was included in its training dataset which was sometime around September 2021.","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/oIr0AXy.gif\")
5.3 Runway announces Gen-2 for Text-To-Video ¶
","metadata":{}},{"cell_type":"markdown","source":" In March 2023, Runway (one of the companies behind Stable Diffusion), announced Gen-2, a text to video model. Gen-2 was able to create videos either by applying the composition and style of an image or text prompt to the structure of a source video (Video to Video) or, using nothing but words (Text to Video). Runway showcases a number of examples [here](https://research.runwayml.com/gen2), and one such example is shown below.","metadata":{}},{"cell_type":"markdown","source":"![](\"https://duet-cdn.vox-cdn.com/thumbor/0x0:768x448/1440x840/filters:focal(384x224:385x225):no_upscale():format(webp)/cdn.vox-cdn.com/uploads/chorus_asset/file/24522127/An_aerial_shot_of_a_mountain_landscape.2023_03_20_13_47_36.gif\")
5.4 QLoRA ¶
","metadata":{}},{"cell_type":"markdown","source":" In May 2023, Dettmers et al., 2023, released Quantized LLMs with Low-Rank Adapters [QLoRA](https://arxiv.org/abs/2305.14314). This work was built on the previous work done on LoRA that was released in 2021 by [Hu et al](https://arxiv.org/abs/2106.09685). LoRA is a technique that reduces the memory footprint of large language models (LLMs) by adding a small number of trainable parameters, called adapters, to each layer of the LLM. The original parameters of the LLM are frozen, and only the adapter weights are updated during fine-tuning. This significantly reduces the memory footprint of the model, while still maintaining its performance.\n
QLoRa is an extension of LoRA that goes three steps further by introducing 4-bit quantization, double quantization and paged optmizers which exploits Nvidia's unified memory for paging. QLoRA reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.\n
Instruction finetuning is an essential step on top of pretrained LLMs which can give LLM's ChatGPT-like capabilities. QLoRA enables finetuning for a wider audience that don't have too many compute resources, making it a big win for the accessibility of state of the art NLP technology. ","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/BCrFdEN.png\")
5.5 OpenAI Releases GPT4 ¶
","metadata":{}},{"cell_type":"markdown","source":" In May 2023, OpenAI released the highly anticipated GPT4. GPT4 was multimodal i.e. accepting image and text inputs, generating text outputs. OpenAI mentioned that they spent 6 months making GPT-4 safer and more aligned. GPT-4 was 82% less likely to respond to requests for disallowed content and 40% more likely to produce factual responses than GPT-3.5 on their internal evaluations. It exhibited human-level performance on various professional and academic benchmarks. For example, it passed a simulated bar exam with a score around the top 10% of test takers. The chart below shows how it does on different exams compared to GTP3.5. It showed impressive results across multiple exams.\n
There was little revealed about the architecture of GPT4, but recent rumours suggest that it is based on eight models, each with 220 billion parameters, which are linked in the Mixture of Experts (MoE) architecture making it a model with 1.76 Trillion parameters. The MoE model is a type of ensemble learning that combines different models, called \"experts,\" to make a decision. ","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/fUjdJGc.png\")
📌 2023 saw an explosion in Generative AI fuelled by the ChatGPT launch in late 2022. \n\n
","metadata":{}},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/Xl9q1Tu.png\")
GitHub Copilot is a game-changer in how AI assists software developers in their coding endeavors. A [survey](https://gizmodo.com/github-survey-92-percent-programmers-are-using-ai-tools-1850534570) by Github, showed that an astounding 92% of the 500 programmers surveyed, use AI tools to assist them in their workflows. Another [study](https://arxiv.org/pdf/2306.15033.pdf) focused on the productivity gained by GitHub Copilot, where half the developers were permitted access to Copilot while the other half were not. The results were striking, with Copilot users completing their projects 55% faster compared to the control group. The study even suggests that the adoption of AI coding tools like Copilot could potentially contribute to a remarkable $1.5 trillion boost in the global GDP. While this was just one study, it is evident that tools like Copilot can be used to write code faster, more efficiently, and with fewer errors. \n\n Industry giants including Google, Meta, and OpenAI played pivotal roles in text Generative AI with significant releases such as PaLM, Galactica, ChatGPT and GPT4, respectively. ChatGPT was a tipping point in text Generative AI which brought LLMs to the center of the worlds attention. LLMs have the ability to generate coherent and contextually relevant text, making them invaluable for content creation, writing assistance, and creative storytelling. They can effectively analyze and organize vast amounts of textual data, enabling more efficient search engines and knowledge extraction systems. Meta's release of LLaMa set the stage for a series of impressive open-source alternatives to ChatGPT. Responding to growing demand, Google introduced Bard, an AI chatbot, as critics clamored for the search giant's response to the popular ChatGPT. Furthermore, game-changing advancements like QLoRA showcased the feasibility of fine-tuning Language Models (LLMs) on consumer-grade GPUs, extending the accessibility of this technology to a broader community.\n","metadata":{}},{"cell_type":"markdown","source":" 📌 Conclusion:\n \n
","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"![](\"https://i.imgur.com/8kmUi3i.png\")
Language models have witnessed remarkable progress in recent years, with Large Language Models (LLMs) at the forefront of this advancement. While LLMs offer great promise, several pitfalls must be acknowledged and addressed before deploying such stytems broadly. One significant concern is the potential for bias in training data, which can perpetuate societal inequalities and reinforce existing biases. Ethical considerations surrounding privacy and data security must also be taken into account, as LLMs require vast amounts of data to train effectively. The copyright concerns seen in text-to-image models are also applicable to LLMs. In a [lawsuit](https://storage.courtlistener.com/recap/gov.uscourts.cand.403220/gov.uscourts.cand.403220.1.0.pdf) against GitHub, Microsoft, and OpenAI the plaintiffs allege that these companies violated copyright, contract, privacy, and business laws, among others, by using public source code pulled from GitHub to create GitHub's Copilot programming assistant. While this case is also pending, it highlights the need to carefully look at data that the model was trained on, before it is put into commercial applications. Model hallucination is also a concern where the model generates text that is incorrect, nonsensical, or not real. In a recent [case](https://apnews.com/article/artificial-intelligence-chatgpt-fake-case-lawyers-d6ae9fa79d0542db9e1455397aef381c), a lawyer used ChatGPT to submit fictitious legal research in an aviation injury claim. This serves as an illustration of the potential dangers associated with relying on technology that is still undergoing evaluation, highlighting the need for appropriate safeguards and checks to ensure its safe and responsible use. Furthermore, the issue of explainability arises, as LLMs can be seen as black boxes, making it challenging to understand how they arrive at their conclusions.\n
Despite the pitfalls, LLMs hold tremendous potential and offer numerous promising possibilities. One of the key promises lies in their ability to enhance human-computer interaction. LLMs can empower individuals to access and navigate vast amounts of information, providing personalized insights and recommendations. Looking forward, addressing issues like model hallucination with techniques such as grounding, might be key to help improve the quality of model outputs. Grounding ensures that the model is not only using the data on which it is trained, but it is also able to access external data sources to provide additional context and 'ground' the model to factual data. One approach to ground the model reponses with factual data was presented in the Retrieval Augmented Generation (RAG) [paper](https://arxiv.org/abs/2005.11401). RAG provides an architecture that adds an additional step where relevant documents on the topic are retrieved from an additional source like the internet, along with the original text that is input to the model. These two sources integrate all the information and give the model the capability to generate more factually correct answers while citing the relevant sources. \n
The rapid advancement of LLMs have also resulted in the emergence of popular tools built on top of these LLMs like LangChain and AutoGPT. [LangChain](https://python.langchain.com/docs/get_started/introduction.html) is an open-source framework that allows developers to build applications using LLMs. It provides a number of features that make it easier to chain together different LLMs and data sources. LangChain's features include prompt templates that makes it easier to create different types of applications, such as chatbots, question-answering systems, and summarization tools. It also supports \"Agents\" which are components that use LLMs to decide what actions should be taken. This makes it possible to build more complex applications that can reason about the world and take appropriate actions. [AutoGPT](https://github.com/Significant-Gravitas/Auto-GPT) is an open source attempt to make LLMs like GPT4 fully autonomous. It was designed to provide targeted, goal-oriented solutions by segmenting larger objectives into a sequence of manageable, smaller subtasks. It then tirelessly executes these subtasks – either indefinitely or until you choose to halt it, which is where the autonomous part comes in. Tools like LangChain and AutoGPT will continue to emerge in this ever evolving LLM landscape, and will play a signficant role in shaping how these models are used in the future. \n
These past few years have witnessed an era of rapid innovation and transformative breakthroughs in Generative AI. As novel models and techniques continue to be developed, we can eagerly anticipate even more astonishing advancements in the years to come, further propelling the field into uncharted territory.","metadata":{}},{"cell_type":"markdown","source":"📌 As we move forward, collaboration between researchers, policymakers, and society at large will be essential to shape the future of text and image generative AI and LLMs. Together, we can unlock the full potential of these technologies, unleashing a new era of creativity, productivity, and human-computer symbiosis. By embracing this promise responsibly, we can create a future where LLMs enrich our lives, inspire innovation, and contribute to the betterment of society as a whole.","metadata":{}},{"cell_type":"markdown","source":"___","metadata":{}},{"cell_type":"markdown","source":"