{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

The Generative AI Hype Cycle

","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"

Introduction

\n

Generative AI refers to a class of artificial intelligence models designed to create new data by learning patterns and structures from existing data. These models can generate content, simulate human-like behavior, and make predictions based on their understanding of the input data.
\nGenerative AI has made incredible advancements in recent years, with language models like GPT-3 and GPT-4 capturing the attention of researchers, businesses, and the general public alike.
\nThe technology is certainly going through the hype cycle, especially while we are all grappling with what the technology is, how it works and establishing its limitations. Indeed, according to the Gartner Hype Cycle, a pattern that often emerges when a new brand, product, or technology is released, the visibility of the new innovation rises or declines over time based on the pattern in the figure below.

\n\n

The graph illustrates how, at the outset, our confidence levels about what can be done with the technology are high. This is not due to our understanding, but rather due to the opposite – our limited understanding combined with the hype, which makes us over-confident.\nAs we understand more about the technology our confidence in its power to impact everything diminishes. But, as our understanding grows we begin to truly understand where it can be effectively used, with the result being that our confidence begins to rise again but this time grounded less on hype and more on reality.
\nLet's ride the wave of hype cycle and try to understand the different phases of Generative AI hype cycle!

\n

1.Generative AI innovation trigger : \"Attention is All you Need\" !

\n

The release of the -now famous- \"Attention is All you Need\" paper by Google researchers in 2017 has driven a paradigm shift in AI. The paper introduced a new class of models called \"Transformers\".Since then, They have been driving a wave of advances in machine learning. Some have even dubbed transformer AI. The paper has accumulated a whopping 38k citations in only 5 years.
\nAnother breakthrough technique was the \"Diffusion models\", which can be regarded as the main factor for the recent breakthrough in the AI generative artworks field.
\nLet's have a deep dive into these 2 classes of models.

\n

a. What are Transformers ?

\n

\nTransformers are a type of neural network architecture that leverages a self-attention mechanisms to process and generate text efficiently and effectively.\nAttention mechanisms allow AI models to selectively focus on relevant parts of the input data while generating output, resulting in more coherent and context-aware text.\nWhile original Transformers were designed for language tasks, the same Transformer architecture has been applied to many other applications like the generation of images, audio, music, or even actions.
\nThe main components of a transformer architecture are the encoder and the decoder, each composed of multiple stacked layers.

\n\n

Encoder (left): The encoder receives an input and builds a (embedding) representation of it. \nEach encoder layer has two sub-layers. The encoder’s inputs first flow through a self-attention sub-layer which encodes each word, and the outputs are fed to the second sub-layer, a point-wise, fully connected feed-forward neural network.\nThe fully connected feed-forward network is applied to each position separately and identically. It consists of two linear transformations with a ReLU activation in between.
\nDecoder (right): The decoder uses the encoder’s representation (features) to generate an output sequence of words one token at a time. At each step, it is auto-regressive, consuming the previously generated token as additional input when generating the next.
\nIn addition to the two sub-layers in each encoder layer, the decoder has an additional encoder-decoder attention sub-layer, which performs multi-head attention over the output of the encoder stack.\n

\n

a. What are Diffusion models ?

\n

\nDiffusion models in deep learning were first introduced by Sohl-Dickstein et al. in the seminal 2015 paper “Deep Unsupervised Learning using Nonequilibrium Thermodynamics.” Unfortunately, it remained behind the curtains for a while. \nBut, in 2019, Song et al. published a paper, “Generative Modeling by Estimating Gradients of the Data Distribution,” using the same principle but a different approach. In 2020, Ho et al. published the paper, now-popular “Denoising Diffusion Probabilistic Models” (DDPM for short). \nAfter 2020, research in diffusion models took off. Much progress has been made in creating, training, and improving diffusion-based generative modeling in a relatively short time. \nThe structure (distribution) of the original image is gradually destroyed by adding noise and then using a neural network model to reconstruct the image, i.e., remove the noise at each step. Doing so enough times and with good data, the model eventually learns to estimate the underlying (original) data distribution. Then, we can simply start with just noise and use the trained neural network to generate a new image representative of the original training dataset.”\n

\n \n

2. Generative AI Peak of Inflated Expectations

\n

\nGenerative models in different areas have followed different paths but eventually intersected with the transformer architecture. This architecture has become the backbone for many generative models in various domains, offering advantages over previous building blocks like LSTM and GRU. \n\nThe transformer architecture has been applied to NLP, resulting in large language models like BERT and GPT. In Computer Vision (CV), Vision Transformers and Swin Transformers have combined transformer architecture with visual components, allowing them to be applied to image-based tasks. \n\nTransformers have also enabled models from different fields to be fused for multimodal tasks, like CLIP, which combines vision and language to generate text and image data.\nThe graph below represents some of the most significant developments in generative AI over the past few years, but they are by no means the only ones. With continued research and development, we can expect to see even more exciting advances in the field of generative AI in the years to come.\n

\n\n

3. Generative AI Trough of Disillusionment

\n

Early vibes of Disillusionment towards generative AI start to appear. Indeed, as generative AI continues to grow in influence, ethical considerations and responsible AI development become increasingly important. Let's discuss some key ethical aspects in the context of generative AI:

\n

a. Bias and Fairness

\n

Ensuring that AI-generated text is free from biases and is fair is crucial for creating equitable AI systems.
\nTechniques like re-sampling, re-weighting, and adversarial training can help mitigate biases in AI-generated text and promote fairness.

\n

b. Data Privacy and Security

\n

Protecting user data and ensuring privacy is a key ethical consideration for AI development.
\nTechniques like federated learning, differential privacy, and secure multi-party computation can help safeguard user data and ensure privacy compliance.

\n

a. Accountability and Responsibility

\n

AI developers and organizations should be held accountable for the impacts of their AI systems, including any unintended consequences.
\nEstablishing clear guidelines, policies, and mechanisms for AI governance can help ensure responsible AI development and deployment.

\n

a. Environmental Sustainability

\n

Reducing the environmental impact of AI development, including energy consumption and carbon emissions, is essential for sustainable AI.
\nTechniques like model pruning, knowledge distillation, and more efficient hardware can help lower energy consumption and reduce the carbon footprint of AI development.

\n

4. For when the Generative AI plateau of Productivity ?

\n

The 2022 Gartner Hype Cycle™ for Artificial Intelligence (AI) identifies must-know innovations in AI technology and techniques that go beyond the everyday AI already being used to add intelligence to previously static business applications, devices and productivity tools.
\nConsidering the Gartner AI Hype Cycle, there are arguably five technologies which are related to Language Technology and Conversational AI, constituting 15% of the report.\nThese five language related technologies in the hype cycle are:\n

\n

\n\n\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nsubmission = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\n\nvals = [\"Generative AI\", \n \"https://www.kaggle.com/yossra/the-generative-ai-hype-cycle\",\n \"'http://www.kaggle.com/.../your_peer_feedback'\",\n \"'http://www.kaggle.com/.../your_peer_feedback'\",\n \"'http://www.kaggle.com/.../your_peer_feedback'\",\n ]\nsubmission.value = vals\n\nsubmission.to_csv('submission.csv', index=False)\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-07-05T22:58:33.563576Z","iopub.execute_input":"2023-07-05T22:58:33.563997Z","iopub.status.idle":"2023-07-05T22:58:33.581732Z","shell.execute_reply.started":"2023-07-05T22:58:33.563968Z","shell.execute_reply":"2023-07-05T22:58:33.580899Z"},"trusted":true},"execution_count":null,"outputs":[]}]}