","metadata":{}},{"cell_type":"markdown","source":"Introduction: Importance, Current Trends, and Key Areas of AIGC
\n\nThe advancements in Artificial Intelligence Generative Content (AIGC) technologies have introduced extensive capabilities, from generating human-like text to creating music compositions. The current state-of-the-art models have achieved impressive feats, thanks to their large-scale pretraining and fine-tuning mechanisms. They can leverage vast amounts of data to learn patterns and produce incredibly realistic and diverse outputs, ranging from text and images to music and videos. These AIGC models have found their use-cases in various domains, including but not limited to, natural language processing, image synthesis, music generation, and video generation. **Lets review what are the most relevant developments on each field by category and model**.","metadata":{}},{"cell_type":"markdown","source":"## Transformers: A Backbone of AI\n\n[Transformers](#References) are the underlying architecture for several state-of-the-art models, like GPT-3, DALL-E-2, Codex, and Gopher. They were designed to overcome the limitations of traditional models like Recurrent Neural Networks (RNNs) in handling variable-length sequences and context-awareness. [**Transformers rely on a self-attention mechanism that lets the model attend to different parts of an input sequence**](#References), making them especially good at handling long-term dependencies. They comprise of an encoder (which generates hidden representations from the input) and a decoder (which uses these hidden representations to generate output). This architecture's parallelizability and learning capabilities have made Transformers ideal for large-scale pre-training and adaptable for various tasks.\n\n## Pre-trained Language Models: Dominating Natural Language Processing\n\nWith the rise of the Transformer architecture, pre-trained language models based on it have become dominant in natural language processing. These models generally fall into two categories based on their training tasks: [**autoregressive language modeling**](#References) and [**masked language modeling**](#References). \n\nModels like [**BERT and RoBERTa use masked language modeling**](#References) where they predict the probability of a masked token in a sentence given the context. XL-Net, another BERT-based model, uses permutation operations to change the prediction order, allowing it to learn more information across tokens.\n\nOn the other hand, models like [**GPT-3 and OPT use autoregressive language modeling**](#References) to predict the probability of the next token given previous tokens. Autoregressive models are particularly suited for generative tasks. \n\n## Revisiting Reinforcement Learning from Human Feedback\n\nDespite being trained on large-scale data, AI-Generated Content (AIGC) might not always produce outputs that align with the user's intent. To better align AIGC output with human preferences, RLHF has been applied to fine-tune models in various applications. This process involves [**pre-training a language model, training a reward model to encode human preferences, and fine-tuning the language model to maximize the learned reward function using reinforcement learning**. Proximal Policy Optimization (PPO) is often used as the RL algorithm](#References). \n\nRecently, the dialogue agent Claude uses Constitutional AI, where the reward model learns via RL from AI Feedback (RLAIF). In Claude, both critiques and AI feedback are guided by a \"constitution,\" which aids in controlling the outputs to be less harmful.\n\n## Hardware, Distributed Training and Cloud Computing\n\nSignificant advancements in hardware have facilitated the training of large-scale models. Computing resources, like [**NVIDIA's A100 GPU and Google's Tensor Processing Units (TPUs), have accelerated model training**](#References).\n\nThe adoption of distributed training, where the training workload is split among multiple processors or machines, has also led to quicker model training. Furthermore, cloud computing services like AWS and Azure now provide access to powerful computing resources, enabling the training of large-scale models.","metadata":{}},{"cell_type":"markdown","source":"## Unimodal Models\n\nUnimodal generative models accept a specific raw data modality as input, such as text or images, and generate predictions in the same modality. These models include **generative language models** such as GPT-3, BART, T5, and **generative vision models** like GAN, VAE, and normalizing flow.\n\n### Generative Language Models\n\nGenerative Language Models (GLMs) generate readable human language based on input data patterns and structures. These models find utility in diverse NLP tasks such as dialogue systems, translation, and question answering. \n\nPre-trained language models have become a cornerstone in the field of NLP. Current state-of-the-art pre-trained language models can be categorized into masked language models (encoders), autoregressive language models (decoders), and encoder-decoder language models.\n\n### Decoder Models\n\nProminent examples of autoregressive decoder-based language models include **GPT, GPT-2, and GPT-3**, which are Transformer-based models utilizing self-attention mechanisms. Models such as [**Gopher**](https://huggingface.co/transformers/v3.3.1/pretrained_models.html) use a GPT-like structure but replace LayerNorm with RSNorm, where a residual connection is added to the original layernorm structure to maintain information. \n\n[**BLOOM**](https://huggingface.co/transformers/v3.3.1/pretrained_models.html) shares the same structure as GPT-3 but replaces sparse attention with a full attention network, better suited for modeling long dependencies. Megatron extends commonly used architectures like GPT-3, BERT, and T5 with distributed training objectives to process a large amount of data.\n\n### Encoder-Decoder Models\n\nOne of the main encoder-decoder methods is the Text-to-Text Transfer Transformer [(**T5**)](https://huggingface.co/docs/transformers/model_doc/t5), combining Transformer-based encoders and decoders for pre-training. T5 employs a \"text-to-text\" approach, transforming both the input and output data into a standardized text format. This allows T5 to be trained on a wide range of NLP tasks.\n\n[**Switch Transformer**](#References) utilizes \"switching\", a simplified MoE routing algorithm, for parallelized training on T5. It obtained larger scale and better performance with the same computational resources compared to the base model. \n\n[**BART**](https://huggingface.co/docs/transformers/model_doc/bart) blends the bidirectional encoder from BERT and the autoregressive decoder from GPT, allowing it to leverage the bidirectional modeling abilities of the encoder while retaining the autoregressive properties for generation tasks.\n\n## Vision Generative Models\n\n### GAN\n\nGenerative Adversarial Networks (GANs) have become popular in the field of image generation research. GANs consist of a generator and a discriminator. The generator learns the distribution of real examples to generate new data, while the discriminator determines whether the input is from the real data space or not.\n\n[**LAPGAN**](#References) generates high-quality images in a coarse-to-fine fashion using a cascade of convolutional networks. [**DCGANs**](#References) structure, a class of CNNs with architectural constraints, has been proposed as a powerful solution for unsupervised learning. [**Progressive GAN**](#References) progressively grows the generator and discriminator, starting from low resolution and adding layers to model finer details. \n\nTo address mode collapse and improve the generative model, both the [**D2GAN**](#References) and [**GMAN**](#References) methods extend the traditional GANs by combining extra discriminators.\n\n### Representative variants\n\nAs the latent vector 𝑧 of the generator is highly unstructured, [**InfoGAN**](#References) proposes another latent code 𝑐 to extract the significant structured features of the actual data space. In [**CGANs**](#References), the generator and discriminator are conditioned on additional information, such as class labels or data from other modalities, to generate samples conditioned on specific attributes.","metadata":{}},{"cell_type":"markdown","source":"## Multimodal Models\n\n### Text Audio Generation ###\n\nSimilar to vision language generation, text audio generation also requires an understanding of **cross-modal interaction**. The goal is to synthesize a segment of audio that aligns with the textual instructions given. This is commonly known as **text-to-speech (TTS)** synthesis. Over the years, several models have been developed that leverage deep learning techniques for TTS. \n\n#### Text Audio Encoders ###\n\nThe encoder in a text audio generation model is responsible for converting the input text into a suitable representation for the decoder to use. Traditional TTS systems often used hand-crafted features and concatenative speech synthesis. However, with the advent of deep learning, end-to-end models like [**Tacotron and Transformer TTS**](#References) have gained popularity. These models utilize character or phoneme sequences as input and generate a mel-spectrogram, which is then converted into waveform by a vocoder.\n\n#### Text Audio Decoders ### \n\nWaveNet was the first model to successfully generate raw speech waveforms. Following its success, many models have been proposed, including Parallel WaveGAN and MelGAN which further accelerated the synthesis process by employing a GAN-based architecture. More recently, models such as [**HiFi-GAN and Parallel WaveGAN v2**](#References) have been proposed that not only improve the quality of generated speech but also ensure real-time performance on CPUs. \n\n### Text Graph Generation \n\nText graph generation is an essential component in the process of creating visual representations from textual descriptions. \n\n#### Text Graph Encoders ####\n\nSimilar to text audio and vision language generation, the first step in text graph generation involves the conversion of the input text into a suitable representation using an encoder. This is typically done using language models like [**GPT or transformer-based models**](#References) that can effectively capture the semantic meaning of the text. \n\n#### Text Graph Decoders ####\n\nOnce the text is encoded, a decoder is used to generate a visual graph based on the encoded representation. Different models have been proposed for this purpose, such as [**GNN-Transformer and GraphRNN**](#References) which generate a graph step-by-step by predicting the adjacencies of new nodes based on the previous graph structure and the encoded text. \n\n### Text Code Generation ###\n\nText code generation or code synthesis is a task where a piece of code is generated based on a given text description. \n\n#### Text Code Encoders ####\n\nThe encoding process in text code generation models involves converting the input text into a suitable representation. For this purpose, models like [**CodeBERT**](#References) and **GPT-C** have been proposed, which are trained on large-scale code datasets and can generate context-aware representations. \n\n#### Text Code Decoders ####\n\nThe decoding process involves generating a piece of code based on the encoded representation. A variety of decoders have been used for this purpose, including autoregressive models, transformer-based models, and even sequence-to-sequence models. A prominent model in this domain is **OpenAI's Codex**, which uses a transformer-based architecture and has been trained on a vast amount of publicly available code. Codex can generate functional code from a given text prompt, making it a significant advancement in the field of AI-generated code.\n\n\n### Text-Music Generation\n\nDeep cross-modal correlation learning architecture for audio and lyrics, where intermodal canonical correlation analysis is used to calculate the similarity of temporal structures between audio and lyrics. Some works focus on generating text information, such as descriptions and captions, given the audio as input. \n\n\n### Text Graph Generation ###\n\nNatural language text is intrinsically vague as it carries various redundant information and is also weakly organized in logic. Knowledge graph (KG) is structural meaning representation which reflects relationships among semantic internal states as graph structure in a language processing system. Text-graph generation is an essential multi-modal topic which could largely free the potential of NLP systems. \n\n#### Text To Knowledge Graph Generation ####\n\nLi et al. treat text-to-KG construction as a process of knowledge graph completion (KGC), where missing terms are progressively covered by inference. [**KG-BERT**](#References) utilizes the power of pre-trained language models to capture more contextualized information during KGC. \n\n#### Knowledge Graph To Text Generation ####\nGTR-LSTM is a sequence-to-sequence encoder-decoder framework which generates text from linearized KG triples. [**DUALENC**](#References), a dual encoding model, is proposed to bridge the structural discrepancy between input graph and output text.\n\n### Semantic Parsing ###\n\nEarly works formulate semantic parsing as sequence-to-sequence generation problems. However, Lyu et al. model semantic parsing as a graph prediction problem by expressing AMR as a root labeled directed acyclic graph (DAG).\n\n### Text Molecule Generation ###\nText2Mol is a cross-modal information retrieval system to retrieve molecule graph based on language description. [**MolT5**](#References) proposes a self-supervised learning framework for text-conditioned de-novo molecule generation and molecule captioning.\n\n### Text Code Generation ###\nText-code generation aims to automatically generate valid programming code from natural language description or provide coding assist. LLMs have recently exhibited great potential in programming language (PL) code generation from natural language (NL) descriptions.\n\n### Text-conditioned Programming Code Generation ###\nCodeBERT is a bimodal Transformer-based pre-trained text-code model which could capture the semantic connection between NL and PL. [**CuBERT**](#References) shares similar model architecture with CodeBERT meanwhile it is not required to perform sentence separation between the natural-language description of a function and its body for sentence-pair representation.\n\n### Interactive Programming System ###\nCODEGEN propose a multi-turn program synthesis approach which factorizes program synthesis conditioned on a single complicated NL specification into progressive generation controlled by a series of user intents. [**TDUIF**](#References) extends interactive programming framework by formalizing the user intent and providing more understandable user feedback.","metadata":{}},{"cell_type":"markdown","source":"***(Note: For a full list of available models, applications & benchmarks we can review the appendix with the references)***","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"\n 🌟 2.4 Reviewing Applications in Open Source\n
","metadata":{}},{"cell_type":"markdown","source":"Most of the mentioned methods & techniques have been vastly utilized in **open source technologies**. The most popular place for releasing models relating to Generative AI (but not only) is Hugging Face. Below we have extracted the most popular models for each category together with several metrics. The snapshot is borrowed from [**Hugging Face Models**](https://huggingface.co/models) on **1 July, 2023**","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install --upgrade kaggle\n!pip install GoogleNews\n!pip install --upgrade accelerate\n!pip install --upgrade transformers\n!pip install newsapi-python\n!pip install transformers torch\n!pip install tweepy\n!pip install praw\n!pip install google-api-python-client","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:43:55.115355Z","iopub.execute_input":"2023-07-12T03:43:55.116011Z","iopub.status.idle":"2023-07-12T03:45:46.871442Z","shell.execute_reply.started":"2023-07-12T03:43:55.115979Z","shell.execute_reply":"2023-07-12T03:45:46.870200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Read the CSV file into a dataframe\nhuggingface_models_df = pd.read_csv('/kaggle/input/huggingface-models/Hugging Face Best Models (07-2023).csv')\n\n# Remove commas from the 'Downloads' column and convert it to integer\nhuggingface_models_df['Downloads'] = huggingface_models_df['Downloads'].str.replace(',', '').astype(int)\n\n# Review snapshot of best models\nhuggingface_models_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:45:46.873481Z","iopub.execute_input":"2023-07-12T03:45:46.873790Z","iopub.status.idle":"2023-07-12T03:45:46.932544Z","shell.execute_reply.started":"2023-07-12T03:45:46.873760Z","shell.execute_reply":"2023-07-12T03:45:46.931712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above dataframe output, we have displayed the best performing models on Hugging Face, by **Type, Task, Model, Downloads and Liked metrics**. To get a better understanding of what is occuring as of recently, and more insights lets visualize the information.","metadata":{}},{"cell_type":"code","source":"from bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet\n\noutput_notebook()\n\n# Colors from the new color palette (We are using Bain Palette)\ncolor_palette = [\n '#8A0F0F', # R900 - Blood Clot\n '#CC0000', # R700 - Bill\n '#D63333', # R500 - Lip Smacker\n '#E16666', # R250 - Sunset\n '#EB9999', # R100 - Salmon\n '#F5CDCC', # R080 - Sunburn\n '#FCE3E3' # R020 - Rosie\n]\n\nmodels = huggingface_models_df['Type'].value_counts().sort_values(ascending=False).reset_index()\nmodels.columns = ['Type', 'Count']\n\n# Create a 'Color' column in the models DataFrame\nmodels['Color'] = color_palette * (len(models) // len(color_palette)) + color_palette[:len(models) % len(color_palette)]\n\n# Create a ColumnDataSource from the models DataFrame\nsource = ColumnDataSource(models)\n\n# Create the Bokeh figure\np = figure(\n width=800,\n height=600,\n toolbar_location=None,\n title=\"Counts of Models by Type\",\n y_range=models['Type'],\n min_border_top=50, # Add space between title and chart\n background_fill_color='#ffffff' # White color\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Count'\np.yaxis.axis_label = 'Type'\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars\nbars = p.hbar(\n y='Type',\n right='Count',\n height=0.5,\n source=source,\n color='Color'\n)\n\n# Create a label set\nlabels = LabelSet(x='Count', y='Type', text='Count', level='glyph', \n x_offset=5, y_offset=-10, source=source, \n text_color=\"black\", text_font=\"Arial\", text_font_size=\"10pt\")\n\np.add_layout(labels)\n\n# Customize the figure\np.title.text_font_size = '20px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:45:46.933935Z","iopub.execute_input":"2023-07-12T03:45:46.934243Z","iopub.status.idle":"2023-07-12T03:45:47.632101Z","shell.execute_reply.started":"2023-07-12T03:45:46.934217Z","shell.execute_reply":"2023-07-12T03:45:47.631072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From what we can see above the model types are heavily **dominated by NLP**, and then by **Computer Vision** - represented by 12, and 8 respectively. Important to note that **Multimodal Methods** are also increasing at fast rate.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.colors as mcolors\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet, NumeralTickFormatter\n\noutput_notebook()\n\ntop_downloads = huggingface_models_df.sort_values('Downloads', ascending=False)[:10].reset_index(drop=True)\n\n# Normalize Downloads\ntop_downloads['Downloads_norm'] = (top_downloads['Downloads'] - top_downloads['Downloads'].min()) / (top_downloads['Downloads'].max() - top_downloads['Downloads'].min())\n\n# Define the color palette as a matplotlib colormap\ncmap = plt.get_cmap('Oranges')\n\n# Assign colors to the normalized downloads values\ntop_downloads['Color'] = [mcolors.rgb2hex(cmap(x)) for x in top_downloads['Downloads_norm']]\n\n# Format Downloads for labels\ntop_downloads['Downloads_str'] = top_downloads['Downloads'].apply(lambda x: '{:,}'.format(x))\n\n# Truncate Model names to a maximum of 40 characters\ntop_downloads['Model'] = top_downloads['Model'].apply(lambda x: (x[:20] + '...') if len(x) > 20 else x)\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(top_downloads)\n\n# Create the Bokeh figure\np = figure(\n width=1100, # Increase the width to make the chart wider\n height=800,\n toolbar_location=None,\n title=\"Top 10 Models by Downloads\",\n x_range=(0, top_downloads['Downloads'].max() * 1.1),\n y_range=list(reversed(top_downloads['Model'])),\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Downloads'\np.yaxis.axis_label = 'Model'\n\n# Format the x-axis ticks to be more readable\np.xaxis[0].formatter = NumeralTickFormatter(format=\"0,0\")\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars\nbars = p.hbar(\n y='Model',\n right='Downloads',\n height=0.5,\n source=source,\n color='Color'\n)\n\n# Create a label set\nlabels = LabelSet(x='Downloads', y='Model', text='Downloads_str', level='glyph', \n x_offset=5, y_offset=-10, source=source, \n text_color=\"black\", text_font=\"Arial\", text_font_size=\"12pt\")\n\np.add_layout(labels)\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:45:47.634393Z","iopub.execute_input":"2023-07-12T03:45:47.634711Z","iopub.status.idle":"2023-07-12T03:45:47.784163Z","shell.execute_reply.started":"2023-07-12T03:45:47.634683Z","shell.execute_reply":"2023-07-12T03:45:47.783246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.colors as mcolors\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet, NumeralTickFormatter\n\noutput_notebook()\n\ntop_liked = huggingface_models_df.sort_values('Liked', ascending=False)[:10].reset_index(drop=True)\n\n# Normalize Liked\ntop_liked['Liked_norm'] = (top_liked['Liked'] - top_liked['Liked'].min()) / (top_liked['Liked'].max() - top_liked['Liked'].min())\n\n# Define the color palette as a matplotlib colormap\ncmap = plt.get_cmap('Greens')\n\n# Assign colors to the normalized likes values\ntop_liked['Color'] = [mcolors.rgb2hex(cmap(x)) for x in top_liked['Liked_norm']]\n\n# Format Liked for labels\ntop_liked['Liked_str'] = top_liked['Liked'].apply(lambda x: '{:,}'.format(x))\n\n# Truncate Model names to a maximum of 40 characters\ntop_liked['Model'] = top_liked['Model'].apply(lambda x: (x[:37] + '...') if len(x) > 40 else x)\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(top_liked)\n\n# Create the Bokeh figure\np = figure(\n width=1000, \n height=800,\n toolbar_location=None,\n title=\"Top 10 Models by Likes\",\n x_range=(0, top_liked['Liked'].max() * 1.1),\n y_range=list(reversed(top_liked['Model'])),\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Likes'\np.yaxis.axis_label = 'Model'\n\n# Format the x-axis ticks to be more readable\np.xaxis[0].formatter = NumeralTickFormatter(format=\"0,0\")\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars\nbars = p.hbar(\n y='Model',\n right='Liked',\n height=0.5,\n source=source,\n color='Color'\n)\n\n# Create a label set\nlabels = LabelSet(x='Liked', y='Model', text='Liked_str', level='glyph', \n x_offset=5, y_offset=-10, source=source, \n text_color=\"black\", text_font=\"Arial\", text_font_size=\"12pt\")\n\np.add_layout(labels)\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\np.xgrid.grid_line_color = None # Remove grid lines\np.ygrid.grid_line_color = None\np.background_fill_color = '#FFFFFF' # White color\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:45:47.785548Z","iopub.execute_input":"2023-07-12T03:45:47.785975Z","iopub.status.idle":"2023-07-12T03:45:47.938524Z","shell.execute_reply.started":"2023-07-12T03:45:47.785930Z","shell.execute_reply":"2023-07-12T03:45:47.937571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the two charts, we can derive the following: \n \na) **Automatic Speech Recognition Model** (especially *jonatasgrosman/wav2vec2-large-xlsr-53-english*) is the most popular model in hugging face, with **64,679,679 downloads**).\n\nb) **Fill Mask Model** (*bert-base-uncased* is the second most popular open source model, with **46,409,423 downloads**).\n\nc) By far the most loved model is the **Text-to-Image** *runwayml/stable-diffusion-v1-5*, with **8380 liked reactions**. \n\nIn a way or another, all these are involved with Generative AI - especially the stable-diffusion model!","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":" \n 🌍 2.5 The Open Source Large Language Model Landscape\n
","metadata":{}},{"cell_type":"markdown","source":"**In the rapidly evolving world of large language models (LLMs) and chatbots, it can be a daunting task to identify truly groundbreaking developments and the latest state-of-the-art models.** Amidst the influx of new releases, some boasting impressive capabilities, it's crucial to have a system to differentiate true progress from mere hype. \n\n**The open-source community conducts robust evaluations on LLMs using four key benchmarks from the Eleuther AI Language Model Evaluation Harness.** This harness serves as a unified platform for assessing the performance of generative language models across an array of evaluation tasks. \n\nThe benchmarks comprise:\n\n1. **AI2 Reasoning Challenge (25-shot):** This evaluation is based on grade-school science questions.\n2. **HellaSwag (10-shot):** A commonsense inference test that is simple for humans, but often challenging for top-tier models.\n3. **MMLU (5-shot):** This is a multi-task test assessing a model's performance across 57 diverse tasks, such as elementary mathematics, US history, computer science, and law.\n4. **TruthfulQA (0-shot):** This test measures a model's tendency to propagate inaccuracies commonly found online.\n\nThe selection of these benchmarks is strategic; **they examine an array of reasoning abilities and general knowledge across multiple disciplines in both 0-shot and few-shot settings**. Therefore, they provide a comprehensive performance evaluation of the LLMs.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nLLMs = pd.read_csv('/kaggle/input/hugging-face-llm-leaderboard-07-2023/Hugging Face LLM Leaderboard (07-2023).csv')\n\nLLMs","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:45:47.939954Z","iopub.execute_input":"2023-07-12T03:45:47.940335Z","iopub.status.idle":"2023-07-12T03:45:47.971757Z","shell.execute_reply.started":"2023-07-12T03:45:47.940298Z","shell.execute_reply":"2023-07-12T03:45:47.970898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below we have some **Key Observations:**\n\n- The model '**tiiuae/falcon-40b-instruct**' had the **highest overall average score (63.2)**, and it also performed particularly well on the HellaSwag and MMLU tests. It is a finetuned model with 40 billion parameters and licensed under Apache 2.0.\n\n- The model '**timdettmers/guanaco-65b-merged**' comes second in the average score with 62.2. Despite having a larger parameter size (65 billion) than the highest-scoring model, it did not outperform it.\n\n- The **highest scoring model in the TruthfulQA test** was '**CalderaAI/30B-Lazarus**' with a score of 58.3, though its average score was 60.7.\n\n- '**tiiuae/falcon-40b**', a pretrained model with 40 billion parameters, performed notably well in the HellaSwag test (85.3), the **highest score in this category**.\n\n- Interestingly, some models with a lower parameter size, such as '**ausboss/llama-30b-supercot**' (30 billion) and '**digitous/Alpacino30b**' (30 billion), **outperformed larger models** in certain tests, indicating that **size is not always directly proportional to performance**.","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"\n 🔍 3. Hands on Research: Where Research is Heading According to Data \n
","metadata":{}},{"cell_type":"markdown","source":"The next step will be to review the direction in which Generative AI is heading, through basic **EDA and NLP techniques**. This will help us confirm or improve the findings in the literature review. Initially, we will start by using the **Arxiv metadata** and analyze some basic trends related to Generative AI. Since the keyword \"Generative AI\" in itself is not comprehensive enough, we have experimented and included keywords from the main models, methods and topics related in a substantial way to Generative AI. Below we start by filtering the data to include the appropriate topics and then at first display the number of papers over time published for these topics.","metadata":{}},{"cell_type":"markdown","source":"\n🔢 3.1 Number of Research Papers Each Month for the Past 2 Years \n
","metadata":{}},{"cell_type":"code","source":"import json\nimport pandas as pd\nfrom bokeh.io import output_notebook, show\nfrom bokeh.models import ColumnDataSource\nfrom bokeh.plotting import figure\nfrom bokeh.transform import linear_cmap\nimport warnings\n\n# Load the data\ndata = []\n\nwith open('/kaggle/input/arxiv/arxiv-metadata-oai-snapshot.json', 'r') as f:\n for line in f:\n data.append(json.loads(line))\n\ndf = pd.json_normalize(data)\n\n# Drop the unused columns\ndf = df.drop(columns=['submitter', 'journal-ref', 'doi', 'report-no', 'license', 'versions'])\n\n# Preprocessing - make all text lower case for consistent matching\ndf['title'] = df['title'].str.lower()\ndf['abstract'] = df['abstract'].str.lower()\n\n# Convert the update_date to datetime and filter for last two years\ndf['update_date'] = pd.to_datetime(df['update_date'])\nstart_date = '2021-05-01'\nend_date = '2023-06-26'\nmask = (df['update_date'] >= start_date) & (df['update_date'] <= end_date)\ndf = df.loc[mask]\n\n# Filter by keywords\nkeywords = [\n 'generative ai',\n 'generative adversarial networks',\n 'autoencoders',\n 'gpt-3',\n 'large language models',\n 'bert',\n 'diffusion models',\n 'Generative AI'\n 'Generative Adversarial Networks',\n 'GAN',\n 'Autoencoders'\n 'Variational Autoencoders',\n 'VAEs',\n 'Generative Models',\n 'Deep Generative Models',\n 'Image Generation',\n 'Text Generation',\n 'Synthetic Data',\n 'Data Augmentation',\n 'CycleGAN',\n 'StyleGAN'\n 'DCGAN',\n 'Deep Convolutional Generative Adversarial Networks',\n 'GAN Training Techniques',\n 'GAN Applications',\n 'GAN Variants',\n 'ProGAN',\n 'BigGAN',\n 'Conditional GANs',\n 'Pix2Pix',\n 'Neural Style Transfer',\n 'Unsupervised Learning',\n 'Music Generation',\n 'Voice Generation',\n 'Large Language Models (LLMs)',\n 'GPT-3',\n 'GPT',\n 'BERT',\n 'Transformer Models',\n 'Transformer-based Generative Models',\n 'Diffusion Models',\n 'Diffusion-based Generative Models',\n 'Data Diffusion',\n 'GPT-4 ',\n 'T5',\n 'ELECTRA',\n 'Sequence Generation',\n 'Text-to-Text Generation Models'\n 'Seq2Seq Models',\n 'Latent Variable Models',\n 'Generative Pre-training',\n 'Masked Language Models',\n 'Cross-Lingual Language Models (XLMs)',\n 'Transfer Learning in Generative Models', \n 'Zero-Shot Learning',\n 'Few-Shot Learning',\n 'Meta Learning',\n '3D Object Generation',\n 'Video Generation',\n 'Inverse Graphics Networks (IGNs)',\n 'Multi-Agent Generative Adversarial Networks',\n 'MAGANs',\n 'Style Transfer'\n]\nwith warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n df_filtered = df[df['title'].str.contains('|'.join(keywords)) | df['abstract'].str.contains('|'.join(keywords))]\n\ndf_filtered = df_filtered.copy()\ndf_filtered['month_year'] = df_filtered['update_date'].dt.to_period('M')\npapers_per_month = df_filtered['month_year'].value_counts().sort_index()\n\npapers_per_month_df = papers_per_month.reset_index()\npapers_per_month_df.columns = ['Month_Year', 'Count']\n\n# Convert 'Month_Year' column to string\npapers_per_month_df['Month_Year'] = papers_per_month_df['Month_Year'].astype(str)\n\n# Define the colors from the new color palette\ncolor_palette = [\n '#E6FCFF', # C020 - Cyanish\n '#B3F5FF', # C080 - Soap\n '#79E2F2', # C100 - Lagoon\n '#00C7E6', # C250 - Sea Breeze\n '#00B8D9', # C500 - Party\n '#00A3BF', # C700 - Fish Tank\n '#008DA6', # C900 - Doona\n]\n\n# Sort papers_per_month_df DataFrame by 'Month_Year'\npapers_per_month_df = papers_per_month_df.sort_values('Month_Year')\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(papers_per_month_df)\n\n# Create the Bokeh figure\np = figure(\n x_range=papers_per_month_df['Month_Year'],\n y_range=(0, papers_per_month_df['Count'].max() * 1.1),\n width=1000,\n height=800,\n toolbar_location=None,\n tools='',\n title=\"Number of Papers Published Per Month\"\n)\n\n# Mapper for the 'Count' column and the color palette\nmapper = linear_cmap(field_name='Count', palette=color_palette, low=min(papers_per_month_df['Count']), high=max(papers_per_month_df['Count']))\n\n# Add a bar glyph to the figure p\np.vbar(\n x='Month_Year',\n top='Count',\n width=0.9,\n source=source,\n fill_color=mapper\n)\n\n# Customizing Visualization\np.xgrid.grid_line_color = None\np.y_range.start = 0\np.y_range.end = papers_per_month_df['Count'].max() * 1.1\np.axis.minor_tick_line_color = None\np.outline_line_color = None\np.title.align = \"center\"\np.title.text_font_size = \"24px\"\np.xaxis.major_label_orientation = 1.0\n\nfrom bokeh.models.annotations import LabelSet\n\nlabels = LabelSet(x='Month_Year', y='Count', text='Count', level='glyph',\n x_offset=-13.5, y_offset=0, source=source, angle=0.785398) # 45 degrees\n\np.add_layout(labels)\n\n\nshow(p)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:45:47.973275Z","iopub.execute_input":"2023-07-12T03:45:47.973578Z","iopub.status.idle":"2023-07-12T03:48:32.511656Z","shell.execute_reply.started":"2023-07-12T03:45:47.973550Z","shell.execute_reply":"2023-07-12T03:48:32.510604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, towards the past 2 months the output of research papers in fields related to Generative AI has **more than doubled in comparison to 2 years ago**. This shows an exponential increase towards resources allocated to writing and developing towards such field of machine learning.","metadata":{}},{"cell_type":"markdown","source":"\n 📗 3.2 Applying Topic Modeling to the Relevant Data\n
","metadata":{}},{"cell_type":"markdown","source":"The next step would be to find a theme commonly found among most research papers towards Generative AI. For that we will utilize topic modeling with tools like spacy and gensim, where at first we will preprocess the data accordingly and then build an LDA model - which we will display in an interactive pyLDAvis chart.","metadata":{}},{"cell_type":"code","source":"%%capture\nimport spacy\nimport gensim\n\n# Your code block is here\nnlp = spacy.load('en_core_web_lg')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-12T03:48:32.513448Z","iopub.execute_input":"2023-07-12T03:48:32.514195Z","iopub.status.idle":"2023-07-12T03:48:49.093561Z","shell.execute_reply.started":"2023-07-12T03:48:32.514157Z","shell.execute_reply":"2023-07-12T03:48:49.092455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Applies lemmatization to the given text by reducing words to their base or root form\ndef lemmatize_stemming(text):\n return [token.lemma_ for token in nlp(text)]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:48:49.094850Z","iopub.execute_input":"2023-07-12T03:48:49.095672Z","iopub.status.idle":"2023-07-12T03:48:49.100343Z","shell.execute_reply.started":"2023-07-12T03:48:49.095634Z","shell.execute_reply":"2023-07-12T03:48:49.099599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Performs preprocessing on the given text by removing stopwords and filtering out short lemmas, returning the resulting list of lemmas.\ndef preprocess(text):\n result = []\n for token in nlp(text):\n if token.lemma_ not in gensim.parsing.preprocessing.STOPWORDS and len(token.lemma_) > 3:\n result.append(token.lemma_)\n return result","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:48:49.104038Z","iopub.execute_input":"2023-07-12T03:48:49.104575Z","iopub.status.idle":"2023-07-12T03:48:49.113222Z","shell.execute_reply.started":"2023-07-12T03:48:49.104546Z","shell.execute_reply":"2023-07-12T03:48:49.112456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Running the previous steps on the abstract\nprocessed_docs = df_filtered['abstract'].map(preprocess)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T03:48:49.114717Z","iopub.execute_input":"2023-07-12T03:48:49.115444Z","iopub.status.idle":"2023-07-12T04:00:18.229816Z","shell.execute_reply.started":"2023-07-12T03:48:49.115407Z","shell.execute_reply":"2023-07-12T04:00:18.228820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gensim.corpora as corpora\n\n# Create Dictionary\ndictionary = corpora.Dictionary(processed_docs)\n\n# Filter out words that occur less than 20 documents, or more than 50% of the documents.\ndictionary.filter_extremes(no_below=20, no_above=0.5)\n\n# Create Corpus\ncorpus = [dictionary.doc2bow(doc) for doc in processed_docs]\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:00:18.231359Z","iopub.execute_input":"2023-07-12T04:00:18.231687Z","iopub.status.idle":"2023-07-12T04:00:21.244963Z","shell.execute_reply.started":"2023-07-12T04:00:18.231658Z","shell.execute_reply":"2023-07-12T04:00:21.243911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from gensim.models import LdaModel\n\n# Build LDA model (The following tuning setup yielded the highest results)\nlda_model = LdaModel(corpus=corpus, id2word=dictionary, num_topics=20, random_state=41,\n update_every=20, chunksize=30, passes=20,\n alpha='auto', per_word_topics=True)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:00:21.246247Z","iopub.execute_input":"2023-07-12T04:00:21.246546Z","iopub.status.idle":"2023-07-12T04:03:22.640279Z","shell.execute_reply.started":"2023-07-12T04:00:21.246522Z","shell.execute_reply":"2023-07-12T04:03:22.639238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Unstructured Output of Topics
","metadata":{}},{"cell_type":"markdown","source":"Lets first review the unstructured output we derived from running the LDA model for the given topics - then we will proceed to review, visualize and interpret the findings accordingly!","metadata":{}},{"cell_type":"code","source":"# Print the Keyword in the 20 topics\nfrom gensim.models import LdaModel\nfrom pprint import pprint\n\n\n# Print the topics\npprint(lda_model.print_topics())","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-12T04:03:22.641586Z","iopub.execute_input":"2023-07-12T04:03:22.641910Z","iopub.status.idle":"2023-07-12T04:03:22.652393Z","shell.execute_reply.started":"2023-07-12T04:03:22.641862Z","shell.execute_reply":"2023-07-12T04:03:22.651209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can see several displayed groups for an optimal number of 20 topics related to Generative AI. First lets review how the perplexity score and coherence score fare in the given algorithm and then dig deeper to finding insights on the Topic Modeling for the field.","metadata":{}},{"cell_type":"markdown","source":" Measuring Model Performance via Perplexity and Coherence Score
","metadata":{}},{"cell_type":"markdown","source":"To better review model performance we can first check perplexity score, which will help us review the optimal way to organize the keywords found and balance accordinly with the topics given - from there we can gain better understanding and interpretability on the main themes across Generative AI in research papers.","metadata":{}},{"cell_type":"code","source":"from gensim.models.coherencemodel import CoherenceModel\n\n# Compute Perplexity\nprint('\\nPerplexity: ', lda_model.log_perplexity(corpus)) # a measure of how good the model is. lower the better.\n\n# Compute Coherence Score\ncoherence_model_lda = CoherenceModel(model=lda_model, texts=processed_docs, dictionary=dictionary, coherence='c_v')\ncoherence_lda = coherence_model_lda.get_coherence()\nprint('\\nCoherence Score: ', coherence_lda)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:03:22.653947Z","iopub.execute_input":"2023-07-12T04:03:22.654366Z","iopub.status.idle":"2023-07-12T04:03:51.761179Z","shell.execute_reply.started":"2023-07-12T04:03:22.654318Z","shell.execute_reply":"2023-07-12T04:03:51.759932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on other comparable benchmarks, a perplexity score of **-9.16** can suggest that the **model performs well on unseen documents**\n\nGiven the coherence score of **0.4595** suggests that the model generates topics with a **moderate level of coherence**, relating to the quality and interpretability of the topics generated by the LDA model.","metadata":{}},{"cell_type":"markdown","source":" Visualizing the Output of the Model in an Interactive Way
","metadata":{}},{"cell_type":"markdown","source":"To better visualize the output we will use PyLDAvis, which is a Python library that provides an interactive visualization tool for exploring and interpreting topic models, such as Latent Dirichlet Allocation (LDA) in a more accesible manner. This will help us explore the, themes behind Generative AI theme in research papers.","metadata":{}},{"cell_type":"code","source":"def format_topics_sentences(ldamodel=None, corpus=corpus, texts=processed_docs):\n # Init output\n sent_topics_list = []\n\n # Get main topic in each document\n for i, row_list in enumerate(ldamodel[corpus]):\n row = row_list[0] if ldamodel.per_word_topics else row_list \n row = sorted(row, key=lambda x: (x[1]), reverse=True)\n # Get the Dominant topic, Perc Contribution and Keywords for each document\n for j, (topic_num, prop_topic) in enumerate(row):\n if j == 0: # => dominant topic\n wp = ldamodel.show_topic(topic_num)\n topic_keywords = \", \".join([word for word, prop in wp])\n sent_topics_list.append([int(topic_num), round(prop_topic,4), topic_keywords])\n else:\n break\n\n sent_topics_df = pd.DataFrame(sent_topics_list, columns=['Dominant_Topic', 'Perc_Contribution', 'Topic_Keywords'])\n\n # Add original text to the end of the output\n contents = pd.Series(texts)\n sent_topics_df = pd.concat([sent_topics_df, contents.rename('Original_Text')], axis=1)\n\n return(sent_topics_df)\n\n\ndf_topic_sents_keywords = format_topics_sentences(ldamodel=lda_model, corpus=corpus, texts=processed_docs)\n\n# Format\ndf_dominant_topic = df_topic_sents_keywords.reset_index()\ndf_dominant_topic.columns = ['Document_No', 'Dominant_Topic', 'Topic_Perc_Contrib', 'Keywords', 'Original_Text']\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:03:51.763436Z","iopub.execute_input":"2023-07-12T04:03:51.764145Z","iopub.status.idle":"2023-07-12T04:04:45.809920Z","shell.execute_reply.started":"2023-07-12T04:03:51.764097Z","shell.execute_reply":"2023-07-12T04:04:45.809084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Adding dictionary to the processed docs\nid2word = gensim.corpora.Dictionary(processed_docs)\n\n# Filter out words that occur less than 20 documents, or more than 50% of the documents.\nid2word.filter_extremes(no_below=20, no_above=0.5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:04:45.811661Z","iopub.execute_input":"2023-07-12T04:04:45.812412Z","iopub.status.idle":"2023-07-12T04:04:47.779748Z","shell.execute_reply.started":"2023-07-12T04:04:45.812374Z","shell.execute_reply":"2023-07-12T04:04:47.778649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\n\n\n#import pyLDAvis.gensim_models\nimport pyLDAvis.gensim\nfrom IPython.display import display\n\n\n\n# Visualize the topics\npyLDAvis.enable_notebook()\nvis = pyLDAvis.gensim.prepare(lda_model, corpus, id2word, sort_topics=False)\nvis","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:04:47.781072Z","iopub.execute_input":"2023-07-12T04:04:47.781375Z","iopub.status.idle":"2023-07-12T04:04:57.882764Z","shell.execute_reply.started":"2023-07-12T04:04:47.781348Z","shell.execute_reply":"2023-07-12T04:04:57.881420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### GPT-4 Based Interpretation for Topic Group ","metadata":{}},{"cell_type":"markdown","source":"| Topic No. | Description |\n| --- | --- |\n| 0 | Methods and algorithms used in autoencoders, focusing on distribution and data sampling, possibly about generative modeling. |\n| 1 | Kernel methods, space, and convergence problems, possibly related to research into kernel methods in machine learning or optimization algorithms. |\n| 2 | Generative adversarial networks (GANs), a class of AI algorithms used in unsupervised machine learning. |\n| 3 | Theory of phase spaces, symmetries, and lattices, likely relating to physics. |\n| 4 | Mathematical methods related to algebra, functions, and polynomials. |\n| 5 | Computer vision, focusing on views, flow, shape synthesis, distance measurement, and alignment. |\n| 6 | Language processing with a focus on large datasets and diffusion techniques. |\n| 7 | Machine translation, specifically into English. |\n| 8 | Graph theory, tree structures, and bindings. |\n| 9 | Quantum theory, possibly relating to quantum computing. |\n| 10 | Testing, categories, types, and attempts at relevance. |\n| 11 | Operator theory in Hilbert spaces, a subject in functional analysis. |\n| 12 | Time dynamics, equations, solutions, boundary evolution, and related concepts, possibly related to dynamical systems. |\n| 13 | Research tools, analysis, topic recognition, and motion processing. |\n| 14 | Materials, artificial intelligence, and fields, possibly physics or electronics-related. |\n| 15 | Cosmological theories, energy, gravity, and the universe - possibly relating to scientific computing or simulations. |\n| 16 | Questions, reasoning, biases, feedback, and causal relations, suggesting research in understanding and improving AI reasoning capabilities. |\n| 17 | Users, visualization, instructions, findings, search algorithms, and policy design. |\n| 18 | Corpus linguistics or computational linguistics. |\n| 19 | Tasks, training, datasets, and performance, indicating research into improving training techniques and measuring AI performance. |","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.colors as mcolors\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet\n\noutput_notebook()\n\n# Create a dataframe with topic counts\ntopic_counts = df_dominant_topic['Dominant_Topic'].value_counts().reset_index()\ntopic_counts.columns = ['Topic Number', 'Document Count']\n\n# Normalize Document Count\ntopic_counts['Document Count_norm'] = (topic_counts['Document Count'] - topic_counts['Document Count'].min()) / (topic_counts['Document Count'].max() - topic_counts['Document Count'].min())\n\n# Define the color palette as a matplotlib colormap\ncmap = plt.get_cmap('Blues')\n\n# Assign colors to the normalized document count values\ntopic_counts['Color'] = [mcolors.rgb2hex(cmap(x)) for x in topic_counts['Document Count_norm']]\n\n# Sort values by document count\ntopic_counts.sort_values('Document Count', ascending=False, inplace=True)\ntopic_counts.reset_index(drop=True, inplace=True)\n\n# Convert the 'Topic Number' to string\ntopic_counts['Topic Number'] = topic_counts['Topic Number'].astype(str)\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(topic_counts)\n\n# Create the Bokeh figure\np = figure(\n width=900,\n height=800,\n toolbar_location=None,\n title=\"Distribution of Dominant Topics Across Documents\",\n x_range=(0, topic_counts['Document Count'].max() * 1.1),\n y_range=list(reversed(topic_counts['Topic Number'])),\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Document Count'\np.yaxis.axis_label = 'Topic Number'\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars\nbars = p.hbar(\n y='Topic Number',\n right='Document Count',\n height=0.5,\n source=source,\n color='Color'\n)\n\n# Create a label set\nlabels = LabelSet(x='Document Count', y='Topic Number', text='Document Count', level='glyph', \n x_offset=5, y_offset=-10, source=source, \n text_color=\"black\", text_font=\"Arial\", text_font_size=\"12pt\")\n\np.add_layout(labels)\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:04:57.885244Z","iopub.execute_input":"2023-07-12T04:04:57.885575Z","iopub.status.idle":"2023-07-12T04:04:58.113843Z","shell.execute_reply.started":"2023-07-12T04:04:57.885541Z","shell.execute_reply":"2023-07-12T04:04:58.112936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can discern, the **Latent Dirichlet Allocation (LDA)** model has identified **20 different topics** in the given text corpus. These topics highlight a range of subjects, with the most dominant ones being:\n\n1. **Topic 6**: A significant focus is on **language processing**, particularly involving large datasets and diffusion techniques, which points to advancements in handling and processing extensive linguistic data.\n\n2. **Topic 19**: This topic relates to **AI performance and training techniques**. It emphasizes tasks, training, datasets, and performance, suggesting that considerable attention is being given to refine training techniques and improve AI performance.\n\n3. **Topic 1**: Kernel methods, space, and convergence problems are the focus here, indicating extensive research into **kernel methods** in machine learning and optimization algorithms.\n\n4. **Topic 18**: It appears to be centred around **corpus linguistics or computational linguistics**, suggesting an interest in language data and computational methods for linguistic analysis.\n\n5. **Topic 11**: It involves **operator theory** in Hilbert spaces, indicating a deep interest in functional analysis.\n\n6. **Topic 0**: Methods and algorithms used in **autoencoders** form the crux of this topic. It indicates the ongoing exploration of generative modeling, particularly involving distribution and data sampling.\n\n\nThese insights reflect the current trends and focus areas in the dataset, which appears to include a mix of machine learning, natural language processing and quantum computing.","metadata":{}},{"cell_type":"markdown","source":" \n 🌐 3.3 Shifting to Google Scholar for a More Comprehensive Analysis\n
","metadata":{}},{"cell_type":"code","source":"df_filtered.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:04:58.115163Z","iopub.execute_input":"2023-07-12T04:04:58.116186Z","iopub.status.idle":"2023-07-12T04:04:58.138640Z","shell.execute_reply.started":"2023-07-12T04:04:58.116148Z","shell.execute_reply":"2023-07-12T04:04:58.137941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Identifying Key Researchers
","metadata":{}},{"cell_type":"markdown","source":"The data we have gathered here is a mix of SerpAPI, manual copy pasting and scraping. As we can see from the code comments below and description we have filtered and reviewed (while manually double checking) the top researchers for the field of Generative AI, as based on citations and papers published on the topic.","metadata":{}},{"cell_type":"code","source":"from bokeh.io import show, output_notebook\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, FactorRange\nfrom bokeh.transform import factor_cmap\nfrom collections import Counter\n\noutput_notebook()\n\n# Create a list of all authors\nauthors_list = []\nfor authors in df_filtered['authors_parsed']:\n for author in authors:\n authors_list.append(author[0])\n\n# Count the number of publications for each author\nauthor_counts = Counter(authors_list)\n\n# Get the top 10 most published authors\nmost_published_authors = author_counts.most_common(10)\n\n# Create a DataFrame for the most published authors\nauthors_df = pd.DataFrame(most_published_authors, columns=[\"Author\", \"Publications\"])\n\n# Define your color palette\ncolor_palette = [\"#403294\", \"#5243AA\", \"#6554C0\", \"#8777D9\", \"#998DD9\", \"#C0B6F2\", \"#EAE6FF\"]\n\n# Create a ColumnDataSource\nsource = ColumnDataSource(data=dict(\n authors=authors_df['Author'].tolist(),\n publications=authors_df['Publications'].tolist(),\n))\n\n# Create a new plot\np = figure(y_range=authors_df['Author'].tolist(), height=800, width=1000, \n title=\"Top 10 most published authors in recent years\",\n x_axis_label='Publications', y_axis_label='Author')\n\n# Add a hbar glyph to the plot\np.hbar(y='authors', right='publications', height=0.5, source=source, color=factor_cmap('authors', palette=color_palette, factors=authors_df['Author'].tolist()))\n\n# Customize the plot\np.xgrid.grid_line_color = None\np.y_range.range_padding = 0.1\n\nshow(p)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:04:58.139783Z","iopub.execute_input":"2023-07-12T04:04:58.140592Z","iopub.status.idle":"2023-07-12T04:04:58.392237Z","shell.execute_reply.started":"2023-07-12T04:04:58.140563Z","shell.execute_reply":"2023-07-12T04:04:58.391111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"However this doesnt say much - since being published is different then being impactful. Also we hit the first roadblock, where we have similar names used for research which presents a challenge to find the actual scholars.\n\nSince the current metadata limits us, we will need to use external data. One way we can impact is through citations. For that lets review scraped data from Google Scholar (Note: Using external APIs can lead to IP ban, since mass scraping violates Google's Policy - therefore we have manually requested to use only a small portion of data for our research)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.colors as mcolors\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet\n\noutput_notebook()\n\n# Read the data\nscholar_data = pd.read_csv('/kaggle/input/google-scholars/Google_Scholars.csv')\n\n# Calculate total citations for each author\nscholar_data['Total Citations'] = scholar_data['2021 (Citations)'] + scholar_data['2022 (Citations)'] + scholar_data['2023 (Citations)']\n\n# Sort the dataframe by total citations and keep only the top 10\nscholar_sorted = scholar_data.sort_values(by='Total Citations', ascending=False).head(10)\n\n# Normalize Total Citations\nscholar_sorted['Citations_norm'] = (scholar_sorted['Total Citations'] - scholar_sorted['Total Citations'].min()) / (scholar_sorted['Total Citations'].max() - scholar_sorted['Total Citations'].min())\n\n# Define the color palette as a matplotlib colormap\ncmap = plt.get_cmap('tab10')\n\n# Assign colors to the normalized citations values\nscholar_sorted['Color'] = [mcolors.rgb2hex(cmap(x)) for x in scholar_sorted['Citations_norm']]\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(scholar_sorted)\n\n# Create the Bokeh figure\np = figure(\n width=900,\n height=800,\n toolbar_location=None,\n title=\"Top 10 Authors with Most Total Citations\",\n x_range=(0, scholar_sorted['Total Citations'].max() * 1.1),\n y_range=list(scholar_sorted['Author']), # Adjusted to be in descending order\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Total Citations'\np.yaxis.axis_label = 'Author'\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars\nbars = p.hbar(\n y='Author',\n right='Total Citations',\n height=0.5,\n source=source,\n color='Color'\n)\n\n# Create a label set\nlabels = LabelSet(x='Total Citations', y='Author', text='Total Citations', level='glyph', \n x_offset=5, y_offset=-10, source=source, \n text_color=\"black\", text_font=\"Arial\", text_font_size=\"12pt\")\n\np.add_layout(labels)\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:04:58.395523Z","iopub.execute_input":"2023-07-12T04:04:58.395844Z","iopub.status.idle":"2023-07-12T04:04:58.596280Z","shell.execute_reply.started":"2023-07-12T04:04:58.395816Z","shell.execute_reply":"2023-07-12T04:04:58.595124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.colors as mcolors\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet, Legend, LegendItem\nfrom bokeh.palettes import Category10\n\noutput_notebook()\n\n# Melt the dataframe to long format\ndf_melted = scholar_data.melt(id_vars=['Author', 'Affiliation'], value_vars=['2021 (Citations)', '2022 (Citations)', '2023 (Citations)'], var_name='Year', value_name='Citations')\n\n# Clean the Year column to keep only the year\ndf_melted['Year'] = df_melted['Year'].str[:4]\n\n# Create the Bokeh figure\np = figure(\n width=900,\n height=800,\n toolbar_location=None,\n title=\"Citation counts over the years for each author\",\n x_range=list(df_melted['Year'].unique()),\n min_border_top=100 # Add space between title and chart\n)\n\n# Create a color palette\npalette = Category10[10]\n\n# Initialize a list to store the legend items\nlegend_items = []\n\n# Plot the line for each author\nfor i, author in enumerate(df_melted['Author'].unique()):\n data = df_melted[df_melted['Author'] == author]\n line = p.line(data['Year'], data['Citations'], line_width=2, color=palette[i%10])\n legend_items.append(LegendItem(label=author, renderers=[line]))\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Year'\np.yaxis.axis_label = 'Citations'\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the legend\nlegend = Legend(items=legend_items, location=\"center\")\np.add_layout(legend, 'right')\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:04:58.597987Z","iopub.execute_input":"2023-07-12T04:04:58.598329Z","iopub.status.idle":"2023-07-12T04:05:10.999748Z","shell.execute_reply.started":"2023-07-12T04:04:58.598301Z","shell.execute_reply":"2023-07-12T04:05:10.997971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.colors as mcolors\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet\nfrom bokeh.palettes import Category20\n\noutput_notebook()\n\n# Calculate total citations for each author\nscholar_data['Total Citations'] = scholar_data[['2021 (Citations)', '2022 (Citations)', '2023 (Citations)']].sum(axis=1)\n\n# Group by affiliation and calculate total citations for each\ngrouped = scholar_data.groupby('Affiliation')['Total Citations'].sum().reset_index()\n\n# Sort the dataframe by Total Citations in descending order\ngrouped_sorted = grouped.sort_values('Total Citations', ascending=False)\n\n# Create the Bokeh figure\np = figure(\n width=900,\n height=800,\n toolbar_location=None,\n title=\"Institutions With Max Nr of Citations From Top Gen AI Scientists\",\n x_range=(0, grouped_sorted['Total Citations'].max() * 1.1),\n y_range=list(reversed(grouped_sorted['Affiliation'])),\n min_border_top=100 # Add space between title and chart\n)\n\n# Create a color palette\npalette = Category20[len(grouped_sorted)]\n\n# Assign the color palette to a new column in the DataFrame\ngrouped_sorted['color'] = palette\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(grouped_sorted)\n\n# Add the horizontal bars\nbars = p.hbar(\n y='Affiliation',\n right='Total Citations',\n height=0.5,\n source=source,\n color='color'\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Total Citations'\np.yaxis.axis_label = 'Affiliation'\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:11.002543Z","iopub.execute_input":"2023-07-12T04:05:11.003447Z","iopub.status.idle":"2023-07-12T04:05:11.226671Z","shell.execute_reply.started":"2023-07-12T04:05:11.003385Z","shell.execute_reply":"2023-07-12T04:05:11.225938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we've seen, having more papers published doesnt mean more impact in the field of Generative AI. Below are some interesting findings from the authors data:\n\n1. **Citation count is increasing every year for each author**, indicating that their work is continuously gaining recognition and being used as a reference by others. This suggests the ongoing relevance and significance of their research.\n\n2. Some researchers, such as **Luke Zettlemoyer, Jianfeng Gao, and Yu Qiao, are leading in terms of citation counts**, suggesting that their work may have had significant influence in the field of generative AI.\n\n3. The citation counts vary significantly among different researchers, implying that **the impact and reach of their work vary**. Some researchers like Luke Zettlemoyer and Jianfeng Gao have citation counts in the tens of thousands, while others, like Raviraj Joshi, have citation counts in the hundreds. This might be due to the difference in their research areas, publication count, or the span of their careers.\n\n4. Among affiliations, **Microsoft and University of Washington have higher total citations from top scientists and researchers in the field** than others, suggesting that these institutions may have a high impact in the field of Generative AI. We could potentially expand this to total citations overall - however we kept it brief only for the most impactful citations from impactful scientists.\n\n5. Despite the lower citation count, the **increase in the citation count of Raviraj Joshi** from 2021 to 2022 was quite noticeable, indicating that the influence of his work could be growing.\n\nIn conclusion, these findings underscore the importance of continuous, quality research in the rapidly evolving field of generative AI. They also highlight the significant contributions of certain individuals and institutions to advancing this field. However, more comprehensive analysis that considers other factors (like the number of citations for the publications of each year, h-index, etc.) could give a more balanced understanding of the researcher's impact.","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"\n⚡ 4. From Conception to Action: A Dive into Kaggle's Generative AI Journey\n
","metadata":{}},{"cell_type":"markdown","source":"Next step would be to see how **Generative AI** has been utilized in the most efficient way in **Kaggle competitions**. In order to do that, we will follow the given steps:\n\na) Compile a new list of keywords for which we will filter the dataset, for **Generative AI competitions**. The reason we are doing that is because the keywords in research paper abstract or papers are usually more heavily associated with the models, techniques, or methods. In **Kaggle competitions** we usually have a more high level, friendly set of keywords used. While we will finetune and filter out only the relevant subject, this will be an important first step.\n\nb) Create a given range for the **past 2 years**, for which we will research the competitions. More specifically, the range we are using is: **2021-05-01** to **2023-06-26**.\n\nc) Perform some **cleaning**, exploratory data **analysis**, and review findings related to the data from the **Generative AI** perspective.\n\nd) We manually check several times the title and description and consult with **GPT4** whether the competition can be classified as **Generative AI**, and filter it out (or select new competitions).\n\ne) We check the **winner solution**, 2nd best solution for each of the **Generative AI** tasks and extract **insights** as to what has been working best for the given competitions.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport warnings\n\n# Load the dataset\ncompetition_data = pd.read_csv('/kaggle/input/meta-kaggle/Competitions.csv')\n\n# Convert the date string to datetime\nstart_date = pd.to_datetime('2021-05-01')\nend_date = pd.to_datetime('2023-06-26')\n\n# Convert the 'EnabledDate' column to datetime\ncompetition_data['EnabledDate'] = pd.to_datetime(competition_data['EnabledDate'])\n\n# Filter the dataframe to only include competitions that were enabled between the start and end dates\nfiltered_competitions = competition_data[(competition_data['EnabledDate'] >= start_date) & (competition_data['EnabledDate'] <= end_date)].copy()\n\n# As there could be some competitions without a 'Subtitle', we will fill missing values with an empty string\nfiltered_competitions['Subtitle'] = filtered_competitions['Subtitle'].fillna('')\n\n# Create a single text column combining 'Title' and 'Subtitle'\nfiltered_competitions['text'] = filtered_competitions['Title'] + ' ' + filtered_competitions['Subtitle']\n\n# Create a lowercase version for comparison with lowercase keywords\nfiltered_competitions['text_lower'] = filtered_competitions['text'].str.lower()\n\n# Modified the keywords to suit better the theme here (f.ex research papers have more technical vocab)\nkeywords = [\n 'generative ai',\n 'generative adversarial networks',\n 'autoencoders',\n 'GAN',\n 'Reconstruct',\n 'regenerating',\n 'generate',\n 'generative models',\n 'deep generative models',\n 'AI agent',\n 'computer vision image',\n 'image generation',\n 'image matching',\n 'images to text',\n 'synthetic image',\n 'deepfake',\n 'text generation',\n 'music generation',\n 'voice generation',\n 'large language models',\n 'llm',\n 'gpt',\n 'diffusion',\n '3d object generation',\n 'video generation',\n]\n\n# Convert the keywords to lower case for matching\nkeywords = [keyword.lower() for keyword in keywords]\n\n# Identify competitions that have any of the keywords in their 'text' field\nfiltered_competitions['is_generative_ai'] = filtered_competitions['text_lower'].apply(lambda x: any(keyword in x for keyword in keywords))\n\n# Display the first few rows of the dataframe\nfiltered_competitions.head(2)\n\n# Suppress the SettingWithCopyWarning\npd.set_option('mode.chained_assignment', None) # Ignore the warning\n\n# Calculate the number of competitions related to Generative AI\nnum_generative_ai_competitions = filtered_competitions['is_generative_ai'].sum()\nprint(f'Number of competitions related to Generative AI: {num_generative_ai_competitions}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:11.228205Z","iopub.execute_input":"2023-07-12T04:05:11.228548Z","iopub.status.idle":"2023-07-12T04:05:11.962236Z","shell.execute_reply.started":"2023-07-12T04:05:11.228520Z","shell.execute_reply":"2023-07-12T04:05:11.961185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Even though the number seems significant (for the past 2 years), we still need to manually review the context of the competitions (not all keywords with generative ai can be related to a competition in the field). A more comprehensive analysis is required.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\n# Convert 'EnabledDate' to datetime format\nfiltered_competitions['EnabledDate'] = pd.to_datetime(filtered_competitions['EnabledDate'])\n\n# Extract the year from 'EnabledDate'\nfiltered_competitions['Year'] = filtered_competitions['EnabledDate'].dt.year\n\n# Count the number of generative AI competitions each year\ndf_generative_ai = filtered_competitions[filtered_competitions['is_generative_ai']].groupby('Year').size().reset_index()\ndf_generative_ai.columns = ['Year', 'Count']\n\n# Create the bar plot\nplt.figure(figsize=(10, 6))\nplt.bar(df_generative_ai['Year'], df_generative_ai['Count'], color='skyblue', edgecolor='black')\nplt.xlabel('Year')\nplt.ylabel('Number of Generative AI Competitions')\nplt.title('Generative AI Competitions by Year')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:11.963584Z","iopub.execute_input":"2023-07-12T04:05:11.963920Z","iopub.status.idle":"2023-07-12T04:05:12.313135Z","shell.execute_reply.started":"2023-07-12T04:05:11.963894Z","shell.execute_reply":"2023-07-12T04:05:12.312035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we see a drop in relevant competetions for Generative AI from 2021. We have certain suspicions on this, since it doesnt match the higher number of research papers, and output devoted to that direction. Lets explore the data further.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.colors as mcolors\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource, LabelSet, NumeralTickFormatter\n\noutput_notebook()\n\n# Count the number of generative AI competitions for each segment\ngenerative_ai_competitions_by_segment = filtered_competitions[filtered_competitions['is_generative_ai']]['HostSegmentTitle'].value_counts().reset_index()\ngenerative_ai_competitions_by_segment.columns = ['HostSegmentTitle', 'Count']\n\n# Normalize Count\ngenerative_ai_competitions_by_segment['Count_norm'] = (generative_ai_competitions_by_segment['Count'] - generative_ai_competitions_by_segment['Count'].min()) / (generative_ai_competitions_by_segment['Count'].max() - generative_ai_competitions_by_segment['Count'].min())\n\n# Define the color palette as a matplotlib colormap\ncmap = plt.get_cmap('Oranges')\n\n# Assign colors to the normalized count values\ngenerative_ai_competitions_by_segment['Color'] = [mcolors.rgb2hex(cmap(x)) for x in generative_ai_competitions_by_segment['Count_norm']]\n\n# Format Count for labels\ngenerative_ai_competitions_by_segment['Count_str'] = generative_ai_competitions_by_segment['Count'].apply(lambda x: '{:,}'.format(x))\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(generative_ai_competitions_by_segment)\n\n# Create the Bokeh figure\np = figure(\n width=1100, \n height=800,\n toolbar_location=None,\n title=\"Generative AI Competitions by Host Segment\",\n x_range=(0, generative_ai_competitions_by_segment['Count'].max() * 1.1),\n y_range=list(reversed(generative_ai_competitions_by_segment['HostSegmentTitle'])),\n min_border_top=100 \n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Number of Generative AI Competitions'\np.yaxis.axis_label = 'Host Segment Title'\n\n# Format the x-axis ticks to be more readable\np.xaxis[0].formatter = NumeralTickFormatter(format=\"0,0\")\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars\nbars = p.hbar(\n y='HostSegmentTitle',\n right='Count',\n height=0.5,\n source=source,\n color='Color'\n)\n\n# Create a label set\nlabels = LabelSet(x='Count', y='HostSegmentTitle', text='Count_str', level='glyph', \n x_offset=5, y_offset=-10, source=source, \n text_color=\"black\", text_font=\"Arial\", text_font_size=\"12pt\")\n\np.add_layout(labels)\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:12.315061Z","iopub.execute_input":"2023-07-12T04:05:12.315714Z","iopub.status.idle":"2023-07-12T04:05:12.550041Z","shell.execute_reply.started":"2023-07-12T04:05:12.315676Z","shell.execute_reply":"2023-07-12T04:05:12.548712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, by far the biggest number of Generative AI competitions is connected to community initiations. The low barrier to entry, narrower scope and more beginner friendly themes make it easier to host competitions in that regard.","metadata":{}},{"cell_type":"markdown","source":" Lets review step by step all the given titles, submissions and descriptions together with the models and check whether they indeed can be classified as Generative AI problems.
","metadata":{}},{"cell_type":"code","source":"generative_ai_competitions = filtered_competitions[filtered_competitions['is_generative_ai']]\nfor title, subtitle in zip(generative_ai_competitions['Title'], generative_ai_competitions['Subtitle']):\n print(f'Title: {title}\\nSubtitle: {subtitle}\\n')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:12.559045Z","iopub.execute_input":"2023-07-12T04:05:12.559401Z","iopub.status.idle":"2023-07-12T04:05:12.566613Z","shell.execute_reply.started":"2023-07-12T04:05:12.559370Z","shell.execute_reply":"2023-07-12T04:05:12.565473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***Based on the titles and descriptions we can see that we have many irrelevant competitions being filtered as related to Generative AI. Therefore, we will follow a different a approach***\n\na) We manually scan the titles of all competitions filtered for the past 2 years based on our keywords.
\nb) Read and comprehend the description & title, and assess whether task is related to Generative AI.
\nc) Consult with GPT4 and review again the given title and description.
\nd) Double check again (Regenerating Twice the Given Description)
\ne) Proceed for further analysis of the Generative AI competition
","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"In order to classify something a competition as **Generative AI**, we would need the models learn to capture the underlying patterns, structure, and characteristics of the training data and use this knowledge to generate new instances of data that resemble the training data. **The goal is not to reproduce exact instances from the training set, but rather to produce novel data samples that could plausibly have been part of the training set**. Examples could incorporate within themselves solutions with models like **GAN**, **VAE** etc. \n\nTherefore, after following the mentioned approach and strictly following the definition of Generative AI, we could find the following competitions for the **past 2 years**:\n\na) [**I’m Something of a Painter Myself (2021)**](https://www.kaggle.com/competitions/gan-getting-started)\n\nb) [**Bristol-Myers Squibb – Molecular Translation (2021)**](https://www.kaggle.com/competitions/bms-molecular-translation)\n\nc) [**Wikipedia - Image/Caption Matching**](https://www.kaggle.com/competitions/wikipedia-image-caption)\n\nd) [**AI Village CTF**](https://www.kaggle.com/competitions/ai-village-ctf)\n\ne) [**Stable Diffusion - Image to Prompts (2023)**](https://www.kaggle.com/competitions/stable-diffusion-image-to-prompts)\n\n(*Currently in Progress*)\n\n- [**Kaggle AI Report (2023)**](https://www.kaggle.com/competitions/2023-kaggle-ai-report)\n\n- [**Image Matching Challenge (2023)**](https://www.kaggle.com/competitions/image-matching-challenge-2023)\n\n---\n\n**Other competitions which could involve elements of Generative AI, but are not necessarily Generative AI competitions are:**\n\n(1) [Halite IV Playground Edition](https://www.kaggle.com/competitions/halite-iv-playground-edition)\n\n(2) [Hungry Geese](https://www.kaggle.com/competitions/hungry-geese)\n\n(3) [Chaii Hindi and Tamil Question Answering](https://www.kaggle.com/competitions/chaii-hindi-and-tamil-question-answering)\n\n(4) [Kore 2022](https://www.kaggle.com/competitions/kore-2022)\n\n(5) [Feedback Prize 2021](https://www.kaggle.com/competitions/feedback-prize-2021)","metadata":{}},{"cell_type":"markdown","source":"**Lets review the available solutions and discussions surrounding the mentioned competitions
**","metadata":{}},{"cell_type":"markdown","source":"| Competition Name | Description | Evaluation | Model(s) |\n|---|---|---|---|\n| I’m Something of a Painter Myself | Build a GAN that generates 7,000 to 10,000 Monet-style images. | MiFID (Memorization-informed Fréchet Inception Distance) | CycleGAN with U-Net and ResNet generator architectures and a PatchGAN discriminator. |\n| Bristol-Myers Squibb – Molecular Translation | Develop a machine learning model capable of converting scanned images of chemical structures into a machine-readable format, specifically, the International Chemical Identifier (InChI) text. | Mean Levenshtein distance | Three-phase solution: image captioning training with noise augmentation, InChI candidates generation using models like Swin Transformer, BERT Decoder, EfficientNet-v2, and others, followed by a reranking phase to pick the optimal InChI candidate. |\n| Wikipedia - Image/Caption Matching | Build a model to automatically retrieve the most contextually relevant text for a given image. | NDCG@5 (Normalized Discounted Cumulative Gain) | EfficientNet B0 to generate image embeddings and XLM-Roberta for generating text embeddings. |\n| AI Village Capture the Flag @ DEFCON | A Capture-the-Flag (CTF) style challenge involving various machine learning security tasks. | Weighted classification accuracy | Brute-force solutions, grid searches, locally linear embedding algorithms, basic logistic regression, and adversarial attacks. |\n| Stable Diffusion - Image to Prompts | Develop a model that predicts the text prompt from a given generated image. | Mean cosine similarity score between the predicted and actual prompt embedding vectors | ViT-Large, ConvNeXt-XXLarge, and BLIP-2 models trained on approximately 10.6 million images generated from around 8.6 million prompts. LLM component was removed from BLIP-2 and LORA was used for fine-tuning. |","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":" 1. I’m Something of a Painter Myself\nUse GANs to create art - will you be the next Monet?
\n\n***Description***: In this paper the task was to build a GAN that **generates 7,000 to 10,000 Monet-style images**.
\n \n***Evaluation***: The evaluation used is [**MiFID** (Memorization-informed Fréchet Inception Distance)](https://www.kaggle.com/competitions/gan-getting-started/overview/evaluation).
\n \n***Solution***: For this task the best available solution revolves around the application of a type of Generative Adversarial Network (GAN) known as **CycleGAN**, combined with **U-Net** and **ResNet** generator architectures and a PatchGAN discriminator. ","metadata":{"execution":{"iopub.status.busy":"2023-06-28T15:16:06.011992Z","iopub.execute_input":"2023-06-28T15:16:06.012848Z","iopub.status.idle":"2023-06-28T15:16:06.048986Z","shell.execute_reply.started":"2023-06-28T15:16:06.012795Z","shell.execute_reply":"2023-06-28T15:16:06.047290Z"}}},{"cell_type":"markdown","source":" 2. Bristol-Myers Squibb – Molecular Translation; Can you translate chemical images to text?
\n\n***Description***: In **this task**, the goal is to develop a machine learning model capable of converting scanned images of chemical structures into a machine-readable format, specifically, the International Chemical Identifier (**InChI**) text. The challenge is motivated by the need to digitize and automate the search of historical chemistry literature. This task utilizes synthetic image data generated by **Bristol-Myers Squibb**.
\n\n***Evaluation***:The evaluation of the submitted solutions will be based on the mean **Levenshtein distance**, which measures the difference between the predicted InChi strings and the actual (ground truth) InChi values.
\n\n***Solution***: The highest scored available solution used a three-phase solution: image captioning training with noise augmentation, **InChI candidates generation using models like Swin Transformer, BERT Decoder, EfficientNet-v2**, and others, followed by a reranking phase to pick the optimal InChI candidate based on validation and average loss across models.","metadata":{}},{"cell_type":"markdown","source":" 3. Wikipedia - Image/Caption Matching\nRetrieve captions based on images
\n\n***Description***: In **this competition**, participants are tasked with building a model to automatically retrieve the most contextually relevant text for a given image. The model should be capable of associating images with corresponding **article titles** or **complex captions**, and must be capable of doing so in **multiple languages**. The objective is to improve the **accessibility** and **searchability** of images on **Wikipedia**. \n\n***Evaluation***:Submissions will be evaluated using **NDCG@5** (**Normalized Discounted Cumulative Gain**), a measure that assigns higher importance to the correct prediction of top-ranked results.
\n\n***Solution***:While not a winning solution, the available notebook with the highest score involves using **EfficientNet B0** to generate image embeddings and **XLM-Roberta** for generating text embeddings. Additionally, it employs **Weights & Biases** to streamline model development and evaluation by tracking experiments, versioning datasets, evaluating model performance, reproducing models, visualizing results, and sharing findings.","metadata":{}},{"cell_type":"markdown","source":" 4. AI Village Capture the Flag @ DEFCON
\n\n***Description***: This **DEFCON30 competition**, hosted by the **AI Village**, is a **Capture-the-Flag (CTF)** style challenge involving various **machine learning security tasks**. Unlike conventional competitions, competitors will interact with **API endpoints** or **code/objects**, obtaining unique **flags** upon successful completion of each challenge. These flags are then submitted in a **.csv** to update the **scoreboard**, with scores weighted by **challenge difficulty**. \n\n***Evaluation***: The evaluation metric is a **weighted classification accuracy**, treating the challenge as a **multiclass classification** problem where each flag represents a unique class label.
\n\n***Solution***: The winner's strategy involved **brute-force solutions**, **grid searches**, **locally linear embedding algorithms**, **basic logistic regression**, and **adversarial attacks**. Some challenges were solved creatively, using tools like an **image editor**, **Excel**, and **handwriting digit recognition**.","metadata":{}},{"cell_type":"markdown","source":"5. Stable Diffusion - Image to Prompts\nDeduce the prompts that generated our \"highly detailed, sharp focus, illustration, 3d renders of majestic, epic\" images
\n\n***Description***: In **this competition**, the goal is to develop a model that predicts the **text prompt** from a given generated image, effectively reversing the typical **text-to-image generative model** process. These predictions will be made on a dataset containing numerous pairs of **text prompts** and **images** generated by **Stable Diffusion 2.0**. Rather than predicting the prompt directly, competitors will submit **embeddings** of their predicted prompts. \n\n***Evaluation***: The evaluation metric is the **mean cosine similarity score** between the predicted and actual prompt embedding vectors.\n\n***Solution***: The winning solution involved training **ViT-Large**, **ConvNeXt-XXLarge**, and **BLIP-2 models** on approximately **10.6 million images** generated from around **8.6 million prompts**, with **Large Language Model (LLM)** component removed from BLIP-2 and **LORA** used for fine-tuning. Various strategies were employed to accelerate the image generation process and to diversify the dataset, including the use of **DiffusionDB_2M**, **COCO Captions**, **VizWiz**, **Open Image**, **ADE20K**, **Flickr30K**, and **TextCaps**. The final submission comprised a weighted ensemble of embeddings from the three aforementioned models, which improved the overall score.","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"\n 📣 4.1 Reviewing Forums: What Are Kagglers Discussing About When it Comes to Generative AI\n
","metadata":{}},{"cell_type":"markdown","source":"Besides notebooks, and competitions another useful subset of data would be to see the relevant discussions and topics related to Generative AI. In a similar fashion, we obtain the data from the Kaggle Metadata dataset.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n# Read the datasets\nforum_topics = pd.read_csv('/kaggle/input/meta-kaggle/ForumTopics.csv')\nforums = pd.read_csv('/kaggle/input/meta-kaggle/Forums.csv')\n\n# Select the necessary columns and rename\nforums = forums[['Id', 'Title']].rename(columns={'Title': 'ForumTitle'})\nforum_topics = forum_topics.rename(columns={'Title': 'ForumTopicTitle'})\n\n# Merge the datasets\nmerged_forums = pd.merge(forums, forum_topics, left_on='Id', right_on='ForumId', how='inner')\n\n# Display the merged dataset\n\nmerged_forums.head()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:12.568107Z","iopub.execute_input":"2023-07-12T04:05:12.568594Z","iopub.status.idle":"2023-07-12T04:05:14.855846Z","shell.execute_reply.started":"2023-07-12T04:05:12.568557Z","shell.execute_reply":"2023-07-12T04:05:14.854845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# List of columns to drop which are not useful\ncolumns_to_drop = ['KernelId', 'LastForumMessageId', 'FirstForumMessageId', 'Id_y', 'Id_x', 'LastCommentDate']\n\n# Drop the columns\nmerged_forums = merged_forums.drop(columns_to_drop, axis=1)\n\n# Display the updated dataframe\nmerged_forums.head()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:14.857365Z","iopub.execute_input":"2023-07-12T04:05:14.857676Z","iopub.status.idle":"2023-07-12T04:05:14.893515Z","shell.execute_reply.started":"2023-07-12T04:05:14.857651Z","shell.execute_reply":"2023-07-12T04:05:14.892383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Futher refinement of keywords, adapting to previous competitions\n\nkeywords = [\n 'generative ai',\n 'generative adversarial networks',\n 'autoencoders',\n 'gpt-3',\n 'large language models',\n 'diffusion models',\n 'Stable Diffusion'\n 'VAEs',\n 'Diffusion',\n 'Generative Models',\n 'Deep Generative Models',\n 'Image Generation',\n 'Text Generation',\n 'Synthetic Data',\n 'wikipedia image',\n 'CycleGAN',\n 'StyleGAN'\n 'DCGAN',\n 'Deep Convolutional Generative Adversarial Networks',\n 'AI Village',\n 'BMS molecular translation',\n 'GAN Variants',\n 'Conditional GANs',\n 'Neural Style Transfer',\n 'Music Generation',\n 'Voice Generation',\n 'Large Language Models (LLMs)',\n 'GPT',\n 'Transformer-based Generative Models',\n 'Diffusion Models',\n 'Diffusion-based Generative Models',\n 'Sequence Generation',\n 'Generative Pre-training',\n 'Zero-Shot Learning',\n 'Few-Shot Learning',\n '3D Object Generation',\n 'Video Generation',\n]\n\n# Convert all keywords to lowercase\nkeywords = [keyword.lower() for keyword in keywords]\n\ndef check_keywords(title):\n \"\"\"Checks if any keyword is a substring of the title\"\"\"\n title = str(title).lower() # Convert title to lowercase\n return any(keyword in title for keyword in keywords)\n\n# Apply the function to the 'ForumTopicTitle' column to filter the dataframe\nmerged_forums['ForumTopicTitle'] = merged_forums['ForumTopicTitle'].astype(str) # Ensure all titles are strings\ngenerative_ai_topics = merged_forums[merged_forums['ForumTopicTitle'].apply(check_keywords)]\n\n# Sort by total views\ngenerative_ai_topics_sorted = generative_ai_topics.sort_values('TotalViews', ascending=False)\n\n# Convert the 'CreationDate' column to datetime format\ngenerative_ai_topics_sorted['CreationDate'] = pd.to_datetime(generative_ai_topics_sorted['CreationDate'])\n\n# Set start and end dates\nstart_date = '2021-05-01'\nend_date = '2023-06-26'\n\n# Filter the data between these dates\nfiltered_generative_ai_topics = generative_ai_topics_sorted[\n (generative_ai_topics_sorted['CreationDate'] >= start_date) &\n (generative_ai_topics_sorted['CreationDate'] <= end_date)\n]\n\n\n# Convert all keywords to lowercase\nkeywords = [keyword.lower() for keyword in keywords]\n\ndef check_keywords(title):\n \"\"\"Checks if any keyword is a substring of the title\"\"\"\n title = str(title).lower() # Convert title to lowercase\n return any(keyword in title for keyword in keywords)\n\n# Apply the function to the 'ForumTopicTitle' column to filter the dataframe\nmerged_forums['ForumTopicTitle'] = merged_forums['ForumTopicTitle'].astype(str) # Ensure all titles are strings\ngenerative_ai_topics = merged_forums[merged_forums['ForumTopicTitle'].apply(check_keywords)]\n\n# Sort by total views\ngenerative_ai_topics_sorted = generative_ai_topics.sort_values('TotalViews', ascending=False)\n\n# Convert the 'CreationDate' column to datetime format\ngenerative_ai_topics_sorted['CreationDate'] = pd.to_datetime(generative_ai_topics_sorted['CreationDate'])\n\n# Set start and end dates\nstart_date = pd.to_datetime('2021-05-01')\nend_date = pd.to_datetime('2023-06-26')\n\n# Filter the data between these dates\nfiltered_generative_ai_topics = generative_ai_topics_sorted[\n (generative_ai_topics_sorted['CreationDate'] >= start_date) &\n (generative_ai_topics_sorted['CreationDate'] <= end_date)\n]\n\nfiltered_generative_ai_topics","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:14.895178Z","iopub.execute_input":"2023-07-12T04:05:14.895490Z","iopub.status.idle":"2023-07-12T04:05:18.392971Z","shell.execute_reply.started":"2023-07-12T04:05:14.895463Z","shell.execute_reply":"2023-07-12T04:05:18.391899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nfrom bokeh.plotting import figure, show\nfrom bokeh.models import ColumnDataSource, NumeralTickFormatter, DatetimeTickFormatter\nfrom bokeh.io import output_notebook\nfrom bokeh.layouts import gridplot\n\noutput_notebook()\n\n# Create a copy to avoid SettingWithCopyWarning\nfiltered_generative_ai_topics = filtered_generative_ai_topics.copy()\n\n# Convert 'CreationDate' to datetime\nfiltered_generative_ai_topics['CreationDate'] = pd.to_datetime(filtered_generative_ai_topics['CreationDate'])\n\n# Extract year and month from 'CreationDate'\nfiltered_generative_ai_topics['YearMonth'] = filtered_generative_ai_topics['CreationDate'].dt.to_period('M')\n\n# Group by year and month, then count the number of topics\ntemporal_analysis = filtered_generative_ai_topics.groupby('YearMonth').size().reset_index()\ntemporal_analysis.columns = ['YearMonth', 'Count']\n\n# Convert 'YearMonth' to datetime type\ntemporal_analysis['YearMonth'] = temporal_analysis['YearMonth'].dt.to_timestamp()\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(temporal_analysis)\n\n# Create the Bokeh figure\np = figure(\n width=800, \n height=600,\n toolbar_location=None,\n title=\"Number of Generative AI Topics on Kaggle Over Time\",\n x_axis_type=\"datetime\",\n x_range=(temporal_analysis['YearMonth'].min(), temporal_analysis['YearMonth'].max()),\n y_range=(0, temporal_analysis['Count'].max()),\n min_border_top=100 \n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Year-Month'\np.yaxis.axis_label = 'Number of Topics'\n\n# Format the x-axis ticks to be more readable\np.xaxis[0].formatter = DatetimeTickFormatter(months=\"%Y-%m\")\np.yaxis[0].formatter = NumeralTickFormatter(format=\"0,0\")\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the line plot\np.line(x='YearMonth', y='Count', line_width=2, source=source, color=\"#000000\")\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:18.394796Z","iopub.execute_input":"2023-07-12T04:05:18.395537Z","iopub.status.idle":"2023-07-12T04:05:18.668513Z","shell.execute_reply.started":"2023-07-12T04:05:18.395504Z","shell.execute_reply":"2023-07-12T04:05:18.667266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After cleaning and preprocessing the data, filtered and fine tuned specifically for the Generative AI topics, and subtopics we can definitely notice a sharp increase from the end of 2022, and continuing to increase at a very fast rate. This means that Kagglers are putting more time and effort into discussing models, solutions, competitions and everything related to Generative AI.","metadata":{}},{"cell_type":"code","source":"from bokeh.plotting import figure, show\nfrom bokeh.models import ColumnDataSource, NumeralTickFormatter\nfrom bokeh.io import output_notebook\nfrom bokeh.palettes import Category10\nfrom sklearn.preprocessing import MinMaxScaler\n\noutput_notebook()\n\ndf = filtered_generative_ai_topics.copy()\n\n# Features to be normalized\nfeatures_to_normalize = ['TotalViews', 'Score', 'TotalMessages', 'TotalReplies']\n\n# Apply Min-Max scaling\nscaler = MinMaxScaler()\ndf[features_to_normalize] = scaler.fit_transform(df[features_to_normalize])\n\n# Compute the 'AttentionScore' for each topic by summing the normalized 'TotalViews', 'Score', 'TotalMessages', and 'TotalReplies'\ndf['AttentionScore'] = df['TotalViews'] + df['Score'] + df['TotalMessages'] + df['TotalReplies']\n\n# Sort the dataframe by 'AttentionScore' in descending order\nsorted_topics = df.sort_values('AttentionScore', ascending=False)\n\n# Truncate the topic titles to the first 50 characters\ntop_topics = sorted_topics.head(10).copy() # Create an explicit copy to avoid the SettingWithCopyWarning\ntop_topics['TruncatedTitle'] = top_topics['ForumTopicTitle'].str[:35]\n\n# Reset the index\ntop_topics.reset_index(drop=True, inplace=True)\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(top_topics)\n\n# Create the Bokeh figure\np = figure(\n width=1000, \n height=600,\n toolbar_location=None,\n title=\"Top 10 Most Popular Generative AI Topics\",\n x_range=(0, top_topics['AttentionScore'].max()),\n y_range=list(reversed(top_topics['TruncatedTitle'])),\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Attention Score'\np.yaxis.axis_label = 'Forum Topic Title'\n\n# Format the x-axis ticks to be more readable\np.xaxis[0].formatter = NumeralTickFormatter(format=\"0.0\")\n\n# Customize the Y-axis labels\np.yaxis.major_label_text_font_size = '12pt'\np.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars\nbars = p.hbar(\n y='TruncatedTitle',\n right='AttentionScore',\n height=0.5,\n source=source,\n color=Category10[10][0] # Use the first color in the Bokeh Category10 palette\n)\n\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:18.670353Z","iopub.execute_input":"2023-07-12T04:05:18.670781Z","iopub.status.idle":"2023-07-12T04:05:18.923616Z","shell.execute_reply.started":"2023-07-12T04:05:18.670744Z","shell.execute_reply":"2023-07-12T04:05:18.922807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since we have several metrics, such as 'TotalViews', 'Score', 'TotalMessages', 'TotalReplies' - we decided to follow a more balanced approach where we normalize all these metrics then add them together. Even though it could be furtherly fine tuned it is a nice start to see what is currently drawing the most attention of kagglers. Based on the chart we can see that exclusively **chatgpt** dominates the discussions and we also have **diffussion models** together with **GAN** displayed at the top. ","metadata":{}},{"cell_type":"code","source":"from plotly.offline import init_notebook_mode\nimport plotly.graph_objs as go\nfrom collections import Counter\nimport nltk\nfrom nltk.corpus import stopwords\nimport warnings\nimport plotly_express as px\n\ninit_notebook_mode(connected=True)\n\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) # Ignore DeprecationWarning\n\n# Define a function to clean and split text into words\ndef clean_and_split(text):\n # Make all text lowercase\n text = text.lower()\n # Split text into words\n words = text.split()\n # Remove stopwords\n stop_words = set(stopwords.words('english'))\n words = [word for word in words if word not in stop_words]\n return words\n\n# Apply the function to 'ForumTopicTitle' and flatten the list of lists\nwords = [word for title in filtered_generative_ai_topics['ForumTopicTitle'] for word in clean_and_split(title)]\n\n# Count the occurrence of each word\nword_counts = Counter(words)\n\n# Get the 10 most common words\ntop_words = word_counts.most_common(10)\n\n# Convert to DataFrame for easier plotting\ntop_words_df = pd.DataFrame(top_words, columns=['Word', 'Count'])\n\n# Generate a list of colors\ncolors = px.colors.qualitative.Plotly[:len(top_words_df)]\n\n# Create the bar plot using plotly.graph_objs\nfig = go.Figure(data=[go.Bar(\n x=top_words_df['Count'],\n y=top_words_df['Word'],\n marker_color=colors, # different colors for each bar\n orientation='h' # horizontal bar chart\n)])\n\nfig.update_layout(\n autosize=False,\n width=1000,\n height=800,\n margin=dict(l=50, r=50, b=150, t=100, pad=4),\n title={'text':'Top 10 Most Common Words in Generative AI Topic Titles', 'x':0.5, 'xanchor': 'center'},\n xaxis_title='Count',\n yaxis_title='Word',\n template='plotly_dark'\n)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:18.924795Z","iopub.execute_input":"2023-07-12T04:05:18.925827Z","iopub.status.idle":"2023-07-12T04:05:22.041311Z","shell.execute_reply.started":"2023-07-12T04:05:18.925792Z","shell.execute_reply":"2023-07-12T04:05:22.040275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The main keywords associated with the most engaging forum topics are again related to **chatgpt, gpt and diffusion models**. This is expected, given the aforementioned topics before.","metadata":{}},{"cell_type":"markdown","source":"Note: The next step would normally be to review the comments for each topic, together with the previous filtered data - however there is a **discrepancy** between the following:\n**(ForumTopicId)** column in ForumMessages.csv doesnt stand for the equivalent in ForumTopics.csv\n**(ForumId)** column or for the Forums.csv (Id) Column.\n\nTherefore we will proceed with running an individual assessment separately for the text mentioned! This time we use the Kaggle API for additional data!","metadata":{}},{"cell_type":"code","source":"import os\n\n# Path to the directory where kaggle.json is uploaded\ncredentials_dir = '/kaggle/input/kaggle/'\n\n# Set the Kaggle configuration directory\nos.environ['KAGGLE_CONFIG_DIR'] = credentials_dir","metadata":{"execution":{"iopub.status.busy":"2023-07-12T04:05:22.042713Z","iopub.execute_input":"2023-07-12T04:05:22.043062Z","iopub.status.idle":"2023-07-12T04:05:22.048053Z","shell.execute_reply.started":"2023-07-12T04:05:22.043032Z","shell.execute_reply":"2023-07-12T04:05:22.046960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nfrom IPython.core.display import display as ICD\nfrom datetime import datetime\nimport subprocess\n\ndef convert_to_df(cmd_output):\n lines = cmd_output.split('\\n')\n columns = lines[1].split(',')\n data = [line.split(',')[:len(columns)] for line in lines[2:]] # skip the header line\n return pd.DataFrame(data, columns=columns)\n\n\nkeywords = [\n 'generative ai',\n 'generative adversarial networks',\n 'autoencoders',\n 'gpt-3',\n 'large language models',\n 'diffusion models',\n 'Generative AI'\n 'Generative Adversarial Networks',\n 'Variational Autoencoders',\n 'Generative Models',\n 'Deep Generative Models',\n 'Image Generation',\n 'Text Generation',\n 'CycleGAN',\n 'StyleGAN'\n 'DCGAN',\n 'Deep Convolutional Generative Adversarial Networks',\n 'GAN Training Techniques',\n 'GAN Applications',\n 'GAN Variants',\n 'Conditional GANs',\n 'Music Generation',\n 'Voice Generation',\n 'Large Language Models (LLMs)',\n 'GPT',\n 'Diffusion Models',\n 'Diffusion-based Generative Models',\n 'Sequence Generation',\n 'Text-to-Text Generation Models'\n 'Generative Pre-training',\n 'Masked Language Models',\n 'Cross-Lingual Language Models (XLMs)',\n 'Transfer Learning in Generative Models', \n 'Zero-Shot Learning',\n 'Few-Shot Learning',\n 'Meta Learning',\n '3D Object Generation',\n 'Video Generation',\n 'Inverse Graphics Networks (IGNs)',\n 'Multi-Agent Generative Adversarial Networks',\n 'MAGANs',\n 'Style Transfer'\n]\n\ndf = pd.DataFrame(columns=['ref', 'title', 'author', 'lastRunTime', 'totalVotes'])\n\ndef get_kernels(keyword):\n global df\n cmd_output = subprocess.getoutput(f'kaggle kernels list -s \"{keyword}\" --csv')\n kernels_df = convert_to_df(cmd_output)\n kernels_df['Keyword'] = keyword\n df = pd.concat([df, kernels_df], ignore_index=True)\n\nfor keyword in keywords:\n get_kernels(keyword)\n\nprint('Listing of Kernels on Kaggle for Generative AI Topics:')\ndisplay(df)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:22.049952Z","iopub.execute_input":"2023-07-12T04:05:22.050686Z","iopub.status.idle":"2023-07-12T04:05:54.641165Z","shell.execute_reply.started":"2023-07-12T04:05:22.050646Z","shell.execute_reply":"2023-07-12T04:05:54.640045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the initial dataframe we can distinguish 648 different kernels with topics related to Generative AI (or subtopics related to it). Now we can perform additional preprocessing, where we clean data and filter to top 20 kernels by highest number of votes.","metadata":{}},{"cell_type":"code","source":"unique_authors = df['author'].nunique()\nprint(\"Number of unique authors:\", unique_authors)","metadata":{"execution":{"iopub.status.busy":"2023-07-12T04:05:54.642557Z","iopub.execute_input":"2023-07-12T04:05:54.642900Z","iopub.status.idle":"2023-07-12T04:05:54.649983Z","shell.execute_reply.started":"2023-07-12T04:05:54.642852Z","shell.execute_reply":"2023-07-12T04:05:54.648703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Convert 'lastRunTime' column to datetime\ndf['lastRunTime'] = pd.to_datetime(df['lastRunTime'], errors='coerce')\n\n# Specify the date range\nstart_date = '2021-05-01'\nend_date = '2023-06-26'\n\n# Filter the DataFrame for the specified date range\nfiltered_df = df[(df['lastRunTime'] >= start_date) & (df['lastRunTime'] <= end_date)]\n\n# Display the filtered DataFrame\ndisplay(filtered_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:54.651295Z","iopub.execute_input":"2023-07-12T04:05:54.651660Z","iopub.status.idle":"2023-07-12T04:05:54.683789Z","shell.execute_reply.started":"2023-07-12T04:05:54.651628Z","shell.execute_reply":"2023-07-12T04:05:54.682777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above we can see the topcs with the highest number of votes for each kernel. For easier display, we can create a chart to review the findings!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.font_manager as fm\n\n# Specify a font that supports the required glyphs\nfont_path = fm.findfont(fm.FontProperties(family='DejaVu Sans'))\n\n# Set the font for Matplotlib\nplt.rcParams['font.family'] = fm.FontProperties(fname=font_path).get_name()\n\n\n# Convert 'lastRunTime' column to datetime\ndf['lastRunTime'] = pd.to_datetime(df['lastRunTime'], errors='coerce')\n\n# Specify the date range\nstart_date = '2021-05-01'\nend_date = '2023-06-26'\n\n# Filter the DataFrame for the specified date range\nfiltered_df = df[(df['lastRunTime'] >= start_date) & (df['lastRunTime'] <= end_date)]\n\n# Convert 'totalVotes' column to numeric dtype\nfiltered_df['totalVotes'] = pd.to_numeric(filtered_df['totalVotes'], errors='coerce')\n\n# Sort the filtered DataFrame by totalVotes in descending order\ntop_20_kernels = filtered_df.nlargest(20, 'totalVotes')\n\n# Display the top 20 kernels by vote count\ndisplay(top_20_kernels)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:54.687291Z","iopub.execute_input":"2023-07-12T04:05:54.687628Z","iopub.status.idle":"2023-07-12T04:05:55.084042Z","shell.execute_reply.started":"2023-07-12T04:05:54.687600Z","shell.execute_reply":"2023-07-12T04:05:55.082966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.express as px\n\n# Create a bar plot using Plotly Express\nfig = px.bar(top_20_kernels, \n x='totalVotes', \n y='title', \n title='Top 20 Kernels by Vote Count', \n color='title', # Specify the column for color coding\n labels={'totalVotes': 'Total Votes', 'title': 'Kernel Title'},\n color_continuous_scale=px.colors.qualitative.Set3 # Provide color map\n )\n\nfig.update_layout(\n autosize=False,\n width=1000,\n height=800,\n margin=dict(l=50, r=50, b=150, t=100, pad=4),\n title={'x':0.5, 'xanchor': 'center'},\n template='plotly_white',\n showlegend=False # Hide the legend\n)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:55.085653Z","iopub.execute_input":"2023-07-12T04:05:55.085999Z","iopub.status.idle":"2023-07-12T04:05:55.484083Z","shell.execute_reply.started":"2023-07-12T04:05:55.085970Z","shell.execute_reply":"2023-07-12T04:05:55.482947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Regarding kernels, the picture is a bit different. We can see that the topics with the highest votes, do not necessarily connext directly to generative AI - however they have connecting points through other methods related (f.ex **synthetic data**). **Covid Kernel & Transformer** related topics here take the cake 🍰!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport plotly.express as px\n\n# Create a copy of the filtered DataFrame\nfiltered_df_copy = filtered_df.copy()\n\n# Convert 'lastRunTime' column to datetime\nfiltered_df_copy['lastRunTime'] = pd.to_datetime(filtered_df_copy['lastRunTime'])\n\n# Calculate the sum of total votes per month\nvotes_per_month = filtered_df_copy.groupby(pd.Grouper(key='lastRunTime', freq='M'))['totalVotes'].sum().reset_index()\n\n# Convert 'lastRunTime' to string format\nvotes_per_month['lastRunTime'] = votes_per_month['lastRunTime'].dt.strftime('%Y-%m')\n\n# Create a bar plot using Plotly Express\nfig = px.bar(votes_per_month, x='lastRunTime', y='totalVotes',\n color='totalVotes',\n color_continuous_scale='sunset',\n labels={'lastRunTime': 'Month', 'totalVotes': 'Total Votes'},\n title='Total Votes per Month')\n\nfig.update_layout(\n autosize=False,\n width=800,\n height=500,\n margin=dict(l=50, r=50, b=100, t=100, pad=4),\n title={'x': 0.5, 'xanchor': 'center'},\n template='plotly_white'\n)\n\nfig.update_traces(text=votes_per_month['totalVotes'], textposition='auto') # Add data labels\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:55.485714Z","iopub.execute_input":"2023-07-12T04:05:55.486080Z","iopub.status.idle":"2023-07-12T04:05:55.636863Z","shell.execute_reply.started":"2023-07-12T04:05:55.486028Z","shell.execute_reply":"2023-07-12T04:05:55.635795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Total most votes received per month were obtained in October of the previous year. While this doesnt say much we can observe a minor seasonality - however we would need to review additional data (which is outside of scope)!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport plotly.express as px\n\n# Group the data by author and calculate the sum of votes\ntop_authors_votes = filtered_df.groupby('author')['totalVotes'].sum().reset_index()\n\n# Sort the authors based on votes in descending order\ntop_authors_votes = top_authors_votes.sort_values(by='totalVotes', ascending=False).head(10)\n\n# Define a custom color sequence\ncustom_colors = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd',\n '#8c564b', '#e377c2', '#7f7f7f', '#bcbd22', '#17becf']\n\n# Create a bar plot using Plotly Express with custom colors\nfig = px.bar(top_authors_votes, x='totalVotes', y='author',\n color='author',\n color_discrete_sequence=custom_colors,\n labels={'totalVotes': 'Total Votes', 'author': 'Author'},\n title='Top 10 Authors by Number of Votes')\n\nfig.update_layout(\n autosize=False,\n width=800,\n height=500,\n margin=dict(l=50, r=50, b=100, t=100, pad=4),\n title={'x': 0.5, 'xanchor': 'center'},\n template='plotly_white'\n)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:55.638398Z","iopub.execute_input":"2023-07-12T04:05:55.638722Z","iopub.status.idle":"2023-07-12T04:05:55.770839Z","shell.execute_reply.started":"2023-07-12T04:05:55.638694Z","shell.execute_reply":"2023-07-12T04:05:55.769864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Among the Kagglers, the most beloved Generative AI authors are: @Rohit, @Maxim & @Jeremy (**Congrats to the guys,and big thank you for all the comprehensive notebooks they share**) 🥇!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom bokeh.models import ColumnDataSource, FactorRange, Range1d\nfrom bokeh.transform import factor_cmap\nfrom bokeh.palettes import Spectral10\nfrom bokeh.plotting import figure, show\n\n# Group the data by keyword and calculate the sum of votes\ntop_keywords_votes = filtered_df.groupby('Keyword')['totalVotes'].sum().reset_index()\n\n# Sort the keywords based on votes in descending order and select the top 10\ndf2 = top_keywords_votes.sort_values(by='totalVotes', ascending=False).head(10)\n\n# Reset the index\ndf2.reset_index(drop=True, inplace=True)\n\n# Create a ColumnDataSource from the DataFrame\nsource2 = ColumnDataSource(df2)\n\n# Create the Bokeh figure\np2 = figure(\n width=1000, \n height=600,\n toolbar_location=None,\n title=\"Top 10 Keywords by Number of Votes\",\n x_range=(0, df2['totalVotes'].max() * 1.05), # Add some padding for the labels\n y_range=list(reversed(df2['Keyword'])), # Reverse the y-axis order\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np2.grid.grid_line_color=None\np2.xaxis.axis_label = 'Total Votes'\np2.yaxis.axis_label = 'Keyword'\n\n# Customize the Y-axis labels\np2.yaxis.major_label_text_font_size = '8pt'\np2.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars with different colors for each bar\nbars2 = p2.hbar(\n y='Keyword',\n right='totalVotes',\n height=0.5,\n source=source2,\n fill_color=factor_cmap('Keyword', palette=Spectral10, factors=df2['Keyword'].unique())\n)\n\n# Add labels to the bars\nlabels2 = LabelSet(x='totalVotes', y='Keyword', text='totalVotes', level='glyph',\n x_offset=5, y_offset=-10, source=source2)\n\np2.add_layout(labels2)\n\n# Customize the figure\np2.title.text_font_size = '24px'\np2.title.align = 'center' # Centered title\np2.xaxis.axis_line_color = None\np2.yaxis.axis_line_color = None\np2.xaxis.major_tick_line_color = None\np2.yaxis.major_tick_line_color = None\n\nshow(p2)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:55.772487Z","iopub.execute_input":"2023-07-12T04:05:55.772914Z","iopub.status.idle":"2023-07-12T04:05:56.027991Z","shell.execute_reply.started":"2023-07-12T04:05:55.772858Z","shell.execute_reply":"2023-07-12T04:05:56.026717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Without a surprise - keywords related to **Large Language, Masked Language** and **Diffusion models** are on top together with **video generation**!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom bokeh.models import ColumnDataSource, FactorRange\nfrom bokeh.transform import factor_cmap\nfrom bokeh.palettes import Spectral10\nfrom bokeh.plotting import figure, show\n\n# Group the data by keyword and calculate the count of notebooks\ntop_keywords_notebooks = filtered_df.groupby('Keyword')['ref'].count().reset_index()\n\n# Sort the keywords based on notebook count in descending order and select the top 10\ndf3 = top_keywords_notebooks.sort_values(by='ref', ascending=False).head(10)\n\n# Reset the index\ndf3.reset_index(drop=True, inplace=True)\n\n# Create a ColumnDataSource from the DataFrame\nsource3 = ColumnDataSource(df3)\n\n# Create the Bokeh figure\np3 = figure(\n width=1000, \n height=600,\n toolbar_location=None,\n title=\"Top 10 Keywords by Number of Notebooks\",\n x_range=(0, df3['ref'].max() * 1.05), # Add some padding for the labels\n y_range=list(reversed(df3['Keyword'])), # Reverse the y-axis order\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np3.grid.grid_line_color=None\np3.xaxis.axis_label = 'Number of Notebooks'\np3.yaxis.axis_label = 'Keyword'\n\n# Customize the Y-axis labels\np3.yaxis.major_label_text_font_size = '12pt'\np3.yaxis.major_label_text_color = 'black'\n\n# Add the horizontal bars with different colors for each bar\nbars3 = p3.hbar(\n y='Keyword',\n right='ref',\n height=0.5,\n source=source3,\n fill_color=factor_cmap('Keyword', palette=Spectral10, factors=df3['Keyword'].unique())\n)\n\n# Add labels to the bars\nlabels3 = LabelSet(x='ref', y='Keyword', text='ref', level='glyph',\n x_offset=5, y_offset=-10, source=source3)\n\np3.add_layout(labels3)\n\n# Customize the figure\np3.title.text_font_size = '24px'\np3.title.align = 'center' # Centered title\np3.xaxis.axis_line_color = None\np3.yaxis.axis_line_color = None\np3.xaxis.major_tick_line_color = None\np3.yaxis.major_tick_line_color = None\n\nshow(p3)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:56.029210Z","iopub.execute_input":"2023-07-12T04:05:56.029535Z","iopub.status.idle":"2023-07-12T04:05:56.290854Z","shell.execute_reply.started":"2023-07-12T04:05:56.029500Z","shell.execute_reply":"2023-07-12T04:05:56.289794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here again in a similar way, we can distinguish the dominance of **Diffusion models** and **GAN** training techniques throughout main kernels. Music generation is also a trending topic among Kaggle notebooks for Generative AI!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport subprocess\nfrom IPython.core.display import display as ICD\n\ndef convert_to_df(cmd_output):\n lines = cmd_output.split('\\n')\n \n # Skip lines starting with \"Warning\"\n lines = [line for line in lines if not line.startswith('Warning')]\n \n # Extract column names from the first line and remove the warning message\n columns = lines[0].split(',')\n columns = [column.replace('\\'', '').strip() for column in columns]\n \n # Extract data from subsequent lines\n data = [line.split(',')[:len(columns)] for line in lines[1:] if line]\n \n return pd.DataFrame(data, columns=columns)\n\n\n# Fetch the datasets list from Kaggle\ndatasets_list_csv = subprocess.getoutput('kaggle datasets list --csv')\ndatasets_list_df = convert_to_df(datasets_list_csv)\n\n# Check the column names in the DataFrame\nprint(datasets_list_df.columns)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:56.292221Z","iopub.execute_input":"2023-07-12T04:05:56.292527Z","iopub.status.idle":"2023-07-12T04:05:57.011904Z","shell.execute_reply.started":"2023-07-12T04:05:56.292500Z","shell.execute_reply":"2023-07-12T04:05:57.011127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"
\n🙌 5. Implementation to Impression: Decoding Social Media Perceptions of Generative AI Developments\n
","metadata":{}},{"cell_type":"markdown","source":"Next step, as mentioned above - would be to review how the impact beyond experts of the field has been perceived. First we start with **news surrounding Generative AI** and the general **perception** (together with the **main agents** and **topics** discussed).","metadata":{}},{"cell_type":"code","source":"from GoogleNews import GoogleNews\n\ngooglenews = GoogleNews(lang='en', period='2y')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:57.013136Z","iopub.execute_input":"2023-07-12T04:05:57.013428Z","iopub.status.idle":"2023-07-12T04:05:57.660779Z","shell.execute_reply.started":"2023-07-12T04:05:57.013403Z","shell.execute_reply":"2023-07-12T04:05:57.659659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from GoogleNews import GoogleNews\nimport time\n\n# Initialize the GoogleNews object\ngooglenews = GoogleNews(lang='en')\n\n# Define the keywords and date range\nkeywords = [\n \"generative ai\",\n \"deepfake\",\n \"image generation\",\n \"text generation\",\n \"video generation\",\n \"Music Generation\",\n]\n\nstart_date = '06/06/2021'\nend_date = '06/06/2023'\n\n# Store all news items in this list\nall_news_items = []\n\n# Perform a search for each keyword and retrieve news articles\nfor keyword in keywords:\n googlenews.search(keyword)\n \n # Fetch the first 5 pages of search results\n for page in range(1, 6):\n googlenews.get_page(page)\n \n # Retrieve the search results\n news_items = googlenews.results()\n \n # Append the news_items to all_news_items\n all_news_items.extend(news_items)\n \n # Delay between API requests to avoid rate limit\n time.sleep(5)\n \n # Clear the GoogleNews object\n googlenews.clear()\n\n# Print the number of news articles retrieved\nprint(f\"Total news articles: {len(all_news_items)}\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:05:57.663565Z","iopub.execute_input":"2023-07-12T04:05:57.665234Z","iopub.status.idle":"2023-07-12T04:08:54.563193Z","shell.execute_reply.started":"2023-07-12T04:05:57.665200Z","shell.execute_reply":"2023-07-12T04:08:54.561973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Convert the list of dictionaries to a DataFrame\ndf = pd.DataFrame(all_news_items)\n\n# Display the first few rows of the DataFrame\ndf","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:08:54.565141Z","iopub.execute_input":"2023-07-12T04:08:54.565913Z","iopub.status.idle":"2023-07-12T04:08:54.590636Z","shell.execute_reply.started":"2023-07-12T04:08:54.565843Z","shell.execute_reply":"2023-07-12T04:08:54.589545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.sort_values(by='datetime', ascending=True)\ndf","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:08:54.592398Z","iopub.execute_input":"2023-07-12T04:08:54.592830Z","iopub.status.idle":"2023-07-12T04:08:54.611658Z","shell.execute_reply.started":"2023-07-12T04:08:54.592792Z","shell.execute_reply":"2023-07-12T04:08:54.610608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from plotly.offline import init_notebook_mode\ninit_notebook_mode(connected=True)\n\n# Get top 10 media sources\ntop_media_sources = df['media'].value_counts().head(10).reset_index()\ntop_media_sources.columns = ['Media Source', 'Number of Articles']\n\n# Create bar chart\nfig = px.bar(top_media_sources, x='Media Source', y='Number of Articles', color='Media Source', \n title='Top 10 Media Sources', labels={'Number of Articles':'Number of Articles'},\n color_discrete_sequence=px.colors.qualitative.Set3)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:08:54.613225Z","iopub.execute_input":"2023-07-12T04:08:54.613529Z","iopub.status.idle":"2023-07-12T04:08:54.730335Z","shell.execute_reply.started":"2023-07-12T04:08:54.613504Z","shell.execute_reply":"2023-07-12T04:08:54.729052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, we see that the main media sources discussing and sharing the most news recently about Generative AI are **City Life & Mark Tech Post** - Surprisingly no major brands such as Yahoo Finance made it through the top 10!","metadata":{}},{"cell_type":"code","source":"from transformers import AutoTokenizer, AutoModelForSequenceClassification\nimport torch\nimport pandas as pd\nimport plotly.graph_objects as go\nimport plotly.express as px\nimport re\n\n# Load the model and tokenizer\nmodel_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\nmodel = AutoModelForSequenceClassification.from_pretrained(model_name)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Define a cleaning function\ndef clean_text(text):\n text = re.sub(r'https?:\\/\\/\\S*', '', text, flags=re.MULTILINE) # remove URLs\n text = re.sub(r'\\s+', ' ', text) # remove extra whitespace\n return text\n\n# Define function for sentiment analysis\ndef sentiment_score(text):\n inputs = tokenizer.encode_plus(text, return_tensors='pt')\n output = model(**inputs)\n scores = output[0][0].detach().numpy()\n scores = torch.nn.functional.softmax(torch.from_numpy(scores), dim=0)\n # Normalize score to a scale from 0 to 100\n return (scores.argmax() + 1) * 20 \n\n# Apply cleaning function to title and description\ndf['title'] = df['title'].apply(clean_text)\ndf['desc'] = df['desc'].apply(clean_text)\n\n# Apply sentiment analysis to title and description\ndf['title_sentiment'] = df['title'].apply(sentiment_score)\ndf['desc_sentiment'] = df['desc'].apply(sentiment_score)\n\n# Create a combined sentiment score\ndf['combined_sentiment'] = df[['title_sentiment', 'desc_sentiment']].mean(axis=1)\n\n# Convert datetime to date\ndf['date'] = pd.to_datetime(df['datetime']).dt.date\n\n# Group by date and calculate average sentiment score\ndf_grouped = df.groupby('date').agg({'combined_sentiment': 'mean'}).reset_index()\n\n# Line chart of average sentiment scores over time\nfig = px.line(df_grouped, x='date', y='combined_sentiment', title='Average Combined Sentiment Score Over Time')\nfig.update_layout(\n xaxis_title='Time',\n yaxis_title='Sentiment',\n autosize=False,\n width=800,\n height=600 \n)\nfig.show()\n\n# Output the processed DataFrame\n# df.to_csv('processed_data.csv', index=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:08:54.731962Z","iopub.execute_input":"2023-07-12T04:08:54.732283Z","iopub.status.idle":"2023-07-12T04:13:01.634869Z","shell.execute_reply.started":"2023-07-12T04:08:54.732254Z","shell.execute_reply":"2023-07-12T04:13:01.633647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Regarding public sentiment, we can see that as for the **last month we are having the lowest sentiment score** just now (perhaps a balancing act of many doomsday articles in circulation). However it can still be considered **overall as a good sentiment**, as we have balanced it to be normalized from 0 to 100. ","metadata":{}},{"cell_type":"code","source":"from datetime import datetime, timedelta\n\n# Convert 'date' column to datetime\ndf['date'] = pd.to_datetime(df['date'])\n\n# Get date one week ago\nweek_ago = datetime.now() - timedelta(weeks=1)\n\n# Filter for the last week\ndf_week = df[df['date'] > week_ago]\n\n# Group by date and count the number of articles\ndf_count = df_week.groupby(df_week['date'].dt.date).size().reset_index(name='count')\n\n# Bar chart of the number of articles over time with color scale\nfig = px.bar(df_count, x='date', y='count', title='Number of Articles Over Time',\n color='count', color_continuous_scale='sunset', template='plotly_dark')\n\nfig.update_layout(xaxis_title='Time', yaxis_title='Number of Articles')\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:13:01.637011Z","iopub.execute_input":"2023-07-12T04:13:01.637479Z","iopub.status.idle":"2023-07-12T04:13:01.718977Z","shell.execute_reply.started":"2023-07-12T04:13:01.637436Z","shell.execute_reply":"2023-07-12T04:13:01.717921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also notice a trend of **more and more articles commenting on Generative AI news**. This shows that we havent reached the peak of hype in the technology yet (as far as mainstream media is concerned). However this does match the increasing number of research papers, kaggle discussion and well as notebooks - so again we are not surprised!","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"\n 👓 5.1 Using Reddit to Obtain Addional Insights\n
","metadata":{"execution":{"iopub.status.busy":"2023-07-02T03:24:16.807415Z","iopub.execute_input":"2023-07-02T03:24:16.808049Z","iopub.status.idle":"2023-07-02T03:24:16.815483Z","shell.execute_reply.started":"2023-07-02T03:24:16.807990Z","shell.execute_reply":"2023-07-02T03:24:16.813547Z"}}},{"cell_type":"markdown","source":"Since we couldnt get data from Twitter for this task (or at least the data we obtained with the current limitations, was not enough), we proceed with **Reddit to obtain additional insights** on users perception on Generative AI. We derive data from the main query keyword (Generative AI) - as well as related subreddits mentioned below!","metadata":{}},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nimport praw\nimport pandas as pd\n\n# Initialize UserSecretsClient\nuser_secrets = UserSecretsClient()\n\n# Get secrets\nclient_id = user_secrets.get_secret(\"REDDIT_CLIENT_ID\")\nclient_secret = user_secrets.get_secret(\"REDDIT_SECRET\")\npassword = user_secrets.get_secret(\"REDDIT_PASSWORD\")\n\nreddit = praw.Reddit(\n client_id=client_id,\n client_secret=client_secret,\n password=password,\n user_agent=\"NFTSentiment by u/XhoniShollaj\", \n username=\"XhoniShollaj\", \n)\n\nsubreddits = ['generative', 'generativeAI', 'GenerativeDesign', 'generativeart', 'generativelinguistics']\n\ndata = {}\nfor subreddit_name in subreddits:\n subreddit = reddit.subreddit(subreddit_name)\n data[subreddit_name] = []\n for post in subreddit.new(limit=1000): # limit set to 1000 due to Reddit's limitation\n data[subreddit_name].append({\n 'title': post.title,\n 'score': post.score,\n 'id': post.id,\n 'url': post.url,\n 'created': post.created,\n 'body': post.selftext\n })\n\n# Convert data dictionary to a DataFrame\ndf = pd.DataFrame.from_dict({(i, j): data[i][j] for i in data.keys() for j in range(len(data[i]))}, orient='index')\ndf.index.names = ['subreddit', 'index']\n\n# Print the DataFrame\ndf\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:13:01.720415Z","iopub.execute_input":"2023-07-12T04:13:01.721074Z","iopub.status.idle":"2023-07-12T04:13:26.739911Z","shell.execute_reply.started":"2023-07-12T04:13:01.721042Z","shell.execute_reply":"2023-07-12T04:13:26.738814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Sort DataFrame by 'score' column in descending order\ndf_sorted = df.sort_values('score', ascending=False)\n\n# Get the top 10 titles and their corresponding scores\ntop_10_titles = df_sorted['title'].head(10)\ntop_10_scores = df_sorted['score'].head(10)\n\n# Truncate titles to a certain number of characters (e.g., 50 characters)\ntruncated_titles = [title[:20] + '...' if len(title) > 50 else title for title in top_10_titles]\n\n# Create a bar chart\nplt.figure(figsize=(10, 6))\nplt.bar(truncated_titles, top_10_scores)\nplt.xticks(rotation=90)\nplt.xlabel('Titles')\nplt.ylabel('Number of Votes')\nplt.title('Top 10 Titles by Number of Votes')\nplt.tight_layout()\n\n# Display the plot\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:13:26.741394Z","iopub.execute_input":"2023-07-12T04:13:26.741708Z","iopub.status.idle":"2023-07-12T04:13:27.120756Z","shell.execute_reply.started":"2023-07-12T04:13:26.741681Z","shell.execute_reply":"2023-07-12T04:13:27.119645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The top 10 titles related to Generative AI, for the top amount of votes are hard to be distinguished or classified on the purpose. Therefore we dig in deeper, to see what the users think of anything related to Generative AI.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\nimport torch\nimport re\nimport plotly.graph_objects as go\nimport plotly.express as px\n\n# Load the model and tokenizer\nmodel_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\nmodel = AutoModelForSequenceClassification.from_pretrained(model_name)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Define a cleaning function\ndef clean_text(text):\n text = re.sub(r'https?:\\/\\/\\S*', '', text, flags=re.MULTILINE) # remove URLs\n text = re.sub(r'\\s+', ' ', text) # remove extra whitespace\n return text\n\n# Define function for sentiment analysis\ndef sentiment_score(text):\n if pd.isnull(text): # Skip over NaN values\n return None\n inputs = tokenizer.encode_plus(text, return_tensors='pt', truncation=True, max_length=512)\n output = model(**inputs)\n scores = output[0][0].detach().numpy()\n scores = torch.nn.functional.softmax(torch.from_numpy(scores), dim=0)\n # Normalize score to a scale from 0 to 100\n return (scores.argmax() + 1) * 20 \n\n\n# Apply cleaning function to title and body\ndf['title'] = df['title'].apply(clean_text)\ndf['body'] = df['body'].apply(clean_text)\n\n# Apply sentiment analysis to title and body\ndf['title_sentiment'] = df['title'].apply(sentiment_score)\ndf['body_sentiment'] = df['body'].apply(sentiment_score)\n\n# Create a combined sentiment score\ndf['combined_sentiment'] = df[['title_sentiment', 'body_sentiment']].mean(axis=1)\n\n# Convert Unix timestamp to datetime\ndf['date'] = pd.to_datetime(df['created'], unit='s')\n\n# Filter date range\nstart_date = '2021-06-01'\nend_date = '2023-07-01'\nmask = (df['date'] > start_date) & (df['date'] <= end_date)\ndf = df.loc[mask]\n\n# Group by month and calculate average sentiment score\ndf_grouped = df.resample('M', on='date').agg({'combined_sentiment': 'mean'}).reset_index()\n\n# Line chart of average sentiment scores over time\nfig = px.line(df_grouped, x='date', y='combined_sentiment', title='Average Combined Sentiment Score Over Time')\nfig.update_layout(\n xaxis_title='Time',\n yaxis_title='Sentiment',\n autosize=False,\n width=800,\n height=600 \n)\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:13:27.122688Z","iopub.execute_input":"2023-07-12T04:13:27.123443Z","iopub.status.idle":"2023-07-12T04:16:58.557670Z","shell.execute_reply.started":"2023-07-12T04:13:27.123400Z","shell.execute_reply":"2023-07-12T04:16:58.556378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Overall, the sentiment of Reddit users matches the one from the Google news - **overall balanced to good**, with minor decline mid 2022. ","metadata":{}},{"cell_type":"code","source":"# Sum the scores by subreddit.\ngrouped = df.groupby('subreddit').sum()\n\n# Sort by score and take top 10.\ntop_subreddits = grouped.sort_values('score', ascending=False).head(10)\n\n# Reset the index.\ntop_subreddits = top_subreddits.reset_index()\n\n# Define data source for the plot.\nsource = ColumnDataSource(data=dict(subreddits=top_subreddits['subreddit'], scores=top_subreddits['score']))\n\n# Create a figure.\np = figure(y_range=source.data['subreddits'], width=800, height=400, \n toolbar_location=None, title=\"Top 10 Subreddits by Score\")\n\n# Add horizontal bars to the figure.\np.hbar(y='subreddits', right='scores', height=0.5, source=source, color='navy')\n\n# Customize the plot.\np.x_range.start = 0\np.xaxis.axis_label = 'Score'\np.yaxis.axis_label = 'Subreddits'\np.title.text_font_size = '14pt'\n\n# Display the plot.\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:16:58.559120Z","iopub.execute_input":"2023-07-12T04:16:58.559456Z","iopub.status.idle":"2023-07-12T04:16:58.822010Z","shell.execute_reply.started":"2023-07-12T04:16:58.559426Z","shell.execute_reply":"2023-07-12T04:16:58.821001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generative - as an all encompassing subreddit (not just for Generative AI) is expected. The score is a combined for all posts within the subreddit!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport plotly.express as px\n\n# Convert the 'created' column from Unix time to a normal timestamp\ndf['created'] = pd.to_datetime(df['created'], unit='s')\n\n# Resample to get counts per month\ndf_time = df.resample('M', on='created').size().reset_index(name='posts')\n\n# Rename the columns\ndf_time.columns = ['Date', 'Number of Posts']\n\n# Create a line chart with Plotly Express\nfig = px.line(df_time, x='Date', y='Number of Posts', title='Total Posts Over Time')\n\n# Customize x and y axis labels\nfig.update_layout(\n xaxis_title='Date',\n yaxis_title='Number of Posts',\n autosize=False,\n width=800,\n height=600 \n)\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:16:58.823312Z","iopub.execute_input":"2023-07-12T04:16:58.823625Z","iopub.status.idle":"2023-07-12T04:16:58.897034Z","shell.execute_reply.started":"2023-07-12T04:16:58.823597Z","shell.execute_reply":"2023-07-12T04:16:58.896008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this chart - we have combined the total posts over time for the search term \"Generative AI\" - and as we can see, the **total number of posts has exponentially increased**!","metadata":{}},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nimport praw\nimport pandas as pd\n\n# Initialize UserSecretsClient\nuser_secrets = UserSecretsClient()\n\n# Get secrets\nclient_id = user_secrets.get_secret(\"REDDIT_CLIENT_ID\")\nclient_secret = user_secrets.get_secret(\"REDDIT_SECRET\")\npassword = user_secrets.get_secret(\"REDDIT_PASSWORD\")\n\nreddit = praw.Reddit(\n client_id=client_id,\n client_secret=client_secret,\n password=password,\n user_agent=\"NFTSentiment by u/XhoniShollaj\", \n username=\"XhoniShollaj\", \n)\n\n# Keyword to search for\nkeyword = 'generative ai'\n\n# Query Reddit for the keyword\nposts = reddit.subreddit('all').search(keyword, limit=1000) # limit set to 1000 due to Reddit's limitation\n\n# Initialize data list\ndata = []\n\n# Filter posts based on creation date (June 2021 to July 2023)\nstart_date = pd.Timestamp('2021-06-01')\nend_date = pd.Timestamp('2023-07-31')\n\n# Collect data\nfor post in posts:\n created_date = pd.Timestamp(post.created_utc, unit='s').date()\n if start_date <= created_date <= end_date:\n data.append({\n 'subreddit': post.subreddit.display_name,\n 'title': post.title,\n 'score': post.score,\n 'id': post.id,\n 'url': post.url,\n 'created': post.created,\n 'body': post.selftext\n })\n\n# Convert data list to a DataFrame\nfull_df = pd.DataFrame(data)\n\n# Print the DataFrame\nfull_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:16:58.898613Z","iopub.execute_input":"2023-07-12T04:16:58.898948Z","iopub.status.idle":"2023-07-12T04:17:04.819319Z","shell.execute_reply.started":"2023-07-12T04:16:58.898918Z","shell.execute_reply":"2023-07-12T04:17:04.818312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from bokeh.plotting import figure, show\nfrom bokeh.models import ColumnDataSource, NumeralTickFormatter\nfrom bokeh.io import output_notebook\nfrom bokeh.palettes import Spectral10\n\n# Copy to stop warning\ndf = full_df.copy()\n\n# Truncate the topic titles to the first 30 characters\ndf['truncated_title'] = df['title'].str[:30]\n\n# Sort the DataFrame by 'score' in descending order and select the top 10\ntop_10_df = df.sort_values('score', ascending=False).head(10)\n\n# Truncate the title column in the top_10_df DataFrame\ntop_10_df['truncated_title'] = top_10_df['title'].str[:30]\n\n# Create a ColumnDataSource from the DataFrame\nsource = ColumnDataSource(top_10_df)\n\n# Create the Bokeh figure\np = figure(\n width=800,\n height=400,\n toolbar_location=None,\n title=\"Top 10 Topics by Votes\",\n x_range=list(top_10_df['truncated_title']),\n y_range=(0, top_10_df['score'].max() * 1.1),\n min_border_top=100 # Add space between title and chart\n)\n\n# Customize the grid\np.grid.grid_line_color=None\np.xaxis.axis_label = 'Topic'\np.yaxis.axis_label = 'Votes'\n\n# Rotate x-axis labels for better readability\np.xaxis.major_label_orientation = 1.2\n\n# Customize the Y-axis labels\np.yaxis[0].formatter = NumeralTickFormatter(format=\"0\")\n\n# Add the vertical bars with different colors for each bar\nbars = p.vbar(\n x='truncated_title',\n top='score',\n width=0.8,\n source=source,\n color=Spectral10[:len(top_10_df)],\n hatch_color='black', # Add hatch color for bars\n fill_color='white', # Add fill color for bars\n line_color='black', # Add line color for bars\n)\n\n# Customize the figure\np.title.text_font_size = '24px'\np.title.align = 'center' # Centered title\np.xaxis.axis_line_color = None\np.yaxis.axis_line_color = None\np.xaxis.major_tick_line_color = None\np.yaxis.major_tick_line_color = None\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:17:04.820644Z","iopub.execute_input":"2023-07-12T04:17:04.820957Z","iopub.status.idle":"2023-07-12T04:17:05.094634Z","shell.execute_reply.started":"2023-07-12T04:17:04.820922Z","shell.execute_reply":"2023-07-12T04:17:05.093632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on the dataframe and display of topic engagement ranked, we can see some of the **biggest names (Apple, Adobe, Google etc.) being related to Generative AI** - which means we can expect big moves behind the scenes","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\nimport torch\nimport re\nimport plotly.graph_objects as go\nimport plotly.express as px\n\n# Load the model and tokenizer\nmodel_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\nmodel = AutoModelForSequenceClassification.from_pretrained(model_name)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Define a cleaning function\ndef clean_text(text):\n text = re.sub(r'https?:\\/\\/\\S*', '', text, flags=re.MULTILINE) # remove URLs\n text = re.sub(r'\\s+', ' ', text) # remove extra whitespace\n return text\n\n# Define function for sentiment analysis\ndef sentiment_score(text):\n if pd.isnull(text): # Skip over NaN values\n return None\n inputs = tokenizer.encode_plus(text, return_tensors='pt', truncation=True, max_length=512)\n output = model(**inputs)\n scores = output[0][0].detach().numpy()\n scores = torch.nn.functional.softmax(torch.from_numpy(scores), dim=0)\n # Normalize score to a scale from 0 to 100\n return (scores.argmax() + 1) * 20 \n\n\n# Apply cleaning function to title and body\nfull_df['title'] = full_df['title'].apply(clean_text)\nfull_df['body'] = full_df['body'].apply(clean_text)\n\n# Apply sentiment analysis to title and body\nfull_df['title_sentiment'] = full_df['title'].apply(sentiment_score)\nfull_df['body_sentiment'] = full_df['body'].apply(sentiment_score)\n\n# Create a combined sentiment score\nfull_df['combined_sentiment'] = full_df[['title_sentiment', 'body_sentiment']].mean(axis=1)\n\n# Convert Unix timestamp to datetime\nfull_df['date'] = pd.to_datetime(full_df['created'], unit='s')\n\n# Filter date range\nstart_date = '2021-06-01'\nend_date = '2023-07-01'\nmask = (full_df['date'] > start_date) & (full_df['date'] <= end_date)\nfull_df = full_df.loc[mask]\n\n# Group by month and calculate average sentiment score\ndf_grouped = full_df.resample('M', on='date').agg({'combined_sentiment': 'mean'}).reset_index()\n\n# Line chart of average sentiment scores over time\nfig = px.line(df_grouped, x='date', y='combined_sentiment', title='Average Combined Sentiment Score Over Time')\nfig.update_layout(\n xaxis_title='Time',\n yaxis_title='Sentiment',\n autosize=False,\n width=800,\n height=600 \n)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:17:05.096099Z","iopub.execute_input":"2023-07-12T04:17:05.096432Z","iopub.status.idle":"2023-07-12T04:18:14.167452Z","shell.execute_reply.started":"2023-07-12T04:17:05.096404Z","shell.execute_reply":"2023-07-12T04:18:14.166428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A combined and normalized score for the sentiment for the search query shows an overall **more positive sentiment towards Generative AI**!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport plotly.express as px\n\n# Convert the 'created' column from Unix time to a normal timestamp\nfull_df['created'] = pd.to_datetime(full_df['created'], unit='s')\n\n# Resample to get counts per month\ndf_time = full_df.resample('M', on='created').size().reset_index(name='posts')\n\n# Rename the columns\ndf_time.columns = ['Date', 'Number of Posts']\n\n# Create a line chart with Plotly Express\nfig = px.line(df_time, x='Date', y='Number of Posts', title='Total Posts Over Time')\n\n# Customize x and y axis labels\nfig.update_xaxes(title_text='Date')\nfig.update_yaxes(title_text='Number of Posts')\n\nfig.update_layout(\n xaxis_title='Date',\n yaxis_title='Number of Posts',\n autosize=False,\n width=800,\n height=600 \n)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:18:14.169114Z","iopub.execute_input":"2023-07-12T04:18:14.169685Z","iopub.status.idle":"2023-07-12T04:18:14.242705Z","shell.execute_reply.started":"2023-07-12T04:18:14.169654Z","shell.execute_reply":"2023-07-12T04:18:14.241624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, we have a **sharp increased engagement over time** for Generative AI related keywords specifically (Reddit is no exception)","metadata":{}},{"cell_type":"markdown","source":"\n💻 5.2 YouTube Data Collection: What is the Sentiment Towards Generative AI on Videos\n
","metadata":{}},{"cell_type":"markdown","source":"In order to have a more encompassing overview of the direction in which Generative AI is heading, we also need to take into account **video content** - and YouTube has the largest amount of data. For us, it can be an opportunity to review what content is trending related to the topic and what are the main themes surrounding it.","metadata":{}},{"cell_type":"code","source":"import json\nimport pandas as pd\nfrom kaggle_secrets import UserSecretsClient\nfrom googleapiclient.discovery import build\n\n# Access the secret containing the YouTube API key\nuser_secrets = UserSecretsClient()\napi_key = user_secrets.get_secret(\"YOUTUBE_API_KEY\")\n\n# Creating a YouTube API client\nyoutube = build('youtube', 'v3', developerKey=api_key)\n\n# Defining the search query and parameters\nkeyword = \"Generative AI\"\nmax_results = 50 # Set a default value for maximum results per request\nresults_per_page = 50 # Number of results to retrieve per API request\n\n# Variables to track pagination\nnext_page_token = None\ntotal_results = []\n\n# Loop until all results are retrieved\nwhile True:\n # Make a search request to the API\n search_response = youtube.search().list(\n q=keyword,\n part='id,snippet',\n maxResults=results_per_page,\n pageToken=next_page_token\n ).execute()\n\n # Process the search results\n video_ids = []\n for search_result in search_response.get('items', []):\n if search_result['id']['kind'] == 'youtube#video':\n video_ids.append(search_result['id']['videoId'])\n\n # Get video statistics and details for each video\n videos = []\n for video_id in video_ids:\n video_response = youtube.videos().list(\n part='snippet,statistics',\n id=video_id\n ).execute()\n\n if video_response['items']:\n video_data = video_response['items'][0]\n title = video_data['snippet']['title']\n description = video_data['snippet']['description']\n published_at = video_data['snippet']['publishedAt']\n view_count = video_data['statistics'].get('viewCount', None)\n like_count = video_data['statistics'].get('likeCount', None)\n dislike_count = video_data['statistics'].get('dislikeCount', None)\n comment_count = video_data['statistics'].get('commentCount', None)\n\n video_info = {\n 'video_id': video_id,\n 'title': title,\n 'description': description,\n 'published_at': published_at,\n 'view_count': view_count,\n 'like_count': like_count,\n 'dislike_count': dislike_count,\n 'comment_count': comment_count\n }\n videos.append(video_info)\n\n # Filter videos with titles or descriptions containing \"Generative AI\"\n generative_ai_videos = [video for video in videos if \"Generative AI\" in video['title'] or \"Generative AI\" in video['description']]\n\n # Add the filtered videos to the total results list\n total_results.extend(generative_ai_videos)\n\n # Check if there are more pages of results\n next_page_token = search_response.get('nextPageToken')\n if not next_page_token:\n break\n\n# Convert video information to a DataFrame\ngenerative_ai_videos_df = pd.DataFrame(total_results)\n\n# Print the DataFrame\nprint(generative_ai_videos_df)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-12T04:18:14.244700Z","iopub.execute_input":"2023-07-12T04:18:14.245595Z","iopub.status.idle":"2023-07-12T04:18:38.750416Z","shell.execute_reply.started":"2023-07-12T04:18:14.245553Z","shell.execute_reply":"2023-07-12T04:18:38.749264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generative_ai_videos_df.head(3)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-12T04:18:38.751995Z","iopub.execute_input":"2023-07-12T04:18:38.752423Z","iopub.status.idle":"2023-07-12T04:18:38.768027Z","shell.execute_reply.started":"2023-07-12T04:18:38.752383Z","shell.execute_reply":"2023-07-12T04:18:38.766969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nltk.download('wordnet', quiet=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:18:38.769547Z","iopub.execute_input":"2023-07-12T04:18:38.769866Z","iopub.status.idle":"2023-07-12T04:18:38.924749Z","shell.execute_reply.started":"2023-07-12T04:18:38.769839Z","shell.execute_reply":"2023-07-12T04:18:38.923945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After filtering out for Generative AI, and clearing the data of unnecessary we **initiate topic modeling** in a similar way as before, to review the main topics discussed around the field (or relevant topics)","metadata":{}},{"cell_type":"code","source":"import spacy\nimport pandas as pd\nfrom gensim import corpora\nfrom gensim.models import LdaModel\nimport pyLDAvis.gensim\nfrom IPython.display import display\nimport string\n\nTOKENIZERS_PARALLELISM = True\n\n# Create a Spacy model\nnlp = spacy.load(\"en_core_web_sm\")\n\n# List of stopwords\nfrom spacy.lang.en.stop_words import STOP_WORDS\n\n# Set up punctuation set\nexclude = set(string.punctuation) \n\n# Define a function to clean the document\ndef clean(doc):\n # Parse the document with Spacy\n doc = nlp(doc)\n\n # Use Spacy's built-in stop word list\n stop_free = \" \".join([token.text for token in doc if not token.is_stop])\n\n # Remove punctuation\n punc_free = ''.join(ch for ch in stop_free if ch not in exclude)\n\n # Use Spacy's built-in lemmatizer\n normalized = \" \".join(token.lemma_ for token in nlp(punc_free))\n\n return normalized\n\n# Apply the cleaning function to each description\ndoc_clean = [clean(doc).split() for doc in generative_ai_videos_df['description']]\n\n# Create the term dictionary of our courpus, where every unique term is assigned an index\ndictionary = corpora.Dictionary(doc_clean)\n\n# Convert list of documents (corpus) into Document Term Matrix using dictionary prepared above.\ndoc_term_matrix = [dictionary.doc2bow(doc) for doc in doc_clean]\n\n# Create the object for LDA model using gensim library\nLda = LdaModel\n\n# Train LDA model on the document term matrix. Reduced the number of topics to 5.\nyoutube_ldamodel = Lda(doc_term_matrix, num_topics=5, id2word = dictionary, passes=50)\n\n# Prepare the visualization\npyLDAvis.enable_notebook()\nvis = pyLDAvis.gensim.prepare(youtube_ldamodel, doc_term_matrix, dictionary, sort_topics=False)\n\n# Show the visualization\ndisplay(vis)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:18:38.927130Z","iopub.execute_input":"2023-07-12T04:18:38.928177Z","iopub.status.idle":"2023-07-12T04:19:30.200638Z","shell.execute_reply.started":"2023-07-12T04:18:38.928136Z","shell.execute_reply":"2023-07-12T04:19:30.199117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from gensim.models.coherencemodel import CoherenceModel\n\nTOKENIZERS_PARALLELISM=True\n\n# Compute Perplexity\nprint('\\nPerplexity: ', youtube_ldamodel.log_perplexity(corpus)) # a measure of how good the model is. lower the better.","metadata":{"execution":{"iopub.status.busy":"2023-07-12T04:19:30.202752Z","iopub.execute_input":"2023-07-12T04:19:30.203229Z","iopub.status.idle":"2023-07-12T04:19:38.017318Z","shell.execute_reply.started":"2023-07-12T04:19:30.203178Z","shell.execute_reply":"2023-07-12T04:19:38.016087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that perplexity score is low - which indicates an overall **good model performance**!","metadata":{}},{"cell_type":"code","source":"pprint(youtube_ldamodel.print_topics())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:19:38.018832Z","iopub.execute_input":"2023-07-12T04:19:38.019236Z","iopub.status.idle":"2023-07-12T04:19:38.027026Z","shell.execute_reply.started":"2023-07-12T04:19:38.019208Z","shell.execute_reply":"2023-07-12T04:19:38.025936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The topics can be tentatively interpreted as follows:\n\n0. **Applications of AI in Gaming and Entertainment**: Terms like \"AI\", \"Generative\", \"NVIDIA\", \"ChatGPT\", and \"Games\" suggest that this topic is about the use of artificial intelligence in gaming and generative tasks, possibly in relation to NVIDIA's technology and OpenAI's ChatGPT.\n\n1. **Generative AI and Video Content Creation**: This topic, featuring words like \"AI\", \"Generative\", \"video\", \"content\", \"create\", and \"generate\", appears to be about the use of AI, particularly generative models, in creating and generating video content.\n\n2. **Google's AI and Generative Content**: The presence of \"AI\", \"generative\", \"Google\", and \"content\" suggest that this topic is about Google's involvement in AI and generative content, possibly related to applications like Google's \"Bard\".\n\n3. **Learning with Generative AI**: This topic, with words like \"AI\", \"Generative\", \"learn\", and \"use\", seems to focus on the learning aspects of generative AI, possibly covering how AI can learn to generate new content or features.\n\n4. **AI, Generative Models, and Data**: The words \"AI\", \"Generative\", \"Data\", and \"learn\" indicate that this topic is about the intersection of AI and data, particularly with a focus on learning from data using generative models.","metadata":{}},{"cell_type":"markdown","source":"After reviewing the themes - we move on to see engagement metrics surrounding the topics. Here we can distinguish the distribution based on metrics for the **hottest topics surrounding Generative AI**.","metadata":{}},{"cell_type":"code","source":"# Convert columns to numeric\ngenerative_ai_videos_df['like_count'] = pd.to_numeric(generative_ai_videos_df['like_count'], errors='coerce')\ngenerative_ai_videos_df['comment_count'] = pd.to_numeric(generative_ai_videos_df['comment_count'], errors='coerce')\ngenerative_ai_videos_df['view_count'] = pd.to_numeric(generative_ai_videos_df['view_count'], errors='coerce')\n\n# Calculate Engagement Score and add it to the DataFrame\ngenerative_ai_videos_df['engagement_score'] = (generative_ai_videos_df['like_count'] + generative_ai_videos_df['comment_count']) / generative_ai_videos_df['view_count']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:19:38.029136Z","iopub.execute_input":"2023-07-12T04:19:38.029675Z","iopub.status.idle":"2023-07-12T04:19:38.043663Z","shell.execute_reply.started":"2023-07-12T04:19:38.029637Z","shell.execute_reply":"2023-07-12T04:19:38.042529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from bokeh.io import show, output_notebook\nfrom bokeh.models import ColumnDataSource\nfrom bokeh.palettes import Reds\nfrom bokeh.plotting import figure\nfrom bokeh.transform import factor_cmap\n\noutput_notebook()\n\n# Sort by 'view_count'\ntop_views = generative_ai_videos_df.sort_values('view_count', ascending=False).head(10)\n\n# Append index to truncated title to ensure uniqueness\ntop_views['title'] = top_views['title'].str[:35] + \" (\" + top_views.index.astype(str) + \")\"\n\n\nsource = ColumnDataSource(data=dict(\n x=top_views['title'],\n y=top_views['view_count'],\n))\n\np = figure(x_range=top_views['title'], height=350, toolbar_location=None, title=\"Top Videos by Views\")\np.vbar(x='x', top='y', width=0.9, source=source, line_color='white',\n fill_color=factor_cmap('x', palette=Reds[7], factors=top_views['title']))\n\np.xgrid.grid_line_color = None\np.y_range.start = 0\np.xaxis.major_label_orientation = 1.2\n\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:34:14.830896Z","iopub.execute_input":"2023-07-12T04:34:14.831419Z","iopub.status.idle":"2023-07-12T04:34:15.277571Z","shell.execute_reply.started":"2023-07-12T04:34:14.831381Z","shell.execute_reply":"2023-07-12T04:34:15.276638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Deriving from the dataframe and visualization, we can discern that main topics driving Generative AI videos by views are related to **instuctional use, new technologies or Politics and AI** all else aside. Now lets extend this to view which videos rank highest by likes and by total engagement (likes + views + comments)","metadata":{}},{"cell_type":"code","source":"from bokeh.io import show, output_notebook\nfrom bokeh.models import ColumnDataSource\nfrom bokeh.plotting import figure\nfrom bokeh.transform import factor_cmap\nfrom bokeh.palettes import Blues, Reds, YlOrBr\n\noutput_notebook()\n\n# Sort by 'view_count'\ntop_views = generative_ai_videos_df.sort_values('view_count', ascending=False).head(10)\n\n# Truncate title to 35 characters\ntop_views['title'] = top_views['title'].str[:35]\n\nsource_views = ColumnDataSource(data=dict(\n x=top_views['title'],\n y=top_views['view_count'],\n))\n\np_views = figure(x_range=top_views['title'], height=350, toolbar_location=None, title=\"Top Videos by Views\")\np_views.vbar(x='x', top='y', width=0.9, source=source_views, line_color='white',\n fill_color=factor_cmap('x', palette=Blues[7], factors=top_views['title']))\n\np_views.xgrid.grid_line_color = None\np_views.y_range.start = 0\np_views.xaxis.major_label_orientation = 1.2\n\nshow(p_views)\n\n# Sort by 'like_count'\ntop_likes = generative_ai_videos_df.sort_values('like_count', ascending=False).head(10)\n\n# Truncate title to 35 characters\ntop_likes['title'] = top_likes['title'].str[:35] + \" (\" + top_likes.index.astype(str) + \")\"\n\n\nsource_likes = ColumnDataSource(data=dict(\n x=top_likes['title'],\n y=top_likes['like_count'],\n))\n\np_likes = figure(x_range=top_likes['title'], height=350, toolbar_location=None, title=\"Top Videos by Likes\")\np_likes.vbar(x='x', top='y', width=0.9, source=source_likes, line_color='white',\n fill_color=factor_cmap('x', palette=Reds[7], factors=top_likes['title']))\n\np_likes.xgrid.grid_line_color = None\np_likes.y_range.start = 0\np_likes.xaxis.major_label_orientation = 1.2\n\nshow(p_likes)\n\n# Calculate Engagement Score\ngenerative_ai_videos_df['engagement_score'] = generative_ai_videos_df['like_count'] + generative_ai_videos_df['comment_count']\n\n# Sort by 'engagement_score'\ntop_engagement = generative_ai_videos_df.sort_values('engagement_score', ascending=False).head(10)\n\n# Truncate title to 40 characters\ntop_engagement['title'] = top_engagement['title'].str[:35] + \" (\" + top_engagement.index.astype(str) + \")\"\n\nsource_engagement = ColumnDataSource(data=dict(\n x=top_engagement['title'],\n y=top_engagement['engagement_score'],\n))\n\np_engagement = figure(x_range=top_engagement['title'], height=350, toolbar_location=None, title=\"Top Videos by Engagement Score\")\np_engagement.vbar(x='x', top='y', width=0.9, source=source_engagement, line_color='white',\n fill_color=factor_cmap('x', palette=YlOrBr[7], factors=top_engagement['title']))\n\np_engagement.xgrid.grid_line_color = None\np_engagement.y_range.start = 0\np_engagement.xaxis.major_label_orientation = 1.2\n\nshow(p_engagement)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:35:48.312633Z","iopub.execute_input":"2023-07-12T04:35:48.313145Z","iopub.status.idle":"2023-07-12T04:35:49.643755Z","shell.execute_reply.started":"2023-07-12T04:35:48.313108Z","shell.execute_reply":"2023-07-12T04:35:49.642728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In a more comprehensive overview - we can see that for all the mentioned metrics and visuals, the titles still have barely changed - worth to note that **NVIDIA above all companies is making magic** in the back; truly inspiring to see all the innovation! Now, lets review the total videos over time for the topic.","metadata":{}},{"cell_type":"code","source":"from bokeh.plotting import figure\nfrom bokeh.io import show, output_notebook\nfrom bokeh.palettes import Category10\nfrom bokeh.models import DatetimeTickFormatter\n\n# Convert 'published_at' column to datetime\ngenerative_ai_videos_df['published_at'] = pd.to_datetime(generative_ai_videos_df['published_at'])\n\n# Sort the DataFrame by 'published_at'\ngenerative_ai_videos_df = generative_ai_videos_df.sort_values('published_at')\n\n# Count the number of videos by date\nvideo_counts = generative_ai_videos_df['published_at'].value_counts().sort_index().cumsum()\n\n# Create a line plot\np = figure(x_axis_type='datetime', title='Total Posts over Time')\np.line(x=video_counts.index, y=video_counts.values, line_width=2, color=Category10[10][0])\n\n# Format the x-axis ticks as dates\np.xaxis.formatter = DatetimeTickFormatter(days=['%Y-%m-%d'])\n\n# Add axis labels\np.xaxis.axis_label = 'Date'\np.yaxis.axis_label = 'Total Videos'\n\n# Show the plot\noutput_notebook()\nshow(p)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:19:39.222280Z","iopub.execute_input":"2023-07-12T04:19:39.223126Z","iopub.status.idle":"2023-07-12T04:19:39.580560Z","shell.execute_reply.started":"2023-07-12T04:19:39.223093Z","shell.execute_reply":"2023-07-12T04:19:39.579451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In a similar fashion, the way in which Generative AI has been incorporated in video content is increasing in an exponential way. This shows that the topics mentioned ","metadata":{}},{"cell_type":"code","source":"from transformers import AutoTokenizer, AutoModelForSequenceClassification\nimport torch\nimport pandas as pd\nimport plotly.graph_objects as go\nimport plotly.express as px\nimport re\n\n# Load the model and tokenizer\nmodel_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\nmodel = AutoModelForSequenceClassification.from_pretrained(model_name)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Define a cleaning function\ndef clean_text(text):\n text = re.sub(r'https?:\\/\\/\\S*', '', text, flags=re.MULTILINE) # remove URLs\n text = re.sub(r'\\s+', ' ', text) # remove extra whitespace\n return text\n\n# Define function for sentiment analysis\ndef sentiment_score(text):\n inputs = tokenizer.encode_plus(text, return_tensors='pt', truncation=True, padding=True)\n output = model(**inputs)\n scores = output.logits.softmax(dim=1)\n sentiment_score = (scores.argmax(dim=1) + 1).item() * 20\n return sentiment_score\n\n# Apply cleaning function to title and description\ngenerative_ai_videos_df['title'] = generative_ai_videos_df['title'].apply(clean_text)\ngenerative_ai_videos_df['description'] = generative_ai_videos_df['description'].apply(clean_text)\n\n# Convert published_at to datetime\ngenerative_ai_videos_df['published_at'] = pd.to_datetime(generative_ai_videos_df['published_at'])\n\n# Extract year and month from the published_at column\ngenerative_ai_videos_df['year'] = generative_ai_videos_df['published_at'].dt.year\ngenerative_ai_videos_df['month'] = generative_ai_videos_df['published_at'].dt.month\n\n# Apply sentiment analysis to title and description\ngenerative_ai_videos_df['title_sentiment'] = generative_ai_videos_df['title'].apply(sentiment_score)\ngenerative_ai_videos_df['description_sentiment'] = generative_ai_videos_df['description'].apply(sentiment_score)\n\n# Create a combined sentiment score\ngenerative_ai_videos_df['combined_sentiment'] = generative_ai_videos_df[['title_sentiment', 'description_sentiment']].mean(axis=1)\n\n# Group by year and month, calculate average sentiment score\ndf_grouped = generative_ai_videos_df.groupby(['year', 'month']).agg({'combined_sentiment': 'mean'}).reset_index()\n\n# Convert year and month columns to datetime\ndf_grouped['date'] = pd.to_datetime(df_grouped[['year', 'month']].assign(day=1))\n\n# Line chart of average sentiment scores over time\nfig = px.line(df_grouped, x='date', y='combined_sentiment', title='Average Combined Sentiment Score Over Time')\nfig.update_layout(\n xaxis_title='Time',\n yaxis_title='Sentiment',\n autosize=False,\n width=800, # adjust these as needed\n height=600 # adjust these as needed\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:19:39.582123Z","iopub.execute_input":"2023-07-12T04:19:39.582461Z","iopub.status.idle":"2023-07-12T04:22:41.561981Z","shell.execute_reply.started":"2023-07-12T04:19:39.582431Z","shell.execute_reply":"2023-07-12T04:22:41.560915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":" \n 🚩 6.1 Limitations of Generative AI\n
","metadata":{}},{"cell_type":"markdown","source":"Even though the technology is accelerating at an incredible rate of development, its important to understand some of the main limitations which are **currently halting certain aspects of Generative AI** (where we can also add Large Language Models). Below we have several categories where we discuss the main limitations:","metadata":{}},{"cell_type":"markdown","source":" \n 6.1.1 Data Limitations & Risks\n
\n\n- Training Language or Diffusion models comes with huge volumes of data - most which is not usually **the right data we need** (could require a lot of efforts and computational power in **pre-processing** to the right format).\n\n- Also, a lot of the data gathered by many models includes data gathered from sites which do not necessarily agree with the terms, conditions and usage of data (f.ex Stack Overflow). Private data without user consent has been utilized, which could lead to **huge legal implications**, which are underway!\n\n- Finally, training data gathered online, usually involves **discriminatory and toxic data** (that could further enhance **exclusion and divisive views**). There is huge ongoing debate to how to produce more neutral views from the output coming from Generative AI models. With the rise of DeepFakes, and audio models, there are also many implications in breaching privacy and cyberbullying (topics which need to be addressed further)!","metadata":{}},{"cell_type":"markdown","source":" \n 6.1.2 Bad Actors and Misuse\n
\n\n- Generative AI, has the potential to greatly **enhance the impact of bad actors** through misuse of the models. There are several ways in which Generative AI can be used for bad, such as:\n\n - **Information Hazard** (Greatly enhancing the spread of misinformation in form of fake images, audio, videos and written articles)\n \n - **Malicious Uses** (F.ex exploitive manipulative agents in social media, scams and overall human-computer interaction harm)\n \n - **Cyber-Bullying** (F.ex using deepfakes to attach reputation, online presence or simply blackmailing certain individuals)","metadata":{}},{"cell_type":"markdown","source":" \n 6.1.3 Negative Impact in Society\n
\n\n- As we will see also later in a more comprehensive breakdown below, Generative AI will massively affect the job market (initially for numerous white collar jobs), creating a **huge imbalance in demand for certain skillsets** (while rendering some almost completely useless). This imbalance can be further enhanced for **specific markets and countries** relying on certain skillsets.\n\n- Its worth noting that Generative AI has huge impact in the environment and carbon footprint. The computing resources utilized for training a base transformer is estimated at 284 tonnes of CO2 (Bender et al 2021).\n\n- **Financial cost** of to train from scratch is about 1USD for 1k parameters - to train ChatGPT with estimated 175 billion parameters can be approximately 175 Million USD. The value of training generative ai models with substantial impact cannot be underestimated.","metadata":{}},{"cell_type":"markdown","source":" \n 6.1.4 Hallucinations\n
","metadata":{}},{"cell_type":"markdown","source":"A hallucination is a generally novel concept, arising from the latest developments of Generative AI, and can be defined as **Generated content that is nonsensical or \nunfaithful to the provided source content** (Ji at al, 2022). \n\nUsually these models output give the impression that the answer is fluent and natural - even though it can be fatally erroneous!\n\nWe can classify hallucinations as Intrinsic or Extrinsic.\n\n- **Intrinsic**: Output in this case contradicts the source (f.ex giving the wrong date of an event). An example can be shown below:\n\n\"Source:\nThe first Ebola vaccine was approved \nby the FDA in 2019, five years after the \ninitial outbreak in 2014\"\n \nSummary output: \nThe first Ebola vaccine was approved \nin 2021\n\n\n- **Extrinsic**: In this case we cannot verify output from the source, but it might not be wrong (f.ex adding unnecessary information on top of a given fact). An example can be displayed as follows:\n \nSource:\nAlice won first prize in fencing last week\n \nOutput:\nAlice won first prize fencing for the \nfirst time last week ***and she was \necstatic.\n","metadata":{}},{"cell_type":"markdown","source":"The problem of hallucination can be traced to some of the following issues (though not limited to):\n\n- Data Collection\n- Imperfect Encoder Learning\n- Erroneous Decoding\n- Exposure Bias\n- Parametric Knowledge Bias\n- Open ended nature of GenAI","metadata":{}},{"cell_type":"markdown","source":"Lets review below a more comprehensive view on how Generative AI is **currently having an impact in society** (quatifyiable by sources such as McKinsey). Focus is mainly on the economic side!","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"\n💢 6.2 Gauging the Real-World Impact and Value Add of Generative AI\n
\n","metadata":{}},{"cell_type":"markdown","source":"There are many reports currently trying to effectively measure the impact of Generative AI in numbers - the most reliable and relevant one we could find was the [The economic potential of generative AI: The next productivity frontier](#References).(Review the mentioned report for more details). The findings summary are reported as below:\n\n1. **Generative AI** could add between **2.6 trillion and 4.4 trillion dollars annually** across 63 use cases, thereby increasing the impact of all artificial intelligence by **15 to 40 percent**.\n\n2. **75 percent** of the value that generative AI use cases could deliver is across four areas: customer operations, marketing and sales, software engineering, and R&D.\n\n3. Generative AI has the potential to **automate 60 to 70 percent** of employees' work activities, marking an increase from previous estimates.\n\n4. Given the potential for technical automation, the pace of workforce transformation is set to accelerate with estimates that **half of today's work activities** could be automated between **2030 and 2060**.\n\n5. Generative AI could enable labor productivity growth of **0.1 to 0.6 percent annually** through 2040, adding 0.2 to 3.3 percentage points to productivity growth annually when combined with other technologies.\n\n6. When applied in 63 use cases across industries, generative AI could deliver economic benefits ranging from **2.6 trillion to 4.4 trillion annually**.\n\n7. The total economic benefits of generative AI, including major use cases and increased productivity across knowledge workers’ activities, could amount to **6.1 trillion to 7.9 trillion annually**.\n\n8. **Generative AI** has potential in opening new avenues for creativity and innovation, and is especially impactful in four main business functions: customer operations, marketing and sales, software engineering, and research and development. These areas could account for approximately **75 percent** of the total annual value from generative AI use cases.\n\n9. In the field of internal knowledge management, Generative AI can enhance efficiency, considering that knowledge workers used to spend **a fifth of their time** searching for information. By taking over such tasks, Generative AI can make this process more efficient and effective.\n\n10. A case study demonstrated that generative AI increased issue resolution by **14 percent** per hour and reduced time spent handling an issue by **9 percent** in customer operations. It also led to a decrease in agent attrition and managerial requests by **25 percent**.\n\n11. Further, Generative AI could reduce the volume of human-serviced customer contacts by up to **50 percent**, depending on a company’s existing level of automation.\n\n12. Generative AI application in customer care functions could increase productivity at a value ranging from **30 to 45 percent** of current function costs. The estimated impacts do not account for potential improvements in customer satisfaction and retention arising from an enhanced experience.\n\nSo Overall:\n\n**Generative AI** holds the potential to generate a value of **2.6 trillion to 4.4 trillion** across industries. The technology could contribute roughly **310 billion** in additional value for the retail industry by enhancing functions like marketing and customer interactions. Meanwhile, for the high tech industry, the primary value comes from improving the speed and efficiency of software development. \n\nIn the **banking industry**, generative AI can enhance efficiencies by performing lower-value tasks in risk management, such as required reporting, monitoring regulatory developments, and collecting data. \n\nIn the **retail and consumer packaged goods (CPG) industry**, the technology can increase productivity by 1.2 to 2.0 percent of annual revenues, or an additional 400 billion to 660 billion, by automating key functions like customer service, marketing and sales, and inventory and supply chain management.\n\nIn the **banking industry**, generative AI could increase productivity by 2.8 to 4.7 percent of the industry’s annual revenues, or an additional $200 billion to $340 billion. It could also enhance customer satisfaction, improve decision making and employee experience, and decrease risks through better monitoring of fraud and risk.","metadata":{}},{"cell_type":"markdown","source":"**Also, what we think should be highlighted is how fast the technology is moving and killing many websites (f.ex below we can see a sharp drop in websites such as Mathematic or Stack Overflow - analysis derived from stack exchange data query)**:","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom bokeh.plotting import figure, show, output_file\nfrom bokeh.io import output_notebook\nfrom bokeh.models import Span, Grid\nfrom bokeh.palettes import Spectral4\n\n# load data\ndata = pd.read_csv(\"/kaggle/input/stackdata/StackData.csv\")\n\n# Create a date from Year, Month columns\ndata['date'] = pd.to_datetime(data[['Year', 'Month']].assign(Day=1))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:22:41.563330Z","iopub.execute_input":"2023-07-12T04:22:41.563646Z","iopub.status.idle":"2023-07-12T04:22:41.583868Z","shell.execute_reply.started":"2023-07-12T04:22:41.563617Z","shell.execute_reply":"2023-07-12T04:22:41.582935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nfrom bokeh.plotting import figure, show, output_file\nfrom bokeh.io import output_notebook\nfrom bokeh.models import Span, Grid\nfrom bokeh.palettes import Spectral4\n\n# load data\ndata = pd.read_csv(\"/kaggle/input/stackdata/StackData.csv\")\n\n# Create a date from Year, Month columns\ndata['date'] = pd.to_datetime(data[['Year', 'Month']].assign(Day=1))\n\noutput_notebook()\n\n# Create the first figure and line chart\np1 = figure(x_axis_type=\"datetime\", title=\"Mathematics Post Count Over Time\", height=350, width=800, background_fill_color='#ffffff')\n\np1.grid.grid_line_color=None\np1.xaxis.axis_label = 'Time'\np1.yaxis.axis_label = 'Post Count'\n\np1.line(data['date'], data['Mathematics Post Count'], color='#CC0000', line_width=2)\n\n# Output to file\noutput_file(\"MathematicsPostCount.html\", title=\"Mathematics Post Count Over Time\")\n\n# Show the first plot\nshow(p1)\n\n# Create the second figure and line chart\np2 = figure(x_axis_type=\"datetime\", title=\"Stack Overflow Post Count Over Time\", height=350, width=800, background_fill_color='#ffffff')\n\np2.grid.grid_line_color=None\np2.xaxis.axis_label = 'Time'\np2.yaxis.axis_label = 'Post Count'\n\np2.line(data['date'], data['Stack Overflow Post Count'], color='#000000', line_width=2)\n\n# Output to file\noutput_file(\"StackOverflowPostCount.html\", title=\"Stack Overflow Post Count Over Time\")\n\n# Show the second plot\nshow(p2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T04:22:41.585235Z","iopub.execute_input":"2023-07-12T04:22:41.585539Z","iopub.status.idle":"2023-07-12T04:22:42.515796Z","shell.execute_reply.started":"2023-07-12T04:22:41.585506Z","shell.execute_reply":"2023-07-12T04:22:42.514893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can derive as a lesson that more websites need to start utilizing Generative AI, in a way in which can bring value to the overall public (or they will be left behind - and face potential collapse)!","metadata":{}},{"cell_type":"markdown","source":"***","metadata":{}},{"cell_type":"markdown","source":"# Summary\n","metadata":{}},{"cell_type":"markdown","source":"- Generative AI is a new technology which has started developing, arguably since the inception of machine learning.🗓\n- For the past 2 years we have experience at rapid growth in research papers being devoted to Generative AI, more than doubling from 2 years ago. 📈\n- The dominant themes across most research papers revolve around **Language Processing, Diffusion Techniques**, and overall **Optimization/Finetuning** of AI Models 📚\n- Among the top scientists publishing high impact research papers in the field are [Luke Zettlemoyer](https://www.cs.washington.edu/people/faculty/lsz), and [Jianfeng Gao](https://www.microsoft.com/en-us/research/people/jfgao/) among other top scientists mentioned in the list. 👨🔬\n- The main institutions from which these high impact scientists are from include, but are not limited to **Microsoft** & **Washington University**; noted for their high active contribution to the field of Generative AI 🏪\n- On **Kaggle** we found 5 relevant competitions related to Generative AI (with 2 still ongoing) - the main winning models were **ensemble techniques**, combining different methods and finetuning them. 🧪\n- The number of **active topics on Kaggle** around Generative AI or any subtopics has increased to more than x7 times then 2 years ago. 💹\n- From the most Kaggle active topics, the main themes and titles are mostly related to **ChatGPT, GPT Models, or Stable Diffusion.** 🏛\n- Regarding **Kaggle Notebooks**, the most voted and engaged notebooks are usually introductory, end-to-end, tutorial style notebooks connected to **State of the Art Models** (f.ex BERT) or **underlying themes** (f.ex COVID). 📓\n- Among the most distinguished Kagglers in Generative AI, by number of votes are [Rohit Singh](https://www.kaggle.com/rhtsingh), [Maksim Ekin Eren](https://github.com/MaksimEkin) and [Jeremy Howard](https://www.kaggle.com/jhoward). 🥇\n- The main themes and keywords surrounding Generative AI, in Kaggle notebooks are **Large Language Models, Diffusion Models & GAN**. 🗺\n- Regarding **social media, and news** - the topic has gained considerable traction for the past 2 years, and has an overall positive sentiment throughout. 🙂\n- **Themes** in social media and news sources revolve around **moves by big companies, video/music content creation and tutorials** teaching about Generative AI. 📰\n- With regards to real impact, we can say that Generative AI has been pushing companies to **considerable drop in users if they dont adapt**, and economic projections show potential to generate a value of **2.6 trillion to 4.4 trillion** dollars across industries. 💰\n- However, Generative AI will also **kill, or push to adapt many companies** (f.ex Stack Overflow, Mathematica etc.) - especially in the field of education. ☠\n- All in all, Generative AI is an exciting field of research, discussion and innovation - as long as we keep it safe, follow the ethica guidelines and respect privacy, this tech holds the potential to make our world a better place :) ","metadata":{}},{"cell_type":"markdown","source":"# References","metadata":{}},{"cell_type":"markdown","source":"1. McKinsey & Company. (2023, January 19). What is generative AI? Retrieved from https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-generative-ai\n2. Anyoha, R. (2017, August 28). The History of Artificial Intelligence. SITN. Retrieved from https://sitn.hms.harvard.edu/flash/2017/history-artificial-intelligence/\n3. Baum, L. E., & Petrie, T. (1966). Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics, 37(6), 1554–1563. doi:10.1214/aoms/1177699147\n4. Amari, Shun-Ichi (1972). \"Learning patterns and pattern sequences by self-organizing nets of threshold elements\". IEEE Transactions. C (21): 1197–1206.\n5. Hochreiter, Sepp; Schmidhuber, Jürgen (1997-11-01). \"Long Short-Term Memory\". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276. S2CID 1915014.\n6. Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (1986a). \"Learning representations by back-propagating errors\". Nature. 323 (6088): 533–536. Bibcode:1986Natur.323..533R. doi:10.1038/323533a0. S2CID 205001834.\n7. Jesuthasan, T. (2021, July 31). Autoregressive (AR) Language Modeling. Medium. Retrieved from https://tonyjesuthasan.medium.com/autoregressive-ar-language-modelling-c9fe5c20aa6e\n8. Kumar, A. (2021, May 20). The Illustrated BERT Masked Language Modeling. Retrieved from https://ankur3107.github.io/blogs/masked-langauge-modeling/#:~:text=Masked%20Language%20Modeling%20is%20a%20fill-in-the-blank%20task%2C%20where,will%20generate%20the%20most%20likely%20substitution%20for%20each.\n9. Muller, B. (2022, March 2). BERT 101 🤗 State Of The Art NLP Model Explained. Retrieved from https://huggingface.co/blog/bert-101\n10. \"GPT-3.\" (n.d.). In Wikipedi from https://en.wikipedia.org/wiki/GPT-3#cite_note-2018_Attention_Paper-2\n11. Goodfellow, Ian; Pouget-Abadie, Jean; Mirza, Mehdi; Xu, Bing; Warde-Farley, David; Ozair, Sherjil; Courville, Aaron; Bengio, Yoshua (2014). Generative Adversarial Nets (PDF). Proceedings of the International Conference on Neural Information Processing Systems (NIPS 2014). pp. 2672–2680.\n12. Fedus, W., Zoph, B., & Shazeer, N. (2021, January 11). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv preprint arXiv:2101.03961 [cs.LG]. Retrieved from arXiv:2101.03961\n13. Denton, E., Chintala, S., Szlam, A., & Fergus, R. (2015, June 18). Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. arXiv preprint arXiv:1506.05751 [cs.CV]. Retrieved from arXiv:1506.05751\n14. Radford, A., Metz, L., & Chintala, S. (2015, November 19). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv preprint arXiv:1511.06434 [cs.LG]. Retrieved from arXiv:1511.06434\n15. Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017, October 27). Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv preprint arXiv:1710.10196 [cs.NE]. Retrieved from arXiv:1710.10196\n16. Nguyen, T. D., Le, T., Vu, H., & Phung, D. (n.d.). Dual Discriminator Generative Adversarial Nets. arXiv preprint arXiv:1709.03831 [cs.LG]. Retrieved from arXiv:1709.03831\n17. Zheng, C., Fan, X., Wang, C., & Qi, J. (2019, November 11). GMAN: A Graph Multi-Attention Network for Traffic Prediction. arXiv preprint arXiv:1911.08415 [eess.SP]. Retrieved from arXiv:1911.08415\n18. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016, June 12). InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. arXiv preprint arXiv:1606.03657 [cs.LG]. Retrieved from arXiv:1606.03657\n19. Mirza, M., & Osindero, S. (2014, November 6). Conditional Generative Adversarial Nets. arXiv preprint arXiv:1411.1784 [cs.LG]. Retrieved from arXiv:1411.1784\n20. Fedus, W., Zoph, B., & Shazeer, N. (2021, January 11). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv preprint arXiv:2101.03961 [cs.LG]. Retrieved from arXiv:2101.03961\n21. Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A. (2017, March 29). Tacotron: Towards End-to-End Speech Synthesis. arXiv preprint arXiv:1703.10135 [cs.CL]. Retrieved from arXiv:1703.10135\n22. Li, N., Liu, S., Liu, Y., Zhao, S., Liu, M., & Zhou, M. (2018, September 19). Neural Speech Synthesis with Transformer Network. arXiv preprint arXiv:1809.08895 [cs.CL]. Retrieved from arXiv:1809.08895\n23. Kong, J., Kim, J., & Bae, J. (2020, October 12). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. arXiv preprint arXiv:2010.05646 [cs.SD]. Retrieved from arXiv:2010.05646\n24. Yamamoto, R., Song, E., & Kim, J. (2019, October 25). Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. arXiv preprint arXiv:1910.11480 [eess.AS]. Retrieved from arXiv:1910.11480\n25. Yang, J., Liu, Z., Xiao, S., Li, C., Lian, D., Agrawal, S., Singh, A., Sun, G., & Xie, X. (2021, May 6). GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. arXiv preprint arXiv:2105.02605 [cs.CL]. Retrieved from arXiv:2105.02605\n26. You, J., Ying, R., Ren, X., Hamilton, W. L., & Leskovec, J. (2018, February 24). GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models. arXiv preprint arXiv:1802.08773 [cs.LG]. Retrieved from arXiv:1802.08773\n27. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., & Zhou, M. (2020, February 19). CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv preprint arXiv:2002.08155 [cs.CL]. Retrieved from arXiv:2002.08155\n28. Kanade, A., Maniatis, P., Balakrishnan, G., & Shi, K. (2019, December 21). Learning and Evaluating Contextual Embedding of Source Code. arXiv preprint arXiv:2001.00059 [cs.SE]. Retrieved from arXiv:2001.00059\n29. Lahiri, S. K., Naik, A., Sakkas, G., Choudhury, P., von Veh, C., Musuvathi, M., Inala, J. P., Wang, C., & Gao, J. (2022, August 11). Interactive Code Generation via Test-Driven User-Intent Formalization. arXiv preprint arXiv:2208.05950 [cs.SE]. Retrieved from arXiv:2208.05950\n","metadata":{}},{"cell_type":"markdown","source":"**Note: ChatGPT4 was used for initial plotting and summarizing text. The initial code for furtherly extended, modified or completely changed accordingly!**","metadata":{}},{"cell_type":"markdown","source":"**(1) Below we have a list (though not complete) - of most metrics associated with Generative AI tasks (mostly mentioned throughout section 2)**","metadata":{}},{"cell_type":"markdown","source":"| Metric Name | Formula |\n|---|---|\n| Inception Score (IS) | $$IS(Y) = \\exp \\left(\\mathbb{E}_{y \\sim p_{data}} [D_{KL}(p_{model}(y \\mid Y) \\| p_{model}(Y))]\\right)$$ |\n| Fréchet Inception Distance (FID) | $$FID(x, g) = \\| \\mu_x - \\mu_g \\|_2^2 + \\text{Tr}(\\Sigma_x + \\Sigma_g - 2(\\Sigma_x \\Sigma_g)^{1/2})$$ |\n| Precision, Recall, F1 Score (PRF) | $$Precision = \\frac{TP}{TP + FP}$$
$$Recall = \\frac{TP}{TP + FN}$$
$$F1 = 2 \\cdot \\frac{Precision \\cdot Recall}{Precision + Recall}$$ |\n| Kernel Inception Distance (KID) | $$KID = \\frac{1}{m(m-1)} \\sum_{i=1}^m \\sum_{j \\neq i} k(x_i, x_j) + \\frac{1}{n(n-1)} \\sum_{i=1}^n \\sum_{j \\neq i} k(y_i, y_j) - \\frac{2}{mn} \\sum_{i=1}^m \\sum_{j=1}^n k(x_i, y_j)$$ |\n| Perceptual Path Length (PPL) | $$PPL(w) = \\mathbb{E}[\\|\\frac{d \\phi(G(w))}{dw}\\|_2]$$ |\n| Log-Likelihood | $$LL = \\frac{1}{N} \\sum_{i=1}^N \\log p_{model}(x_i)$$ |\n| BLEU Score | $$BLEU = BP \\cdot \\exp(\\sum_{n=1}^N w_n \\log p_n)$$ |\n| ROUGE Score | $$ROUGE-N = \\frac{\\sum_{S \\in \\text{ref}}\\sum_{\\text{gram}_n \\in S} \\text{Count}_{\\text{clip}}(\\text{gram}_n)}{\\sum_{S \\in \\text{ref}}\\sum_{\\text{gram}_n \\in S} \\text{Count}(\\text{gram}_n)}$$ |\n| Word Error Rate (WER) | $$WER = \\frac{S+D+I}{N}$$ |\n| Jensen-Shannon Divergence (JSD) | $$JSD(P \\| Q) = \\frac{1}{2}D_{KL}(P \\| M) + \\frac{1}{2}D_{KL}(Q \\| M)$$ |\n| Perplexity | $$Perplexity = 2^{-\\frac{1}{N}\\sum_{i=1}^N \\log p(x_i)}$$ |\n| Peak Signal-to-Noise Ratio (PSNR) | $$PSNR = 10 \\cdot \\log_{10} \\left(\\frac{\\text{MAX}_I^2}{\\text{MSE}}\\right)$$ |\n| Structural Similarity Index Measure (SSIM) | $$SSIM(x, y) = \\frac{(2\\mu_x\\mu_y + c_1)(2\\sigma_{xy} + c_2)}{(\\mu_x^2 + \\mu_y^2 + c_1)(\\sigma_x^2 + \\sigma_y^2 + c_2)}$$ |\n| Mel Cepstral Distortion (MCD) | $$MCD = \\frac{10}{\\ln 10} \\sqrt{2 \\sum_{i=1}^{N} (m_i - m'_i)^2}$$ |\n| Word Error Rate (WER) | $$WER = \\frac{S+D+I}{N}$$ |\n","metadata":{}}]}![\"GenAI](\"https://media4.giphy.com/media/4cxDlUnKK2RrENaLA2/giphy.gif?cid=ecf05e47wv4mlzjuavj52zy3tahi0tu8u2kcs75w797tc4i3&ep=v1_gifs_related&rid=giphy.gif&ct=g\")