","metadata":{}},{"cell_type":"markdown","source":"The artificial intelligence (AI) exists in our lives since the primitive examples of computer programs. It is a cutting-edge field of computer science that has been shaping the way we interact with technology. It uses deterministic or stochastic models to perform tasks that requires human intelligence and making these models capable of reasoning and problem-solving. We will dive into the exciting world of AI (actually a subset of it) and address the latest advancements in the past two years from every angle.","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nArtifical Intelligence has come a long way and its rapid progress has some key reasons;\n\n- One of them is the **exponential growth of data**, called [Big Data](https://www.oracle.com/big-data/what-is-big-data/). With the help of digital devices and the [Internet of Things](https://en.wikipedia.org/wiki/Internet_of_things), everything is connected and we now have access to vast amount of information. It definitely provides valuable data to the AI systems for training.\n- Advancements in computational power and infrastructure have played a crucial role in the development of AI. High-performance computing capabilities allow AI algorithms to process complex tasks at much higher speeds. It enabled us to make our AI models faster and more accurate.\n- **State-of-the-Art deep learning architectures** has shown remarkable success in various applications such as; image recognition, natural language processing and **image-text-speech synthesis**.\n\nAnd we will focus on the topic of **image-text-speech synthesis** which can be defined under the field of **Generative AI**. ","metadata":{}},{"cell_type":"markdown","source":"
\n
It will be a long ride. We will try to cover most of the domain with even providing code examples and sync you with 'the Generative AI of 2023' !
\n
","metadata":{}},{"cell_type":"markdown","source":"You can use the [Table of Contents](#Table-of-Contents) below to navigate freely within the article. We have also prepared an [Emoji Legend](#Emoji-Legend) for you to better understand which section or topic the part you are reading is related to!\n\n---\n\n### Emoji Legend\n\n| Emoji | Meaning |\n|:-----:|:------------:|\n| πΌοΈ | Image Domain |\n| π€ | Text Domain |\n| π΅ | Audio Domain |\n| π§ | Hardware |\n| π | Open-Source |\n| π₯οΈ | UI |","metadata":{}},{"cell_type":"markdown","source":"---\n\n### Table of Contents","metadata":{}},{"cell_type":"code","source":"from IPython.display import display, HTML\njs = '''\n
","metadata":{}},{"cell_type":"markdown","source":"# What is Generative AI?","metadata":{}},{"cell_type":"markdown","source":"
.\n
","metadata":{}},{"cell_type":"markdown","source":"Generative Artificial Intelligence (Generative AI) is a subset of AI and it focuses on creating new media like image, text and audio. These **complex neural-network models** learn from large amounts of data to generate output that **mimics** the **patterns** and **characteristics** of the given training data. It has ability to generate content that **goes beyond copy-paste**. It mostly learns the training data in its best abstract form (which we will be calling it **latent vector** later) and produce things that mostly suprise us with its realistic and unique feeling!\n\n","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\nimport pandas as pd\nimport json\n\n# Took reference for json I/O from: https://www.kaggle.com/code/harshsingh2209/research-paper-recommendation\n\nFILE = \"/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json\"\n\ndef get_data():\n with open(FILE) as f:\n for line in f:\n yield line\n \ndataframe = {\n 'title': [],\n 'year': [],\n 'month': [],\n 'abstract': [],\n}\n\ndata = get_data()\n\nfor i, paper in enumerate(data):\n paper = json.loads(paper)\n try:\n date = paper['update_date'].split('-')\n year, month = int(date[0]), int(date[1])\n if year > 2015:\n dataframe['title'].append(paper['title'])\n dataframe['year'].append(year)\n dataframe['month'].append(month)\n dataframe['abstract'].append(paper['abstract'])\n else:\n pass\n except:\n pass\n\ndf = pd.DataFrame(dataframe)\n\n####\n\nnlp = [\"BERT\", \"GPT\", \"BLOOM\", \"NLP\", \"language modeling\", \"text generation\", \"text synthesis\", \"conversational agent\"]\nvisual = [\"VAE\", \" GAN\", \"ransformer\", \"diffusion\", \"diffuser\", \"CNN\", \"inpaint\", \"outpaint\", \"style transfer\", \"image synthesis\", \"conditional image\", \"image generation\"]\naudio = [\"HuBERT\", \"HUBERT\", \"hubert\", \"contentvec\", \"VITS\", \"voice\", \"audio synthesis\", \"waveform generation\", \"speech synthesis\", \"music generation\", \"audio generation\", \"text to speech\"]\nall_topics = nlp + visual + audio\n\ndate_mention_df = None\n\nfor tpc_lst, tpc_grp in zip([nlp, visual, audio, all_topics], [\"NLP\", \"Image\", \"Audio\", \"All\"]):\n inc_df = df[df.title.str.contains('|'.join(tpc_lst))]\n if not isinstance(date_mention_df, pd.DataFrame):\n date_mention_df = inc_df.groupby([\"year\", \"month\"])[\"title\"].count().reset_index()\n date_mention_df[\"date\"] = date_mention_df[\"year\"].astype(str) + \"-\" + date_mention_df[\"month\"].astype(str)\n date_mention_df[\"type\"] = tpc_grp\n else:\n date_mention_df2 = inc_df.groupby([\"year\", \"month\"])[\"title\"].count().reset_index()\n date_mention_df2[\"date\"] = date_mention_df2[\"year\"].astype(str) + \"-\" + date_mention_df2[\"month\"].astype(str)\n date_mention_df2[\"type\"] = tpc_grp \n date_mention_df = pd.concat([date_mention_df, date_mention_df2], ignore_index=True)\n \nfig = px.scatter(x=date_mention_df[\"date\"].iloc[:-1],\n y=date_mention_df[\"title\"].iloc[:-1],\n template=\"plotly_white\",\n color=date_mention_df[\"type\"].iloc[:-1],\n title=\"Generative AI Paper Count vs Year\",\n labels={'x': 'Year', 'y':'Paper Count'})\nfig.update_traces(marker_size=10)\nfig.show()\n\ndel date_mention_df, date_mention_df2, dataframe, paper, df, inc_df","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-07-04T09:37:18.313480Z","iopub.execute_input":"2023-07-04T09:37:18.313828Z","iopub.status.idle":"2023-07-04T09:39:05.974372Z","shell.execute_reply.started":"2023-07-04T09:37:18.313800Z","shell.execute_reply":"2023-07-04T09:39:05.973441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generative AI gained a significant popularity in recent years. The advancements in AI/ML and computational power as we mentioned above have caused an increase in the adoption of Generative AI models. It has also got the attention of researchers, industry professionals, geeks and enthusiasts. Its applications can be seen in a wide range of domains like art, design, music and consultancy. Many organizations are trying to leverage its capabilities to create influential products.","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nThe open-source tools and frameworks has played a critical role the popularity. Libraries and user-friendly interfaces have made it easier for users with no technical backgrounds to engage with it. This has triggered a wider audience to explore and play with the possibilities of creating new content. Popularity also led the creation of communities and platforms dedicated to sharing knowledge, resources and tricks. Online forums and open-source repositories have been common platforms for users to collaborate and showcase. This cooperation has also supported the exploration and advancements of Generative AI.\n\nNow that we have briefly talked about the concept, we can get into the technical aspects of Generative AI. We will explore the technical advancements in Generative AI separately for the image, audio, text domains and infrastructure. First things first, building blocks of Generative AI!","metadata":{}},{"cell_type":"markdown","source":"
.\n
","metadata":{}},{"cell_type":"markdown","source":"# Building Blocks","metadata":{}},{"cell_type":"markdown","source":"
Before explaining the advancement in the past two years, it's important for us to first touch upon the architectures that serve as the building blocks.
","metadata":{}},{"cell_type":"markdown","source":"[Autoencoders (AE)](https://en.wikipedia.org/wiki/Autoencoder) are neural networks designed to learn efficient representations of input data by **encoding it into a lower-dimensional latent space** and **decoding it back to its original form**. AE models consist of an encoder network that compresses the input data and a decoder network that reconstructs the original data from the latent space representation. This technique is being used in tasks like image denoising, dimensionality reduction and anomaly detection.","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"We now know that Autoencoders (AE) compress information from the original data into a significantly smaller latent space and that directly influences the output. The next experiments focused on sampling this space from a distribution instead of creating it deterministically. This approach gave rise to the [Variational Autoencoders](https://arxiv.org/abs/1312.6114). Variational Autoencoders (VAE) extend the capabilities of traditional autoencoders by introducing **probabilistic modeling**. VAEs learn a latent space that follows a specific probability distribution, usually a Gaussian distribution. By sampling from this distribution, **VAEs can generate new data points that resemble the training data**. This probabilistic approach allows for **controlled generation and interpolation of data**, making VAEs useful for tasks like **image synthesis** and **data augmentation**.","metadata":{}},{"cell_type":"markdown","source":"
These are the examples from a VAE that takes images from MNIST dataset as input and sampling new outputs from similar latent distributions. (Source)
\n \n\n[Generative Adversarial Networks (GAN)](https://arxiv.org/abs/1406.2661) take a different approach to generative AI by employing a game-theoretic framework. GANs consist of **two competing neural networks**: **a generator** and **a discriminator**. The generator generates synthetic data samples, while the discriminator tries to distinguish between real and synthetic data. Through an iterative process, the generator and discriminator improve their performance, leading to the creation of more realistic and high-quality data samples. GANs have achieved remarkable success in **generating images, videos and even realistic human faces**.","metadata":{}},{"cell_type":"markdown","source":"
These are the examples from a GAN that generates modern art. (Source)
\n \n\n\n[U-Net](https://arxiv.org/abs/1505.04597) is a convolutional neural network for segmentation tasks in the biomedical field of image processing. The network architecture is based entirely on convolutional layers and has been modified and expanded to work with fewer training images and provide more precise segmentations. **\"The main idea is to supplement a usual contracting network by successive layers, where pooling operations are replaced by upsampling operators. Hence these layers increase the resolution of the output. A successive convolutional layer can then learn to assemble a precise output based on this information.\" [(Source)](https://arxiv.org/abs/1505.04597)** \n\nThe size of the information is reduced and compressed at each layer in standard architectures like autoencoders. In the latent layer, the information is compressed to its most compact form and then passed through the decoder layers to increase its resolution again. The compression in the latent layer mostly leads to information loss. However, in the U-Net architecture all layers are **convolutional** and **transposed-convolutional**, which **minimizes this information loss**.","metadata":{}},{"cell_type":"markdown","source":"
\n \n\n[Transformers](https://arxiv.org/abs/1706.03762) are deep learning architectures that operate on sequential data **using self-attention mechanism**, enabling them to process data at once and capture the context and relevance. They **overcome the \"vanishing gradient\" problem** that RNNs face with long inputs. It consists of two units: **an encoder** and **a decoder**. Through the self-attention mechanism, **it can focus on specific parts of the input to generate the output**. This approach facilitates the model's ability to capture context and learn the specific relationships between words. Over time, transformers have become increasingly popular and have paved the way for the development of [Generative Pre-Trained Transformers (GPT)](https://en.wikipedia.org/wiki/Generative_pre-trained_transformer) language models. ","metadata":{}},{"cell_type":"markdown","source":"
\n \n\n[Diffusion](https://arxiv.org/abs/2006.11239) models are branch of generative models, which implies their capability to produce new output resembling the training data. The functioning of them involves the alteration of the training data by introducing Gaussian noise gradually, then restoring the original information by denoising. Once the training phase is completed, we can utilize the diffusion models to generate new data by giving a random noise to the denoising process. Diffusion method also exhibits the benefits of scalability and parallelizability. This allows for efficient training on larger datasets and the ability to parallelize the computational workload.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n\n
\n
\n Those were all the essential methods and architectures you needed to know! Now, we will witness how these methods have advanced in recent times and how they have come together to create remarkable things!\n
\n \n\nFor a long time Convolutional Neural Networks (CNNs) such as [ResNet](https://arxiv.org/abs/1512.03385), [EfficientNet](https://arxiv.org/abs/1905.11946) and [Inception](https://arxiv.org/abs/1409.4842) have been the go-to solutions for image tasks. ViTs introduce a novel approach that revolutionizes how our models interpret and understand image inputs. Recognizing the potential of Transformers, researchers quickly adapted this architecture for image processing with slight modifications. \n\nThe process starts with dividing an image into smaller 16x16 patches and extracting their embeddings. These embeddings capture meaningful information about each patch. By **leveraging self-attention mechanisms**, ViTs enable the model to **focus on relevant patches** and understand the relationships between them.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It offers a unique perspective on image processing by combining the power of Transformers with the inherent capabilities of CNNs. \n β’ It enabled us to create state-of-the-art latent embeddings for image inputs. \n β’ It is an encoder backbone standard for the most of the generative image models! \n
\n \n\nIn January 2021, [OpenAI](https://openai.com/) introduced **CLIP**. It is designed to connect text and images in innovative ways. The model described in the paper can do more than just generate text. By training it on a large dataset of images and their associated texts, it can also be used as a classifier. This means it can analyze images and determine what objects or concepts are present in them. The researchers found that this model performs even better than other state-of-the-art models specifically trained for image classification on a popular dataset called [ImageNet](https://ieeexplore.ieee.org/document/5206848). This is a significant achievement and shows the model's potential for various tasks beyond generating text.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ We will be seeing it is used in visual generative models working with prompts since it pairs image and text data on the same latent space very well. \n β’ When you carefully examine the code below, you can also see that the CLIP model includes the ViT architecture mentioned above as its backbone already!\n
We can execute your zero-shot classifier with executing the lines below! We will give the diagram you see above as an input and will get our classification result.
\n \n\nLatent diffusion models are highly promising field of research that offers intriguing possibilities for image generation. This groundbreaking work made diffusion models more accessible and practical. The authors introduced novel formulations and training procedures that led to **significant improvements in \"image fidelity\"**. These advancements brought diffusion models to a level comparable to that of Generative Adversarial Networks (GANs) and validated the effectiveness of these making innovations at generative algorithms.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It started a hype and became a point of interest in non-deterministic image generation with really nice looking results!\n
\n \n\nDisco Diffusion is a technique that transforms text descriptions into visually captivating images. We already mentioned **CLIP**. It employs a **CLIP-guided diffusion model** to generate artwork with vibrant colors, intricate details and impressive compositions. Initially developed by [Katherine Crowson](https://github.com/crowsonkb) and subsequently optimized by other developers in collaboration with projects like **CLIP** and [OpenCLIP](https://github.com/mlfoundations/open_clip), Disco Diffusion has become a popular choice for creating abstract art using AI. You can take advantage of [Google Colab](https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb) to utilize Disco Diffusion. By leveraging the power of Disco Diffusion and the capabilities of cloud providers, you can also create fascinating images without any expensive hardware!","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ After it is combined with text encoders like CLIP, it became a very popular text-2-image (conditional) diffusion model.\n
\n \n\n**DALL-E** is a **GPT-3-based model** developed by the OpenAI team, trained on a dataset of **12 billion text-image pairs**. It gained popularity among the general public for its **image manipulation** and **text-rendering** capabilities. After a year of its release, it was replaced by its newer version in the market.\n\n**DALL-E 2** is an image generation model developed by OpenAI that builds upon DALL-E, capable of producing **more realistic and accurate outputs** with **up to four times higher resolution**. Currently, this model runs on the DALL-E service of [OpenAI](https://openai.com/) platform.\n\nIt is a good alternative to the open-source conditional diffusion models with its ease of use.\n\n- **Website:** https://openai.com/dall-e-2\n- **Service:** https://labs.openai.com/","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nBootstrapping Language-Image Pre-training (BLIP) is an innovative pre-training framework designed to **enhance vision-language understanding and generation**. It offers [a multi-modal approach](https://en.wikipedia.org/wiki/Multimodal_learning) to tackle the challenges of interpreting text and images, leading to impressive results across a diverse set of tasks in the vision-language domain. It also employs **a new dataset bootstrapping technique for learning from noisy web data**.\n\nDuring pre-training, BLIP simultaneously optimizes three objectives: **2 understanding-based objectives (ITC and ITM)** and **1 generation-based objective (LM)**. This comprehensive approach enhances the model's ability to understand and generate meaningful representations of vision and language domain.\n\n- **Image-Text Contrastive Loss (ITC)** is for the uni-modal encoder. It aims to align the feature space of the visual transformer and the text transformer by ensuring positive image-text pairs to have similar representations in contrast to the negative pairs.\n\n- **Image-Text Matching Loss (ITM)** is for the image-grounded text encoder. ITM is a binary classification task. The model is predicting whether an image-text pair is positive (matched) or negative (unmatched) given their multimodal features.\n\n- **Language Modeling Loss (LM)** is for the image-grounded text decoder. It generates textual descriptions on the images.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It is one of the early and most known examples of the multi-modal architectures. \n β’ BLIP has become an essential tool in the field of image generation due to its ability to generate reference prompts and providing synthetic training corpus data for image generation models.\n
\n \n\n\nMidjourney is a **commercial** diffusion model developed by Midjourney Inc., similar to **DALL-E** and **Stable Diffusion**. Users interact with the model **through** [Discord](https://discord.com/) and receive alternative outputs based on the prompts they provide. Midjourney offers the following features to the users:\n\n- Making minor modifications to the selected output\n- Upscaling the image\n- Generating variations of the image\n- Combining two images in terms of style\n- Generating prompts that describe the given image\n\n\n- **Website:** https://docs.midjourney.com/v1/en","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nImagen is a text-to-image diffusion model developed by [Google](https://www.google.com/). Google has released Imagen through an app called [AI Test Kitchen](https://aitestkitchen.withgoogle.com/signup). This is where Google likes to test different AI projects before they are released to the public. The model uses **CLIP** with **Latent Diffusion**. They have released a paper for it but unfortunately there is not an official implementation published by Google. Still, you can find some implementations by open-source developers.\n\n- **Website:** https://imagen.research.google/","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nStable Diffusion is a model that transforms a blurry or low-resolution image into a clear and detailed one. This deep learning model is released in 2022 based on the concept of **Super-Resolution**. It can take a noisy or low-quality image as input and produce a high-resolution image as output. The key idea behind Stable Diffusion was to train a sophisticated AI model to remove noise and enhance the visual details in an image. This is done by teaching the model on a large dataset of images with varying levels of noise. By analyzing the distribution of these training images, the model learns to simulate the missing details and generate a high-quality output image.\n\nStable Diffusion **is not limited to just enhancing images**. It can **generate detailed images based on text descriptions**. It can also be used for tasks like **inpainting (filling in missing parts of an image)**, **outpainting (extending the image beyond its original boundaries)** and even **translating images based on a given text prompt**.","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nThe architecture of Stable Diffusion involves three main components: The variational autoencoder (VAE), U-Net, Text Encoder. **And we already mentioned all of them above!** The process goes like below:\n\n- **VAE** compresses the image into a smaller latent space, capturing the essential meaning of the image. Gaussian noise is then applied iteratively to this compressed representation, simulating the denoising process.\n- **U-Net (with ResNet backbone)** reverses the denoising process and obtains a latent representation.\n- **VAE decoder** reconstructs the final image by converting the latent representation back into pixel space.\n\nOne remarkable feature of Stable Diffusion is **its flexibility in conditioning the denoising process**. It can be guided by **different inputs** such as **text**, **images** or **other types of data**. For text conditioning, a pretrained text encoder called [CLIP ViT-L/14](https://huggingface.co/sentence-transformers/clip-ViT-L-14) is used to transform text prompts into an embedding space.\n\nStable Diffusion is **relatively lightweight compared to similar models**, making it accessible for consumer-level hardware like GPUs. With **860 million parameters in the U-Net** and **123 million in the text encoder**, it strikes a balance between performance and efficiency.","metadata":{}},{"cell_type":"markdown","source":"---\n### Image Inpainting & Outpainting\n\n
\n \n\nBy continuing the diffusion process beyond the edges of the original image, users can create a larger canvas and generate new visual elements that seamlessly blend with the original image. The particles interact with each other and the image data in a controlled manner, creating new patterns and details that are consistent with the original image. Same applies for the inpainting process; users can delete the pixels in the image and make the model fill the blanks with diffusion process!","metadata":{}},{"cell_type":"markdown","source":"---\n### Stable Diffusion 1.5 vs 2.1\n\n
\n \n\nThere have been important changes and updates during the development of Stable Diffusion. The most known are:\n\n- Stable Diffusion 2 replaces **CLIP** with [OpenCLIP](https://github.com/mlfoundations/open_clip) for text encoding, leading to **improved image quality** and **utilizing a publicly available dataset**.\n- **Negative prompting** plays a crucial role in achieving strong performance in Stable Diffusion 2 compared to Stable Diffusion 1.\n- Stable Diffusion 2 has an advantage in **text coherence**, addressing the challenge of **accurately representing complex text**.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It is a milestone in generative image model family. Most of the image generation products or libraries are using pre-trained Stable Diffusion models or they are re-training their own!\n
\n \n\nDreambooth is an innovative approach that uses **a specialized form of fine-tuning** to teach **new concepts to Stable Diffusion** models. It became very popular among users who tried to apply it to their own photos and others have been using it to add new artistic styles into their images.\n\nThe system consists of three main components:\n\n- **The Language Model**: This component uses a transformer-based model (a pre-trained [BERT](https://arxiv.org/abs/1810.04805) tokenizer) to transform the text prompt entered by the user into a representation suitable for the diffusion model. It employs [cross-attention](https://vaclavkosar.com/ml/cross-attention-in-transformer-architecture) mechanisms to establish meaningful connections between the text and image.\n\n- **The Diffusion Model**: Diffusion part uses a conditional U-Net architecture. It takes two inputs: **the representation of the text prompt from the language model** and **a Gaussian noise**. The diffusion model progressively removes the noise from the representation and converges towards the output specified by the prompt.\n\n- **The Decoder**: The decoder's role is to upsample the diffusion model's output from the initial **64x64** resolution to **512x512**.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It allowed users to fine-tune Stable Diffusion models for a style or an instance quite successfully with much less data.\n
Let's generate 4 images with the same prompts as Stable Diffusion. But this time it will be in Disney style thanks to Dreambooth!
","metadata":{}},{"cell_type":"code","source":"from diffusers import StableDiffusionPipeline\nimport torch\n \nmodel_id = \"nitrosocke/mo-di-diffusion\"\npipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)\npipe = pipe.to(\"cuda\")\n\nprompt = \"a photo of an astronaut riding a horse on mars, modern disney style\"\ngenerator = [torch.Generator(device=\"cuda\").manual_seed(i) for i in range(4)]\nimages = pipe(prompt, generator=generator, num_images_per_prompt=4).images\n\ndel pipe","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-04T09:46:37.216790Z","iopub.execute_input":"2023-07-04T09:46:37.217368Z","iopub.status.idle":"2023-07-04T09:49:45.918625Z","shell.execute_reply.started":"2023-07-04T09:46:37.217337Z","shell.execute_reply":"2023-07-04T09:49:45.917612Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid_plotter(images)","metadata":{"execution":{"iopub.status.busy":"2023-07-04T09:49:45.920608Z","iopub.execute_input":"2023-07-04T09:49:45.920961Z","iopub.status.idle":"2023-07-04T09:49:46.863552Z","shell.execute_reply.started":"2023-07-04T09:49:45.920919Z","shell.execute_reply":"2023-07-04T09:49:46.862677Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n \n\n\nMiDaS is **a monocular depth estimation model** that aims to predict the depth information of a scene from a single 2D image or video. It employs **an end-to-end deep learning architecture**, allowing it to learn complex depth representations directly from the input images. The latest version is trained on **12 datasets** which makes it more robust in handling **various scenes**, **lighting conditions** and **object types**. There are also pretrained models are available, it allows us to utilize the model for our own depth estimation tasks without the need for extensive training. Furthermore, **MiDaS will be utilized in video-generating generative AI models to bind consecutive images**, as we will discuss later.","metadata":{}},{"cell_type":"code","source":"import cv2\nimport torch\nimport urllib.request\n\nurl, filename = (\"https://github.com/pytorch/hub/raw/master/images/dog.jpg\", \"dog.jpg\")\nurllib.request.urlretrieve(url, filename)\n\nmodel_type = \"DPT_Large\"\nmidas = torch.hub.load(\"intel-isl/MiDaS\", model_type)\n\ndevice = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\nmidas.to(device)\nmidas.eval()\n\nmidas_transforms = torch.hub.load(\"intel-isl/MiDaS\", \"transforms\")\ntransform = midas_transforms.dpt_transform\n\nimg = cv2.imread(filename)\nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\ninput_batch = transform(img).to(device)\n\nwith torch.no_grad():\n prediction = midas(input_batch)\n\n prediction = torch.nn.functional.interpolate(\n prediction.unsqueeze(1),\n size=img.shape[:2],\n mode=\"bicubic\",\n align_corners=False,\n ).squeeze()\n\noutput = prediction.cpu().numpy()\ndel midas, midas_transforms, prediction","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-04T09:49:46.864984Z","iopub.execute_input":"2023-07-04T09:49:46.865533Z","iopub.status.idle":"2023-07-04T09:51:09.997443Z","shell.execute_reply.started":"2023-07-04T09:49:46.865500Z","shell.execute_reply":"2023-07-04T09:51:09.996432Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid_plotter([img, output])","metadata":{"execution":{"iopub.status.busy":"2023-07-04T09:51:09.999308Z","iopub.execute_input":"2023-07-04T09:51:10.000113Z","iopub.status.idle":"2023-07-04T09:51:10.776936Z","shell.execute_reply.started":"2023-07-04T09:51:10.000078Z","shell.execute_reply":"2023-07-04T09:51:10.776097Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n \n\nDeforum is an open-source library developed for creating animation videos using Stable Diffusion models. By utilizing the **Image2Image** feature that comes with Stable Diffusion, it enables the creation of animations by **applying transformations to intermediate images**, which simulates camera movements. It can create animations based on **2D** and **3D** approaches and uses **MiDaS** for 3D animation generation.\n\n- **Website:** https://deforum.github.io/\n- **Github:** https://github.com/deforum-art/deforum-stable-diffusion\n- **Example Notebook:** https://colab.research.google.com/github/deforum-art/deforum-stable-diffusion/blob/main/Deforum_Stable_Diffusion.ipynb","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nControlNet is a neural network specifically designed to **control the outputs of a pretrained image diffusion model**, such as Stable Diffusion. Its primary purpose is to facilitate the input of a conditioning image. By integrating with Stable Diffusion, it enables the model to incorporate a conditioning input during the image generation process. This integration leads to improved performance and greater control over the generated images compared to using base Stable Diffusion models.\n\nSome examples of condition inputs that can be used with ControlNet are:\n\n- **Scribbles:** Hand-drawn marks or sketches indicating desired shapes or objects in the image.\n- **Edge maps:** Binary representations highlighting the edges and contours of objects within the image.\n- **Pose key points:** Specific points denoting the pose or position of objects or individuals in the image.\n- **Depth maps:** Depth information indicating the relative distance of objects in the scene.\n- **Segmentation maps:** Maps indicating different regions or segments within the image, such as object masks or semantic labels.\n- **Normal maps:** Maps representing surface normals, providing information about the orientation of objects.\n\n---\n\n### ControlNet Processors\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### HED\n
\n \n\n\n**GEN1** is a paid service introduced by [Runway](https://runwayml.com/) in 2023, **allowing video editing using generative AI models**. It primarily offers the following features to users:\n\n- Applying any style to the entire video\n- Selecting and customizing elements within the video\n- Transforming mockup images or objects into high-quality renders using prompts\n\n- **Website:** https://research.runwayml.com/gen1","metadata":{}},{"cell_type":"markdown","source":"
\n \n\n**AUTOMATIC1111 App** is a web browser interface that utilizes GPUs for Stable Diffusion. It allows users to run, train and deploy Stable Diffusion models just with a single click. Setting up the app on your web browser is straightforward. The platform offers a collection of user-contributed models and extensions specifically designed for creating AI art. It is designed to be easy for beginners to use while also offering advanced features for professionals. Some notable features include:\n\n- Converting text to images & processing images\n- The design allows for the integration of additional functionalities with plug-in support\n- Other features like in-painting, color sketching and prompt metrics\n- The active open-source community regularly introduces new features, often making A1111 the first platform to receive them.\n- ControlNet and Deforum!\n\nBest of all, it's open-source and free to use.\n\n- **Github:** https://github.com/AUTOMATIC1111/stable-diffusion-webui","metadata":{}},{"cell_type":"markdown","source":"---\n### AI-Generated QR Code Hype\n
\n \n\nRecently, generative AI-generated images that **look like drawings** but still **can function as** [QR codes](https://en.wikipedia.org/wiki/QR_code) have become quite popular on social media. These images were created **using ControlNet** and you can achieve similar results **with A1111**! Image diffusion models such as Stable Diffusion have the ability to generate a wide variety of images. As we already mentioned, ControlNet allows for the control of Stable Diffusion models using additional conditions such as outlines, subject poses or depth maps.\n\n- The process begins by generating an image that resembles a QR Code using img2img. ControlNet is activated during the sampling steps to imprint the QR Code onto the image.\n- ControlNet is deactivated towards the end of the sampling process to enhance the image's overall consistency. This approach ensures that the final output incorporates the desired QR Code while maintaining a visually coherent result.","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nDragGAN is an innovative photo editing tool that allows users **to reshape and edit images interactively using AI**. Unlike traditional editing tools, DragGAN generates images according to user needs and provides precise control over the pose, shape, expression and layout. It produces realistic outputs by utilizing a Drag-based GAN. It even works for complex scenarios like hallucinating occluded content and deforming shapes and is able to preserve to object structures. DragGAN **promises a user-friendly interface** and the potential to revolutionize image editing.\n\nDragGAN features point-based editing which enables users to make accurate adjustments by dragging and dropping points directly on the image. It goes beyond 2D editing by integrating 3D and allows users to modify object pose, shape, expression and layout while maintaining the depth structure of the image. With its unique capabilities and intuitive interface, DragGAN has the potential to transform the way we approach image editing, for both experienced editors and no-coder beginners.","metadata":{}},{"cell_type":"code","source":"from IPython.display import display, HTML\njs = '''\n\n\n\n
Here is a demo of DragGAN we prepared for you using a StyleGAN2 trained on dog images!
\n \n\nThe ChatGPT service -launched by OpenAI- utilizes an enhanced version of **GPT3** named **GPT3.5**. The model gains the ability to not only understand and interpret human dialogues but also **interpret code snippets** in this version. This service is available to the public. However, some users may not access to the service depending on the usage demand.\n\n- **Website:** https://chat.openai.com/","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nCodegen uses LLMs and [RLHF](https://huggingface.co/blog/rlhf) for code generation. It sees code generation as a sequence prediction task, where the user provides specifications in natural language and the desired code is generated through sampling. The architecture of the model includes a transformer decoder with left-to-right causal masking and uses [rotary position embedding](https://blog.eleuther.ai/rotary-embeddings/). **\"The largest version of the model is pre-trained on BigQuery, a large-scale dataset of multiple programming languages from GitHub repositories. The data consists of 119.2B tokens and includes C, C++, Go, Java, JavaScript, and Python.\"** [(Source)](https://huggingface.co/Salesforce/codegen-2B-multi)\n\n- **HuggingFace Demo:** https://huggingface.co/Salesforce/codegen2-16B","metadata":{}},{"cell_type":"markdown","source":"
Let's give CodeGen a \"torch import line\" and make it continue that sequence.
\n \n\nJasper is another well-known service AI-based text generation. It is effective in creating high-quality content that can be customized to match your brand's voice. It also has templates on content generation for:\n- Long-form assistant\n- Content improver\n- Google Ads headline\n- Video description\n- Email subjects\n\n\n- **Website:** https://www.jasper.ai/","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nOne year after OpenAI released ChatGPT, they introduced GPT4 as a paid service in 2023. This model includes improvements addressing many of the complaints about ChatGPT. The main enhancements include **increased factual accuracy in responses**, **reduced hallucination**, **greater ability to modify behavior based on user requests** and **improved capability to understand and resist illegal requests**. Additionally, users can **provide visual inputs to GPT4 to generate interpretations** or ask questions about the images.\n\n- **Website:** https://chat.openai.com/","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nBLOOM is an **autoregressive Large Language Model (LLM)**. With its extensive training, it excels at generating coherent text in **46 languages** and **13 programming languages**, producing results that are remarkably similar to text written by humans. One of the key strengths of BLOOM is its ability to **generate text continuations based on prompts**. BLOOM utilizes past values to predict and generate the next set of text by leveraging its autoregressive nature and ensures contextually relevant output. Its architecture comprises **70 layers** and **112 attention heads**, resulting in **a hidden layer dimension of 14,336**.\n\nBLOOM is a pretty large model so we won't be executing it in this Kaggle kernel. But you can run its hosted version on HuggingFace!\n\n- **HuggingFace Demo:** https://huggingface.co/bigscience/bloom","metadata":{}},{"cell_type":"markdown","source":"
.\n
","metadata":{}},{"cell_type":"markdown","source":"## π€ | π₯οΈ | Claude 2 (2023)\n
\n \n\n[Anthropic](https://www.anthropic.com/) (an artificial intelligence venture founded collaboratively by former OpenAI employees) announced the release of their new text-generating AI model: **Claude 2**. It is the latest version of Anthropic's initial AI chatbot. It is currently in beta and available for limited access through both the web and a paid API in the United States and the United Kingdom. Several businesses, including the productive AI platform Jasper and Sourcegraph have already started using Claude 2 as a pilot.\n\n- **Website:** https://claude.ai/","metadata":{}},{"cell_type":"markdown","source":"
.\n
","metadata":{}},{"cell_type":"markdown","source":"# Advancements in Audio Domain","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nWe have already provided the first example of using transformers in various domains with vision transformers. Of course it didn't stop there, the transformer approach was also tried in the audio domain. Meta researchers used **the sequential representation of the BERT architecture** in the **audio domain**. HuBERT learns to model the sequential sound input. The model **embeds raw audio inputs in a latent space**. HuBERT offers building models solely trained on audio data in a self-supervised way, eliminating the need for text inputs.\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It enabled us to create robust latent embeddings for continuous audio data. \n β’ Because of that, it will be used in most of the audio generation models as feature extractor! \n
\n \n\nVITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech) incorporates [variational inference](https://en.wikipedia.org/wiki/Variational_Bayesian_methods) and [adversarial training](https://www.mdpi.com/1999-4893/15/8/283) to enhance the generative modeling capability. Additionally, a stochastic duration predictor is introduced to synthesize speech with diverse rhythms from input text, capturing the natural variations in pitch and rhythm that exist in human speech.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It is a big step in text-2-speech modeling and most of the latest architectures include this approach.\n
\n \n\nContentVec extends the HuBERT by proposing a new self-supervised learning method with specifically **focusing on achieving speaker disentanglement without significant loss of content**. It introduces disentangling mechanisms to both the teachers (masked prediction labels) and the students (learned representations) within the HuBERT framework. You can also take a look at [this video](https://www.youtube.com/embed/aiGp1g-dCY4) for more detailed explanation!","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It is the current latent embedding choice for most of the open source audio generation tools and pipelines since it gives reasonably better results over base HuBERT.\n
\n \n\nSoft-VC is a novel approach to voice conversion by focusing on self-supervised representation learning. It uses soft speech units which are learned by predicting a distribution over the discrete units. **\"By modeling uncertainty, soft units capture more content information, improving the intelligibility and naturalness of converted speech.\"** [(Source)](https://github.com/bshall/soft-vc)\n","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n
IMPORTANCE
\n β’ It is adopted by many developers in open-source and commercial voice-conversion tools.\n
\n \n\n\nThe aim of the model is to transfer a given melodic speech as if it is being spoken by another person. That's why the model is specifically defined as a **singing voice conversion model**. The app uses the **Soft-VC** encoder to analyze and extract speech features from the source audio. These features are then used directly in the **VITS model**. It helps preserving the original pitch and intonations of the audio. **\"This project differs fundamentally from VITS, as it focuses on Singing Voice Conversion (SVC) rather than Text-to-Speech (TTS). In this project, TTS functionality is not supported, and VITS is incapable of performing SVC tasks. It's important to note that the models used in these two projects are not interchangeable or universally applicable.\"** [(Source)](https://github.com/svc-develop-team/so-vits-svc)\n\n- **Github:** https://github.com/voicepaw/so-vits-svc-fork","metadata":{}},{"cell_type":"markdown","source":"
\n \n\n\nBark is a transformer-based text-2-audio model developed by Suno. It can generate realistic speechs, music and sound effects. It can also mimic non-verbal sounds like laughing and crying. It is a **GPT-style model** and heavily took implementation references from [nanoGPT](https://github.com/karpathy/nanoGPT). It has an open-source repository, so you can try it yourself on your machine or with using the HuggingFace space below!\n\n- **HuggingFace:** https://huggingface.co/spaces/suno/bark","metadata":{}},{"cell_type":"markdown","source":"
Let's create an example with Bark!
","metadata":{}},{"cell_type":"code","source":"!pip install git+https://github.com/suno-ai/bark.git --quiet\n\nfrom bark import SAMPLE_RATE, generate_audio, preload_models\nfrom bark.generation import clean_models\n\nfrom scipy.io.wavfile import write as write_wav\nfrom IPython.display import Audio, display\n\n# download and load all models\npreload_models()\n\n# generate audio from text\ntext_prompt = \"\"\"\n Hello fellow Kagglers!\n\"\"\"\n\naudio_array = generate_audio(text_prompt, history_prompt=\"v2/en_speaker_1\")\n\n# save audio to disk\nwrite_wav(\"bark_generation.wav\", SAMPLE_RATE, audio_array)\nclean_models()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-04T10:09:29.322741Z","iopub.execute_input":"2023-07-04T10:09:29.323805Z","iopub.status.idle":"2023-07-04T10:18:33.785160Z","shell.execute_reply.started":"2023-07-04T10:09:29.323768Z","shell.execute_reply":"2023-07-04T10:18:33.784045Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(Audio(\"bark_generation.wav\",\n rate=SAMPLE_RATE))","metadata":{"execution":{"iopub.status.busy":"2023-07-04T10:18:33.789236Z","iopub.execute_input":"2023-07-04T10:18:33.789893Z","iopub.status.idle":"2023-07-04T10:18:33.807108Z","shell.execute_reply.started":"2023-07-04T10:18:33.789855Z","shell.execute_reply":"2023-07-04T10:18:33.806143Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n \n\n\nRVC is a simple and easy-to-use voice transformation framework based on **VITS**. You can train your models with at least 10 minutes of low-noise speech data. It executes fast training even on poor graphic cards.\n\n- **Github:** https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI","metadata":{}},{"cell_type":"markdown","source":"
\n \n\n\nMusicGen is a **single stage auto-regressive Transformer model** trained over a **32kHz EnCodec tokenizer**. It doesn't require a self-supervised semantic representation. We won't be providing a demo since it has dependencies incompatible with the Kaggle kernel. But you can try it out with the demo below:\n\n- **HuggingFace:** https://huggingface.co/spaces/facebook/MusicGen","metadata":{}},{"cell_type":"markdown","source":"
.\n
","metadata":{}},{"cell_type":"markdown","source":"# Advancements in Hardware","metadata":{}},{"cell_type":"markdown","source":"
.\n
","metadata":{}},{"cell_type":"markdown","source":"## π§ | Google TPUv4 (2021)\n
\n \n\nTensor Processing Unit (TPU) is a hardware accelerator produced by Google. It is specifically designed for machine learning tasks. TPUs are highly efficient at performing tensor operations. They offer high computational power with low power consumption. It is optimized for parallel processing and can handle large-scale data efficiently. **\"TPU v4 improved performance by more than 2x over TPU v3 chips.\"** [(Source)](https://www.hpcwire.com/2021/05/20/google-launches-tpu-v4-ai-chips/)","metadata":{}},{"cell_type":"markdown","source":"
\n \n\nH100 is an AI computing unit that uses **NVIDIA Hopperβ’ GPU computing architecture** with **Transformer Engine**. It is optimized for training and deploying **generative AI**, **large language models** and **recommendation systems**. ","metadata":{}},{"cell_type":"markdown","source":"
\n \n\n[Deep Learning Super Sampling (DLSS)](https://www.nvidia.com/en-us/geforce/technologies/dlss/) is the name of the deep learning model-based upscaling model family developed by NVIDIA since 2019. It was initially introduced to support the NVIDIA RTX20 series graphics cards and **uses specificially-designed hardware units** for execution. The main purpose of the technology is to render the game at a lower native resolution and then use artificial intelligence to enhance the visuals to a higher resolution.\n\nWith **DLSS 1.0**, NVIDIA provided a framework to game developers, enabling them to **train custom models for their games**. Special DLSS models were released for each game, allowing **real-time spatial image upscaling using convolutional auto-encoders** within the game.\n\nWith **DLSS 2.0**, NVIDIA uses low-resolution images, motion vectors, depth buffers and brightness data to apply **temporal anti-aliasing upsampling**. It also employs **convolutional auto-encoder models** to **fix any potential artifacts** that may occur.\n\nWith **DLSS 3.0**, NVIDIA has added the [optical-flow](https://en.wikipedia.org/wiki/Optical_flow) approach to its existing solution (DLSS 2.0). This allows **the creation of a virtual third frame between two consecutive frames**. With this update, most games supporting the technology **had a significant increase in frames per second (FPS)**.\n\nNVIDIA **does not provide** this as an open-source software-based solution. All relevant models and solutions come pre-installed with graphics card drivers **for the NVIDIA RTX20 series GPUs and above**.\n\n- **Promotion Page:** https://www.nvidia.com/en-us/geforce/technologies/dlss/","metadata":{}},{"cell_type":"markdown","source":"
Generative AI has many potential benefits in many alternative use-cases. But making content generation automated also raises ethical concerns.
","metadata":{}},{"cell_type":"markdown","source":"---\n### Bias\nBiased data makes biased models. Garbage in, garbage out. The model may unintentionally learn and replicate the bias in the training data. The outputs may claim existence of stereotypes, societal inequalities or may target certain groups of people.\n\n**Example Case:** ChatGPT has been found to generate text that is biased against women, people of color and other marginalized groups.\n\n**Example Case 2:** When asked to generate an image of a \"CEO\"; Midjourney was more likely to generate an image of a white man than a woman or a person of color.\n\n---\n### Data Privacy\nGenerative AI models often **rely on Big Data** for training. And it might be including your personal information if it is scraped through web. The models can learn and reproduce private information that exists in the data. They can still produce content that reveals the identity of individuals even if the data is anonymized.\n\n[Example Lawsuit](https://cyberscoop.com/openai-lawsuit-privacy-data-scraping/): Plaintiffs accuse the company of breaching their privacy rights by extracting their personal data from the internet without obtaining their consent. They argued that OpenAI employed this data to train its generative AI models, that can generate images and text that could be used to impersonate them.\n\n---\n### Fake Content\nGenerative AI models have the capability to produce highly realistic images, videos, text and audio. This raises the risk of **creating and spreading false information**. Malicious people can use this technology for creating fake news or even impersonating another individuals. The widespread availability of the tools makes generating fake content easier.\n\n[Example Lawsuit](https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html): A lawyer referenced fake cases produced by ChatGPT in a legal document submitted to a federal court. The lawyer was discovered to have excessively relied on the AI tool and neglected to verify the authenticity of the cited cases.\n\n---\n### Copyright (IP)\nThe ability of mimicking the data of Generative AI raises a challenge in determining the originality of the generated content. AI-generated content could have serious problems with the copyrights, trademarks or patents.\n\n[Example Lawsuit](https://www.theverge.com/2023/1/28/23575919/microsoft-openai-github-dismiss-copilot-ai-copyright-lawsuit): A legal action claims that Microsoft, GitHub, and OpenAI have violated copyright laws by enabling \"GitHub Copilot\" to reproduce licensed code snippets without proper attribution. This lawsuit consists of a group of software developers who argue that Copilot has infringed upon their copyrighted materials.\n\n[Example Lawsuit 2](https://www.theverge.com/2023/1/16/23557098/generative-ai-art-copyright-legal-lawsuit-stable-diffusion-midjourney-deviantart): It's been said that Midjourney and Stability AI have violated the rights of numerous artists by training their tools using images obtained through web scraping. This lawsuit consists of artists who argue that Midjourney and Stability AI have utilized their images without obtaining proper permission.\n\n---\n### Job Displacement\nGenerative AI is becoming more capable of performing tasks traditionally done by humans. It is also a concern for content-creators like copy-writers and artists. There is a common thought about its potential negative impact on employment and job opportunities when its improved.","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"> Nobody phrases it this way, but I think that artificial intelligence is almost a humanities discipline. It is really an attempt to understand human intelligence and human cognition.\n\n*βSebastian Thrun*\n\nGenerative artificial intelligence (Generative AI) has significantly transformed our lives in many ways with its capabilities of text, image and sound generation. Generative AI fundamentally has auto-encoders, GANs and transformers in its core. But it is continuing to progress exponentially with the newest forms of them and surpassing its previous state each month. This remarkable technology has revolutionized the way we create, express and perceive the world around us.\n\nOn the other hand, products like ChatGPT have opened up significant loopholes in the education system. Educators are looking for ways of how to prevent students from having their assignments done by these generative tools. Blog writing has undergone a complete evolution and developers who write code have reduced the time they spend on it by up to a quarter. The comfort these tools bring to people is like a double-edged sword. They allow us to complete tasks much faster and accelerate our progress on one side. On the other side; they tend to make us lazier.\n\nThese tools have a strong mathematical foundation and they ultimately rely on the data we provide. Applications like ChatGPT or Midjourney are capable of gathering that data by scraping the public internet sources. And unfortunately the works of writers, artists and content creators are also included in this data. Also, the models that trained with biased data on the internet can also exhibit unethical or racist discourse.\n\n---\n\n### Fighting with the Threats\nWe have seen that there is a lot of ethical concern about Generative AI. We may propose some actions against these threats:\n\n- **Bias**: Organizations and the individuals training AI models should collect training data that captures a wide range of diversity. They should monitor the bias of their model and create appropriate metrics to track it. The R&D teams should include researchers from different disciplines to make sure there is comprehensive knowledge about the product being built.\n- **Data Privacy:** The teams should declare guidelines about their data collection, storage and usage policies. Their product should inform the users about the data collecting practices and should give them control over their personal information.\n- **Fake content:** There is surely a need for robust verification mechanisms and trustworthy fact-checkers.\n- **Copyright (IP):** It's still a vague area. A lot of forums and panels should be organized on the subject involving AI researchers, legal experts, generative artists and content creators. The findings should be scientifically reported and the rights of content creators should be protected at governmental level.\n- **Job Displacement:** Initiatives can invest in reskilling individuals for the changing job market, providing them opportunities and even educating them to work alongside Generative AI technologies. There may be also collaboration with governments to design policies about supporting job transition.\n\nWe also believe that these issues will also be addressed through future regulations.\n\n---\n\n### Future of ...\n\n#### Image Generation\n\nThe usage of the diffuser architecture is increasingly being preferred in image generation models. Developers have shifted towards multi-modal approaches rather than uni-modal ones and aiming to generate images and videos with using inputs from diverse domains. The fire on the GAN has never extinguished. Developers seem to greatly appreciate the ability of GANs to produce high-quality outputs and their explainability - controllability. We believe that we will continue to hear frequently about diffuser and GAN architectures in the coming years.\n\n\n#### Text Generation\nTransformers continues to be the building block of text-generation models. Researchers constantly derive new architectures containing transformers to improve metrics, while developers quickly adapt these architectures into their libraries and sometimes even better: optimize and accelerate them. In recent years, we have witnessed a significant use of text-generation models as chatbots. Companies aim to create virtual assistants that can assist people in various topics. We have already listed the privimitive versions of these general-assistants. These models are currently quite demanding to run on standard PCs, but we have no doubt that we will have a virtual assistant that can help us in any subject running offline on our phones in the future.\n\n\n#### Audio Generation\nFor audio-generation models, we have also witnessed the widespread use of natural language processing approaches like [BERT](https://arxiv.org/abs/1810.04805) and GPT in recent years. These architectures also utilize signal processing techniques specific to the domain. They are far more advanced compared to their past versions and purposes. While it used to be quite challenging to find an open-source text-to-speech model in the past, now we can use these models to process one person's speech with another person's voice. It's not difficult to imagine a tool in the future that can create a unique voice tone for a character defined by image and text input and perform voice-over.\n\n#### Multi Modality\nWhile the text, audio and image domains have individually shown significant advancements in recent years, much of the research and development efforts have focused on integrating these domains together. [Multi-modal](https://en.wikipedia.org/wiki/Multimodal_learning) approaches like CLIP and BLIP pioneered the search for the ability to work with different domain pairs such as image-text and audio-text within the same model. Many virtual assistant products now leverage these multi-modal approaches, providing users with the ability to input various types of data and generate output in multiple formats. It is highly likely that in the future, we will witness native models capable of supporting all input types seamlessly without the need for any plugins like [ChatGPT's Code Interpreter](https://openai.com/blog/chatgpt-plugins#code-interpreter).\n\n#### AI Hardware\nAs the computational demand of AI tools increase, organizations and researchers are dedicating time to develop more powerful hardware instead of waiting longer. The two major pioneers in this development are NVIDIA and Google as we mentioned above. These companies have been developing specialized hardware to train and run AI models. Over the years, we have witnessed how technologies that were once considered 'cutting edge' have become ordinary and accessible. These AI hardware devices may not easily enter an average user's home for now, but we are sure that they will become standard hardwares of AI in the future.\n\n---\n\nAs people who have been following the advancements of generative AI since its early days, we might have observed certain patterns. When a generative AI algorithm is initially released, it emerges as slow, buggy and produces relatively low-resolution output. It often prone to numerous errors. However, they quickly evolve and show improvements with the support of communities, companies and initiatives. Their output speed is accelerated through optimization and they also begin to produce much higher-quality outputs in various scenarios. When we reach a point of saturation, a new model emerges and continues to surprise all of us. It has become impossible to ignore this technology, it has already entered to our lives. Rather than opposing it; we must understand its dangers, gain awareness and learn how to effectively use it in our daily lives.\n\nGenerative AI will be one of our greatest tools in the future. It is already started to have a significant impact on creative tasks. It will be helping us complete our tasks quicker, automatizing our routines and allowing us to allocate time to new adventures! We can say that it definitely started **a renaissance in creativity**. ","metadata":{}},{"cell_type":"markdown","source":"
.\n
","metadata":{}},{"cell_type":"markdown","source":"**P.S.:** The authors utilized LLMs to accurately translate long sentences during the writing of this article. A bibliography section has not been included at the end of the article since the references are provided as clickable texts along with the respective words.","metadata":{}}]}
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/genai_header.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/datagen.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/timeline-genai.jpg?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/autoencoder.png\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/vautoencoder.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/vae_1.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/vae_2.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/gan_diagram.png?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/architecture_unetV2.png\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/transformer_architecture.png\"){kind=spacer}
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/diff_denoising.jpeg\")
![](\"https://media.giphy.com/media/3owzVTWM0m3bDtEM1i/giphy.gif\"/)
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/vit.png\")
![](\"https://images.pexels.com/photos/6477261/pexels-photo-6477261.jpeg\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/clip.png\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/latent-diffusion.png\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/disco.jpeg?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/dall-e-2.jpeg\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/blip.png?raw=true\")
![](\"https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/gif/midjourney.gif?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/imagen.png\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/sd.jpeg?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/unet-vae.jpeg?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/outpainting.jpeg?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/sd-comparison.jpeg?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/dreambooth.jpeg?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/midas.png\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/gif/deforum_1.gif\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/gif/deforum_2.gif\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/gif/deforum_3.gif\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/gif/deforum_4.gif\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/controlnet.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/hed.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/openpose.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/midas_contnet.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/gif/gen1.gif?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/automatic1111.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/qrcode_controlnet.jpg?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/draggan-arch.png?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/chatgpt.png\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/codegen.png?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/jasper.png\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/gpt4.png\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/bloom.png?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/claude2.png\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/hubert.jpg\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/vits.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/contentvec.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/soft-vc.png?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/so-vits-svc-gui.png\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/bark.jpeg?raw=true\")
![](\"https://raw.githubusercontent.com/nlztrk/kaggle_genai_article/main/img/rvc.png\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/mgen.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/tpu.jpg?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/h100.jpg?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/dlss3.png?raw=true\")
![](\"https://github.com/nlztrk/kaggle_genai_article/blob/main/img/ethics.png?raw=true\")