{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# **Applicational and Technical Advancements in Generative AI** ","metadata":{}},{"cell_type":"markdown","source":"### How did we get here? \n\nThe first significant landmark on the journey towards generative AI was the introduction of the concept of artificial intelligence itself in the mid-20th century. Notable figures such as Alan Turing and John McCarthy played instrumental roles in birthing this field. Turing, with his pioneering work on the \"universal machine\" (Turing, 1936), and McCarthy, through his seminal work in establishing the Dartmouth Conference in 1956, which served as the birthplace of AI as a field of study.
\n\nThe next important leap forward occurred with the advent of machine learning in the 1980s. Machine learning involves algorithms designed to parse through data, learn from this data, and then make decisions or predictions based on these learnings. While early machine learning methods were mostly linear and statistical in nature, the foundation was being laid for the development of more sophisticated learning systems.
\n\nFinally, the concept of generative AI came into being, standing on the shoulders of these AI advancements. The field of generative AI got a significant boost with the advent of Generative Adversarial Networks (GANs), introduced by Ian Goodfellow and his team in 2014. Unlike traditional AI models that focus on classifying or predicting data, GANs represented a different approach. These models involved two neural networks — a generator and a discriminator — competing in a zero-sum game. The generator produces data instances (such as images), and the discriminator evaluates them for authenticity. Over time, the generator learns to create increasingly convincing artificial data instances. This marked a paradigm shift, as AI was now capable of creating new, original content.
\n","metadata":{}},{"cell_type":"markdown","source":"\n# **Applicational Advances** ","metadata":{}},{"cell_type":"markdown","source":"### Chatbots Before Generative AI \n\nThough Generative AI has revolutionized the field of interactive chat bots, it's important to note the difference between the previous generation of chatbots and todays, which will help us imagine what tomorrow's could look like. Here are some notable pioneers:\n\nELIZA: Developed by Joseph Weizenbaum at MIT in 1966, ELIZA was one of the first chatbots and arguably the most famous early example. ELIZA was designed to simulate a conversation with a psychotherapist by using pattern matching and substitution to give the illusion of understanding.\n\nPARRY: Developed by Kenneth Colby in 1972, PARRY was designed to simulate a patient with schizophrenia. It incorporated models of belief systems into its responses.\n\nJabberwacky: Developed in 1981 by Rollo Carpenter, Jabberwacky aimed to offer an interactive, learning chatbot with a sense of humor.\n\nSmarterChild: Developed by ActiveBuddy in 2000 and available on platforms like AOL Instant Messenger and MSN Messenger, SmarterChild marked the dawn of commercially focused chatbots. It could provide users with real-time information such as weather updates, movie showtimes, and trivia.\n\nNotice that before generative AI, most chatbots were either rule-based or retrieval-based:\n\nRule-based chatbots respond to specific commands and have a predefined set of rules. They are simple but can't handle complex requests or adapt to changing user inputs.\n\nRetrieval-based chatbots choose a response from a predetermined set based on the user's input, using a decision tree or a similar mechanism. They can handle more complex queries than rule-based bots but are still limited by the responses pre-programmed into them.\n\nThe advent of generative AI has brought about the third generation of chatbots, generative models, which don't rely on predefined responses. Instead, they use machine learning, specifically sequence-to-sequence (Seq2Seq) neural networks, to generate responses. They are called generative because they can create responses to previously unseen queries. ","metadata":{}},{"cell_type":"markdown","source":"### Revolutionary changes to Chatbots brought forth by Generative AI\n\nIn the past two years, there has been a tremendous leap forward in progress, spearheaded by ChatGPT, LaMDA, and now Bard. Though both generative AI and the previous generation of chatbots can read and respond to messages, there are four major distinctions that separate Generative AI chatbots from the former. \n\nNatural Conversations: Generative AI enables chatbots to produce human-like text that can make conversations more fluid, natural, and engaging. They understand the context and can maintain the thread of the conversation over multiple turns, making interactions more meaningful.\n\nHandling Unseen Queries: Unlike rule-based or retrieval-based chatbots, generative AI models can handle unseen queries and produce relevant responses. This ability dramatically improves the usability and functionality of chatbots.\n\nPersonalization: Generative AI can learn from previous interactions, allowing chatbots to offer a personalized experience to each user.\n\nScalability: Generative AI chatbots can handle a large volume of queries without the need for extensive manual labor to predefine or categorize responses.\n\nAll this makes for machines that are capable of passing the famed Turing Test. While this test was said to be the hallmark for AI's human-like thinking, it's hard to say that generative AI, which works by determining the probability of the next token based on all the previous tokens, is the type of true intelligence in machines that Turing envisioned. These features also allow for generative AI chatbots to become useful in many other fields. Many of the customer support chatbots are being replaced by generative AI because of its much more useful and personable characteristics. ","metadata":{}},{"cell_type":"markdown","source":"\n# **Technical Advances in Generative AI** ","metadata":{}},{"cell_type":"markdown","source":"Of the numerous advances in Generative AI in the past two years, for important functionality purposes they can be broken down into the following:\n\n1. Improved Training Efficiency: One of the most notable improvements has been in the efficiency of training these large models. This has been achieved through a combination of new training methodologies, algorithms, and architectural enhancements. Techniques like gradient accumulation, model parallelism, and more sophisticated forms of transfer learning have significantly decreased the computational requirements and time for training.\n\n2. Variable-Length Token Handling: While transformer models have proven to be incredibly versatile for many tasks, they have traditionally struggled with tasks involving variable-length sequences. However, recent advances have incorporated new methods for handling variable-length tokens, increasing the versatility and efficacy of these models.\n\n3. Data Augmentation for Textual Data: Previously, data augmentation techniques have been more prevalent and effective in vision tasks. However, recent progress in techniques for textual data augmentation like backtranslation, text mixing, and consistency training has resulted in significant improvements in generative models' robustness and generalization capabilities.\n\n4. Conditional Transformers: While transformers are incredibly powerful, they often struggle with controlled generation tasks where the output must satisfy certain conditions. Advances in this field have introduced Conditional Transformers, which use a conditioning vector during training and generation to improve performance on controlled tasks.\n\n5. Latent Variable Models: Recent developments have made significant strides in latent variable models for generative tasks. These models, which involve learning a compressed representation of the input data, have resulted in more efficient and robust generative models.\n\n6. Self-Supervised Learning: In 2021, large language models like GPT-3 had already shown the power of self-supervised learning. In the two years since, the use of self-supervised learning has expanded even further, with new architectures and loss functions developed to maximize the information that can be gleaned from unlabeled data.\n\n7. Explainability in AI: Interpretability and transparency of AI models have also taken center stage. Understanding why an AI makes a particular decision is increasingly important. Techniques like layer-wise relevance propagation or saliency mapping, which were traditionally used for vision tasks, have been adapted for NLP tasks, increasing the explainability of generative models.\n\n8. Multimodal Models: There's a growing trend toward models capable of understanding and generating multiple forms of data, such as text and images simultaneously. Newer architectures and training methodologies have made these models more effective, leading to exciting applications like generating text descriptions for images or even generating images from textual descriptions.\n\n9. Few-Shot and Zero-Shot Learning: Few-shot and zero-shot learning, where a model generalizes to new tasks with minimal or no training examples, have seen significant advancements. Innovations in training strategies and model architectures have led to models that can understand task descriptions better and generate more accurate output for tasks they were not explicitly trained on.\n","metadata":{}},{"cell_type":"markdown","source":"\n# **Generative AI Apps/Tools** ","metadata":{}},{"cell_type":"markdown","source":" \n\n*Generative AI Apps, link to image here: https://www.datacamp.com/cheat-sheet/the-generative-ai-tools-landscape*\n","metadata":{}},{"cell_type":"markdown","source":"These tools have almost all emerged in the past two years with the explosion in generative AI. Considering how new this technology is, imagine the possibilities that can emerge. Predicting the future of the AI field is almost impossible, but there are a few insights that may give us some ideas to what the future landscape looks like. ","metadata":{}},{"cell_type":"markdown","source":"\n# **Challenges** ","metadata":{}},{"cell_type":"markdown","source":"Generative AI is not a magic box that will fix all of our problems, but understanding the drawbacks of this technology as it stands today will allow us to improve it for the future. Here are some major challenges we face today with generative AI: \n\n1. Interpretability and Transparency: Generative models, particularly deep learning models, are often referred to as \"black boxes\" due to their lack of interpretability. This means it's difficult to understand how these models make decisions or generate outputs. This can be problematic in sensitive domains where understanding the reasoning behind outputs is crucial.\n\n2. Data Bias and Fairness: The outputs of generative AI are only as good as the data they've been trained on. If the training data contains biases, the model will likely generate biased outputs. Ensuring fairness and eliminating bias is a significant challenge, as it requires careful data selection and sometimes manual intervention.\n\n3. Resource-Intensive Training: Training generative AI models, especially large ones, can be computationally intensive and require significant amounts of data. This not only has implications for costs but also for the environmental impact of AI research.\n\n4. Regulation and Ethical Use: With AI becoming increasingly influential, the challenge of ensuring that its use is regulated and ethical is ever more pressing. This includes considerations about privacy (given that models can inadvertently generate sensitive information if they've been trained on it), as well as concerns about AI-generated deepfakes and misinformation.\n\n5. Control and Safety: With powerful generative models, there's a risk of generating harmful or inappropriate content. Designing models and systems that can reliably avoid generating such content while still maintaining usefulness and generality is a difficult task.\n\n6. Evaluation Metrics: Evaluating the quality of generative models is non-trivial. Traditional metrics such as BLEU for text and Inception Score for images have known limitations, and human evaluation can be time-consuming and subjective.\n\n7. Limited Real-world Interaction: Most generative models currently operate based on static input and lack the ability to interact dynamically with the real world. Creating models that can interact with humans and environments over time is a significant research challenge.\n\n8. Long-Term Coherence: While generative AI has become impressively good at generating plausible-sounding or looking content, maintaining coherence and consistency over longer pieces of text or larger images is still challenging.\n\n9. Dependency on Large Datasets: Current state-of-the-art generative models usually rely on large amounts of data for training. However, for many tasks and languages, such data may not be available or accessible. This poses a challenge in expanding the applicability of generative AI.","metadata":{}},{"cell_type":"markdown","source":"\n# **Future Outlook?**","metadata":{}},{"cell_type":"markdown","source":"The whole idea of AI hinges upon the idea of learning from past iterations and improving upon it. Likewise this field of generative AI will do the same, learning and improving upon past implementations. GPT 4 has far surpassed the abilities of GPT 3, and in such a short time. Likewise the wave of generative AI has rode on the backs of prior research and AI technology. ","metadata":{}},{"cell_type":"markdown","source":"\n*The difference between GPT 3.5 and GPT 4. Noteable improvements are in the Bar Exam (from ~10th percentile to ~90th percentile, and STEM subjects such as the AP Statistics exam (from ~40th percentile to ~85th percentile and the USABO semifinalist exam, which is an exam for the top biology students in the USA(from 31st percentile to 99th percentile). Image is from a table in [OpenAI's GPT-4 Technical Report.](https://cdn.openai.com/papers/gpt-4.pdf)*","metadata":{}},{"cell_type":"markdown","source":"As generative AI continues to evolve, it's shaping a future ripe with potential and new possibilities. The past two years have seen remarkable technical and applicational advancements, each contributing to the momentum of generative AI and influencing its future trajectory.\n\nWe've seen strides in efficiency of training large models, advances in token handling, and improved augmentation techniques. These breakthroughs have enabled generative AI models to produce increasingly sophisticated content across a wide range of tasks. This suggests a future where generative AI models will continue to evolve and enhance their performance, tackling more complex tasks across diverse domains.\n\nThe development of conditional transformers and latent variable models also hold significant promise. These advancements pave the way for controlled generation tasks and better data representation, indicating a future where we can exert more control over the output of generative models. This could lead to AI systems that are not only more capable but also more customizable and adaptable to specific tasks.\n\nChallenges such as achieving interpretability, fairness, control, and safety in AI systems remain critical. However, these challenges also present opportunities for future growth. They prompt researchers to innovate in ways that align AI with societal values, ethical considerations, and practical needs. We can foresee advancements in AI ethics, regulation, and methodologies to address these concerns.\n\nFurthermore, the need to optimize resource-intensive training processes and reduce AI's environmental impact has prompted innovation in resource efficiency. This could potentially lead to greener and more sustainable AI practices in the future.\n\nIn the face of limited interaction with the real world and difficulties in maintaining coherence over longer pieces of generated content, the emphasis might shift towards creating models that can interact with humans and environments over time. This could result in AI systems that are more dynamic, interactive, and responsive, thereby improving the user experience and applicability in real-world scenarios.\n\nWhile we anticipate these advancements, the dependency of generative AI on large datasets continues to be a hurdle, especially for tasks and languages with limited data. We might see increased efforts towards developing models that can learn efficiently from smaller datasets or developing methods for synthetic data generation.\n\nThe rapid pace of advancements and the range of challenges to overcome paint an exciting future for generative AI. It's a future characterized by more capable, customizable, ethical, and interactive AI systems that can generate valuable insights, drive innovation, and contribute to a multitude of applications across different sectors.","metadata":{}},{"cell_type":"markdown","source":"\n# **References:** ","metadata":{}},{"cell_type":"markdown","source":"1.\tRichie Cotton, (2023). *The Generative AI Tools Landscape.* Datacamp.\n2.\tTim Murphy, (2023). *The evolution of chatbots and generative AI.\n* Techtarget.\n3.\tMatt White, (2023). *A Brief History of Generative AI.\n* medium.\n4.\tJohn Moore, (2023). *7 generative AI challenges that businesses should consider\n.* Techtarget.\n5.\tMichelle Lee, Lukas Kruger, (2023). *Risks and ethical considerations of generative AI\n.* Deloitte.\n6.\tCem Dilmegani, (2023). *Generative AI Ethics: Top 6 Concerns.* AI Multiple.\n7.\t(2023). *GPT-4 Technical Report.* OpenAI\n8.\tOermann, Eric Karl, Kondziolka, Douglas, (2023). *On Chatbots and Generative Artificial Intelligence\n.* LWW journal.","metadata":{}}]}