{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# *Introduction*","metadata":{}},{"cell_type":"markdown","source":"*In this essay, I have endeavored to illuminate a few of new perspectives on the employment of generative AI, which have emerged as a result of the evolution of Transformer architectures. In addition, the essey encompasses state-of-the-art (SotA)  achievements in various areas of Hi-Tech that may not initially seem directly related to generative AI technologies. However, these advancements provide valuable insights into potential future AI applications.*\n\n*We will start with the background of transformer technology in the context of the entire development of AI and try to give an intuition about how it works and in which areas of machine learning it is applicable. Next, we will digress from the subject of generative AI, touching on the topic of robotics, discussing its limitations in using SotA AI advances to the fullest and what advanced technologies will affect robotics in the future. Then we will talk about reinforcement learning and about the applications of transformers in this area, where intelligence is required to act. Finally, we will move into territory where advances in natural language processing, code generation, reinforcement learning, and robotics seem to converge.*\n\n\n*It is assumed that the reader has an understanding of the basics of ML and enough enthusiasm for learning to follow the suggested links.*","metadata":{}},{"cell_type":"markdown","source":"# **The Rise of the Transformers**","metadata":{}},{"cell_type":"markdown","source":"The progress of generative neural networks has opened before us the prospects for the development of AI far beyond the generation of pictures or texts, in sometimes unanticipated fields, partly due to numerous emergence effects, partly due to its multimodal nature, and sometimes simply because of the generation of demands to other domains, thereby stimulating their growth.","metadata":{}},{"cell_type":"markdown","source":"The last day of November 2022 was a milestone in human history. On this day, ChatGPT technology went public [[1](https://openai.com/blog/chatgpt)].\n\nIt is worth mentioning that from a scientific standpoint, the public release of ChatGPT in November 2022 did not come as a surprise. By that time, the GPT-3 model, which the chatbot was built upon, had already been available in limited capacity for over a year. Furthermore, since the release of the influential article \"[Attention is all you need](https://arxiv.org/abs/1706.03762)\" in 2017 [[2](https://arxiv.org/abs/1706.03762)], the field of AI, with its precursor models and rival systems, had been undergoing progressive improvements, particularly in the realm of transformers. What is of greater significance is that the rapid popularization of ChatGPT brought about irreversible changes in society. Artificial intelligence ceased to be a science fiction concept, and humanity encountered a technology that rapidly disseminate worldwide, profoundly influencing various aspects of life.\n\nFurthermore, a positive feedback loop emerges: the progress of the AI technologies are the result of big data generated through the digitalization of society. However, the introduction of these technologies entails even greater digitalization, subsequently driving further advancements in AI.\n\nThe Internet revolution also exhibited a comparable self-accelerating growth pattern. However, it still took a couple of decades for long-standing [computer networks](https://en.wikipedia.org/wiki/ARPANET), initially limited to specialized institutes, to transform into a tool that fulfills the fundamental needs of every individual, saturating them with [Web content](https://en.wikipedia.org/wiki/World_Wide_Web).","metadata":{}},{"cell_type":"markdown","source":"The term 'Artificial Intelligence' has a history spanning half a century, enduring multiple cycles of high expectations and subsequent periods of reduced enthusiasm known as 'artificial intelligence winters.' Its meaning has undergone numerous transformations over time. In recent decades, the term has been prominently featured in popular media and advertising, primarily referring to machine learning techniques for data analysis and advanced algorithms for game behavior. Within academic circles, the phrase 'narrow artificial intelligence' emerged to distinguish these tools from the conceptualization of an idealized 'general artificial intelligence'([AGI](https://en.wikipedia.org/wiki/Artificial_general_intelligence)). While the advent of deep learning (convolutional neural networks) has facilitated substantial progress in practical domains like image recognition and machine translation, many advancements still appeared as captivating novelties in the hands of scientists and engineers.\n\nHowever, by the end of 2022, computers demonstrated a semblance of what we intuitively associate with \"intelligence\", bringing elements of the science fiction stories we once read closer to reality. Although this may not yet qualify as \"general artificial intelligence\" we are inclined to overlook this distinction, recognizing that the limitation lies more in our understanding of \"intelligence\" rather than the technical possibilities to achieve such alignment.","metadata":{}},{"cell_type":"markdown","source":"The emergence of AI over the past year holds major scientific significance. It has blurred the conventional boundaries between 'narrow' and 'general' AI, as we witness AI tools exhibiting properties that can perplex researchers studying biological intelligence.\n\nHistorically, advancements in different scientific fields have often fueled each other's progress. Increased public demand attracts more researchers to the given domain, leading to breakthroughs that have far-reaching effects in other domains.\n\nResearch on artificial neural networks, from the early [perceptron](https://en.wikipedia.org/wiki/Perceptron)[[5](https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf)] to [convolutional networks](https://en.wikipedia.org/wiki/Convolutional_neural_network)[[6](https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf)], has drawn inspiration from neuroscience[[4](https://www.cse.chalmers.se/~coquand/AUTOMATA/mcp.pdf)]. [The origins of artificial intelligence](https://en.wikipedia.org/wiki/History_of_artificial_intelligence) as a scientific discipline can be traced back to mythology, philosophy and science fiction. At the same time, it is no longer a secret that human thinking is closely connected with the speech apparatus [[7](https://www.sryahwapublications.com/annals-of-language-and-literature/pdf/v2-i4/3.pdf)]. Recent advances in AI, especially in the form of NLP and multimodal transformers, have once again raised deep questions among neuroscientists and philosophers regarding the functioning of human cognition and the nature of intelligence itself, and have provided new methodologies for further research. ","metadata":{}},{"cell_type":"markdown","source":"## **Attention, Transformer!**","metadata":{}},{"cell_type":"markdown","source":"In natural language processing problems, there have long been statistical approaches that ignore word order, but it is necessary to take into account their relative position, since the meaning of the phrase directly depends on it. The use of convolutional networks here did not give such a significant gain in efficiency as in computer vision, since, unlike image pixels, the words most related in meaning are not necessarily located nearby in the text. We need to learn how to evaluate the relationship of each current word with its context. \n \nA way to solve this problem was found and it was called the [attention](https://arxiv.org/ftp/arxiv/papers/1902/1902.02181.pdf) mechanism [[3](https://arxiv.org/abs/1902.02181)]. “[Attention is all you need](https://arxiv.org/abs/1706.03762)” is the slogan that became the headline of an article [[2](https://arxiv.org/abs/1706.03762)] that signified a pivotal milestone in Artificial Intelligence research.\n\nFor the particular case when semantic correlations between words of the same phrase are calculated, it is called \"[self-attention](https://en.wikipedia.org/wiki/Self-attention#:~:text=Self%20Attention%2C%20also%20called%20intra,representation%20of%20the%20same%20sequence.)\". A great illustration of what's going on can be found at https://huggingface.co/spaces/exbert-project/exbert","metadata":{}},{"cell_type":"markdown","source":"\n \nHaving learned the semantic correlations between words (more precisely, [NLP](https://en.wikipedia.org/wiki/Natural_language_processing) operates with [tokens](https://en.wikipedia.org/wiki/Lexical_analysis#Token) - essential lexical units of the language), we can parallelize this operation to simultaneously calculate different semantic aspects (so-called \"[multi-head attention](https://paperswithcode.com/method/multi-head-attention)\") and then repeat the operation at a higher level and, possibly, adding nonlinearities, thus build more and more generalizing the general meaning of the phrase. And in the end, we come to a compressed representation of the meaning of the phrase in the form of a numerical sequence - the so-called [embeddings](https://en.wikipedia.org/wiki/Word_embedding). Moreover, the calculated embedding for a service token (CLS, \"class token\"), for example, added to the beginning of a phrase, can be interpreted as a generalized \"meaning\" of the phrase.\n\nInitially, embeddings were regarded as a form of compression technology. However, the encoder approach associated with embeddings has proven to be remarkably powerful. We have now achieved the capability to map externally perceived information into an internal representation that not only stores the information but also bears resemblance to the extraction of meanings. Does this not, albeit remotely, resemble the processes unfolding within our own minds?","metadata":{}},{"cell_type":"markdown","source":"![Transformer model architecture](https://upload.wikimedia.org/wikipedia/commons/thumb/8/8f/The-Transformer-model-architecture.png/800px-The-Transformer-model-architecture.png)\n\n*Transformer model architecture, author: Yuening Jia - DOI:10.1088/1742-6596/1314/1/012186 https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)*","metadata":{}},{"cell_type":"markdown","source":"So far, we have learned how to encode a natural language phrase by turning a sequence of tokens into a set of embeddings, but what should we do to decode it?\n\nFirstly, while the encoder receives tokens (represented by their respective numbers in a dictionary) as input and converts them into embeddings that capture meaning in context, the decoder must receive embeddings as input and convert them back into token numbers.\n\nSecondly, the decoder functions as an autoregressive model, predicting each subsequent token one by one. The tokens already predicted by the decoder are also fed back into the input, allowing the model to maintain contextual awareness.\n\nDecoding can be likened to encoding by calculating the correlation between the encoder output and the current state of the decoder (referred to as \"cross-attention\"). Moreover, the decoder output sequence is masked (\"masked attention\") to prevent inadvertent leakage of input data into the output. Essentially, at each step, the decoder performs a classification task, selecting the most probable next token from the dictionary based on the initial context received from the encoder and the portion of the output already generated.","metadata":{}},{"cell_type":"markdown","source":"You can read more about attention layers and transformer architecture, as well as how it can be used in applications, [here](https://huggingface.co/learn/nlp-course/chapter1/4?fw=pt#attention-layers).","metadata":{}},{"cell_type":"markdown","source":"## **Multimodality**","metadata":{}},{"cell_type":"markdown","source":"The transformer architecture revolutionized not only natural language processing but also proved surprisingly effective in computer vision tasks. In [ViT](https://en.wikipedia.org/wiki/Vision_transformer) (Vision Transformer), the image is divided into parts that are converted into flattened patches, similar to natural language tokens, and then encoded accordingly. This approach not only enables obtaining embeddings for specific portions of the image but also facilitates deriving an encoded \"meaning\" for the entire image (an equivalent of a service token is used for this operation).\n\n\n\n\n![Vision Transformer Architecture for Image Classification (author: Davide Coccomini, https://en.wikipedia.org/wiki/Vision_transformer)](https://upload.wikimedia.org/wikipedia/commons/3/3e/Vision_Transformer.gif)\n\n*Vision Transformer Architecture for Image Classification (author: Davide Coccomini, https://en.wikipedia.org/wiki/Vision_transformer)*","metadata":{}},{"cell_type":"markdown","source":"The versatility offered by the transformer architecture is precisely what we need. For the first time, we possess the technical capacity to process visual, textual, and audio information in a cohesive manner, closely resembling the functioning of our own cognitive processes.\n\nThe expression \"just like it happens in our head\" has been circulating for 70 years since the introduction of the first [artificial perceptron](https://en.wikipedia.org/wiki/Perceptron) by Rosenblatt, and it has been associated with the onset of the 'AI winter' period. Therefore, it should be approached with caution. However, it is worth mentioning that transformers, in comparison to previous levels of AI technologies, represent a notable step toward our intuitive understanding of intelligence.","metadata":{}},{"cell_type":"markdown","source":"## **Codegeneration**","metadata":{}},{"cell_type":"markdown","source":"Alongside text, images, and other multimedia, transformers are capable of processing program code. The AI's ability to generate code has caused some unease among programmers, but the significance of this development reaches beyond impending changes in the software development profession.\n\nFirstly, code generation systems have already surpassed the boundaries of processing a user's straightforward request. Now, they can acquire a set of additional documentation, potentially multimedia, as context[[24](https://arxiv.org/pdf/2207.05987.pdf)], or a demonstrative pool[[25](https://people.ece.ubc.ca/amesbah/resources/papers/cedar-icse23.pdf)]. Secondly, code generation models have learned to decompose the given task into several simpler steps[[26](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf)]. Thirdly, there are so-called Program-aided language models which combine reasoning using Large Language Models (LLMs) with an algorithmic approach using a Python interpreter[[27](https://arxiv.org/pdf/2211.10435.pdf)]. Lastly, the self-refine approach [[28](https://arxiv.org/pdf/2303.17651.pdf)] is used, where the model receives feedback on the results of the previous output and takes necessary steps for its automatic improvement.\n\nThe fact that ideas are implemented extremely quickly speaks volumes, for example, today [[29](https://help.openai.com/en/articles/6825453-chatgpt-release-notes)] ChatGPT Code Interpreter allows to upload contextual documents up to hundred of megabytes and download the results of work[[30](https://openai.com/blog/chatgpt-plugins#code-interpreter)], that is, we are seeing how artificial intelligence is merging with cloud storage and a virtual execution environment for algorithmic tasks\n\nAll these developments indicate that we are on the horizon of a paradigm shift in automation, transitioning from programmed devices to self-programming agents.\n\n","metadata":{}},{"cell_type":"markdown","source":"# **It's time to act!**","metadata":{}},{"cell_type":"markdown","source":"Undeniably, when we explore works of science fiction literature, we discover that [since ancient times](https://en.wikipedia.org/wiki/Timeline_of_artificial_intelligence), people envisioned artificial intelligence manifesting primarily as intelligent actors capable of executing real-world tasks, rather than solely focusing on writing or painting. There has long been a persistent demand for integrating artificial intelligence into robotics. The question that remains is: how close are we to realizing this demand?\n\nIt suddenly turned out that Transformers can help here, but the world of robots has its own nuances, so let's delve into the details step by step.","metadata":{}},{"cell_type":"markdown","source":"## **Robotics**","metadata":{}},{"cell_type":"markdown","source":"In the realm of robotics, progress has been comparatively slower compared to the wider AI industry. This is primarily due to the immense challenges posed by interacting with the real world. These challenges encompass the vast dimensionality of input data, the difficulty of efficient learning within a practical timeframe in real environments, and the stringent requirements for real-time performance and energy efficiency.","metadata":{}},{"cell_type":"markdown","source":"### **Energy Efficiency**","metadata":{}},{"cell_type":"markdown","source":"Existing computer architectures exhibit substantial inefficiency in comparison to the human brain. To process an equivalent amount of information, the human brain consumes much less energy (the difference is several orders of magnitude). This dissimilarity originates from the contrasting principles of operation between biological neurons, which communicate through short-lived spikes, and modern electronics, which based on transistor logic governed by changes in potential or conductivity.\n\nTowards the end of the 20th century, concepts for creating [neuromorphic computers](https://en.wikipedia.org/wiki/Neuromorphic_engineering) based on the elements that mimic the behavior of biological neurons were introduced. However, it wasn't until the early 21st century that hardware implementations began to emerge within [research projects](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3181466)[[23](https://www.frontiersin.org/articles/10.3389/fnins.2011.00108/full)].\n\nMore recently, Intel introduced commercial versions of its neuromorphic processors, [Loihi 1](https://www.researchgate.net/publication/322548911_Loihi_A_Neuromorphic_Manycore_Processor_with_On-Chip_Learning) and [Loihi 2, accompanied by the Lava framework ](https://www.intel.com/content/www/us/en/newsroom/news/intel-unveils-neuromorphic-loihi-2-lava-software.html)for programming such systems. A subsequent advancement involved the development of the [Kapoho Point board](https://www.intel.com/content/www/us/en/research/neuromorphic-computing.html), which incorporates Loihi 2 and serves as a self-contained solution for portable robotics applications[[8](https://www.intel.com/content/www/us/en/newsroom/news/intel-unveils-neuromorphic-loihi-2-lava-software.html)].\n\nThese [neuromorphic processors](https://en.wikipedia.org/wiki/Cognitive_computer#Intel_Loihi_chip) have demonstrated a significant reduction in [power consumption, up to two orders of magnitude](https://download.intel.com/newsroom/2022/2022innovation/neuromorphic-backgrounder.pdf) compared to traditional chips[[9](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9395703)], while also offering [support for Continual Learning](https://www.intel.com/content/www/us/en/newsroom/news/neuromorphic-computing-helps-robots-keep-learning.html)[[10](https://www.intel.com/content/www/us/en/newsroom/news/neuromorphic-computing-helps-robots-keep-learning.html)][[11](http://www.sandamirskaya.eu/resources/Interactive_Continual_Learning_for_Robots__Neuromorphic_Approach__ICONS_.pdf )] in interactive environments. This marks a notable stride towards the future of robotics. Furthermore, the compatibility of Loihi 2 and Lava with traditional computing instills hope that existing AI solutions can seamlessly transition into the realm of robotics.","metadata":{}},{"cell_type":"markdown","source":"### **Internet of Things**","metadata":{}},{"cell_type":"markdown","source":"Over the past few years, the excitement surrounding the Internet of Things ([IoT](https://en.wikipedia.org/wiki/Internet_of_things)) may have been somewhat overshadowed by the recent breakthroughs in AI. Nevertheless, it is crucial to acknowledge two significant aspects. Firstly, IoT, in conjunction with robotics, fuels the need for the emergence of more energy-efficient computer technologies. Secondly, the pervasive integration of IoT devices enables the accumulation of data from various real-world physical systems. This data accumulation lays the groundwork for the development of virtual reality systems and creates the necessary conditions for a significant leap forward in robotics.\n\nThus, we can say that the Internet of Things will be able to fully show its emergence only after breakthrough achievements in Artificial Intelligence and Robotics, but neither Artificial Intelligence nor Robotics will be able to make a technological revolution without the co-development of the Internet of Things.","metadata":{}},{"cell_type":"markdown","source":"### **Virtual Reality and Real Virtuality**","metadata":{}},{"cell_type":"markdown","source":"The issue at hand is that robots often [face constraints](https://analyticsindiamag.com/do-reinforcement-learning-algorithms-need-new-simulators-to-solve-real-world-problems/) when it comes to learning in real-world environments. The process is often time-consuming, expensive, and even hazardous. Simulating reality emerges as a potential Brian Ichter and Karol Hausman, Research Scientists, Google Research, Brain Teamsolution to this challenge. To facilitate the simulation of reality, it is crucial to develop efficient tools for creating virtual environments that possess equivalent physical properties.  \n\nNotably, [NVidia's Omniverse technology](https://www.nvidia.com/en-us/omniverse/) has emerged as a trendsetter in the field of [digital twins](hhttps://blogs.nvidia.com/blog/2021/12/14/what-is-a-digital-twin/) creation, while Unity is also making efforts [in this domain](https://unity.com/solutions/digital-twin-definition) and offers its own [solutions for 3D modeling](https://unity.com/solutions/architecture-engineering-construction/rt3d-explained). Furthermore, AWS provides [digital twin hosting services](https://aws.amazon.com/what-is/digital-twin/), allowing the deployment and management of virtual replicas and interactions with them.\n\n\nIt is worth noting the intriguing connection between generative artificial intelligence and the generation of 3D models from images. AI can be employed not only for the task of generating 3D models but also for autocompleting their physical properties. Furthermore, the collection of data from these models serves as a catalyst for the continued development of AI. This creates a positive feedback loop that propels advancements in these technologies.","metadata":{}},{"cell_type":"markdown","source":"### **Find a common language with robot**","metadata":{}},{"cell_type":"markdown","source":"The success of large language models has opened up prospects for improving robot control interfaces using natural languages. This task is more difficult than voice control of an agent in a virtual environment, because in a physical environment, more attention must be paid to security and the feasibility of the planned action. [PaLM-SayCan](https://sites.research.google/palm-saycan) is a joint research project of [Google Research](https://g.co/robotics) and [Everyday Robots](https://everydayrobots.com/), that focused on search for effective way to combine advanced language models with robot learning algorithms to leverage the benefits of both[[21](https://ai.googleblog.com/2022/08/towards-helpful-robots-grounding.html)]. \"Do as I can, not as I say\"[[22](https://arxiv.org/pdf/2204.01691.pdf)] - the motto of their approach. It leverages advanced language model knowledge to enable a physical agent, such as a robot, to follow high-level textual instructions for physically-grounded tasks, while grounding the language model in tasks that are feasible within a specific real-world context.","metadata":{}},{"cell_type":"markdown","source":"Robotics also gives rise to a demand for new algorithms in a relatively isolated domain of machine learning, initially  focused on optimizing agent behavior within specific environments - Reinforcement Learning.","metadata":{}},{"cell_type":"markdown","source":"## **Reinforcement Learning**","metadata":{}},{"cell_type":"markdown","source":"[Reinforcement Learning](https://en.wikipedia.org/wiki/Reinforcement_learning) is grounded in the idea of teaching an agent to exhibit rational behavior within a specified environment by using rewards for \"good\" actions and penalties for \"bad\" ones. This concept draws parallels with how living organisms reinforce behavior through the external stimuli.\n\nHowever, a significant challenge lies in defining what constitutes \"correct\" behavior as it involves devising appropriate reward functions, which can be a non-trivial task. One of the approaches is imitation learning, where the behavoir policy that maximizes the reward function is constructed by observing expert (human) behavior. For a brief yet informative overview of reinforcement learning, see [this link](https://www.natolambert.com/writing/intro-to-rl-in-30-minutes).","metadata":{}},{"cell_type":"markdown","source":"### **What's next step?**","metadata":{}},{"cell_type":"markdown","source":"It is important to note that in Reinforcement Learning, the task at hand is generative in nature: given an arbitrary state of the agent and the environment, the objective is to generate a decision regarding the next action to take. Although the assumptions of the [Markov process](https://en.wikipedia.org/wiki/Markov_chain) (\"What happens next depends only on the state of affairs now.\") often allow to consider only the current state, thereby reducing the choice of the next action to an optimization problem (the Markov Property underpins the existence and success of all modern reinforcement learning algorithms [[20](https://www.natolambert.com/writing/intro-to-rl-in-30-minutes)]), the generative aspect remains decisive in determining the behavior of an agent, especially in real physical systems, where an exhaustive description of the external environment is difficult.","metadata":{}},{"cell_type":"markdown","source":"And why not try to use the Transformer here too?\n\nThe Online Decision Transformer [[12](https://arxiv.org/abs/2106.01345)][[13](https://arxiv.org/abs/2202.05607)] presents a compelling solution by combining the strengths of Transformers and Reinforcement Learning. This approach to enable agents to generate action sequences that aim to achieve long-term goals and maximize cumulative rewards. By leveraging the capabilities of Transformers and their attention mechanisms, agents can make informed decisions based on current states and expected future rewards, leading to more sophisticated and strategic decision-making in complex environments. Overall, this approach enhances the performance and effectiveness of Reinforcement Learning agents by tailoring action sequences towards long-term objectives.\n\n\"This is a complete shift in the Reinforcement Learning paradigm since we use generative trajectory modeling (modeling the joint distribution of the sequence of states, actions, and rewards) to replace conventional RL algorithms. This means that in Decision Transformers, we don’t maximize the return but rather generate a series of future actions that achieve the desired return\", Edward Beeching said [[14](https://huggingface.co/learn/deep-rl-course/unitbonus3/decision-transformers)].","metadata":{}},{"cell_type":"markdown","source":"### **Environments**","metadata":{}},{"cell_type":"markdown","source":"There are several popular frameworks that provide environment for Reinforcement Learning such as [Gymnasium](https://gymnasium.farama.org/) or [Unity ML Agents](https://unity.com/products/machine-learning-agents).\n\nHowever, Reinforcement Learning applications go far beyond the emulator of some physical environment. There's no need to look far for examples: ChatGPT is a chatbot trained on human interaction [[15](https://openai.com/research/instruction-following)] using Reinforcement Learning (on top of LLM GPT-3 or GPT-4). The idea was further developed in the form of [AutoGPT](https://autogpt.net/), capable of automating the interaction with ChatGPT and thus providing the user with a much more versatile and independent assistant. In fact, AutoGPT concept is an intermediate stage in the evolution of intelligent assistants that gradually begin to acquire the characteristics of autonomous agents [[16](https://autogpt.net/the-rise-of-autonomous-agents-preparing-for-the-ai-revolution/)].\n\n\nIndeed, we have seen before how reinforcement learning has come to the aid of large language models to make their progress available to mankind. However, we will now look at how these large language models can in turn help reinforcement learning agents.","metadata":{}},{"cell_type":"markdown","source":"## **Algorithm as a Service**","metadata":{}},{"cell_type":"markdown","source":"We have witnessed remarkable advancements in code generation with GPT-4 [[17](https://www.nature.com/articles/d41586-023-01833-0)]. However, imagine if we could automate the generation of algorithms for an agent's behavior in unfamiliar environments. Why not allow the agent, when faced with challenging decisions, to seek advice from a more advanced AI service?\n\nInterestingly, these concepts are being applied in both robotics and virtual environments for Reinforcement Learning agents concurrently. The capability of code generation presents exciting possibilities for programming systems that lack a comprehensive predefined description.","metadata":{}},{"cell_type":"markdown","source":"[ChatGPT for Robotics](https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/) is a project from Microsoft Research that allows the user to control a robot through natural language intercommunication. Human's commands are received at the input of ChatGPT, which, possibly by asking clarifying questions, determines the algorithm of the robot's behavior and, based on a predefined control API, generates the appropriate scripts.\n\n\"Our goal with this research is to see if ChatGPT can think beyond text, and reason about the physical world to help with robotics tasks. We want to help people interact with robots more easily, without needing to learn complex programming languages or details about robotic systems. The key challenge here is teaching ChatGPT how to solve problems considering the laws of physics, the context of the operating environment, and how the robot’s physical actions can change the state of the world.\", Microsoft Research said [[18](https://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf)]\n\n\n![Robotics with ChatGPT: design principles](https://www.microsoft.com/en-us/research/uploads/prod/2023/02/steps-1536x342.png)\n*Robotics with ChatGPT: design principles (image from [[18](https://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf)]  \"ChatGPT for Robotics: Design Principles and Model Abilities\")*","metadata":{}},{"cell_type":"markdown","source":"The [Voyager project](https://voyager.minedojo.org/) is dedicated to teaching an agent to play Minecraft in the [MineDojo](https://minedojo.org/) environment. The agent is provided with a GPT-4 generated world exploration plan that maximizes curiosity. Agent training is iterative: at each iteration, Voyager uses GPT-4 to generate a script to achieve the goal and uses feedback from the game environment and possible programming errors to improve the code on the next iteration. A successfully executed script is stored in a special “skills library”, which is constantly updated. This library supports the search for suitable skills, and, if necessary, their combination for a more complex action. The authors of the study note a funny emergency that maintaining such a skill base not only improves the behavior of the agent, but also improves the quality of the AutoGPT itself (“external” fine-tuning).\n\nVoyager consists of three key components: 1) an automatic curriculum that maximizes exploration, 2) an ever-growing skill library of executable code for storing and retrieving complex behaviors, and 3) a new iterative prompting mechanism that incorporates environment feedback, execution errors, and self-verification for program improvement [[19](https://voyager.minedojo.org/)]. \n\n![Key components of the Voyager agent](https://voyager.minedojo.org/assets/images/components.png)\n\n*Key components of the Voyager agent (image from  [[19](https://voyager.minedojo.org/)] \"Voyager: An Open-Ended Embodied Agent with Large Language Models\"*)","metadata":{}},{"cell_type":"markdown","source":"# **Where does Kaggle fit into all of this?**","metadata":{}},{"cell_type":"markdown","source":"SotA AI technologies operate with datasets and computing power far beyond what an individual researcher or small group can afford. Nevertheless, from the above review, it is clear that the initially islolated domains of AI are increasingly merging and recombining with each other.\n\nTherefore, we should expect the emergence of new needs and new opportunities for using medium-sized datasets and “classical” machine learning techniques. The promising prospects for AI in robotics and IoT will require portable and energy efficient ML solutions. For example, synthetic datasets and the distillation knowledge technique, or fundamentally new approaches, may come in handy. But there is no doubt that the need for specialists and data will only increase.\n\nThe Kaggle platform concentrates a variety of high quality datasets and people with the skills to process them, which has value no matter the scale.\n\nWhatever the case, despite the dizzying progress of the last decade, AI technology has consistently based on a Data Science foundation, and the Kaggle community is still the largest provider of professional practitioners in the field.","metadata":{}},{"cell_type":"markdown","source":"# **Bibliography & References**","metadata":{}},{"cell_type":"markdown","source":"[1] Introducing ChatGPT (2022) https://openai.com/blog/chatgpt ","metadata":{}},{"cell_type":"markdown","source":"[2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. arXiv preprint arXiv:1706.03762. https://arxiv.org/abs/1706.03762","metadata":{}},{"cell_type":"markdown","source":"[3] Galassi, A., Lippi, M., & Torroni, P. (2019). Attention in Natural Language Processing. arXiv preprint arXiv:1902.02181. https://arxiv.org/abs/1902.02181","metadata":{}},{"cell_type":"markdown","source":"[4] McCulloch, W., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5(4), 115-133. https://www.cse.chalmers.se/~coquand/AUTOMATA/mcp.pdf","metadata":{}},{"cell_type":"markdown","source":"[5] Rosenblatt, F. (1957). The Perceptron — A Perceiving and Recognizing Automaton. Cornell Aeronautical Laboratory Inc. https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf","metadata":{}},{"cell_type":"markdown","source":"[6] Fukushima, Kunihiko (1980). \"Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position\" (PDF). Biological Cybernetics. 36 (4): 193–202. doi:10.1007/BF00344251. PMID 7370364. S2CID 206775608. https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf","metadata":{}},{"cell_type":"markdown","source":"[7] Md Mozaffor Hossain. “Reciprocity between Mind and Language: A Psycholinguistic Reasoning\".\nAnnals of Language and Literature, 2(4), 2018, pp. 13-17. https://www.sryahwapublications.com/annals-of-language-and-literature/pdf/v2-i4/3.pdf","metadata":{}},{"cell_type":"markdown","source":"[8] Intel Advances Neuromorphic with Loihi 2, New Lava Software Framework and New Partners (September 30, 2021) https://www.intel.com/content/www/us/en/newsroom/news/intel-unveils-neuromorphic-loihi-2-lava-software.html","metadata":{}},{"cell_type":"markdown","source":"[9] Davies et al.: Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (Vol. 109, No. 5, May 2021)\nhttps://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9395703","metadata":{}},{"cell_type":"markdown","source":"[10] Intel Labs Improves Interactive, Continual Learning for Robots with Neuromorphic Computing (August 31, 2022) https://www.intel.com/content/www/us/en/newsroom/news/neuromorphic-computing-helps-robots-keep-learning.html","metadata":{}},{"cell_type":"markdown","source":"[11] Sandamirskaya, Y., Neftci, E. O., & Schöner, G. (2011). Interactive continual learning for robots: a neuromorphic approach. http://www.sandamirskaya.eu/resources/Interactive_Continual_Learning_for_Robots__Neuromorphic_Approach__ICONS_.pdf ","metadata":{}},{"cell_type":"markdown","source":"[12] Chen, L., Dai, Z., Dai, B., Song, Y., Zhang, Y., & Artzi, Y. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. arXiv preprint arXiv:2106.01345.\nhttps://arxiv.org/abs/2106.01345","metadata":{}},{"cell_type":"markdown","source":"[13] Zheng, Q., Zeng, X., Li, X., Zhou, M., & Choi, Y. (2022). Online Decision Transformer. arXiv preprint arXiv:2202.05607.\nhttps://arxiv.org/abs/2202.05607","metadata":{}},{"cell_type":"markdown","source":"[14] Edward Beeching from Hugging Face \"Decision Transformers and Offline RL\" https://huggingface.co/learn/deep-rl-course/unitbonus3/decision-transformers","metadata":{}},{"cell_type":"markdown","source":"[15] Ouyang L. et al (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155\nhttps://arxiv.org/pdf/2203.02155.pdf","metadata":{}},{"cell_type":"markdown","source":"[16] Masa (May 5, 2023). The Rise Of Autonomous Agents: Preparing For Ther AI Revolution https://autogpt.net/the-rise-of-autonomous-agents-preparing-for-the-ai-revolution/","metadata":{}},{"cell_type":"markdown","source":"[17] Perkel, Jeffrey M. (June 5, 2023). \"Six tips for better coding with ChatGPT\". Nature. 618 (7964): 422–423. doi:10.1038/d41586-023-01833-0 https://www.nature.com/articles/d41586-023-01833-0","metadata":{}},{"cell_type":"markdown","source":"[18] Vemprala, S., Bonatti, R., Bucker, A., & Kapoor, A. (2023). ChatGPT for Robotics: Design Principles and Model Abilities. \nhttps://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf","metadata":{}},{"cell_type":"markdown","source":"[19] Wang G. et al (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv preprint arXiv:2305.16291. \nhttps://arxiv.org/abs/2305.16291","metadata":{}},{"cell_type":"markdown","source":"[20] A Different Intro to RL in 30 Minutes: Nathan Lambert https://www.natolambert.com/writing/intro-to-rl-in-30-minutes","metadata":{}},{"cell_type":"markdown","source":"[21] Ichter, B., & Hausman, K. (2022, August). Towards Helpful Robots: Grounding Language in Robotic Affordances. Google AI Blog. \nhttps://ai.googleblog.com/2022/08/towards-helpful-robots-grounding.html","metadata":{}},{"cell_type":"markdown","source":"[22] Ahn, M., Hesse, C., & Wermter, S. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.\nhttps://arxiv.org/pdf/2204.01691.pdf","metadata":{}},{"cell_type":"markdown","source":"[23] Poon, C.-S., & Zhou, K. (2011). Neuromorphic silicon neurons and large-scale neural networks: challenges and opportunities. Frontiers in Neuroscience, 5, 108. doi:10.3389/fnins.2011.00108\nhttps://www.frontiersin.org/articles/10.3389/fnins.2011.00108/full","metadata":{}},{"cell_type":"markdown","source":"[24] Zhou, S., Dai, Z., Dai, B., & Song, Y. (2022). DocPrompting: Generating Code by retrieving the docs. \nhttps://arxiv.org/pdf/2207.05987.pdf","metadata":{}},{"cell_type":"markdown","source":"[25] Nashid, N., Nejati, S., & Azim, T. (2023). Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning.\nhttps://people.ece.ubc.ca/amesbah/resources/papers/cedar-icse23.pdf","metadata":{}},{"cell_type":"markdown","source":"[26] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: Wei J. et al https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf","metadata":{}},{"cell_type":"markdown","source":"[27] Gao, T., Zhang, Y., Wang, S., Li, W., Long, C., & Wang, T. (2022). PAL: Program-aided language models.\nhttps://arxiv.org/pdf/2211.10435.pdf","metadata":{}},{"cell_type":"markdown","source":"[28] Madaan, A., Kumar, V., Sharma, S., & Chaudhary, Y. (2023). Self-Refine: Iterative Refinement with Self-Feedback. \nhttps://arxiv.org/pdf/2303.17651.pdf\n","metadata":{}},{"cell_type":"markdown","source":"[29] ChatGPT — Release Notes: Code interpreter is now rolling out in beta on web (July 6), 2023 https://help.openai.com/en/articles/6825453-chatgpt-release-notes","metadata":{}},{"cell_type":"markdown","source":"[30] Code interpreter: An experimental ChatGPT model that can use Python, handle uploads and downloads https://openai.com/blog/chatgpt-plugins#code-interpreter","metadata":{}},{"cell_type":"markdown","source":"### *Usage of AI generation of text*\n\n*The selection of material and the creation of the text of this essay was carried out by me personally. However, since I am not an English native-speaker, AI services were actively used for translation and grammar correction*","metadata":{}},{"cell_type":"markdown","source":"______________________________________________________________________________________________________________________________________","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nsubmission = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.set_index('type', inplace=True)\nsubmission.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.loc['essay_category'].value = 'Generative AI'\nsubmission.loc['essay_url'].value = 'https://www.kaggle.com/kkiryll/generative-ai-rise-of-the-transformers'\n\nsubmission.loc['feedback1_url'].value = 'https://www.kaggle.com/code/mistylight/mini-giants-small-language-models/comments#2332524'\nsubmission.loc['feedback2_url'].value = 'https://www.kaggle.com/code/yossra/the-generative-ai-hype-cycle/comments#2332717'\nsubmission.loc['feedback3_url'].value = 'https://www.kaggle.com/code/shraddhabarke/automated-programming-with-generative-ai/comments#2345508'\nsubmission.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=True)","metadata":{},"execution_count":null,"outputs":[]}]}