![\"drawing\"/](\"https://miro.medium.com/v2/resize:fit:640/format:webp/1*n1AZU6IkpfjC0l22Md2x0Q.png\")
\nAuthor: Bhargav M gowda
\nCategory: Other
","metadata":{"papermill":{"duration":0.00313,"end_time":"2023-07-05T19:56:26.045945","exception":false,"start_time":"2023-07-05T19:56:26.042815","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Introduction\n\nDarwin and spencer wrote 'survival of the fittest' implying those who were most fit would survive the social world due to some biological mechanism that made them superior. In evolution theory, organisms undergo a process of natural selection, where those with advantageous traits or behaviors have a higher likelihood of survival and reproduction. Over generations, these favorable traits become more prevalent in the population. Among them, humans stand tall as the most triumphant species on Mother Earth. What makes humans special is their ability to explore, exploit, generalise and adapt to changing environment or circumstances. If this process is that good why don't we make use of it?? This led to the birth of a new field of machine learning which will become a pillar of the 21st century's AI revolution.\n\n","metadata":{"papermill":{"duration":0.002132,"end_time":"2023-07-05T19:56:26.050652","exception":false,"start_time":"2023-07-05T19:56:26.048520","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## History of reinforcement learning\n\nThe history of reinforcement learning (RL) can be traced back to the mid-20th century. The early foundation for RL was laid by the work on Learning Automata (1950s-1960s), explored the idea of using probabilistic models to make adaptive decisions. It was followed by the Dynamic Programming (1957), a theoretical framework for solving optimization problems by breaking them down into smaller subproblems. Q-learning (1980) a model-free RL algorithm that learns the optimal action-value function by iteratively updating Q-values based on observed rewards and state transitions, is one of the foundational algorithms in RL. Temporal Difference Learning (1992) combined ideas from dynamic programming and Monte Carlo methods to enable RL agents to learn from incomplete sequences of experiences and estimate the value functions. Deep Q-Network (DQN) (2013) proposed by DeepMind, demonstrated the successful integration of deep neural networks with RL. DQN achieved superhuman performance in playing a variety of Atari 2600 games solely based on pixel inputs. AlphaGo (2016) DeepMind's AlphaGo, a RL-based system, made significant headlines when it defeated the world champion Go player, Lee Sedol [1] ","metadata":{}},{"cell_type":"markdown","source":"RL algorithms aim to optimize decision-making by iteratively learning from interactions with an environment. Let us understand this with an example of a Dog, ofcourse I'm team dog too :) In the process of training the dogs, we use food as a reward to reinforce desired responses and behaviours from it. When a dog performs a desired action, such as sitting or following a command, they are rewarded with a treat. This reward serves as a positive reinforcement, signaling to the dog that their behavior was correct and increasing the likelihood of them repeating that behavior in the future. \n\n[](https://postimg.cc/CdmWPWdM)\n\n\n Fig.2. The process of Training Data Development, referenced from [2]\n
\nIn the dog training example, the agent (the dog) interacts with the environment (the training area) based on a specific policy (commands given by the trainer). The dog's actions (sitting, lying down, coming when called) are influenced by its current state (position, trainer's command) and are reinforced through rewards (treats or praise) provided by the trainer. By associating the actions with the rewards, the dog learns to adjust its behavior and improve its performance over time.\n\nThe concepts of agent, reward, policy, action, state, and environment provide a framework for understanding the dynamics of dog training and how the principles of reinforcement learning can be applied to shape and reinforce desired behaviors in dogs [3]\n\n\n \n
\n Fig.2. Reinforcement learning, referenced from [4]\n
\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## A Taxonomy of Reinforcement Learning Algorithms\n\nThe taxonomy of algorithms in the modern RL space is to highlight the most foundational design choices in deep RL algorithms about what to learn and how to learn it,\nto expose the trade-offs in those choices, and to place a few prominent modern algorithms into context with respect to those choice.\n\nOne of the most important branching points in an RL algorithm is the question of whether the agent has access to (or learns) a model of the environment. By a model of the environment, we mean a function which predicts state transitions and rewards. Algorithms which use a model are called model-based methods, and those that don’t are called model-free. While model-free methods forego the potential gains in sample efficiency from using a model, they tend to be easier to implement and tune.[5]\n\n\n ![](\"https://spinningup.openai.com/en/latest/_images/rl_algorithms_9_15.svg\"/)
\n
\n Fig.2. Taxonomy of Reinforcement learning, referenced from [5]\n
","metadata":{}},{"cell_type":"markdown","source":"\n\n\n Fig.2. Chatgpt three step cycle, referenced from [8]\n
","metadata":{}},{"cell_type":"markdown","source":"## Robotics Transformer for real-world control at scale: RT1\n\nMajor recent advances in multiple subfields of machine learning (ML) research, such as computer vision and natural language processing, have been enabled by a shared common approach that leverages large, diverse datasets and expressive models that can absorb all of the data effectively. Although there have been various attempts to apply this approach to robotics, robots have not yet leveraged highly-capable models as well as other subfields.\n\nSeveral factors contribute to this challenge. First, there’s the lack of large-scale and diverse robotic data, which limits a model’s ability to absorb a broad set of robotic experiences. Data collection is particularly expensive and challenging for robotics because dataset curation requires engineering-heavy autonomous operation, or demonstrations collected using human teleoperations. A second factor is the lack of expressive, scalable, and fast-enough-for-real-time-inference models that can learn from such datasets and generalize effectively.\n\nTo address these challenges, Google propose the Robotics Transformer 1 (RT-1), a multi-task model that tokenizes robot inputs and outputs actions (e.g., camera images, task instructions, and motor commands) to enable efficient inference at runtime, which makes real-time control feasible. This model is trained on a large-scale, real-world robotics dataset of 130k episodes that cover 700+ tasks, collected using a fleet of 13 robots from Everyday Robots (EDR) over 17 months. Google demonstrates that RT-1 can exhibit significantly improved zero-shot generalization to new tasks, environments and objects compared to prior techniques, and Google has open-sourced the RT-1 code, with a hope that will provide a valuable resource for future research on scaling up robot learning. [9]","metadata":{}},{"cell_type":"markdown","source":"## Pre-Training for Robots\nProgress in deep learning highlights the tremendous potential of utilizing diverse robotic datasets for attaining effective generalization and makes it enticing to consider leveraging broad datasets for attaining robust generalization in robotic learning as well. However, in practice, we often want to learn a new skill in a new environment that is unlikely to be contained in the prior data. Therefore we ask: how can we leverage existing diverse offline datasets in combination with small amounts of task-specific data to solve new tasks, while still enjoying the generalization benefits of training on large amounts of data? Researchers from UC Berkely demonstrate that end-to-end offline RL can be an effective approach for doing this, without the need for any representation learning or vision-based pre-training. \n \n Pre-training for robots (PTR), a framework based on offline RL that attempts to effectively learn new tasks by combining pre-training on existing robotic datasets with rapid fine-tuning on a new task, with as few as 10 demonstrations. PTR utilizes an existing offline RL method, conservative Q-learning (CQL), but extends it to include several crucial design decisions that enable PTR to actually work and outperform a variety of prior methods. PTR is the one of the first RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demonstrations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in a variety of toy kitchens. They also demonstrate that PTR can enable effective autonomous fine-tuning and improvement in a handful of trials, without needing any demonstrations.[10]","metadata":{}},{"cell_type":"markdown","source":"## Interfacing language models with Robots : SayCan \n\nLarge language models can encode a wealth of semantic knowledge about the world. Such knowledge could in principle be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack contextual grounding, which makes it difficult to leverage them for decision making within a given real-world context. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. SayCan is proposed to provide this grounding by means of pretrained behaviors, which are used to condition the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model’s “hands and eyes,” while the language model supplies high-level semantic knowledge about the task. \n\nImagine a robot operating in a kitchen that is capable of executing skills such as \"pick up the coffee cup\" or \"go to the sink\". To get the robot to use these skills to perform a complex task (e.g. \"I spilled my drink, can you help?\"), the user could manually break it up into steps consisting of these atomic commands. However,this would be exceedingly tedious. A language model can split the high-level instruction (\"I spilled my drink, can you help?\") into sub-tasks, but it cannot do that effectively unless it has the context of what the robot is capable of given the abilities, current state of the robot and its environment[11]\n\n[](https://postimg.cc/LYmbvTpZ)\n\n\n Fig.2. SayCan, referenced from [12]\n
\n","metadata":{}},{"cell_type":"markdown","source":"## Magnetic Control of nuclear fusion Plasmas in Tokamak \n\n\nA tokamak is the most advanced of current fusion machine designs in order to get fusion reactions going. Although there are many designs, the Tokamak a Soviet design is the best way we have found so far. Tokamak stands for toroidal chamber magnetic coils, it's a ring doughnut shaped vessel with superconducting magnetic coils around it and uses a blend of deuterium and tritium as fuel mix. The fuel should be kept hot enough, dense enough and long enough like the center of a star for fusion reactions to occur. Inside the tokamak the fuel is heated to produce a plasma, a superheated gas in the fourth state of matter. . The hot plasma(150 million degree celcius) is kept away from the walls of the tokamak by a magnetic field produced primarily by two sets of coils that contain the plasma horizontally and vertically. Fast control systems can change the magnetic field produced by these magnets to keep the plasma trapped and stable. The conventional approaches is to precompute a set of feedforward coil currents and voltages, Then designing single-input single-output PID controllers to stabilize the plasma vertical position and control the radial position and plasma current. But, the controllers are designed on the basis of linearized model dynamics, and gain scheduling is required to track time-varying control targets.\n\n\n ![\"drawing\"/](\"https://i.postimg.cc/kGJhn8Qf/ezgif-com-gif-to-flif.gif\")
\n
\"I thought AlphaGo was based on probability calculation and that it was merely a machine. But when I saw its moves, I changed my mind. Surely, AlphaGo is creative.\" - Lee Sedol
\n\nTwo years later, its successor - AlphaZero - learned from scratch to master Go, chess and shogi with a single algorithm. Expanding upon this foundation, **MuZero**, an advanced iteration of AlphaZero, has extended and enhanced these abilities even further. The ability to plan is an important part of human intelligence, allowing us to solve problems and make decisions about the future. For example, if we see dark clouds forming we might predict it will rain and decide to take an umbrella with us before we go out. Researchers have tried to tackle this major challenge in AI by using two main approaches: lookahead search or model-based planning. Systems that use lookahead search, such as AlphaZero, have achieved remarkable success in classic games such as checkers, chess and poker, but rely on being given knowledge of their environment’s dynamics. Model-based systems aim to address this issue by learning an accurate model of an environment’s dynamics, and then using it to plan. However, the complexity of modelling every aspect of an environment has meant these algorithms are unable to compete in visually rich domains, such as Atari. MuZero uses a different approach to overcome the limitations of previous approaches. Instead of trying to model the entire environment, MuZero just models aspects that are important to the agent’s decision-making process. After all, knowing an umbrella will keep you dry is more useful to know than modelling the pattern of raindrops in the air.\n\n \n