{"cells":[{"metadata":{},"cell_type":"markdown","source":"This is a two part tutorial series on TF-Agents \n\nPart 1 : https://www.kaggle.com/usharengaraju/tfagents-environment-policy-driver\n\nPart 2 : https://www.kaggle.com/usharengaraju/tf-agents-replay-buffer-network-checkpointer\n\nCredit : The article series has been adapted from the official tensorflow documentation."},{"metadata":{},"cell_type":"markdown","source":"## Reinforcement Learning\n\nReinforcement learning (RL) is a machine learning framework where agents takes action in an environment in order to maximize the cumulative reward .\n\n![](https://drive.google.com/uc?id=1ayosPHB3DIbmgnuu0kkN-dpA1TfnurNo)\n\nPic Credit : ai.googleblog.com\n\n### TF-Agents :\n\nA reliable, scalable and easy to use TensorFlow library for Contextual Bandits and Reinforcement Learning.\n\n![](https://drive.google.com/uc?id=1zjzi0OY7G-QusVpYppjdVvbAP-pX0-RZ)\n\nPic Credit : Tensorflow Dev Summit\n\n### Advantages of TF-Agents :\n\n🎯 Great Resources to get started with Reinforcement Learning like colab , documentation and resources\n\n🎯 Well suited to handle complex RL problems\n\n🎯 Helps in developing RL algorithms quickly\n\n🎯 Can be configured easily with gin-config\n\n![](https://drive.google.com/uc?id=1QBCM4TbPlaoftfpXZVoQ7uUScvDtqga0)\n\nPic Credit : Tensorflow Dev Summit\n\nIn this notebook , we will be discussing three concepts - Environment , Policy and Driver\n\n### Environment\n\n<div class=\"alert alert-block alert-info\">\nThe agent receives an observation and chooses an action which it applies on the environment and gets back reward\nand a observation from the environment. The goal of the agent is to train a policy to choose actions which will\nmaximize the cumulative rewards.\n</div>\n\nTF-Agents has both Python and TensorFlow implentation of environments .\n"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install -q tf-agents\n!pip install -q gym","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"from __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport abc\nimport tensorflow as tf\nimport numpy as np\n\nfrom tf_agents.environments import py_environment\nfrom tf_agents.environments import tf_environment\nfrom tf_agents.environments import tf_py_environment\nfrom tf_agents.environments import utils\nfrom tf_agents.specs import array_spec\nfrom tf_agents.environments import wrappers\nfrom tf_agents.environments import suite_gym\nfrom tf_agents.trajectories import time_step as ts\n\n\nfrom tf_agents.specs import array_spec\nfrom tf_agents.specs import tensor_spec\nfrom tf_agents.networks import network\n\nfrom tf_agents.policies import py_policy\nfrom tf_agents.policies import random_py_policy\nfrom tf_agents.policies import scripted_py_policy\nfrom tf_agents.policies import tf_policy\nfrom tf_agents.policies import random_tf_policy\nfrom tf_agents.policies import actor_policy\nfrom tf_agents.policies import q_policy\nfrom tf_agents.policies import greedy_policy\n\nfrom tf_agents.environments import suite_gym\nfrom tf_agents.metrics import py_metrics\nfrom tf_agents.metrics import tf_metrics\nfrom tf_agents.drivers import py_driver\nfrom tf_agents.drivers import dynamic_episode_driver\n\ntf.compat.v1.enable_v2_behavior()\n\nfrom tf_agents.trajectories import time_step as ts\n\ntf.compat.v1.enable_v2_behavior()\n\ntf.compat.v1.enable_v2_behavior()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Python Environments :\n\n<div class=\"alert alert-block alert-info\">\nThe interface that all python environments must implement is in environments/py_environment.PyEnvironment.\n</div>\n\nAfter interaction with the environment the following information about the next step are available \n\n📌 **observation:** This is the part of the environment state that the agent can observe to choose its actions at the next step.\n\n📌 **reward:** The agent is learning to maximize the sum of these rewards across multiple steps.\n\n📌 **step_type:** Interactions with the environment are usually part of a sequence/episode. e.g. multiple moves in a game of chess. step_type can be either FIRST, MID or LAST to indicate whether this time step is the first, intermediate or last step in a sequence.\n\n📌 **discount:** This is a float representing how much to weight the reward at the next time step relative to the reward at the current time step.\n\n"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"class PyEnvironment(object):\n\n  def reset(self):\n    \"\"\"Return initial_time_step.\"\"\"\n    self._current_time_step = self._reset()\n    return self._current_time_step\n\n  def step(self, action):\n    \"\"\"Apply action and return new time_step.\"\"\"\n    if self._current_time_step is None:\n        return self.reset()\n    self._current_time_step = self._step(action)\n    return self._current_time_step\n\n  def current_time_step(self):\n    return self._current_time_step\n\n  def time_step_spec(self):\n    \"\"\"Return time_step_spec.\"\"\"\n\n  @abc.abstractmethod\n  def observation_spec(self):\n    \"\"\"Return observation_spec.\"\"\"\n\n  @abc.abstractmethod\n  def action_spec(self):\n    \"\"\"Return action_spec.\"\"\"\n\n  @abc.abstractmethod\n  def _reset(self):\n    \"\"\"Return initial_time_step.\"\"\"\n\n  @abc.abstractmethod\n  def _step(self, action):\n    \"\"\"Apply action and return new time_step.\"\"\"\n    self._current_time_step = self._step(action)\n    return self._current_time_step","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Standard Environments :\n\n<div class=\"alert alert-block alert-info\">\nTF Agents has built-in wrappers for many standard environments like the OpenAI Gym, DeepMind-control and Atari, which can be easily loaded using our environment suites. The code below loads the CartPole environment from the OpenAI gym\n</div>"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"environment = suite_gym.load('CartPole-v0')\nprint('action_spec:', environment.action_spec())\nprint('time_step_spec.observation:', environment.time_step_spec().observation)\nprint('time_step_spec.step_type:', environment.time_step_spec().step_type)\nprint('time_step_spec.discount:', environment.time_step_spec().discount)\nprint('time_step_spec.reward:', environment.time_step_spec().reward)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Policy\n\n<div class=\"alert alert-block alert-info\">\nPolicies map an observation from the environment to an action or a distribution over actions.Most policies have a neural network to compute actions and/or distributions over actions from TimeSteps.Policies can be saved/restored, and can be used indepedently of the agent for data collection, evaluation etc.\n</div>\n\nPolicies contain the following information \n\n📌 **action:** The action to be applied to the environment.\n\n📌 **state:** The state of the policy (e.g. RNN state) to be fed into the next call to action.\n\n📌 **info:** Optional side information such as action log probabilities.\n\n"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"class Base(object):\n\n  @abc.abstractmethod\n  def __init__(self, time_step_spec, action_spec, policy_state_spec=()):\n    self._time_step_spec = time_step_spec\n    self._action_spec = action_spec\n    self._policy_state_spec = policy_state_spec\n\n  @abc.abstractmethod\n  def reset(self, policy_state=()):\n    # return initial_policy_state.\n    pass\n\n  @abc.abstractmethod\n  def action(self, time_step, policy_state=()):\n    # return a PolicyStep(action, state, info) named tuple.\n    pass\n\n  @abc.abstractmethod\n  def distribution(self, time_step, policy_state=()):\n    # Not implemented in python, only for TF policies.\n    pass\n\n  @abc.abstractmethod\n  def update(self, policy):\n    # update self to be similar to the input `policy`.\n    pass\n\n  @property\n  def time_step_spec(self):\n    return self._time_step_spec\n\n  @property\n  def action_spec(self):\n    return self._action_spec\n\n  @property\n  def policy_state_spec(self):\n    return self._policy_state_spec","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Random Python Policy\n\nRandomPyPolicy generates random actions for the discrete/continuous given action_spec. "},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"action_spec = array_spec.BoundedArraySpec((2,), np.int32, -10, 10)\nmy_random_py_policy = random_py_policy.RandomPyPolicy(time_step_spec=None,\n    action_spec=action_spec)\ntime_step = None\naction_step = my_random_py_policy.action(time_step)\nprint(action_step)\naction_step = my_random_py_policy.action(time_step)\nprint(action_step)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Drivers\n\n<div class=\"alert alert-block alert-info\">\nDrivers are abstraction for the process of executing a policy in an environment for a specified number of steps  during data collection, evaluation and generating a video of the agent.The data encountered by the driver at each step like observation , action , reward , current and next step is saved in Trajectory and broadcast to a set of observers such as replay buffers and metrics. \n</div>\n\nImplementations for drivers are available both in Python and TensorFlow\n\n**Python Drivers :**\n\nThe PyDriver class takes a python environment, a python policy and a list of observers to update at each step. "},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"class PyDriver(object):\n\n  def __init__(self, env, policy, observers, max_steps=1, max_episodes=1):\n    self._env = env\n    self._policy = policy\n    self._observers = observers or []\n    self._max_steps = max_steps or np.inf\n    self._max_episodes = max_episodes or np.inf\n\n  def run(self, time_step, policy_state=()):\n    num_steps = 0\n    num_episodes = 0\n    while num_steps < self._max_steps and num_episodes < self._max_episodes:\n\n      # Compute an action using the policy for the given time_step\n      action_step = self._policy.action(time_step, policy_state)\n\n      # Apply the action to the environment and get the next step\n      next_time_step = self._env.step(action_step.action)\n\n      # Package information into a trajectory\n      traj = trajectory.Trajectory(\n         time_step.step_type,\n         time_step.observation,\n         action_step.action,\n         action_step.info,\n         next_time_step.step_type,\n         next_time_step.reward,\n         next_time_step.discount)\n\n      for observer in self._observers:\n        observer(traj)\n\n      # Update statistics to check termination\n      num_episodes += np.sum(traj.is_last())\n      num_steps += np.sum(~traj.is_boundary())\n\n      time_step = next_time_step\n      policy_state = action_step.state\n\n    return time_step, policy_state\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below runs a random policy on the CartPole environment, saving the results to a **replay buffer**.\n"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"env = suite_gym.load('CartPole-v0')\npolicy = random_py_policy.RandomPyPolicy(time_step_spec=env.time_step_spec(), \n                                         action_spec=env.action_spec())\nreplay_buffer = []\nmetric = py_metrics.AverageReturnMetric()\nobservers = [replay_buffer.append, metric]\ndriver = py_driver.PyDriver(\n    env, policy, observers, max_steps=20, max_episodes=1)\n\ninitial_time_step = env.reset()\nfinal_time_step, _ = driver.run(initial_time_step)\n\nprint('Replay Buffer:')\nfor traj in replay_buffer:\n  print(traj)\n\nprint('Average Return: ', metric.result())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**References :**\n\nhttps://www.tensorflow.org/agents\n\nhttps://www.youtube.com/watch?v=-TTziY7EmUA&t=286s\n\nhttps://www.youtube.com/watch?v=U7g7-Jzj9qo&t=1479s"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}