{
  "id": 414487,
  "title": "Using TPUs in kaggle",
  "url": "/competitions/benetech-making-graphs-accessible/discussion/414487",
  "author_name": "",
  "post_date": "2023-06-01T21:01:09.426646900Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>First time in a computer vision competition.  I was trying to use the TPUs available on kaggle with pytorch, but could not get it started.  I tried to use </p>\n<p><code>!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py</code><br>\n<code>!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev</code></p>\n<p>But get the following error</p>\n<blockquote>\n  <p>tead: <a href=\"https://pip.pypa.io/warnings/venv\" target=\"_blank\">https://pip.pypa.io/warnings/venv</a> class=\"ansi-yellow-fg\"&gt;<br>\n  Traceback (most recent call last):<br>\n    File \"pytorch-xla-env-setup.py\", line 182, in <br>\n      run_setup(args)<br>\n    File \"pytorch-xla-env-setup.py\", line 156, in run_setup<br>\n      install_vm(version, args.apt_packages, is_root=not args.tpu)<br>\n    File \"pytorch-xla-env-setup.py\", line 140, in install_vm<br>\n      subprocess.call(cmd)<br>\n    File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call<br>\n      with Popen(*popenargs, *<em>kwargs) as p:\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in <strong>init</strong>\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: 'gsutil'tead: <a href=\"https://pip.pypa.io/warnings/venv\" target=\"_blank\">https://pip.pypa.io/warnings/venv</a> class=\"ansi-yellow-fg\"&gt;\nTraceback (most recent call last):\n  File \"pytorch-xla-env-setup.py\", line 182, in \n    run_setup(args)\n  File \"pytorch-xla-env-setup.py\", line 156, in run_setup\n    install_vm(version, args.apt_packages, is_root=not args.tpu)\n  File \"pytorch-xla-env-setup.py\", line 140, in install_vm\n    subprocess.call(cmd)\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call\n    with Popen(</em>popenargs, **kwargs) as p:<br>\n    File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in <strong>init</strong><br>\n      self._execute_child(args, executable, preexec_fn, close_fds,<br>\n    File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child<br>\n      raise child_exception_type(errno_num, err_msg, err_filename)<br>\n  FileNotFoundError: [Errno 2] No such file or directory: 'gsutil'</p>\n</blockquote>\n<p><strong>How do I install gsutil?</strong></p>",
  "messages": [
    {
      "id": "2284255",
      "postDate": "06/01/2023 21:01:09",
      "content": "<p>First time in a computer vision competition.  I was trying to use the TPUs available on kaggle with pytorch, but could not get it started.  I tried to use </p>\n<p><code>!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py</code><br>\n<code>!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev</code></p>\n<p>But get the following error</p>\n<blockquote>\n  <p>tead: <a href=\"https://pip.pypa.io/warnings/venv\" target=\"_blank\">https://pip.pypa.io/warnings/venv</a> class=\"ansi-yellow-fg\"&gt;<br>\n  Traceback (most recent call last):<br>\n    File \"pytorch-xla-env-setup.py\", line 182, in <br>\n      run_setup(args)<br>\n    File \"pytorch-xla-env-setup.py\", line 156, in run_setup<br>\n      install_vm(version, args.apt_packages, is_root=not args.tpu)<br>\n    File \"pytorch-xla-env-setup.py\", line 140, in install_vm<br>\n      subprocess.call(cmd)<br>\n    File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call<br>\n      with Popen(*popenargs, *<em>kwargs) as p:\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in <strong>init</strong>\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: 'gsutil'tead: <a href=\"https://pip.pypa.io/warnings/venv\" target=\"_blank\">https://pip.pypa.io/warnings/venv</a> class=\"ansi-yellow-fg\"&gt;\nTraceback (most recent call last):\n  File \"pytorch-xla-env-setup.py\", line 182, in \n    run_setup(args)\n  File \"pytorch-xla-env-setup.py\", line 156, in run_setup\n    install_vm(version, args.apt_packages, is_root=not args.tpu)\n  File \"pytorch-xla-env-setup.py\", line 140, in install_vm\n    subprocess.call(cmd)\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call\n    with Popen(</em>popenargs, **kwargs) as p:<br>\n    File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in <strong>init</strong><br>\n      self._execute_child(args, executable, preexec_fn, close_fds,<br>\n    File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child<br>\n      raise child_exception_type(errno_num, err_msg, err_filename)<br>\n  FileNotFoundError: [Errno 2] No such file or directory: 'gsutil'</p>\n</blockquote>\n<p><strong>How do I install gsutil?</strong></p>",
      "rawMarkdown": "First time in a computer vision competition.  I was trying to use the TPUs available on kaggle with pytorch, but could not get it started.  I tried to use \n\n`!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py`\n`!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev`\n\nBut get the following error\n\n>tead: https://pip.pypa.io/warnings/venv class=\"ansi-yellow-fg\">\nTraceback (most recent call last):\n  File \"pytorch-xla-env-setup.py\", line 182, in <module>\n    run_setup(args)\n  File \"pytorch-xla-env-setup.py\", line 156, in run_setup\n    install_vm(version, args.apt_packages, is_root=not args.tpu)\n  File \"pytorch-xla-env-setup.py\", line 140, in install_vm\n    subprocess.call(cmd)\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call\n    with Popen(*popenargs, **kwargs) as p:\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in __init__\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: 'gsutil'tead: https://pip.pypa.io/warnings/venv class=\"ansi-yellow-fg\">\nTraceback (most recent call last):\n  File \"pytorch-xla-env-setup.py\", line 182, in <module>\n    run_setup(args)\n  File \"pytorch-xla-env-setup.py\", line 156, in run_setup\n    install_vm(version, args.apt_packages, is_root=not args.tpu)\n  File \"pytorch-xla-env-setup.py\", line 140, in install_vm\n    subprocess.call(cmd)\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call\n    with Popen(*popenargs, **kwargs) as p:\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in __init__\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: 'gsutil'\n\n**How do I install gsutil?**",
      "votes": null
    },
    {
      "id": "2284755",
      "postDate": "06/02/2023 09:00:11",
      "content": "<p>As far as I remember, xla is already installed on kaggle. if you want to install it yourself, you should install the library from a dedicated location. first of all you need to know the version of python. after that you put libraries like this<br>\n!pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 <a href=\"https://storage.googleapis\" target=\"_blank\">https://storage.googleapis</a>.<br>\ncom/tpu-pytorch/wheels/colab/torch_xla-2.0-cp39-cp39-linux_x86_64.whl<br>\ncp39 - means for python 3.9</p>",
      "rawMarkdown": "As far as I remember, xla is already installed on kaggle. if you want to install it yourself, you should install the library from a dedicated location. first of all you need to know the version of python. after that you put libraries like this\n!pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 https://storage.googleapis.\ncom/tpu-pytorch/wheels/colab/torch_xla-2.0-cp39-cp39-linux_x86_64.whl\ncp39 - means for python 3.9",
      "votes": null
    },
    {
      "id": "2285435",
      "postDate": "06/02/2023 17:50:46",
      "content": "<p>Thanks!  This seems to be working now.</p>",
      "rawMarkdown": "Thanks!  This seems to be working now.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2284755,
      "author_name": "zaakciiru",
      "author_url": "",
      "post_date": "06/02/2023 09:00:11",
      "content": "<p>As far as I remember, xla is already installed on kaggle. if you want to install it yourself, you should install the library from a dedicated location. first of all you need to know the version of python. after that you put libraries like this<br>\n!pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 <a href=\"https://storage.googleapis\" target=\"_blank\">https://storage.googleapis</a>.<br>\ncom/tpu-pytorch/wheels/colab/torch_xla-2.0-cp39-cp39-linux_x86_64.whl<br>\ncp39 - means for python 3.9</p>",
      "votes": null,
      "replies": [
        {
          "id": 2285435,
          "author_name": "danielphalen",
          "author_url": "",
          "post_date": "06/02/2023 17:50:46",
          "content": "<p>Thanks!  This seems to be working now.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2284255": "First time in a computer vision competition.  I was trying to use the TPUs available on kaggle with pytorch, but could not get it started.  I tried to use \n\n`!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py`\n`!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev`\n\nBut get the following error\n\n>tead: https://pip.pypa.io/warnings/venv class=\"ansi-yellow-fg\">\nTraceback (most recent call last):\n  File \"pytorch-xla-env-setup.py\", line 182, in <module>\n    run_setup(args)\n  File \"pytorch-xla-env-setup.py\", line 156, in run_setup\n    install_vm(version, args.apt_packages, is_root=not args.tpu)\n  File \"pytorch-xla-env-setup.py\", line 140, in install_vm\n    subprocess.call(cmd)\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call\n    with Popen(*popenargs, **kwargs) as p:\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in __init__\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: 'gsutil'tead: https://pip.pypa.io/warnings/venv class=\"ansi-yellow-fg\">\nTraceback (most recent call last):\n  File \"pytorch-xla-env-setup.py\", line 182, in <module>\n    run_setup(args)\n  File \"pytorch-xla-env-setup.py\", line 156, in run_setup\n    install_vm(version, args.apt_packages, is_root=not args.tpu)\n  File \"pytorch-xla-env-setup.py\", line 140, in install_vm\n    subprocess.call(cmd)\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 340, in call\n    with Popen(*popenargs, **kwargs) as p:\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 858, in __init__\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File \"/usr/local/lib/python3.8/subprocess.py\", line 1704, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: 'gsutil'\n\n**How do I install gsutil?**",
    "2284755": "As far as I remember, xla is already installed on kaggle. if you want to install it yourself, you should install the library from a dedicated location. first of all you need to know the version of python. after that you put libraries like this\n!pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 https://storage.googleapis.\ncom/tpu-pytorch/wheels/colab/torch_xla-2.0-cp39-cp39-linux_x86_64.whl\ncp39 - means for python 3.9",
    "2285435": "Thanks!  This seems to be working now."
  },
  "source": "meta"
}