{ "id": 583716, "title": "to pip or not to pip, that's the question ", "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/583716", "author_name": "", "post_date": "2025-06-08T19:33:13.345048Z", "votes": 6, "comment_count": 9, "views": 0, "content": "

It's the first time I enter a code competition that does not allow me to connect to the internet and install the packages my model uses with

\n

\"! pip\"

\n

What is the best way around this ?

\n

Can I use the dependency option ?

", "messages": [ { "id": "3220099", "postDate": "06/08/2025 19:33:13", "content": "

It's the first time I enter a code competition that does not allow me to connect to the internet and install the packages my model uses with

\n

\"! pip\"

\n

What is the best way around this ?

\n

Can I use the dependency option ?

", "rawMarkdown": "It's the first time I enter a code competition that does not allow me to connect to the internet and install the packages my model uses with \n\n\"! pip\"\n\nWhat is the best way around this ?\n\nCan I use the dependency option ?", "votes": null }, { "id": "3220105", "postDate": "06/08/2025 19:58:06", "content": "

You can install all your required dependencies in another notebook with internet and save them later as a dataset which you can attach to your notebook without internet. Check this discussion for better references.

", "rawMarkdown": "You can install all your required dependencies in another notebook with internet and save them later as a dataset which you can attach to your notebook without internet. Check this [discussion](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/435153) for better references.", "votes": null }, { "id": "3220126", "postDate": "06/08/2025 20:50:04", "content": "

You need to download the relevant packages and save them in a separate dataset and then load them in your submission kernel.
\nAn example is here @fredericnicholson

", "rawMarkdown": "You need to download the relevant packages and save them in a separate dataset and then load them in your submission kernel.\nAn example is [here](https://www.kaggle.com/code/ravi20076/cmi2025-public-imports-v1) @fredericnicholson", "votes": null }, { "id": "3220143", "postDate": "06/08/2025 22:08:38", "content": "

According to the Kaggle Packages documentation, under the \"Using Dependency Manager to import Python dependencies\" section, we should be able to pip install packages using the dependency manager in the Add-ons section of Kaggle notebooks. However, when I tried this in the context of the competition, it didn’t work as expected. The notebook runs successfully during the save phase, but fails during the scoring phase—even though I followed the documentation exactly.

\n

Now I’ve noticed that two experienced users suggested installing wheel files from a dataset using pip directly within notebook cells. So I’m wondering—is there a known issue with the Kaggle dependency manager not handling packages properly during scoring, especially in offline environments?

", "rawMarkdown": "According to the Kaggle Packages documentation, under the [\"Using Dependency Manager to import Python dependencies\"](https://www.kaggle.com/docs/packages#packages-python-dependencies) section, we should be able to pip install packages using the dependency manager in the Add-ons section of Kaggle notebooks. However, when I tried this in the context of the competition, it didn’t work as expected. The notebook runs successfully during the save phase, but fails during the scoring phase—even though I followed the documentation exactly.\n\nNow I’ve noticed that two experienced users suggested installing wheel files from a dataset using pip directly within notebook cells. So I’m wondering—is there a known issue with the Kaggle dependency manager not handling packages properly during scoring, especially in offline environments?", "votes": null }, { "id": "3220161", "postDate": "06/08/2025 23:07:48", "content": "

Hi Frederic.

\n

I've been trying to get better organised and develop my code in a (private) github repo rather than having everything in kaggle notebooks, thus necessitating pip install. The problem is, as you point out, that since this is a no-internet competition we can't pip install into competition notebooks.

\n

Here's my workflow. Hopefully this is useful for others:

\n
    \n
  1. Package your code as a repository on github.com.
    2. \n
  2. Set up a private access token on github (see this discussion)
    3. \n
  3. Include this as a kaggle \"secret\".
    4. \n
  4. Create a new notebook, set it as a utility script.
    5. \n
  5. Pip install your github repo in this notebook with a command like !pip install git+https://{github_pat}@github.com/username/repo.git --target=/kaggle/working/ where github_pat is your github private access token as a kaggle secret. Note the target option, which is described in this notebook and this notebook, both due to kononenko
    6. \n
  6. Create another notebook (this is your actual competition notebook). Select \"no internet\".
    7. \n
  7. Add the previous notebook as a dataset.
    8. \n
  8. Now, the import module statement should work in your notebook where module is your relevant module name (defined in your code repo).
    9. \n
\n

If your github repo is not private, forget about steps 2, 3, and you don't need the {github_pat} in step 5.

\n

For people who want to explore this method of code organisation and devlopment, I recently wrote a tutorial on setting up a github repo for kaggle.

", "rawMarkdown": "Hi Frederic. \n\nI've been trying to get better organised and develop my code in a (private) github repo rather than having everything in kaggle notebooks, thus necessitating pip install. The problem is, as you point out, that since this is a no-internet competition we can't pip install into competition notebooks.\n\nHere's my workflow. Hopefully this is useful for others:\n\n1. Package your code as a repository on github.com.\n2. Set up a private access token on github ([see this discussion](https://www.kaggle.com/discussions/questions-and-answers/537365))\n3. Include this as a kaggle \"secret\".\n4. Create a new notebook, set it as a utility script.\n5. Pip install your github repo in this notebook with a command like `!pip install git+https://{github_pat}@github.com/username/repo.git --target=/kaggle/working/` where `github_pat` is your github private access token as a kaggle secret. Note the `target` option, which is described in [this notebook](https://www.kaggle.com/code/kononenko/pip-install-no-internet/) and [this notebook, both due to kononenko](https://www.kaggle.com/code/kononenko/pip-install-once)\n6. Create another notebook (this is your actual competition notebook). Select \"no internet\".\n7. Add the previous notebook as a dataset.\n8. Now, the `import module` statement should work in your notebook where `module` is your relevant module name (defined in your code repo).\n\nIf your github repo is not private, forget about steps 2, 3, and you don't need the `{github_pat}` in step 5.\n\nFor people who want to explore this method of code organisation and devlopment, I recently wrote a [tutorial on setting up a github repo for kaggle](https://www.kaggle.com/code/nnjjpp/how-to-reuse-your-code).", "votes": null }, { "id": "3220186", "postDate": "06/09/2025 01:18:37", "content": "

This isn't a \"Kaggle Package\" competition, there were many issues with Kaggle packages in Drawing competition.
\nOnce you submit the notebook for scoring there's no package being built with the dependencies so it fails.

\n

Offline pip install steps, like the approach described above would work best.

", "rawMarkdown": "This isn't a \"Kaggle Package\" competition, there were many issues with Kaggle packages in [Drawing competition](https://www.kaggle.com/competitions/drawing-with-llms/discussion?sort=undefined&search=error). \nOnce you submit the notebook for scoring there's no package being built with the dependencies so it fails. \n\nOffline pip install steps, like the approach described above would work best.", "votes": null }, { "id": "3220198", "postDate": "06/09/2025 02:00:58", "content": "

we have two options:

\n

option 1:

\n

In a new notebook, run the following and save it.

\n
!pip wheel ultralytics --wheel- ./wheels\n! -r wheels. wheels/\n!rm -rf wheels/\n
\n

and then turn the output into a dataset and then import it and install it like so:

\n
!pip install --no-deps -q /kaggle//ultralytics/wheels/ultralytics*.whl\n
\n

option 2:

\n

if it's a basic python package, you can:

\n
    \n
  1. In your notebook, click on Add-ons from the top menu
    2. \n
  2. Select Install Dependencies
    3. \n
  3. Enter the package you want to install: ex - pip install ultralytics
    4. \n
  4. Click run (to install the package (internet on)) or click save to save it.
    5. \n
\n

When you save the notebook for submission or directly use the \"Submit\" button, the packages will be installed itself (internet off).

\n

Option 2 is for quick testing. Option 1 will save you installation time.

", "rawMarkdown": "we have two options:\n\noption 1:\n\nIn a new notebook, run the following and save it.\n\n```python\n!pip wheel ultralytics --wheel-dir ./wheels\n!zip -r wheels.zip wheels/\n!rm -rf wheels/\n```\n\nand then turn the output into a dataset and then import it and install it like so:\n\n```python\n!pip install --no-deps -q /kaggle/input/ultralytics/wheels/ultralytics*.whl\n```\n\noption 2:\n\nif it's a basic python package, you can:\n\n1. In your notebook, click on `Add-ons` from the top menu \n2. Select `Install Dependencies` \n3. Enter the package you want to install: ex - `pip install ultralytics`\n4. Click run (to install the package (internet on)) or click save to save it. \n\nWhen you save the notebook for submission or directly use the \"Submit\" button, the packages will be installed itself (internet off). \n\nOption 2 is for quick testing. Option 1 will save you installation time.", "votes": null }, { "id": "3220397", "postDate": "06/09/2025 08:35:31", "content": "

Thank you very much for the information, I will try with offline pip install steps.

", "rawMarkdown": "Thank you very much for the information, I will try with offline pip install steps.", "votes": null }, { "id": "3221285", "postDate": "06/10/2025 17:33:09", "content": "

@ravi20076 ,

\n

thanks for sharing your example. I understand what you are doing in this notebook, but how are you referencing this content from a different kaggle notebook?

", "rawMarkdown": "ravi20076 , \n\nthanks for sharing your example. I understand what you are doing in this notebook, but how are you referencing this content from a different kaggle notebook?", "votes": null }, { "id": "3221292", "postDate": "06/10/2025 17:44:41", "content": "

@alejopaullier, thank you for sharing this link. I am much closer, but it looks the directory is missing information required for installation,

\n
\n

\"ERROR: Directory '/kaggle/input/autogluon-install-fgn/install_ag' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.\"
\n any idea how to fis that ?

\n
", "rawMarkdown": "alejopaullier, thank you for sharing this link. I am much closer, but it looks the directory is missing information required for installation, \n>\"ERROR: Directory '/kaggle/input/autogluon-install-fgn/install_ag' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.\" \nany idea how to fis that ?", "votes": null } ], "comments": [ { "id": 3220105, "author_name": "alejopaullier", "author_url": "", "post_date": "06/08/2025 19:58:06", "content": "

You can install all your required dependencies in another notebook with internet and save them later as a dataset which you can attach to your notebook without internet. Check this discussion for better references.

", "votes": null, "replies": [ { "id": 3221292, "author_name": "fredericnicholson", "author_url": "", "post_date": "06/10/2025 17:44:41", "content": "

@alejopaullier, thank you for sharing this link. I am much closer, but it looks the directory is missing information required for installation,

\n
\n

\"ERROR: Directory '/kaggle/input/autogluon-install-fgn/install_ag' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.\"
\n any idea how to fis that ?

\n
", "votes": null, "replies": [] } ] }, { "id": 3220126, "author_name": "ravi20076", "author_url": "", "post_date": "06/08/2025 20:50:04", "content": "

You need to download the relevant packages and save them in a separate dataset and then load them in your submission kernel.
\nAn example is here @fredericnicholson

", "votes": null, "replies": [ { "id": 3221285, "author_name": "fredericnicholson", "author_url": "", "post_date": "06/10/2025 17:33:09", "content": "

@ravi20076 ,

\n

thanks for sharing your example. I understand what you are doing in this notebook, but how are you referencing this content from a different kaggle notebook?

", "votes": null, "replies": [] } ] }, { "id": 3220143, "author_name": "pelinkeskin", "author_url": "", "post_date": "06/08/2025 22:08:38", "content": "

According to the Kaggle Packages documentation, under the \"Using Dependency Manager to import Python dependencies\" section, we should be able to pip install packages using the dependency manager in the Add-ons section of Kaggle notebooks. However, when I tried this in the context of the competition, it didn’t work as expected. The notebook runs successfully during the save phase, but fails during the scoring phase—even though I followed the documentation exactly.

\n

Now I’ve noticed that two experienced users suggested installing wheel files from a dataset using pip directly within notebook cells. So I’m wondering—is there a known issue with the Kaggle dependency manager not handling packages properly during scoring, especially in offline environments?

", "votes": null, "replies": [ { "id": 3220186, "author_name": "rashmibanthia", "author_url": "", "post_date": "06/09/2025 01:18:37", "content": "

This isn't a \"Kaggle Package\" competition, there were many issues with Kaggle packages in Drawing competition.
\nOnce you submit the notebook for scoring there's no package being built with the dependencies so it fails.

\n

Offline pip install steps, like the approach described above would work best.

", "votes": null, "replies": [ { "id": 3220397, "author_name": "pelinkeskin", "author_url": "", "post_date": "06/09/2025 08:35:31", "content": "

Thank you very much for the information, I will try with offline pip install steps.

", "votes": null, "replies": [] } ] } ] }, { "id": 3220161, "author_name": "nnjjpp", "author_url": "", "post_date": "06/08/2025 23:07:48", "content": "

Hi Frederic.

\n

I've been trying to get better organised and develop my code in a (private) github repo rather than having everything in kaggle notebooks, thus necessitating pip install. The problem is, as you point out, that since this is a no-internet competition we can't pip install into competition notebooks.

\n

Here's my workflow. Hopefully this is useful for others:

\n
    \n
  1. Package your code as a repository on github.com.
    2. \n
  2. Set up a private access token on github (see this discussion)
    3. \n
  3. Include this as a kaggle \"secret\".
    4. \n
  4. Create a new notebook, set it as a utility script.
    5. \n
  5. Pip install your github repo in this notebook with a command like !pip install git+https://{github_pat}@github.com/username/repo.git --target=/kaggle/working/ where github_pat is your github private access token as a kaggle secret. Note the target option, which is described in this notebook and this notebook, both due to kononenko
    6. \n
  6. Create another notebook (this is your actual competition notebook). Select \"no internet\".
    7. \n
  7. Add the previous notebook as a dataset.
    8. \n
  8. Now, the import module statement should work in your notebook where module is your relevant module name (defined in your code repo).
    9. \n
\n

If your github repo is not private, forget about steps 2, 3, and you don't need the {github_pat} in step 5.

\n

For people who want to explore this method of code organisation and devlopment, I recently wrote a tutorial on setting up a github repo for kaggle.

", "votes": null, "replies": [] }, { "id": 3220198, "author_name": "jaejohn", "author_url": "", "post_date": "06/09/2025 02:00:58", "content": "

we have two options:

\n

option 1:

\n

In a new notebook, run the following and save it.

\n
!pip wheel ultralytics --wheel- ./wheels\n! -r wheels. wheels/\n!rm -rf wheels/\n
\n

and then turn the output into a dataset and then import it and install it like so:

\n
!pip install --no-deps -q /kaggle//ultralytics/wheels/ultralytics*.whl\n
\n

option 2:

\n

if it's a basic python package, you can:

\n
    \n
  1. In your notebook, click on Add-ons from the top menu
    2. \n
  2. Select Install Dependencies
    3. \n
  3. Enter the package you want to install: ex - pip install ultralytics
    4. \n
  4. Click run (to install the package (internet on)) or click save to save it.
    5. \n
\n

When you save the notebook for submission or directly use the \"Submit\" button, the packages will be installed itself (internet off).

\n

Option 2 is for quick testing. Option 1 will save you installation time.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3220099": "It's the first time I enter a code competition that does not allow me to connect to the internet and install the packages my model uses with \n\n\"! pip\"\n\nWhat is the best way around this ?\n\nCan I use the dependency option ?", "3220105": "You can install all your required dependencies in another notebook with internet and save them later as a dataset which you can attach to your notebook without internet. Check this [discussion](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/435153) for better references.", "3220126": "You need to download the relevant packages and save them in a separate dataset and then load them in your submission kernel.\nAn example is [here](https://www.kaggle.com/code/ravi20076/cmi2025-public-imports-v1) @fredericnicholson", "3220143": "According to the Kaggle Packages documentation, under the [\"Using Dependency Manager to import Python dependencies\"](https://www.kaggle.com/docs/packages#packages-python-dependencies) section, we should be able to pip install packages using the dependency manager in the Add-ons section of Kaggle notebooks. However, when I tried this in the context of the competition, it didn’t work as expected. The notebook runs successfully during the save phase, but fails during the scoring phase—even though I followed the documentation exactly.\n\nNow I’ve noticed that two experienced users suggested installing wheel files from a dataset using pip directly within notebook cells. So I’m wondering—is there a known issue with the Kaggle dependency manager not handling packages properly during scoring, especially in offline environments?", "3220161": "Hi Frederic. \n\nI've been trying to get better organised and develop my code in a (private) github repo rather than having everything in kaggle notebooks, thus necessitating pip install. The problem is, as you point out, that since this is a no-internet competition we can't pip install into competition notebooks.\n\nHere's my workflow. Hopefully this is useful for others:\n\n1. Package your code as a repository on github.com.\n2. Set up a private access token on github ([see this discussion](https://www.kaggle.com/discussions/questions-and-answers/537365))\n3. Include this as a kaggle \"secret\".\n4. Create a new notebook, set it as a utility script.\n5. Pip install your github repo in this notebook with a command like `!pip install git+https://{github_pat}@github.com/username/repo.git --target=/kaggle/working/` where `github_pat` is your github private access token as a kaggle secret. Note the `target` option, which is described in [this notebook](https://www.kaggle.com/code/kononenko/pip-install-no-internet/) and [this notebook, both due to kononenko](https://www.kaggle.com/code/kononenko/pip-install-once)\n6. Create another notebook (this is your actual competition notebook). Select \"no internet\".\n7. Add the previous notebook as a dataset.\n8. Now, the `import module` statement should work in your notebook where `module` is your relevant module name (defined in your code repo).\n\nIf your github repo is not private, forget about steps 2, 3, and you don't need the `{github_pat}` in step 5.\n\nFor people who want to explore this method of code organisation and devlopment, I recently wrote a [tutorial on setting up a github repo for kaggle](https://www.kaggle.com/code/nnjjpp/how-to-reuse-your-code).", "3220186": "This isn't a \"Kaggle Package\" competition, there were many issues with Kaggle packages in [Drawing competition](https://www.kaggle.com/competitions/drawing-with-llms/discussion?sort=undefined&search=error). \nOnce you submit the notebook for scoring there's no package being built with the dependencies so it fails. \n\nOffline pip install steps, like the approach described above would work best.", "3220198": "we have two options:\n\noption 1:\n\nIn a new notebook, run the following and save it.\n\n```python\n!pip wheel ultralytics --wheel-dir ./wheels\n!zip -r wheels.zip wheels/\n!rm -rf wheels/\n```\n\nand then turn the output into a dataset and then import it and install it like so:\n\n```python\n!pip install --no-deps -q /kaggle/input/ultralytics/wheels/ultralytics*.whl\n```\n\noption 2:\n\nif it's a basic python package, you can:\n\n1. In your notebook, click on `Add-ons` from the top menu \n2. Select `Install Dependencies` \n3. Enter the package you want to install: ex - `pip install ultralytics`\n4. Click run (to install the package (internet on)) or click save to save it. \n\nWhen you save the notebook for submission or directly use the \"Submit\" button, the packages will be installed itself (internet off). \n\nOption 2 is for quick testing. Option 1 will save you installation time.", "3220397": "Thank you very much for the information, I will try with offline pip install steps.", "3221285": "ravi20076 , \n\nthanks for sharing your example. I understand what you are doing in this notebook, but how are you referencing this content from a different kaggle notebook?", "3221292": "alejopaullier, thank you for sharing this link. I am much closer, but it looks the directory is missing information required for installation, \n>\"ERROR: Directory '/kaggle/input/autogluon-install-fgn/install_ag' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.\" \nany idea how to fis that ?" }, "source": "meta" }