The best performing solutions for this kernels-only competition were the solutions using BigGANs implementations found on GitHub, and there was a little discussion here about exporting Python project files into a single snippet to make it possible to run on Kaggle kernel. We cannot just clone a repo into a kernel, so the goal is to prepare the code from a repository into a single standalone file, or notebook.
\n\nSure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source. Much more convenient would be to work on the project with proper sub-modules and then apply some converter to dump everything into a single file.
\n\nSo the question is, do you know any Python package or snippet, that could traverse the Python project structure and dump into a valid standalone script? How do you usually solve this task? Is there any proven solution, or do you write custom exporting code for this purpose? Or maybe you're ok with working on kernels directly?
\n\nWould be glad to hear your thoughts.
", "messages": [ { "id": "601384", "postDate": "08/17/2019 14:33:18", "content": "The best performing solutions for this kernels-only competition were the solutions using BigGANs implementations found on GitHub, and there was a little discussion here about exporting Python project files into a single snippet to make it possible to run on Kaggle kernel. We cannot just clone a repo into a kernel, so the goal is to prepare the code from a repository into a single standalone file, or notebook.
\n\nSure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source. Much more convenient would be to work on the project with proper sub-modules and then apply some converter to dump everything into a single file.
\n\nSo the question is, do you know any Python package or snippet, that could traverse the Python project structure and dump into a valid standalone script? How do you usually solve this task? Is there any proven solution, or do you write custom exporting code for this purpose? Or maybe you're ok with working on kernels directly?
\n\nWould be glad to hear your thoughts.
", "rawMarkdown": "The best performing solutions for this kernels-only competition were the solutions using BigGANs implementations found on GitHub, and there was a little discussion [here](https://www.kaggle.com/c/generative-dog-images/discussion/104281) about exporting Python project files into a single snippet to make it possible to run on Kaggle kernel. We cannot just clone a repo into a kernel, so the goal is to prepare the code from a repository into a single standalone file, or notebook.\n\nSure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source. Much more convenient would be to work on the project with proper sub-modules and then apply some converter to dump everything into a single file.\n\nSo the question is, do you know any Python package or snippet, that could traverse the Python project structure and dump into a valid standalone script? How do you usually solve this task? Is there any proven solution, or do you write custom exporting code for this purpose? Or maybe you're ok with working on kernels directly?\n\nWould be glad to hear your thoughts.", "votes": null }, { "id": "601427", "postDate": "08/17/2019 15:38:53", "content": "It depends whether the Kaggle kernel-only competition allows datasets or not. Some comps like \"Dog Generative Images\" disallow both datasets and internet. In this case you need to write a script which copies and pastes all the code into a single file which you can then copy and paste into Kaggle. Also config and text files need to be converted to Python string variables. Other kernel-only comps such as Steel Defect Detection disallows internet but doesn't disallow datasets. In this case, you can package the entire repo as a dataset and import the dataset. Finally, if a comp allows both internet and datasets, then you can just use ! pip install.
Yeah, the internet-enabled cases are the most simple ones. Though recently it seems that there were many competitions where internet access is disabled. (Or maybe it only seems so).
\n\nRegarding the dataset-enabled case, it should be a \"dataset\" that includes the repository, right? Or a folder with the *.py files?
Take the entire repository with its files and folder structure and create a zip file (or tar.gz). Then include that single zip file in a Kaggle dataset. When you attach a Kaggle dataset with a zip file, Kaggle will automatically unzip it when you start your kernel. Therefore your kernel will immediately have access to the same files and folder structure as the original GitHub repository.
Ok great, makes sense! Thank you.
", "rawMarkdown": "Ok great, makes sense! Thank you.", "votes": null }, { "id": "602269", "postDate": "08/18/2019 20:57:56", "content": "\n\n\nSure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source.
\n
Thats what I did :D\nI've already done this before with Bert and other huge nlp models. Next time I will try to go the other way
", "rawMarkdown": "> Sure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source.\n\nThats what I did :D\nI've already done this before with Bert and other huge nlp models. Next time I will try to go the other way", "votes": null }, { "id": "602582", "postDate": "08/19/2019 08:23:52", "content": "Yeah, that what I usually do, too :) But it is so time consuming, and one can easily mess things up, I would say. Especially in the competitions like this with GANs, when the key is a model architecture, and not data preprocessing or feature engineering. So many implementations and repositories, I was a bit overwhelmed with manually adapting this solutions to kernels.
", "rawMarkdown": "Yeah, that what I usually do, too :) But it is so time consuming, and one can easily mess things up, I would say. Especially in the competitions like this with GANs, when the key is a model architecture, and not data preprocessing or feature engineering. So many implementations and repositories, I was a bit overwhelmed with manually adapting this solutions to kernels.", "votes": null } ], "comments": [ { "id": 601427, "author_name": "cdeotte", "author_url": "", "post_date": "08/17/2019 15:38:53", "content": "It depends whether the Kaggle kernel-only competition allows datasets or not. Some comps like \"Dog Generative Images\" disallow both datasets and internet. In this case you need to write a script which copies and pastes all the code into a single file which you can then copy and paste into Kaggle. Also config and text files need to be converted to Python string variables. Other kernel-only comps such as Steel Defect Detection disallows internet but doesn't disallow datasets. In this case, you can package the entire repo as a dataset and import the dataset. Finally, if a comp allows both internet and datasets, then you can just use ! pip install.
Yeah, the internet-enabled cases are the most simple ones. Though recently it seems that there were many competitions where internet access is disabled. (Or maybe it only seems so).
\n\nRegarding the dataset-enabled case, it should be a \"dataset\" that includes the repository, right? Or a folder with the *.py files?
Take the entire repository with its files and folder structure and create a zip file (or tar.gz). Then include that single zip file in a Kaggle dataset. When you attach a Kaggle dataset with a zip file, Kaggle will automatically unzip it when you start your kernel. Therefore your kernel will immediately have access to the same files and folder structure as the original GitHub repository.
Ok great, makes sense! Thank you.
", "votes": null, "replies": [] } ] }, { "id": 602269, "author_name": "dvorobiev", "author_url": "", "post_date": "08/18/2019 20:57:56", "content": "\n\n\nSure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source.
\n
Thats what I did :D\nI've already done this before with Bert and other huge nlp models. Next time I will try to go the other way
", "votes": null, "replies": [ { "id": 602582, "author_name": "purplejester", "author_url": "", "post_date": "08/19/2019 08:23:52", "content": "Yeah, that what I usually do, too :) But it is so time consuming, and one can easily mess things up, I would say. Especially in the competitions like this with GANs, when the key is a model architecture, and not data preprocessing or feature engineering. So many implementations and repositories, I was a bit overwhelmed with manually adapting this solutions to kernels.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "601384": "The best performing solutions for this kernels-only competition were the solutions using BigGANs implementations found on GitHub, and there was a little discussion [here](https://www.kaggle.com/c/generative-dog-images/discussion/104281) about exporting Python project files into a single snippet to make it possible to run on Kaggle kernel. We cannot just clone a repo into a kernel, so the goal is to prepare the code from a repository into a single standalone file, or notebook.\n\nSure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source. Much more convenient would be to work on the project with proper sub-modules and then apply some converter to dump everything into a single file.\n\nSo the question is, do you know any Python package or snippet, that could traverse the Python project structure and dump into a valid standalone script? How do you usually solve this task? Is there any proven solution, or do you write custom exporting code for this purpose? Or maybe you're ok with working on kernels directly?\n\nWould be glad to hear your thoughts.", "601427": "It depends whether the Kaggle kernel-only competition allows datasets or not. Some comps like \"Dog Generative Images\" disallow both datasets and internet. In this case you need to write a script which copies and pastes all the code into a single file which you can then copy and paste into Kaggle. Also config and text files need to be converted to Python string variables. Other kernel-only comps such as Steel Defect Detection disallows internet but doesn't disallow datasets. In this case, you can package the entire repo as a dataset and import the dataset. Finally, if a comp allows both internet and datasets, then you can just use `! pip install`.", "601438": "Yeah, the internet-enabled cases are the most simple ones. Though recently it seems that there were many competitions where internet access is disabled. (Or maybe it only seems so).\n\nRegarding the dataset-enabled case, it should be a \"dataset\" that includes the repository, right? Or a folder with the `*.py` files?", "601494": "Take the entire repository with its files and folder structure and create a `zip` file (or `tar.gz`). Then include that single zip file in a Kaggle dataset. When you attach a Kaggle dataset with a zip file, Kaggle will automatically unzip it when you start your kernel. Therefore your kernel will immediately have access to the same files and folder structure as the original GitHub repository.", "601504": "Ok great, makes sense! Thank you.", "602269": "> Sure enough, it is not too convenient to write the code directly into the kernel without any modular structure, or copy it function after function from the original source.\n\nThats what I did :D\nI've already done this before with Bert and other huge nlp models. Next time I will try to go the other way", "602582": "Yeah, that what I usually do, too :) But it is so time consuming, and one can easily mess things up, I would say. Especially in the competitions like this with GANs, when the key is a model architecture, and not data preprocessing or feature engineering. So many implementations and repositories, I was a bit overwhelmed with manually adapting this solutions to kernels." }, "source": "meta" }