Blog Post: We Downloaded 10,000,000 Jupyter Notebooks From Github
\nThere are over millions of Jupyter notebooks open sourced on GitHub. You can use the code snippet to download 10 million Jupyter notebooks from an AWS bucket thanks to JetBrains. It takes around 2400 hours on CPU to download all the notebooks. This could be used as an external dataset for training or retrieval. You could use this for training your models in case you own a data center or have a couple thousand free GPUs lying around.
\nfrom urllib.request import urlretrieve\nimport random\nimport json\n\ndef read_json(filepath): \n return json.load(open(filepath))\n\ndef read_ipynb(file): \n with open(file, mode= 'r', encoding= 'utf-8') as f:\n return json.loads(f.read())\n\n!wget https://github-notebooks-samples.s3-eu-west-1.amazonaws.com/ntbs_list.json\nnotebooks_list = read_json('ntbs_list.json')\nrandom.shuffle(notebooks_list)\n\nfor ipynb_file in tqdm(notebooks_list[:NUM_NOTEBOOKS]):\n ipynb_save_path = TMP_DIR / ipynb_file\n urlretrieve(f'https://github-notebooks-update1.s3-eu-west-1.amazonaws.com/{ipynb_file}', ipynb_save_path)\nBlog Post: We Downloaded 10,000,000 Jupyter Notebooks From Github
\nThere are over millions of Jupyter notebooks open sourced on GitHub. You can use the code snippet to download 10 million Jupyter notebooks from an AWS bucket thanks to JetBrains. It takes around 2400 hours on CPU to download all the notebooks. This could be used as an external dataset for training or retrieval. You could use this for training your models in case you own a data center or have a couple thousand free GPUs lying around.
\nfrom urllib.request import urlretrieve\nimport random\nimport json\n\ndef read_json(filepath): \n return json.load(open(filepath))\n\ndef read_ipynb(file): \n with open(file, mode= 'r', encoding= 'utf-8') as f:\n return json.loads(f.read())\n\n!wget https://github-notebooks-samples.s3-eu-west-1.amazonaws.com/ntbs_list.json\nnotebooks_list = read_json('ntbs_list.json')\nrandom.shuffle(notebooks_list)\n\nfor ipynb_file in tqdm(notebooks_list[:NUM_NOTEBOOKS]):\n ipynb_save_path = TMP_DIR / ipynb_file\n urlretrieve(f'https://github-notebooks-update1.s3-eu-west-1.amazonaws.com/{ipynb_file}', ipynb_save_path)\nThanks! Could official help confirm if we could use this data?
", "rawMarkdown": "Thanks! Could official help confirm if we could use this data?", "votes": 3 }, { "id": 1880839, "postDate": "2022-08-02T04:49:48.513Z", "content": "Thanks! Could official help confirm if we could use this data?
", "rawMarkdown": "Thanks! Could official help confirm if we could use this data?", "votes": 1 }, { "id": 1855864, "postDate": "2022-07-15T01:46:52.790Z", "content": "Very helpful post
", "rawMarkdown": "Very helpful post\n", "votes": 1 }, { "id": 1880268, "postDate": "2022-08-01T15:15:56.720Z", "content": "Hey there - Kaggle cannot provide legal guidance or interpretation on the licensing of external data.
\nCompetition rules state:
\n\n", "rawMarkdown": "Hey there - Kaggle cannot provide legal guidance or interpretation on the licensing of external data. \n\nCompetition rules state:\n>C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).", "votes": 2, "replies": [ { "id": 1880503, "postDate": "2022-08-01T19:06:56.150Z", "content": "C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).
\n
\"Here’s how we used the hundreds of thousands of publicly accessible repos on GitHub to learn more about the current state of data science.\"
\n@addisonhoward These notebooks are publicly accessible, but we could not verify all the notebooks's licence, so not possible to use?
\nIt is very confusing for this external data could help improve your score, I'm fine if the official said we could not use ths dataset, with offical training data still an excellent competition(enough training data already, and allowing using external dataset will make training process hard to reproduce), but if someone used this and others not it is unfair, and we all not 100% sure if this dataset is allowed or not.
thanks bro
", "rawMarkdown": "thanks bro", "votes": 1 } ], "comments": [ { "id": 1879229, "author_name": "gezi", "author_url": "", "post_date": "2022-07-31T23:49:24.327000", "content": "Thanks! Could official help confirm if we could use this data?
", "votes": 3, "replies": [] }, { "id": 1880839, "author_name": "Minh Thanh", "author_url": "", "post_date": "2022-08-02T04:49:48.513000", "content": "Thanks! Could official help confirm if we could use this data?
", "votes": 1, "replies": [] }, { "id": 1855864, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-15T01:46:52.790000", "content": "Very helpful post
", "votes": 1, "replies": [] }, { "id": 1880268, "author_name": "Addison Howard", "author_url": "", "post_date": "2022-08-01T15:15:56.720000", "content": "Hey there - Kaggle cannot provide legal guidance or interpretation on the licensing of external data.
\nCompetition rules state:
\n\n", "votes": 2, "replies": [ { "id": 1880503, "author_name": "gezi", "author_url": "", "post_date": "2022-08-01T19:06:56.150000", "content": "C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).
\n
\"Here’s how we used the hundreds of thousands of publicly accessible repos on GitHub to learn more about the current state of data science.\"
\n@addisonhoward These notebooks are publicly accessible, but we could not verify all the notebooks's licence, so not possible to use?
\nIt is very confusing for this external data could help improve your score, I'm fine if the official said we could not use ths dataset, with offical training data still an excellent competition(enough training data already, and allowing using external dataset will make training process hard to reproduce), but if someone used this and others not it is unfair, and we all not 100% sure if this dataset is allowed or not.
thanks bro
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1853431": "Blog Post: [We Downloaded 10,000,000 Jupyter Notebooks From Github](https://blog.jetbrains.com/datalore/2020/12/17/we-downloaded-10-000-000-jupyter-notebooks-from-github-this-is-what-we-learned/)\n\nThere are over millions of Jupyter notebooks open sourced on GitHub. You can use the code snippet to download 10 million Jupyter notebooks from an AWS bucket thanks to JetBrains. It takes around 2400 hours on CPU to download all the notebooks. This could be used as an external dataset for training or retrieval. You could use this for training your models in case you own a data center or have a couple thousand free GPUs lying around.\n\n```\nfrom urllib.request import urlretrieve\nimport random\nimport json\n\ndef read_json(filepath): \n return json.load(open(filepath))\n\ndef read_ipynb(file): \n with open(file, mode= 'r', encoding= 'utf-8') as f:\n return json.loads(f.read())\n\n!wget https://github-notebooks-samples.s3-eu-west-1.amazonaws.com/ntbs_list.json\nnotebooks_list = read_json('ntbs_list.json')\nrandom.shuffle(notebooks_list)\n\nfor ipynb_file in tqdm(notebooks_list[:NUM_NOTEBOOKS]):\n ipynb_save_path = TMP_DIR / ipynb_file\n urlretrieve(f'https://github-notebooks-update1.s3-eu-west-1.amazonaws.com/{ipynb_file}', ipynb_save_path)\n```", "1879229": "Thanks! Could official help confirm if we could use this data?", "1880839": "Thanks! Could official help confirm if we could use this data?", "1855864": "Very helpful post\n", "1880268": "Hey there - Kaggle cannot provide legal guidance or interpretation on the licensing of external data. \n\nCompetition rules state:\n>C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).", "1880501": "", "1855556": "", "1863698": "thanks bro" } }