I see that my features are often used by people and they generate them each time, so I have decided to calculate these features and share them in a dataset:\nhttps://www.kaggle.com/artgor/molecules-generated-features\nThe data is stored in hdf5 files to save space and for faster loading. You can load them like this:
\ntrain = pd.read_hdf(f'{path}/train.h5', 'table')\n\n\ntest = pd.read_hdf(f'{path}/test.h5', 'table')\n
I see that my features are often used by people and they generate them each time, so I have decided to calculate these features and share them in a dataset:\nhttps://www.kaggle.com/artgor/molecules-generated-features\nThe data is stored in hdf5 files to save space and for faster loading. You can load them like this:
\ntrain = pd.read_hdf(f'{path}/train.h5', 'table')\n\n\ntest = pd.read_hdf(f'{path}/test.h5', 'table')\n
when i try to load them, i get memory overflow. any tips? thanks for sharing your work btw!
", "rawMarkdown": "when i try to load them, i get memory overflow. any tips? thanks for sharing your work btw!", "votes": 1, "replies": [ { "id": 554966, "postDate": "2019-06-18T08:45:27.523Z", "content": "I also only managed to read in the test file. RAM usage 11.7/16 Gb after reading in only the test file.
", "rawMarkdown": "I also only managed to read in the test file. RAM usage 11.7/16 Gb after reading in only the test file." }, { "id": 554981, "postDate": "2019-06-18T09:20:01.650Z", "content": "Yes this files are too big to work with them in the kernels directly. There are some options:\n- open them on a hardware with a lot of RAM;\n- read lines belonging to one type at a time and train a model on them;\n- use subsets of columns;
", "rawMarkdown": "Yes this files are too big to work with them in the kernels directly. There are some options:\n- open them on a hardware with a lot of RAM;\n- read lines belonging to one type at a time and train a model on them;\n- use subsets of columns;", "votes": 2 } ] }, { "id": 554645, "postDate": "2019-06-17T20:13:58.390Z", "content": "I am new to kernels but I am getting fairly good at copying others' work. Can I read the files without uploading them into my notebook?\ntrain = pd.read_hdf(f'{https://www.kaggle.com/artgor/molecules_generated_features#}/train.h5', 'table')\nor without #:\ntrain = pd.read_hdf(f'{https://www.kaggle.com/artgor/molecules_generated_features}/train.h5', 'table')\nboth give FileNotFoundError
**Update:\nAttached files to the notebook. Still can't read them.\nfile_folder = '../input'\nfile_folder2 = '../input/molecules_generated_features'\ntest = pd.read_hdf(f'{file_folder}/test.h5', 'table')\ntest = pd.read_hdf(f'{file_folder2}/test.h5', 'table')\nerror example: FileNotFoundError: File ../input/test.h5 does not exist. I see .h5 files in /input/molecules_generated_features/ subfolder
**Update: I can't read the original test.csv either. It has something to do with the way Andrew structures his kernels I am guessing since I have not seen this problem with other people kernels.\ntest = pd.read_csv(f'{file_folder}/test.csv')\nError: FileNotFoundError: File b'../input/test.csv' does not exist
When you have only one dataset in a kernel, its files are in ../input/ directory directly, but then you have 2 or more datasets, then in ../input/ you'll see separate folders for each dataset.
So run os.listdir(\"../input/\") command and find out the way to the files.
Thank you. This helped me to troubleshoot the problem. Reading the attached file OK now. The problem was that the underscores( '_' ) that I see in the workspace on the right in the browser is actually a dash ('-').\nCorrect:\nos.listdir(\"../input/molecules-generated-features/\")
Wrong:\nos.listdir(\"../input/molecules_generated_features/\")
how can i convert .Dat file to convert .csv so that i will be start working
", "rawMarkdown": "how can i convert .Dat file to convert .csv so that i will be start working " }, { "id": 554406, "postDate": "2019-06-17T11:25:04.863Z", "content": "Sweet.
", "rawMarkdown": "Sweet." }, { "id": 558673, "postDate": "2019-06-22T18:20:00.087Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 554858, "author_name": "fulowa", "author_url": "", "post_date": "2019-06-18T05:39:16.607000", "content": "when i try to load them, i get memory overflow. any tips? thanks for sharing your work btw!
", "votes": 1, "replies": [ { "id": 554966, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2019-06-18T08:45:27.523000", "content": "I also only managed to read in the test file. RAM usage 11.7/16 Gb after reading in only the test file.
", "votes": 0, "replies": [] }, { "id": 554981, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2019-06-18T09:20:01.650000", "content": "Yes this files are too big to work with them in the kernels directly. There are some options:\n- open them on a hardware with a lot of RAM;\n- read lines belonging to one type at a time and train a model on them;\n- use subsets of columns;
", "votes": 2, "replies": [] } ] }, { "id": 554645, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2019-06-17T20:13:58.390000", "content": "I am new to kernels but I am getting fairly good at copying others' work. Can I read the files without uploading them into my notebook?\ntrain = pd.read_hdf(f'{https://www.kaggle.com/artgor/molecules_generated_features#}/train.h5', 'table')\nor without #:\ntrain = pd.read_hdf(f'{https://www.kaggle.com/artgor/molecules_generated_features}/train.h5', 'table')\nboth give FileNotFoundError
**Update:\nAttached files to the notebook. Still can't read them.\nfile_folder = '../input'\nfile_folder2 = '../input/molecules_generated_features'\ntest = pd.read_hdf(f'{file_folder}/test.h5', 'table')\ntest = pd.read_hdf(f'{file_folder2}/test.h5', 'table')\nerror example: FileNotFoundError: File ../input/test.h5 does not exist. I see .h5 files in /input/molecules_generated_features/ subfolder
**Update: I can't read the original test.csv either. It has something to do with the way Andrew structures his kernels I am guessing since I have not seen this problem with other people kernels.\ntest = pd.read_csv(f'{file_folder}/test.csv')\nError: FileNotFoundError: File b'../input/test.csv' does not exist
When you have only one dataset in a kernel, its files are in ../input/ directory directly, but then you have 2 or more datasets, then in ../input/ you'll see separate folders for each dataset.
So run os.listdir(\"../input/\") command and find out the way to the files.
Thank you. This helped me to troubleshoot the problem. Reading the attached file OK now. The problem was that the underscores( '_' ) that I see in the workspace on the right in the browser is actually a dash ('-').\nCorrect:\nos.listdir(\"../input/molecules-generated-features/\")
Wrong:\nos.listdir(\"../input/molecules_generated_features/\")
how can i convert .Dat file to convert .csv so that i will be start working
", "votes": 0, "replies": [] }, { "id": 554406, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-06-17T11:25:04.863000", "content": "Sweet.
", "votes": 0, "replies": [] }, { "id": 558673, "author_name": "", "author_url": "", "post_date": "2019-06-22T18:20:00.087000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "554396": "I see that my features are often used by people and they generate them each time, so I have decided to calculate these features and share them in a dataset:\nhttps://www.kaggle.com/artgor/molecules-generated-features\nThe data is stored in `hdf5` files to save space and for faster loading. You can load them like this:\n\n```\ntrain = pd.read_hdf(f'{path}/train.h5', 'table')\n```\n```\ntest = pd.read_hdf(f'{path}/test.h5', 'table')\n```", "554858": "when i try to load them, i get memory overflow. any tips? thanks for sharing your work btw!", "554645": "I am new to kernels but I am getting fairly good at copying others' work. Can I read the files without uploading them into my notebook?\n`train = pd.read_hdf(f'{https://www.kaggle.com/artgor/molecules_generated_features#}/train.h5', 'table')`\nor without #:\n`train = pd.read_hdf(f'{https://www.kaggle.com/artgor/molecules_generated_features}/train.h5', 'table')`\nboth give FileNotFoundError \n\n**Update:\nAttached files to the notebook. Still can't read them.\n`file_folder = '../input'\nfile_folder2 = '../input/molecules_generated_features'\ntest = pd.read_hdf(f'{file_folder}/test.h5', 'table')\ntest = pd.read_hdf(f'{file_folder2}/test.h5', 'table')`\nerror example: FileNotFoundError: File ../input/test.h5 does not exist. I see .h5 files in /input/molecules_generated_features/ subfolder\n\n****Update: I can't read the original test.csv either. It has something to do with the way Andrew structures his kernels I am guessing since I have not seen this problem with other people kernels.\n`test = pd.read_csv(f'{file_folder}/test.csv')`\nError: FileNotFoundError: File b'../input/test.csv' does not exist", "555094": "how can i convert .Dat file to convert .csv so that i will be start working ", "554406": "Sweet.", "558673": "" } }