Hello, I know the latest data files train_v2 and test_v2 are so big which makes kernel to crash all the time. I have downloaded the previous version of test data and started using the same. I am not sure how can I share my data files with you here in Kaggle. I am new to Kaggle. if you know, please tell me I will be happy to help.
", "rawMarkdown": "Hello, I know the latest data files train_v2 and test_v2 are so big which makes kernel to crash all the time. I have downloaded the previous version of test data and started using the same. I am not sure how can I share my data files with you here in Kaggle. I am new to Kaggle. if you know, please tell me I will be happy to help.", "votes": null }, { "id": "420329", "postDate": "11/13/2018 13:10:13", "content": "Hi deepali, you can upload the files as a public dataset, I think this is the only way you can share your files.
", "rawMarkdown": "Hi deepali, you can upload the files as a public dataset, I think this is the only way you can share your files.", "votes": null }, { "id": "420803", "postDate": "11/14/2018 06:31:19", "content": "I uploaded files as public data, can you try to access it...
", "rawMarkdown": "I uploaded files as public data, can you try to access it...", "votes": null }, { "id": "421427", "postDate": "11/15/2018 01:51:09", "content": "hey, deepali. can we use previous version of data to train and test for competition ?
", "rawMarkdown": "hey, deepali. can we use previous version of data to train and test for competition ?", "votes": null }, { "id": "421524", "postDate": "11/15/2018 05:14:37", "content": "I think 'Yes'. Because to upload your own data option is enabled. However I am new to Kaggle and to Data Science as well. I don't want to give up what I have already started. So, I will complete it with V1 data.
", "rawMarkdown": "I think 'Yes'. Because to upload your own data option is enabled. However I am new to Kaggle and to Data Science as well. I don't want to give up what I have already started. So, I will complete it with V1 data.", "votes": null }, { "id": "422242", "postDate": "11/16/2018 00:25:55", "content": "I give up the hits column in V2 data
", "rawMarkdown": "I give up the hits column in V2 data", "votes": null }, { "id": "422246", "postDate": "11/16/2018 00:43:18", "content": "If you can download the data and store it on your computer, then use command line tools to split the data. I'm not sure of the windows CLI, but on Linux and Mac/Unix, the command is literally \"split\". You can also pipe the data into some programs for incremental reading, parsing and analysis. This is more like what you will have if you work at any of the top cloud service companies. Keep in mind that you do not need to use all columns all the time. Pick a few and/or take a sample of data. I often create learning curves which shows how the volume of data effects learning performance. You will be surprised what you can do either with less data or by stacked generalizations (models built on subsets of data independently and later aggregated). Check out some old papers from Leo Breiman for more info.
\n\nAlso, once you read it in, save it as H5, HDF5 or Parquet to faster IO when you reload the data.
", "rawMarkdown": "If you can download the data and store it on your computer, then use command line tools to split the data. I'm not sure of the windows CLI, but on Linux and Mac/Unix, the command is literally \"split\". You can also pipe the data into some programs for incremental reading, parsing and analysis. This is more like what you will have if you work at any of the top cloud service companies. Keep in mind that you do not need to use all columns all the time. Pick a few and/or take a sample of data. I often create learning curves which shows how the volume of data effects learning performance. You will be surprised what you can do either with less data or by stacked generalizations (models built on subsets of data independently and later aggregated). Check out some old papers from Leo Breiman for more info.\n\nAlso, once you read it in, save it as H5, HDF5 or Parquet to faster IO when you reload the data.", "votes": null }, { "id": "426556", "postDate": "11/23/2018 12:47:47", "content": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd
", "rawMarkdown": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd", "votes": null }, { "id": "426562", "postDate": "11/23/2018 12:56:29", "content": "thanks for your advices
", "rawMarkdown": "thanks for your advices", "votes": null }, { "id": "426564", "postDate": "11/23/2018 12:56:42", "content": "thank you
", "rawMarkdown": "thank you", "votes": null }, { "id": "426598", "postDate": "11/23/2018 14:09:24", "content": "this recommendation is gold. Very good advice!
", "rawMarkdown": "this recommendation is gold. Very good advice!", "votes": null } ], "comments": [ { "id": 420226, "author_name": "deepaliagni", "author_url": "", "post_date": "11/13/2018 10:06:00", "content": "Hello, I know the latest data files train_v2 and test_v2 are so big which makes kernel to crash all the time. I have downloaded the previous version of test data and started using the same. I am not sure how can I share my data files with you here in Kaggle. I am new to Kaggle. if you know, please tell me I will be happy to help.
", "votes": null, "replies": [ { "id": 420329, "author_name": "joaopmpeinado", "author_url": "", "post_date": "11/13/2018 13:10:13", "content": "Hi deepali, you can upload the files as a public dataset, I think this is the only way you can share your files.
", "votes": null, "replies": [] }, { "id": 420803, "author_name": "deepaliagni", "author_url": "", "post_date": "11/14/2018 06:31:19", "content": "I uploaded files as public data, can you try to access it...
", "votes": null, "replies": [] }, { "id": 421427, "author_name": "yanmaitong", "author_url": "", "post_date": "11/15/2018 01:51:09", "content": "hey, deepali. can we use previous version of data to train and test for competition ?
", "votes": null, "replies": [] }, { "id": 421524, "author_name": "deepaliagni", "author_url": "", "post_date": "11/15/2018 05:14:37", "content": "I think 'Yes'. Because to upload your own data option is enabled. However I am new to Kaggle and to Data Science as well. I don't want to give up what I have already started. So, I will complete it with V1 data.
", "votes": null, "replies": [] }, { "id": 422242, "author_name": "kangzhang", "author_url": "", "post_date": "11/16/2018 00:25:55", "content": "I give up the hits column in V2 data
", "votes": null, "replies": [] } ] }, { "id": 422246, "author_name": "ldmtwo", "author_url": "", "post_date": "11/16/2018 00:43:18", "content": "If you can download the data and store it on your computer, then use command line tools to split the data. I'm not sure of the windows CLI, but on Linux and Mac/Unix, the command is literally \"split\". You can also pipe the data into some programs for incremental reading, parsing and analysis. This is more like what you will have if you work at any of the top cloud service companies. Keep in mind that you do not need to use all columns all the time. Pick a few and/or take a sample of data. I often create learning curves which shows how the volume of data effects learning performance. You will be surprised what you can do either with less data or by stacked generalizations (models built on subsets of data independently and later aggregated). Check out some old papers from Leo Breiman for more info.
\n\nAlso, once you read it in, save it as H5, HDF5 or Parquet to faster IO when you reload the data.
", "votes": null, "replies": [ { "id": 426562, "author_name": "kangzhang", "author_url": "", "post_date": "11/23/2018 12:56:29", "content": "thanks for your advices
", "votes": null, "replies": [] }, { "id": 426598, "author_name": "leafar", "author_url": "", "post_date": "11/23/2018 14:09:24", "content": "this recommendation is gold. Very good advice!
", "votes": null, "replies": [] } ] }, { "id": 426556, "author_name": "mlisovyi", "author_url": "", "post_date": "11/23/2018 12:47:47", "content": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd
", "votes": null, "replies": [ { "id": 426564, "author_name": "kangzhang", "author_url": "", "post_date": "11/23/2018 12:56:42", "content": "thank you
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "420050": "", "420226": "Hello, I know the latest data files train_v2 and test_v2 are so big which makes kernel to crash all the time. I have downloaded the previous version of test data and started using the same. I am not sure how can I share my data files with you here in Kaggle. I am new to Kaggle. if you know, please tell me I will be happy to help.", "420329": "Hi deepali, you can upload the files as a public dataset, I think this is the only way you can share your files.", "420803": "I uploaded files as public data, can you try to access it...", "421427": "hey, deepali. can we use previous version of data to train and test for competition ?", "421524": "I think 'Yes'. Because to upload your own data option is enabled. However I am new to Kaggle and to Data Science as well. I don't want to give up what I have already started. So, I will complete it with V1 data.", "422242": "I give up the hits column in V2 data", "422246": "If you can download the data and store it on your computer, then use command line tools to split the data. I'm not sure of the windows CLI, but on Linux and Mac/Unix, the command is literally \"split\". You can also pipe the data into some programs for incremental reading, parsing and analysis. This is more like what you will have if you work at any of the top cloud service companies. Keep in mind that you do not need to use all columns all the time. Pick a few and/or take a sample of data. I often create learning curves which shows how the volume of data effects learning performance. You will be surprised what you can do either with less data or by stacked generalizations (models built on subsets of data independently and later aggregated). Check out some old papers from Leo Breiman for more info.\n\nAlso, once you read it in, save it as H5, HDF5 or Parquet to faster IO when you reload the data.", "426556": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd", "426562": "thanks for your advices", "426564": "thank you", "426598": "this recommendation is gold. Very good advice!" }, "source": "meta" }