All, \nIs it possible in kaggle to save the intermediate output in a csv file and on next trial, read from this file, instead of repeating the analysis from Step 1 ? I was able to save this intermediate file, by committing the notebook and simple df.to_csv cmd. I can see the saved file, in output section. However unable to read from this. The path '../output' does not work! Anybody has done this pls help.
", "messages": [ { "id": "440189", "postDate": "12/17/2018 07:36:01", "content": "All, \nIs it possible in kaggle to save the intermediate output in a csv file and on next trial, read from this file, instead of repeating the analysis from Step 1 ? I was able to save this intermediate file, by committing the notebook and simple df.to_csv cmd. I can see the saved file, in output section. However unable to read from this. The path '../output' does not work! Anybody has done this pls help.
", "rawMarkdown": "All, \nIs it possible in kaggle to save the intermediate output in a csv file and on next trial, read from this file, instead of repeating the analysis from Step 1 ? I was able to save this intermediate file, by committing the notebook and simple df.to_csv cmd. I can see the saved file, in output section. However unable to read from this. The path '../output' does not work! Anybody has done this pls help.", "votes": null }, { "id": "440280", "postDate": "12/17/2018 10:26:23", "content": "You can create new kernel and use output files from previous kernel as data
", "rawMarkdown": "You can create new kernel and use output files from previous kernel as data", "votes": null }, { "id": "441241", "postDate": "12/18/2018 12:59:13", "content": "Not working , getting message - File b'/hist_DecFeb18.csv' does not exist, using cmd pd.read_csv('/'). Any reason why this is coming ?
", "rawMarkdown": "Not working , getting message - File b'/hist_DecFeb18.csv' does not exist, using cmd pd.read_csv('/", "votes": null }, { "id": "441494", "postDate": "12/18/2018 17:54:34", "content": "Anybody has updates on this pls, couldnt find anything on the forums, ie how to read a intermediate file, in the next step of the analysis.
", "rawMarkdown": "Anybody has updates on this pls, couldnt find anything on the forums, ie how to read a intermediate file, in the next step of the analysis.", "votes": null }, { "id": "441725", "postDate": "12/19/2018 00:59:02", "content": "You can download it (from output) and then I think you can insert it as a dataset in your kernel. A nice solution to download directly (without commiting) is described here -->> https://www.kaggle.com/rtatman/download-a-csv-file-from-a-kernel
", "rawMarkdown": "You can download it (from output) and then I think you can insert it as a dataset in your kernel. A nice solution to download directly (without commiting) is described here -->> https://www.kaggle.com/rtatman/download-a-csv-file-from-a-kernel", "votes": null }, { "id": "441733", "postDate": "12/19/2018 01:26:39", "content": "Hi! AnirbanK, have you tried to use pickle or hdf5 file systems, they are great as checkpoints\nhttps://medium.com/@jerilkuriakose/using-hdf5-with-python-6c5242d08773\nhttps://www.datacamp.com/community/tutorials/pickle-python-tutorial
", "rawMarkdown": "Hi! AnirbanK, have you tried to use pickle or hdf5 file systems, they are great as checkpoints\nhttps://medium.com/@jerilkuriakose/using-hdf5-with-python-6c5242d08773\nhttps://www.datacamp.com/community/tutorials/pickle-python-tutorial", "votes": null }, { "id": "441741", "postDate": "12/19/2018 02:02:44", "content": "There is no doubt that my advice is right. Is your file path correct? Kaggle kernels may change folder name of output files when kernel name contains whitespaces or underlines.
", "rawMarkdown": "There is no doubt that my advice is right. Is your file path correct? Kaggle kernels may change folder name of output files when kernel name contains whitespaces or underlines.", "votes": null }, { "id": "441920", "postDate": "12/19/2018 08:53:00", "content": "Yep, I have used pickle file. Its great to store intermediate files in that. Will try for sure. Hd5 i believe is more for model saving. Thanks
", "rawMarkdown": "Yep, I have used pickle file. Its great to store intermediate files in that. Will try for sure. Hd5 i believe is more for model saving. Thanks", "votes": null }, { "id": "441921", "postDate": "12/19/2018 08:54:35", "content": "Well I saved as df.to_csv(), and reading with pd.read_csv() . should i have to give kernel naame for file path ?
", "rawMarkdown": "Well I saved as df.to_csv(", "votes": null }, { "id": "441923", "postDate": "12/19/2018 08:55:31", "content": "Thanks, makes sense. Download and then add as a new data file, will hv to check. Thats ok, if we have to commit .
", "rawMarkdown": "Thanks, makes sense. Download and then add as a new data file, will hv to check. Thats ok, if we have to commit .", "votes": null }, { "id": "442026", "postDate": "12/19/2018 11:39:29", "content": "Yes, it worked now. Created a new kernel from the Output tab, and i can read the files from prev kernel. Great! thanks so much.
", "rawMarkdown": "Yes, it worked now. Created a new kernel from the Output tab, and i can read the files from prev kernel. Great! thanks so much.", "votes": null }, { "id": "443682", "postDate": "12/22/2018 06:15:11", "content": "Tried, didnt work. Code used to save is : \nwith open(TransCt_by_Card_NewMerchMax, \"wb\") as f:\n w = pickle.dump(TransCt_by_Card_NewMerchMax,f)\nTo read:\nTransCt_by_Card_NewMerchMax=pd.read_pickle('../input/elo-cards-basic-eda-2/TransCt_by_Card_NewMerchMax.pkl')
\n\nthere's only the file name in the file after reading, its a string. same code works fine in normal python env. Something wrong, will go with csv files instead
", "rawMarkdown": "Tried, didnt work. Code used to save is : \nwith open(TransCt_by_Card_NewMerchMax, \"wb\") as f:\n w = pickle.dump(TransCt_by_Card_NewMerchMax,f)\nTo read:\nTransCt_by_Card_NewMerchMax=pd.read_pickle('../input/elo-cards-basic-eda-2/TransCt_by_Card_NewMerchMax.pkl')\n\nthere's only the file name in the file after reading, its a string. same code works fine in normal python env. Something wrong, will go with csv files instead", "votes": null } ], "comments": [ { "id": 440280, "author_name": "lumaxy", "author_url": "", "post_date": "12/17/2018 10:26:23", "content": "You can create new kernel and use output files from previous kernel as data
", "votes": null, "replies": [ { "id": 441241, "author_name": "anirbank", "author_url": "", "post_date": "12/18/2018 12:59:13", "content": "Not working , getting message - File b'/hist_DecFeb18.csv' does not exist, using cmd pd.read_csv('/'). Any reason why this is coming ?
", "votes": null, "replies": [] }, { "id": 441741, "author_name": "lumaxy", "author_url": "", "post_date": "12/19/2018 02:02:44", "content": "There is no doubt that my advice is right. Is your file path correct? Kaggle kernels may change folder name of output files when kernel name contains whitespaces or underlines.
", "votes": null, "replies": [] }, { "id": 441921, "author_name": "anirbank", "author_url": "", "post_date": "12/19/2018 08:54:35", "content": "Well I saved as df.to_csv(), and reading with pd.read_csv() . should i have to give kernel naame for file path ?
", "votes": null, "replies": [] }, { "id": 442026, "author_name": "anirbank", "author_url": "", "post_date": "12/19/2018 11:39:29", "content": "Yes, it worked now. Created a new kernel from the Output tab, and i can read the files from prev kernel. Great! thanks so much.
", "votes": null, "replies": [] } ] }, { "id": 441494, "author_name": "anirbank", "author_url": "", "post_date": "12/18/2018 17:54:34", "content": "Anybody has updates on this pls, couldnt find anything on the forums, ie how to read a intermediate file, in the next step of the analysis.
", "votes": null, "replies": [] }, { "id": 441725, "author_name": "soutsios", "author_url": "", "post_date": "12/19/2018 00:59:02", "content": "You can download it (from output) and then I think you can insert it as a dataset in your kernel. A nice solution to download directly (without commiting) is described here -->> https://www.kaggle.com/rtatman/download-a-csv-file-from-a-kernel
", "votes": null, "replies": [ { "id": 441923, "author_name": "anirbank", "author_url": "", "post_date": "12/19/2018 08:55:31", "content": "Thanks, makes sense. Download and then add as a new data file, will hv to check. Thats ok, if we have to commit .
", "votes": null, "replies": [] } ] }, { "id": 441733, "author_name": "cv13j0", "author_url": "", "post_date": "12/19/2018 01:26:39", "content": "Hi! AnirbanK, have you tried to use pickle or hdf5 file systems, they are great as checkpoints\nhttps://medium.com/@jerilkuriakose/using-hdf5-with-python-6c5242d08773\nhttps://www.datacamp.com/community/tutorials/pickle-python-tutorial
", "votes": null, "replies": [ { "id": 441920, "author_name": "anirbank", "author_url": "", "post_date": "12/19/2018 08:53:00", "content": "Yep, I have used pickle file. Its great to store intermediate files in that. Will try for sure. Hd5 i believe is more for model saving. Thanks
", "votes": null, "replies": [] }, { "id": 443682, "author_name": "anirbank", "author_url": "", "post_date": "12/22/2018 06:15:11", "content": "Tried, didnt work. Code used to save is : \nwith open(TransCt_by_Card_NewMerchMax, \"wb\") as f:\n w = pickle.dump(TransCt_by_Card_NewMerchMax,f)\nTo read:\nTransCt_by_Card_NewMerchMax=pd.read_pickle('../input/elo-cards-basic-eda-2/TransCt_by_Card_NewMerchMax.pkl')
\n\nthere's only the file name in the file after reading, its a string. same code works fine in normal python env. Something wrong, will go with csv files instead
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "440189": "All, \nIs it possible in kaggle to save the intermediate output in a csv file and on next trial, read from this file, instead of repeating the analysis from Step 1 ? I was able to save this intermediate file, by committing the notebook and simple df.to_csv cmd. I can see the saved file, in output section. However unable to read from this. The path '../output' does not work! Anybody has done this pls help.", "440280": "You can create new kernel and use output files from previous kernel as data", "441241": "Not working , getting message - File b'/hist_DecFeb18.csv' does not exist, using cmd pd.read_csv('/", "441494": "Anybody has updates on this pls, couldnt find anything on the forums, ie how to read a intermediate file, in the next step of the analysis.", "441725": "You can download it (from output) and then I think you can insert it as a dataset in your kernel. A nice solution to download directly (without commiting) is described here -->> https://www.kaggle.com/rtatman/download-a-csv-file-from-a-kernel", "441733": "Hi! AnirbanK, have you tried to use pickle or hdf5 file systems, they are great as checkpoints\nhttps://medium.com/@jerilkuriakose/using-hdf5-with-python-6c5242d08773\nhttps://www.datacamp.com/community/tutorials/pickle-python-tutorial", "441741": "There is no doubt that my advice is right. Is your file path correct? Kaggle kernels may change folder name of output files when kernel name contains whitespaces or underlines.", "441920": "Yep, I have used pickle file. Its great to store intermediate files in that. Will try for sure. Hd5 i believe is more for model saving. Thanks", "441921": "Well I saved as df.to_csv(", "441923": "Thanks, makes sense. Download and then add as a new data file, will hv to check. Thats ok, if we have to commit .", "442026": "Yes, it worked now. Created a new kernel from the Output tab, and i can read the files from prev kernel. Great! thanks so much.", "443682": "Tried, didnt work. Code used to save is : \nwith open(TransCt_by_Card_NewMerchMax, \"wb\") as f:\n w = pickle.dump(TransCt_by_Card_NewMerchMax,f)\nTo read:\nTransCt_by_Card_NewMerchMax=pd.read_pickle('../input/elo-cards-basic-eda-2/TransCt_by_Card_NewMerchMax.pkl')\n\nthere's only the file name in the file after reading, its a string. same code works fine in normal python env. Something wrong, will go with csv files instead" }, "source": "meta" }