{ "id": 438558, "title": "Working on competition code within Kaggle", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/438558", "author_name": "Nader E Afshar", "post_date": "2023-09-11T17:41:13.080000", "votes": 0, "comment_count": 2, "views": 0, "content": "

It seems data produced as a result of a run, is only kept for a short period of time and deleted after any inactivity, so one would have to re-rum the code to regenerate working datasets; time consuming in this case.

\n

Is there a way to persist data generated for a while, while we work through the development phase? or do we need to complete the work outside Kaggle and then import it in for submission?

\n

Furthermore there does not seem to be sufficient resources always available. I see a number of code examples where people have read both datasets (parquet & csv) into memory, but I run out of memory trying the same code.

\n

I would appreciate some guidance.

", "messages": [ { "id": 2550204, "postDate": "2023-12-05T21:52:23.357Z", "content": "

@nadereafshar My practice for such scenarios is;

\n
    \n
  1. I set the persistence to file only otherwise it takes long to reboot and fill the HDD with variables data. with that I save to files/models after every lengthy process like running and apply or changing datatypes etc., so I can reload later where I left
    \n\"\"
    2. \n
\n

Also, I save modified data in pickle format because it retains the columns datatypes, so saving the time again.

", "rawMarkdown": "@nadereafshar My practice for such scenarios is;\n1. I set the persistence to file only otherwise it takes long to reboot and fill the HDD with variables data. with that I save to files/models after every lengthy process like running and apply or changing datatypes etc., so I can reload later where I left\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16944358%2Fa68ccff1fb0a8df5c796015542d9fd20%2Fkaggle%20discussion.png?generation=1701812964851800&alt=media)\n\nAlso, I save modified data in pickle format because it retains the columns datatypes, so saving the time again." }, { "id": 2433649, "postDate": "2023-09-11T17:41:13.080Z", "content": "

It seems data produced as a result of a run, is only kept for a short period of time and deleted after any inactivity, so one would have to re-rum the code to regenerate working datasets; time consuming in this case.

\n

Is there a way to persist data generated for a while, while we work through the development phase? or do we need to complete the work outside Kaggle and then import it in for submission?

\n

Furthermore there does not seem to be sufficient resources always available. I see a number of code examples where people have read both datasets (parquet & csv) into memory, but I run out of memory trying the same code.

\n

I would appreciate some guidance.

", "rawMarkdown": "It seems data produced as a result of a run, is only kept for a short period of time and deleted after any inactivity, so one would have to re-rum the code to regenerate working datasets; time consuming in this case.\n\nIs there a way to persist data generated for a while, while we work through the development phase? or do we need to complete the work outside Kaggle and then import it in for submission?\n\nFurthermore there does not seem to be sufficient resources always available. I see a number of code examples where people have read both datasets (parquet & csv) into memory, but I run out of memory trying the same code. \n\nI would appreciate some guidance.\n\n" }, { "id": 2435642, "postDate": "2023-09-13T05:47:09.303Z", "content": "

Hey @nadereafshar

\n

I am currently doing exploratory data analysis and building some heuristic models. The approach I have taken is to do these analysis and modelling on a very small subset of the dataset so that I can iterate quickly and once I have something that I want to try out on the larger dataset, there are two options. Both these options take quite a long time for me I let that run and check after an hour or two.

\n
    \n
  1. To run on the training dataset, one can use the Save Version button the top right.
    2. \n
  2. To run on the test dataset, one can Submit to the competition.
    3. \n
\n

I have some notes on how I built the smaller dataset for analysis here:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437667

\n

The dataset is at the link below which you can attach to your notebook on the same lines of the competition dataset:
\nhttps://www.kaggle.com/datasets/bkowshik/detect-sleep-states

", "rawMarkdown": "Hey @nadereafshar \n\nI am currently doing exploratory data analysis and building some heuristic models. The approach I have taken is to do these analysis and modelling on a very small subset of the dataset so that I can iterate quickly and once I have something that I want to try out on the larger dataset, there are two options. Both these options take quite a long time for me I let that run and check after an hour or two.\n\n1. To run on the training dataset, one can use the `Save Version` button the top right.\n2. To run on the test dataset, one can `Submit` to the competition.\n\nI have some notes on how I built the smaller dataset for analysis here:\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437667\n\nThe dataset is at the link below which you can attach to your notebook on the same lines of the competition dataset:\nhttps://www.kaggle.com/datasets/bkowshik/detect-sleep-states", "isDeleted": true } ], "comments": [ { "id": 2550204, "author_name": "Najeeb Z", "author_url": "", "post_date": "2023-12-05T21:52:23.357000", "content": "

@nadereafshar My practice for such scenarios is;

\n
    \n
  1. I set the persistence to file only otherwise it takes long to reboot and fill the HDD with variables data. with that I save to files/models after every lengthy process like running and apply or changing datatypes etc., so I can reload later where I left
    \n\"\"
    2. \n
\n

Also, I save modified data in pickle format because it retains the columns datatypes, so saving the time again.

", "votes": 0, "replies": [] }, { "id": 2435642, "author_name": "Bhargav Kowshik", "author_url": "", "post_date": "2023-09-13T05:47:09.303000", "content": "

Hey @nadereafshar

\n

I am currently doing exploratory data analysis and building some heuristic models. The approach I have taken is to do these analysis and modelling on a very small subset of the dataset so that I can iterate quickly and once I have something that I want to try out on the larger dataset, there are two options. Both these options take quite a long time for me I let that run and check after an hour or two.

\n
    \n
  1. To run on the training dataset, one can use the Save Version button the top right.
    2. \n
  2. To run on the test dataset, one can Submit to the competition.
    3. \n
\n

I have some notes on how I built the smaller dataset for analysis here:
\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437667

\n

The dataset is at the link below which you can attach to your notebook on the same lines of the competition dataset:
\nhttps://www.kaggle.com/datasets/bkowshik/detect-sleep-states

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2550204": "@nadereafshar My practice for such scenarios is;\n1. I set the persistence to file only otherwise it takes long to reboot and fill the HDD with variables data. with that I save to files/models after every lengthy process like running and apply or changing datatypes etc., so I can reload later where I left\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16944358%2Fa68ccff1fb0a8df5c796015542d9fd20%2Fkaggle%20discussion.png?generation=1701812964851800&alt=media)\n\nAlso, I save modified data in pickle format because it retains the columns datatypes, so saving the time again.", "2433649": "It seems data produced as a result of a run, is only kept for a short period of time and deleted after any inactivity, so one would have to re-rum the code to regenerate working datasets; time consuming in this case.\n\nIs there a way to persist data generated for a while, while we work through the development phase? or do we need to complete the work outside Kaggle and then import it in for submission?\n\nFurthermore there does not seem to be sufficient resources always available. I see a number of code examples where people have read both datasets (parquet & csv) into memory, but I run out of memory trying the same code. \n\nI would appreciate some guidance.\n\n", "2435642": "Hey @nadereafshar \n\nI am currently doing exploratory data analysis and building some heuristic models. The approach I have taken is to do these analysis and modelling on a very small subset of the dataset so that I can iterate quickly and once I have something that I want to try out on the larger dataset, there are two options. Both these options take quite a long time for me I let that run and check after an hour or two.\n\n1. To run on the training dataset, one can use the `Save Version` button the top right.\n2. To run on the test dataset, one can `Submit` to the competition.\n\nI have some notes on how I built the smaller dataset for analysis here:\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437667\n\nThe dataset is at the link below which you can attach to your notebook on the same lines of the competition dataset:\nhttps://www.kaggle.com/datasets/bkowshik/detect-sleep-states" } }