{ "id": 438137, "title": "Help to work with so much data?", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/438137", "author_name": "", "post_date": "2023-09-09T17:02:05.824519700Z", "votes": 2, "comment_count": 5, "views": 0, "content": "

Hi kagglers,

\n

I'm having trouble to analyze and to work with so big datasets, my computer is not very powerful, so I'm working with kaggle notebooks.

\n

I would like to know the best approach for this kind of issue, for example:

\n
    \n

  1. Use some specific library or funtion to be able to load all the data. (I don't have much hope on this btw)

    2. \n

  2. Load the data in batches for both EDA and training.

    3. \n

  3. Check this URL on how to do it.

    4. \n

  4. Any other approach.

    5. \n
\n

I've seen some discussion about this but no clear solution for me.

\n

So I would appreciate any advice or tip, because I don't usually work with so big datasets and I'm kind of lost.

", "messages": [ { "id": "2430914", "postDate": "09/09/2023 17:02:05", "content": "

Hi kagglers,

\n

I'm having trouble to analyze and to work with so big datasets, my computer is not very powerful, so I'm working with kaggle notebooks.

\n

I would like to know the best approach for this kind of issue, for example:

\n
    \n

  1. Use some specific library or funtion to be able to load all the data. (I don't have much hope on this btw)

    2. \n

  2. Load the data in batches for both EDA and training.

    3. \n

  3. Check this URL on how to do it.

    4. \n

  4. Any other approach.

    5. \n
\n

I've seen some discussion about this but no clear solution for me.

\n

So I would appreciate any advice or tip, because I don't usually work with so big datasets and I'm kind of lost.

", "rawMarkdown": "Hi kagglers,\n\nI'm having trouble to analyze and to work with so big datasets, my computer is not very powerful, so I'm working with kaggle notebooks.\n\nI would like to know the best approach for this kind of issue, for example:\n\n1. Use some specific library or funtion to be able to load all the data. (I don't have much hope on this btw)\n\n2. Load the data in batches for both EDA and training.\n\n3. Check this URL on how to do it.\n\n4. Any other approach.\n\nI've seen some discussion about this but no clear solution for me.\n\nSo I would appreciate any advice or tip, because I don't usually work with so big datasets and I'm kind of lost.", "votes": null }, { "id": "2430936", "postDate": "09/09/2023 17:13:03", "content": "

I am new with processing large datasets as well. I just had quick thoughts like chunking the data by their series ID (e.g. with tfrecords, npz, etc.), and then load it lazily (by using tf.Dataset or torch DataLoader) series by series when we fit our model. I am curious about other approaches.

", "rawMarkdown": "I am new with processing large datasets as well. I just had quick thoughts like chunking the data by their series ID (e.g. with tfrecords, npz, etc.), and then load it lazily (by using tf.Dataset or torch DataLoader) series by series when we fit our model. I am curious about other approaches.", "votes": null }, { "id": "2442820", "postDate": "09/17/2023 10:27:14", "content": "

@tomasluna , @rimbax : Tomas/Rimba, you may want to review the disccussion thread per Best Practices of Tackling BigData-Scale Tasks to review some of the suggestions.

", "rawMarkdown": "tomasluna , @rimbax : Tomas/Rimba, you may want to review the disccussion thread per [Best Practices of Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047) to review some of the suggestions.", "votes": null }, { "id": "2443382", "postDate": "09/17/2023 17:25:00", "content": "

Thank you @rimbax for your answer!

\n

My idea for now is doing something similar to that, I'll take a look to those functions, will be interesting for sure!

", "rawMarkdown": "Thank you @rimbax for your answer!\n\nMy idea for now is doing something similar to that, I'll take a look to those functions, will be interesting for sure!", "votes": null }, { "id": "2443385", "postDate": "09/17/2023 17:27:49", "content": "

Very useful information! Thank you @gvyshnya !

", "rawMarkdown": "Very useful information! Thank you @gvyshnya !", "votes": null }, { "id": "2443392", "postDate": "09/17/2023 17:30:13", "content": "

Sure, will check your extensive discussion on there.

", "rawMarkdown": "Sure, will check your extensive discussion on there.", "votes": null } ], "comments": [ { "id": 2430936, "author_name": "rimbax", "author_url": "", "post_date": "09/09/2023 17:13:03", "content": "

I am new with processing large datasets as well. I just had quick thoughts like chunking the data by their series ID (e.g. with tfrecords, npz, etc.), and then load it lazily (by using tf.Dataset or torch DataLoader) series by series when we fit our model. I am curious about other approaches.

", "votes": null, "replies": [ { "id": 2443382, "author_name": "tomasluna", "author_url": "", "post_date": "09/17/2023 17:25:00", "content": "

Thank you @rimbax for your answer!

\n

My idea for now is doing something similar to that, I'll take a look to those functions, will be interesting for sure!

", "votes": null, "replies": [] } ] }, { "id": 2442820, "author_name": "gvyshnya", "author_url": "", "post_date": "09/17/2023 10:27:14", "content": "

@tomasluna , @rimbax : Tomas/Rimba, you may want to review the disccussion thread per Best Practices of Tackling BigData-Scale Tasks to review some of the suggestions.

", "votes": null, "replies": [ { "id": 2443385, "author_name": "tomasluna", "author_url": "", "post_date": "09/17/2023 17:27:49", "content": "

Very useful information! Thank you @gvyshnya !

", "votes": null, "replies": [] }, { "id": 2443392, "author_name": "rimbax", "author_url": "", "post_date": "09/17/2023 17:30:13", "content": "

Sure, will check your extensive discussion on there.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2430914": "Hi kagglers,\n\nI'm having trouble to analyze and to work with so big datasets, my computer is not very powerful, so I'm working with kaggle notebooks.\n\nI would like to know the best approach for this kind of issue, for example:\n\n1. Use some specific library or funtion to be able to load all the data. (I don't have much hope on this btw)\n\n2. Load the data in batches for both EDA and training.\n\n3. Check this URL on how to do it.\n\n4. Any other approach.\n\nI've seen some discussion about this but no clear solution for me.\n\nSo I would appreciate any advice or tip, because I don't usually work with so big datasets and I'm kind of lost.", "2430936": "I am new with processing large datasets as well. I just had quick thoughts like chunking the data by their series ID (e.g. with tfrecords, npz, etc.), and then load it lazily (by using tf.Dataset or torch DataLoader) series by series when we fit our model. I am curious about other approaches.", "2442820": "tomasluna , @rimbax : Tomas/Rimba, you may want to review the disccussion thread per [Best Practices of Tackling BigData-Scale Tasks](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/441047) to review some of the suggestions.", "2443382": "Thank you @rimbax for your answer!\n\nMy idea for now is doing something similar to that, I'll take a look to those functions, will be interesting for sure!", "2443385": "Very useful information! Thank you @gvyshnya !", "2443392": "Sure, will check your extensive discussion on there." }, "source": "meta" }