The contest of “Child Mind Institute - Detect Sleep States” imposes us to operate under the significant resource constraints.
\nThe above-mentioned circumstances force us to invent solutions that are
\nEven if you decide to convert your Kaggle notebook into a Google Collab Pro+ one (it is possible but it comes at a price: https://colab.research.google.com/signup), it would not resolve all of the resource constraint-driven problems for your codebase.
\nTherefore, the only viable option is to get your head around the best practices of the effective tackling BigData-scale problems in the online resource-limited kernels provided by Kaggle. Some of the best practices are universal, and others are specific to the type of operations you try to accomplish (like EDA-specific or ML Engineering-specific tricks). We are going to review them in the sections below.
\nPlotting the datasets that are bigger them a RAM memory available to your solution will crash your conventional workflows (like trying to feed a pandas dataframe to a graph lib like plotly or seaborn and then visualize something).
\nTherefore, more elaborate approaches should be used to achieve your goal. One of the popular patterns of doing it is described below
\nThe good case of implementing such a strategy for this contest is provided in Plotting Big Data: Sleeping Series where DuckDB and Plotly were used together to cook the noodles.
\nSQLLite can be also leveraged in such a situation (see here as an example).
\nExercising every general best practice described in the respective section above is essential for the time-efficient ML pipeline built on Kaggle kernels. However, there is one more trick that can help you.
\nYou can
\nOther thing to consider is to unitilize the batch prediction strategy where your model predicts against the subsamples of the test set rather then on the entire test set at once.
\nIf we were not limited to Kaggle kernels in this competition, I would go further to share one more recommendation. It is about taking advantage of lazy evaluation. Lazy evaluation refers to the strategy which delays evaluation of an expression until the value is actually needed. Lazy evaluation is an important concept (used especially in functional programming), and if you want to read more about its different usages in Python you can start here.
\nMany industrial platforms (like Spark/PySpark, Dask, Ray etc.) facilitate it for you out of the box. For example, I stick to Spark/PySpark as a go-to choice to build BigData-scale solutions in the industrial projects.
\nHowever, launching the infrastructure needed for the platforms like Spark/PySpark or Dask requires some resources (RAM, CPU) to be consumed, and it could make it impossible to use on Kaggle kernels with the datasets of a size compared with the data of “Child Mind Institute - Detect Sleep States” contest.
", "messages": [ { "id": 2442819, "postDate": "2023-09-17T10:22:35.420Z", "content": "The contest of “Child Mind Institute - Detect Sleep States” imposes us to operate under the significant resource constraints.
\nThe above-mentioned circumstances force us to invent solutions that are
\nEven if you decide to convert your Kaggle notebook into a Google Collab Pro+ one (it is possible but it comes at a price: https://colab.research.google.com/signup), it would not resolve all of the resource constraint-driven problems for your codebase.
\nTherefore, the only viable option is to get your head around the best practices of the effective tackling BigData-scale problems in the online resource-limited kernels provided by Kaggle. Some of the best practices are universal, and others are specific to the type of operations you try to accomplish (like EDA-specific or ML Engineering-specific tricks). We are going to review them in the sections below.
\nPlotting the datasets that are bigger them a RAM memory available to your solution will crash your conventional workflows (like trying to feed a pandas dataframe to a graph lib like plotly or seaborn and then visualize something).
\nTherefore, more elaborate approaches should be used to achieve your goal. One of the popular patterns of doing it is described below
\nThe good case of implementing such a strategy for this contest is provided in Plotting Big Data: Sleeping Series where DuckDB and Plotly were used together to cook the noodles.
\nSQLLite can be also leveraged in such a situation (see here as an example).
\nExercising every general best practice described in the respective section above is essential for the time-efficient ML pipeline built on Kaggle kernels. However, there is one more trick that can help you.
\nYou can
\nOther thing to consider is to unitilize the batch prediction strategy where your model predicts against the subsamples of the test set rather then on the entire test set at once.
\nIf we were not limited to Kaggle kernels in this competition, I would go further to share one more recommendation. It is about taking advantage of lazy evaluation. Lazy evaluation refers to the strategy which delays evaluation of an expression until the value is actually needed. Lazy evaluation is an important concept (used especially in functional programming), and if you want to read more about its different usages in Python you can start here.
\nMany industrial platforms (like Spark/PySpark, Dask, Ray etc.) facilitate it for you out of the box. For example, I stick to Spark/PySpark as a go-to choice to build BigData-scale solutions in the industrial projects.
\nHowever, launching the infrastructure needed for the platforms like Spark/PySpark or Dask requires some resources (RAM, CPU) to be consumed, and it could make it impossible to use on Kaggle kernels with the datasets of a size compared with the data of “Child Mind Institute - Detect Sleep States” contest.
", "rawMarkdown": "# Introduction\n\nThe contest of “Child Mind Institute - Detect Sleep States” imposes us to operate under the significant resource constraints.\n- Handling the large (really BigData-scale) dataset\n- Running it as a code competition on Kaggle notebooks\n\nThe above-mentioned circumstances force us to invent solutions that are\n- Smart in terms of utilizing RAM, CPU/GPU/TPU time, and disk storage. \n- Fast in terms of the execution time (basic Kaggle kernels have the limit of a session to run up to 9 h, with visually reported execution time not exceeding 6 h).\n\nEven if you decide to convert your Kaggle notebook into a Google Collab Pro+ one (it is possible but it comes at a price: https://colab.research.google.com/signup), it would not resolve all of the resource constraint-driven problems for your codebase.\n\nTherefore, the only viable option is to get your head around the best practices of the effective tackling BigData-scale problems in the online resource-limited kernels provided by Kaggle. Some of the best practices are universal, and others are specific to the type of operations you try to accomplish (like EDA-specific or ML Engineering-specific tricks). We are going to review them in the sections below.\n\n# Universal tricks\n\n- Load your data in fast, time-efficient manner (yes, reading BigData with Pandas directly is a pain as well as the drain of the execution time of your session, even if you utilize a batch reading approach); the good example of a fast data loading strategy with *polar* is provided in https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive ; obviously, this is not the only strategy possible to implement – you should be smart to look at the appropriate options (*pyarrrow* to be one of the things to look at, besides *polar*; btw, the nice and fast *pyarrow*-based data loading strategy demonstrated by @tolgadincer in https://www.kaggle.com/code/tolgadincer/sleep-state-fast-data-access-with-parquet)\n- Optimize the in-memory size of Pandas dataframe (some of the techniques of this sort are shown in https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive and https://www.kaggle.com/code/renatoreggiani/feat-eng-ideas-60-memory-reduction-cmi , for instance)\n- Force regular garbage collection in the certain points of your notebook, using the capabilities of *gc* package\n- Store the preprocessed training data as a separate Kaggle dataset – it will save the execution of your notebooks doing EDA or/and ML experiments down the road (for example, you can refer to Carl Ellis’s [“Zzzs: Lightweight training dataset + target” dataset](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target) as a good implementation of this particular principle)\n\n# EDA-specific tricks\n\nPlotting the datasets that are bigger them a RAM memory available to your solution will crash your conventional workflows (like trying to feed a pandas dataframe to a graph lib like plotly or seaborn and then visualize something). \n\nTherefore, more elaborate approaches should be used to achieve your goal. One of the popular patterns of doing it is described below\n- Launch a memory-efficient lightweight database engine \n- Mount or import your BigData-scale files into it as tables\n- Pre-calculate the data needed for your charts (histograms, boxplots, bar charts etc.) with SQL queries\n- Visualize your pre-calculated data with a graph library of your choice (with minimal client-side calculations)\n\nThe good case of implementing such a strategy for this contest is provided in [Plotting Big Data: Sleeping Series](https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series) where DuckDB and Plotly were used together to cook the noodles.\n\nSQLLite can be also leveraged in such a situation (see [here](https://plotly.com/python/v3/ipython-notebooks/big-data-analytics-with-pandas-and-sqlite/) as an example).\n\n# Specific ML modelling tricks\n\nExercising every general best practice described in the respective section above is essential for the time-efficient ML pipeline built on Kaggle kernels. However, there is one more trick that can help you. \n\nYou can\n- Train your model out of the Kaggle premise\n- Save (serialize) it as a binary file (either pickle or a native TensorFlow model storage format could be used, depending on what a model you train) – in such a way, you get a pre-trained model in your disposal\n- Upload the pre-trained model to a new Kaggle dataset\n- Import such a dataset into your notebook intended to submit predictions to load the pre-trained model from the binary file\n\nOther thing to consider is to unitilize the batch prediction strategy where your model predicts against the subsamples of the test set rather then on the entire test set at once.\n\n# Final notes\n\nIf we were not limited to Kaggle kernels in this competition, I would go further to share one more recommendation. It is about taking advantage of **lazy evaluation**. Lazy evaluation refers to the strategy which delays evaluation of an expression until the value is actually needed. Lazy evaluation is an important concept (used especially in functional programming), and if you want to read more about its different usages in Python you can start [here](https://towardsdatascience.com/what-is-lazy-evaluation-in-python-9efb1d3bfed0).\n\nMany industrial platforms (like Spark/PySpark, Dask, Ray etc.) facilitate it for you out of the box. For example, I stick to Spark/PySpark as a go-to choice to build BigData-scale solutions in the industrial projects. \n\nHowever, launching the infrastructure needed for the platforms like Spark/PySpark or Dask requires some resources (RAM, CPU) to be consumed, and it could make it impossible to use on Kaggle kernels with the datasets of a size compared with the data of “Child Mind Institute - Detect Sleep States” contest.\n", "votes": 43 }, { "id": 2454090, "postDate": "2023-09-24T15:15:50.383Z", "content": "Hi again @gvyshnya
\nI think you should have featured @tolgadincer 's Sleep State: Fast Data Access with Parquet\n in this post. The trick is crazy fast for me.
\nI have adapted this to my notebook as well, boosted with caching system so no need to reload the data when it is cached. My pleasure if you want to feature my notebook in this discussion.
\nMany thanks!
", "rawMarkdown": "Hi again @gvyshnya \n\nI think you should have featured @tolgadincer 's [Sleep State: Fast Data Access with Parquet\n](https://www.kaggle.com/code/tolgadincer/sleep-state-fast-data-access-with-parquet) in this post. The trick is crazy fast for me.\n\nI have adapted this to [my notebook](https://www.kaggle.com/code/rimbax/detection-plots-of-all-277-series-with-flags?scriptVersionId=144052433&cellId=6) as well, boosted with caching system so no need to reload the data when it is cached. My pleasure if you want to feature my notebook in this discussion.\n\nMany thanks!", "votes": 4, "replies": [ { "id": 2454370, "postDate": "2023-09-24T18:29:05.437Z", "content": "@rimbax : thanks, Rimbaudi will digest it shortly!
", "rawMarkdown": "@rimbax : thanks, Rimbaudi will digest it shortly!", "votes": 1, "replies": [ { "id": 2457694, "postDate": "2023-09-27T06:00:39.723Z", "content": "Thank you for your consideration @gvyshnya , all the best!
", "rawMarkdown": "Thank you for your consideration @gvyshnya , all the best!", "votes": 1 } ] }, { "id": 2488788, "postDate": "2023-10-19T13:54:26.687Z", "content": "Good work @rimbax
\nThe memory constraints is a killer.
Does anyone have this issue of getting high scores offline using particular tweaked parameters only to score lower on here? This is even when run multiple times offline showing improved consistent performance only to submit online to end up with no improvements.
", "rawMarkdown": "Good work @rimbax \nThe memory constraints is a killer. \n\nDoes anyone have this issue of getting high scores offline using particular tweaked parameters only to score lower on here? This is even when run multiple times offline showing improved consistent performance only to submit online to end up with no improvements. ", "votes": 1 } ] }, { "id": 2488757, "postDate": "2023-10-19T13:18:17.050Z", "content": "I was struggling on utilizing out-trained model, you nailed the point in \"Specific ML modeling tricks\" section. Thanks for the great article. :)
", "rawMarkdown": "I was struggling on utilizing out-trained model, you nailed the point in \"Specific ML modeling tricks\" section. Thanks for the great article. :)", "votes": 1, "replies": [ { "id": 2488767, "postDate": "2023-10-19T13:31:07.400Z", "content": "@jhjh97 : I am glad to know it was helpful :)
", "rawMarkdown": "@jhjh97 : I am glad to know it was helpful :)" } ] }, { "id": 2488599, "postDate": "2023-10-19T11:16:24.277Z", "content": "Thanks for the article. The question is when I upload data to submit predictions so that my model passes the evaluation. Can I use already processed data and trained models from another program ? Or is it necessary to use only the data that is given in the conditions of the task and send the model for evaluation only with them?
", "rawMarkdown": "Thanks for the article. The question is when I upload data to submit predictions so that my model passes the evaluation. Can I use already processed data and trained models from another program ? Or is it necessary to use only the data that is given in the conditions of the task and send the model for evaluation only with them?", "votes": 1, "replies": [ { "id": 2488619, "postDate": "2023-10-19T11:26:48.610Z", "content": "@rinatbayanov : thanks, Rinat. You can definitely use pre-processed data from either a public dataset created by someone else or make your own dataset (whether private or public, upon your call) to use in the model training down the road.
\nAs for the publicly available datasets for this contest, you can check out the references below
\nThank you! Can I upload the received submission file from my other code to check the result ? If I have a lot of models and I don't want to have memory problems? That is, the code will consist of 3 cells, the 1st is the import of libraries, the 2nd is the import of submission and then the submission record in response. If the submission is in the correct format, will the competition give an assessment to my code? This is my first competition and I haven't been able to upload a solution for a week, sorry
", "rawMarkdown": "Thank you! Can I upload the received submission file from my other code to check the result ? If I have a lot of models and I don't want to have memory problems? That is, the code will consist of 3 cells, the 1st is the import of libraries, the 2nd is the import of submission and then the submission record in response. If the submission is in the correct format, will the competition give an assessment to my code? This is my first competition and I haven't been able to upload a solution for a week, sorry", "votes": 1 }, { "id": 2491035, "postDate": "2023-10-21T09:22:22.193Z", "content": "@rinatbayanov : Hey Rinat,
\nManipulating with the submission files the way you described in the last comment is allowed for non-code competitions without a hidden (private) part of the testing set.
\nHowever, this one is the Code competition. Therefore, it is bound to the respective submission requirements. The essence is, your notebook should read the testing set (its unpublic part) in the runtime to score it online.
\nSo, the bottom line is, you can use pre-trained models, and you are not allowed to use submission files prepared outside the platform.
\nI hope it is helpful.
", "rawMarkdown": "@rinatbayanov : Hey Rinat,\n\nManipulating with the submission files the way you described in the last comment is allowed for non-code competitions without a hidden (private) part of the testing set. \n\nHowever, this one is the [Code competition](https://www.kaggle.com/c/child-mind-institute-detect-sleep-states/overview/code-requirements). Therefore, it is bound to the respective [submission requirements](https://www.kaggle.com/docs/competitions#notebooks-only-FAQ). The essence is, your notebook should read the testing set (its unpublic part) in the runtime to score it online.\n\nSo, the bottom line is, you can use pre-trained models, and you are not allowed to use submission files prepared outside the platform.\n\nI hope it is helpful." } ] } ] }, { "id": 2467313, "postDate": "2023-10-04T12:51:27.810Z", "content": "Thanks for this great resource Georgii. I ran into memory issues almost immediately so it is nice to know some options in how to address it.
", "rawMarkdown": "Thanks for this great resource Georgii. I ran into memory issues almost immediately so it is nice to know some options in how to address it. ", "votes": 1, "replies": [ { "id": 2467324, "postDate": "2023-10-04T12:58:49.273Z", "content": "@louisjbotha : good to know it was helpful.
", "rawMarkdown": "@louisjbotha : good to know it was helpful." } ] }, { "id": 2465505, "postDate": "2023-10-03T03:14:36.450Z", "content": "I guess we can just use pytorch data loaders to create efficient batches
", "rawMarkdown": "I guess we can just use pytorch data loaders to create efficient batches", "votes": 1, "replies": [ { "id": 2466453, "postDate": "2023-10-03T20:24:17.510Z", "content": "@samu2505 : yes, agreed - it could be one of the possible effective approaches.
", "rawMarkdown": "@samu2505 : yes, agreed - it could be one of the possible effective approaches.", "votes": 1 } ] }, { "id": 2450481, "postDate": "2023-09-21T22:48:07.427Z", "content": "There are many helpful tips here. Thanks.
", "rawMarkdown": "There are many helpful tips here. Thanks.", "votes": 1, "replies": [ { "id": 2453539, "postDate": "2023-09-24T05:51:11.663Z", "content": "@jonblanchard : thanks, Jon! I am glad you found this post to be useful.
", "rawMarkdown": "@jonblanchard : thanks, Jon! I am glad you found this post to be useful.", "votes": 1 } ] }, { "id": 2450182, "postDate": "2023-09-21T16:53:06.237Z", "content": "Hi, I'm new here and haven't made any submissions on this challenge yet, but why are people talking so much about the dataset's size? I picked this one (in contrast to for example \"RSNA 2023 Abdominal Trauma Detection\" with 460 GB of data) due to the small data size.
\nIt is just 1 GB on disk, and loading it to pandas dataframe brings the memory consumption to just 4.5 GB (along with 22 GB of used virtual memory). I tried loading on a computer with only about 14 GB of free RAM, and it could be done without problems in about 30 seconds. Well oddly this time the python process uses 7.7 GB of RAM, but still. It is not like tens or hundreds of gigabytes.
\nWhat am I missing here, is the Kaggle's notebook environment really constrained on resources? The dataset's shape is only 127946340 x 5, but I guess the repeated strings may take lots of space if no compression (or de-duplication) is used?
", "rawMarkdown": "Hi, I'm new here and haven't made any submissions on this challenge yet, but why are people talking so much about the dataset's size? I picked this one (in contrast to for example \"RSNA 2023 Abdominal Trauma Detection\" with 460 GB of data) due to the small data size.\n\nIt is just 1 GB on disk, and loading it to pandas dataframe brings the memory consumption to just 4.5 GB (along with 22 GB of used virtual memory). I tried loading on a computer with only about 14 GB of free RAM, and it could be done without problems in about 30 seconds. Well oddly this time the python process uses 7.7 GB of RAM, but still. It is not like tens or hundreds of gigabytes.\n\nWhat am I missing here, is the Kaggle's notebook environment really constrained on resources? The dataset's shape is only 127946340 x 5, but I guess the repeated strings may take lots of space if no compression (or de-duplication) is used?", "votes": 1, "replies": [ { "id": 2450216, "postDate": "2023-09-21T17:29:10.097Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2453544, "postDate": "2023-09-24T05:54:13.790Z", "content": "@nikonyrh : Hey Niko, yes, the resource constraints on the Kaggle notebook end add complexity here since it is a Code completion (that is, you should submit your predictions from a notebook).
", "rawMarkdown": "@nikonyrh : Hey Niko, yes, the resource constraints on the Kaggle notebook end add complexity here since it is a Code completion (that is, you should submit your predictions from a notebook)." } ] }, { "id": 2447801, "postDate": "2023-09-20T08:58:59.787Z", "content": "First of all, thank you for taking time putting it everything together. This information is very helpful for people like me. Learnt quite a few things from this post.
", "rawMarkdown": "First of all, thank you for taking time putting it everything together. This information is very helpful for people like me. Learnt quite a few things from this post. ", "votes": 1 }, { "id": 2446994, "postDate": "2023-09-19T18:50:48.980Z", "content": "It may be helpful to break the data along Series_ids and then use TensorFlow Dataset to read the entire data in chunks and process them. I have been able to plot the data associated with a single series without too much trouble, provided feature-engineered columns do not grow wild.
You can find a framework for doing this here
", "rawMarkdown": "It may be helpful to break the data along `Series_ids` and then use `TensorFlow Dataset` to read the entire data in chunks and process them. I have been able to plot the data associated with a single series without too much trouble, provided feature-engineered columns do not grow wild.\n\nYou can find a framework for doing this [here](https://www.kaggle.com/code/biswajitsahoo1111/reading-multiple-csv-files-in-tensorflow-2)", "votes": 1 }, { "id": 2445807, "postDate": "2023-09-19T05:04:35.570Z", "content": "Good tips @gvyshnya. Thanks for sharing!
", "rawMarkdown": "Good tips @gvyshnya. Thanks for sharing!", "votes": 1, "replies": [ { "id": 2446007, "postDate": "2023-09-19T07:32:38.670Z", "content": "@sudheeshr : thanks, Sudheesh!
", "rawMarkdown": "@sudheeshr : thanks, Sudheesh!" } ] }, { "id": 2444357, "postDate": "2023-09-18T08:58:22.843Z", "content": "Based on the observations, it is clear that the right number of right features with a streamlined efficient classifier model and avoiding memory traps is the key to this. If one can just get the data engineering right, then it becomes just testing the different classifiers to see which gives best results. It looks like LSTMs and 1D CNNs may win this with gradient boosting methods behind (I think).
", "rawMarkdown": "Based on the observations, it is clear that the right number of right features with a streamlined efficient classifier model and avoiding memory traps is the key to this. If one can just get the data engineering right, then it becomes just testing the different classifiers to see which gives best results. It looks like LSTMs and 1D CNNs may win this with gradient boosting methods behind (I think).", "votes": 1 }, { "id": 2444133, "postDate": "2023-09-18T07:18:57.590Z", "content": "Great tips for optimizing BigData-scale tasks in Kaggle notebooks! Efficient data loading, memory optimization, and pre-trained model usage are key to success. Thanks for sharing @gvyshnya
\n\"How do you handle extremely large datasets when using SQLite for data visualization in Kaggle notebooks? Any specific tips or resources you recommend for that?\"
", "rawMarkdown": "Great tips for optimizing BigData-scale tasks in Kaggle notebooks! Efficient data loading, memory optimization, and pre-trained model usage are key to success. Thanks for sharing @gvyshnya \n\n**\"How do you handle extremely large datasets when using SQLite for data visualization in Kaggle notebooks? Any specific tips or resources you recommend for that?\"**", "votes": 1, "replies": [ { "id": 2444144, "postDate": "2023-09-18T07:29:47.543Z", "content": "@nithinreddy90 : Hey Ranamalla, you can try to adapt the flow from https://plotly.com/python/v3/ipython-notebooks/big-data-analytics-with-pandas-and-sqlite/ into a Kaggle notebook.
", "rawMarkdown": "@nithinreddy90 : Hey Ranamalla, you can try to adapt the flow from https://plotly.com/python/v3/ipython-notebooks/big-data-analytics-with-pandas-and-sqlite/ into a Kaggle notebook.", "replies": [ { "id": 2445702, "postDate": "2023-09-19T03:23:55.060Z", "content": "Thank you, @gvyshnya for providing the resources to refer.
", "rawMarkdown": "Thank you, @gvyshnya for providing the resources to refer." } ] } ] }, { "id": 2443730, "postDate": "2023-09-17T23:35:24.467Z", "content": "Very insightful post as always @gvyshnya
\nWill try out the zzz dataset later
", "rawMarkdown": "Very insightful post as always @gvyshnya \n\nWill try out the zzz dataset later", "votes": 1, "replies": [ { "id": 2443794, "postDate": "2023-09-18T02:02:59.850Z", "content": "@ifeanyichukwunwobodo : thanks!
", "rawMarkdown": "@ifeanyichukwunwobodo : thanks!" } ] }, { "id": 2443540, "postDate": "2023-09-17T18:44:05.630Z", "content": "Hello.
\nI actually have trained the models and also loading the training data in batches with no errors and getting an output. But now get an error when doing submission to kaggle, after some time:
\n\"Notebook Threw Exception: While rerunning your code, your notebook hit an unhandled error. Note that the hidden dataset can be larger/smaller/different than the public dataset.\"
\nI have checked the error help on Kaggle but still stumped.
\nThe notebooks run successfully prior to submission but fails when scoring. Any tips?
", "rawMarkdown": "Hello.\n\nI actually have trained the models and also loading the training data in batches with no errors and getting an output. But now get an error when doing submission to kaggle, after some time:\n\n\"Notebook Threw Exception: While rerunning your code, your notebook hit an unhandled error. Note that the hidden dataset can be larger/smaller/different than the public dataset.\"\n\nI have checked the error help on Kaggle but still stumped.\n\nThe notebooks run successfully prior to submission but fails when scoring. Any tips?", "votes": 1, "replies": [ { "id": 2444026, "postDate": "2023-09-18T06:14:06.793Z", "content": "@aidlre001 It's probably a memory error. In your submission notebook, the test data only has a few example rows. When you submit, your code is rerun against the real test data which is a lot larger.
\nTry replacing the test data with the training data in your submission notebook and running it. It's a simple way to simulate what happens after submission and you can usually detect the cause of the error.
", "rawMarkdown": "@aidlre001 It's probably a memory error. In your submission notebook, the test data only has a few example rows. When you submit, your code is rerun against the real test data which is a lot larger. \n\nTry replacing the test data with the training data in your submission notebook and running it. It's a simple way to simulate what happens after submission and you can usually detect the cause of the error. \n", "votes": 3, "replies": [ { "id": 2444276, "postDate": "2023-09-18T08:35:47.793Z", "content": "Hi Rai,
\nThe memory error is different and the message usually indicates that. I used to get those before I finished training and when trying to submit. This one was different.
\nThe weird thing is that it completes on my notebook session with no errors or messages. It happened in the submission stage.
\nHowever, I have figured it out and it is an embarrassing discovery because it was due to trivial initialization issues . It is weird that the executed notebook did not throw any errors when I ran it on my end.
", "rawMarkdown": "Hi Rai,\n\nThe memory error is different and the message usually indicates that. I used to get those before I finished training and when trying to submit. This one was different.\n\nThe weird thing is that it completes on my notebook session with no errors or messages. It happened in the submission stage.\n\nHowever, I have figured it out and it is an embarrassing discovery because it was due to trivial initialization issues@aidlre001 : good to know you resolved the issues!
", "rawMarkdown": "@aidlre001 : good to know you resolved the issues!" }, { "id": 2444528, "postDate": "2023-09-18T10:13:07.717Z", "content": "Thank you Georgii
", "rawMarkdown": "Thank you Georgii", "votes": 1 }, { "id": 2453200, "postDate": "2023-09-23T21:27:32.507Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2453202, "postDate": "2023-09-23T21:28:53.057Z", "content": "Hello @aidlre001
\nCan you add some more details on your \"trivial initialization issues\"?
\nI am having a similar issue (the notebook completes but the submision fails with the exception error) and I am not sure what else to look at!
\nThank you in advance.
", "rawMarkdown": "Hello @aidlre001 \n\nCan you add some more details on your \"trivial initialization issues\"?\n\nI am having a similar issue (the notebook completes but the submision fails with the exception error) and I am not sure what else to look at!\n\nThank you in advance.", "votes": 1 }, { "id": 2454486, "postDate": "2023-09-24T20:36:25.633Z", "content": "Hi David,
\nBreak down your notebook into simple steps and execute each step gradually. You may discover the problem because Kaggle doesn't pick the error, at least from my experience. Check the order of operations if you are calling a function or class before defining it.
", "rawMarkdown": "Hi David,\n\nBreak down your notebook into simple steps and execute each step gradually. You may discover the problem because Kaggle doesn't pick the error, at least from my experience. Check the order of operations if you are calling a function or class before defining it. ", "votes": 2 } ] } ] }, { "id": 2456847, "postDate": "2023-09-26T13:03:11.327Z", "content": "Valuable, substantial material handling large(big) data.
", "rawMarkdown": "Valuable, substantial material handling large(big) data.", "votes": 2, "replies": [ { "id": 2457085, "postDate": "2023-09-26T15:52:48.067Z", "content": "@mpwolke : thanks a lot for your encouraging feedback!
", "rawMarkdown": "@mpwolke : thanks a lot for your encouraging feedback!", "votes": 1 } ] }, { "id": 2444585, "postDate": "2023-09-18T11:01:20.087Z", "content": "This is great
", "rawMarkdown": "This is great", "votes": 2, "replies": [ { "id": 2446010, "postDate": "2023-09-19T07:34:53.910Z", "content": "@vineetkukreti : thanks, Vineet!
", "rawMarkdown": "@vineetkukreti : thanks, Vineet!", "votes": 1 } ] }, { "id": 2443394, "postDate": "2023-09-17T17:31:51.840Z", "content": "This is the most helpful tips by far. I have no experience with Spark but will give it a try later. For now I am planning to split the dataset by the series ID.
", "rawMarkdown": "This is the most helpful tips by far. I have no experience with Spark but will give it a try later. For now I am planning to split the dataset by the series ID.", "votes": 2, "replies": [ { "id": 2444166, "postDate": "2023-09-18T07:41:16.583Z", "content": "@rimbax : Thanks, Rimba! Your idea with splitting the series data by series id could be really helpful. As for Spark/PySpark, it would be a great companion to your next projects!
", "rawMarkdown": "@rimbax : Thanks, Rimba! Your idea with splitting the series data by series id could be really helpful. As for Spark/PySpark, it would be a great companion to your next projects!", "votes": 1, "replies": [ { "id": 2444293, "postDate": "2023-09-18T08:41:43.343Z", "content": "Sure, thank you for your support. My idea is so that one have flexible options which series(es) they would like to use (+ adjusting to their computational power).
", "rawMarkdown": "Sure, thank you for your support. My idea is so that one have flexible options which series(es) they would like to use (+ adjusting to their computational power).", "votes": 2 } ] } ] }, { "id": 2443141, "postDate": "2023-09-17T14:47:10.070Z", "content": "Thanks for sharing your insights! I found that preprocessing the test file was a major challenge, but I recently switched to performing feature engineering and predictions in smaller chunks, and it's made a significant difference. Unfortunately, I've run out of submissions for today.
", "rawMarkdown": "Thanks for sharing your insights! I found that preprocessing the test file was a major challenge, but I recently switched to performing feature engineering and predictions in smaller chunks, and it's made a significant difference. Unfortunately, I've run out of submissions for today.", "votes": 2, "replies": [ { "id": 2443365, "postDate": "2023-09-17T17:14:57.490Z", "content": "@lennarthaupts : Hey Lennart, good to know you found your way through the difficulties!
", "rawMarkdown": "@lennarthaupts : Hey Lennart, good to know you found your way through the difficulties!", "votes": 3 } ] }, { "id": 2442895, "postDate": "2023-09-17T11:30:34.197Z", "content": "Thanks good post
", "rawMarkdown": "Thanks good post ", "votes": 1 } ], "comments": [ { "id": 2454090, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-24T15:15:50.383000", "content": "Hi again @gvyshnya
\nI think you should have featured @tolgadincer 's Sleep State: Fast Data Access with Parquet\n in this post. The trick is crazy fast for me.
\nI have adapted this to my notebook as well, boosted with caching system so no need to reload the data when it is cached. My pleasure if you want to feature my notebook in this discussion.
\nMany thanks!
", "votes": 4, "replies": [ { "id": 2454370, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-24T18:29:05.437000", "content": "@rimbax : thanks, Rimbaudi will digest it shortly!
", "votes": 1, "replies": [ { "id": 2457694, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-27T06:00:39.723000", "content": "Thank you for your consideration @gvyshnya , all the best!
", "votes": 1, "replies": [] } ] }, { "id": 2488788, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-10-19T13:54:26.687000", "content": "Good work @rimbax
\nThe memory constraints is a killer.
Does anyone have this issue of getting high scores offline using particular tweaked parameters only to score lower on here? This is even when run multiple times offline showing improved consistent performance only to submit online to end up with no improvements.
", "votes": 1, "replies": [] } ] }, { "id": 2488757, "author_name": "", "author_url": "", "post_date": "2023-10-19T13:18:17.050000", "content": "I was struggling on utilizing out-trained model, you nailed the point in \"Specific ML modeling tricks\" section. Thanks for the great article. :)
", "votes": 1, "replies": [ { "id": 2488767, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-10-19T13:31:07.400000", "content": "@jhjh97 : I am glad to know it was helpful :)
", "votes": 0, "replies": [] } ] }, { "id": 2488599, "author_name": "Rinat", "author_url": "", "post_date": "2023-10-19T11:16:24.277000", "content": "Thanks for the article. The question is when I upload data to submit predictions so that my model passes the evaluation. Can I use already processed data and trained models from another program ? Or is it necessary to use only the data that is given in the conditions of the task and send the model for evaluation only with them?
", "votes": 1, "replies": [ { "id": 2488619, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-10-19T11:26:48.610000", "content": "@rinatbayanov : thanks, Rinat. You can definitely use pre-processed data from either a public dataset created by someone else or make your own dataset (whether private or public, upon your call) to use in the model training down the road.
\nAs for the publicly available datasets for this contest, you can check out the references below
\nThank you! Can I upload the received submission file from my other code to check the result ? If I have a lot of models and I don't want to have memory problems? That is, the code will consist of 3 cells, the 1st is the import of libraries, the 2nd is the import of submission and then the submission record in response. If the submission is in the correct format, will the competition give an assessment to my code? This is my first competition and I haven't been able to upload a solution for a week, sorry
", "votes": 1, "replies": [] }, { "id": 2491035, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-10-21T09:22:22.193000", "content": "@rinatbayanov : Hey Rinat,
\nManipulating with the submission files the way you described in the last comment is allowed for non-code competitions without a hidden (private) part of the testing set.
\nHowever, this one is the Code competition. Therefore, it is bound to the respective submission requirements. The essence is, your notebook should read the testing set (its unpublic part) in the runtime to score it online.
\nSo, the bottom line is, you can use pre-trained models, and you are not allowed to use submission files prepared outside the platform.
\nI hope it is helpful.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2467313, "author_name": "Louis J Botha", "author_url": "", "post_date": "2023-10-04T12:51:27.810000", "content": "Thanks for this great resource Georgii. I ran into memory issues almost immediately so it is nice to know some options in how to address it.
", "votes": 1, "replies": [ { "id": 2467324, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-10-04T12:58:49.273000", "content": "@louisjbotha : good to know it was helpful.
", "votes": 0, "replies": [] } ] }, { "id": 2465505, "author_name": "samu2505", "author_url": "", "post_date": "2023-10-03T03:14:36.450000", "content": "I guess we can just use pytorch data loaders to create efficient batches
", "votes": 1, "replies": [ { "id": 2466453, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-10-03T20:24:17.510000", "content": "@samu2505 : yes, agreed - it could be one of the possible effective approaches.
", "votes": 1, "replies": [] } ] }, { "id": 2450481, "author_name": "Jon Blanchard", "author_url": "", "post_date": "2023-09-21T22:48:07.427000", "content": "There are many helpful tips here. Thanks.
", "votes": 1, "replies": [ { "id": 2453539, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-24T05:51:11.663000", "content": "@jonblanchard : thanks, Jon! I am glad you found this post to be useful.
", "votes": 1, "replies": [] } ] }, { "id": 2450182, "author_name": "Niko N", "author_url": "", "post_date": "2023-09-21T16:53:06.237000", "content": "Hi, I'm new here and haven't made any submissions on this challenge yet, but why are people talking so much about the dataset's size? I picked this one (in contrast to for example \"RSNA 2023 Abdominal Trauma Detection\" with 460 GB of data) due to the small data size.
\nIt is just 1 GB on disk, and loading it to pandas dataframe brings the memory consumption to just 4.5 GB (along with 22 GB of used virtual memory). I tried loading on a computer with only about 14 GB of free RAM, and it could be done without problems in about 30 seconds. Well oddly this time the python process uses 7.7 GB of RAM, but still. It is not like tens or hundreds of gigabytes.
\nWhat am I missing here, is the Kaggle's notebook environment really constrained on resources? The dataset's shape is only 127946340 x 5, but I guess the repeated strings may take lots of space if no compression (or de-duplication) is used?
", "votes": 1, "replies": [ { "id": 2450216, "author_name": "", "author_url": "", "post_date": "2023-09-21T17:29:10.097000", "content": "", "votes": 0, "replies": [] }, { "id": 2453544, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-24T05:54:13.790000", "content": "@nikonyrh : Hey Niko, yes, the resource constraints on the Kaggle notebook end add complexity here since it is a Code completion (that is, you should submit your predictions from a notebook).
", "votes": 0, "replies": [] } ] }, { "id": 2447801, "author_name": "Vamsi", "author_url": "", "post_date": "2023-09-20T08:58:59.787000", "content": "First of all, thank you for taking time putting it everything together. This information is very helpful for people like me. Learnt quite a few things from this post.
", "votes": 1, "replies": [] }, { "id": 2446994, "author_name": "Nader E Afshar", "author_url": "", "post_date": "2023-09-19T18:50:48.980000", "content": "It may be helpful to break the data along Series_ids and then use TensorFlow Dataset to read the entire data in chunks and process them. I have been able to plot the data associated with a single series without too much trouble, provided feature-engineered columns do not grow wild.
You can find a framework for doing this here
", "votes": 1, "replies": [] }, { "id": 2445807, "author_name": "Sudheesh R", "author_url": "", "post_date": "2023-09-19T05:04:35.570000", "content": "Good tips @gvyshnya. Thanks for sharing!
", "votes": 1, "replies": [ { "id": 2446007, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-19T07:32:38.670000", "content": "@sudheeshr : thanks, Sudheesh!
", "votes": 0, "replies": [] } ] }, { "id": 2444357, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T08:58:22.843000", "content": "Based on the observations, it is clear that the right number of right features with a streamlined efficient classifier model and avoiding memory traps is the key to this. If one can just get the data engineering right, then it becomes just testing the different classifiers to see which gives best results. It looks like LSTMs and 1D CNNs may win this with gradient boosting methods behind (I think).
", "votes": 1, "replies": [] }, { "id": 2444133, "author_name": "Ranamalla Nithin Reddy", "author_url": "", "post_date": "2023-09-18T07:18:57.590000", "content": "Great tips for optimizing BigData-scale tasks in Kaggle notebooks! Efficient data loading, memory optimization, and pre-trained model usage are key to success. Thanks for sharing @gvyshnya
\n\"How do you handle extremely large datasets when using SQLite for data visualization in Kaggle notebooks? Any specific tips or resources you recommend for that?\"
", "votes": 1, "replies": [ { "id": 2444144, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-18T07:29:47.543000", "content": "@nithinreddy90 : Hey Ranamalla, you can try to adapt the flow from https://plotly.com/python/v3/ipython-notebooks/big-data-analytics-with-pandas-and-sqlite/ into a Kaggle notebook.
", "votes": 0, "replies": [ { "id": 2445702, "author_name": "Ranamalla Nithin Reddy", "author_url": "", "post_date": "2023-09-19T03:23:55.060000", "content": "Thank you, @gvyshnya for providing the resources to refer.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2443730, "author_name": "Ifeanyichukwu Nwobodo", "author_url": "", "post_date": "2023-09-17T23:35:24.467000", "content": "Very insightful post as always @gvyshnya
\nWill try out the zzz dataset later
", "votes": 1, "replies": [ { "id": 2443794, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-18T02:02:59.850000", "content": "@ifeanyichukwunwobodo : thanks!
", "votes": 0, "replies": [] } ] }, { "id": 2443540, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-17T18:44:05.630000", "content": "Hello.
\nI actually have trained the models and also loading the training data in batches with no errors and getting an output. But now get an error when doing submission to kaggle, after some time:
\n\"Notebook Threw Exception: While rerunning your code, your notebook hit an unhandled error. Note that the hidden dataset can be larger/smaller/different than the public dataset.\"
\nI have checked the error help on Kaggle but still stumped.
\nThe notebooks run successfully prior to submission but fails when scoring. Any tips?
", "votes": 1, "replies": [ { "id": 2444026, "author_name": "Rai", "author_url": "", "post_date": "2023-09-18T06:14:06.793000", "content": "@aidlre001 It's probably a memory error. In your submission notebook, the test data only has a few example rows. When you submit, your code is rerun against the real test data which is a lot larger.
\nTry replacing the test data with the training data in your submission notebook and running it. It's a simple way to simulate what happens after submission and you can usually detect the cause of the error.
", "votes": 3, "replies": [ { "id": 2444276, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T08:35:47.793000", "content": "Hi Rai,
\nThe memory error is different and the message usually indicates that. I used to get those before I finished training and when trying to submit. This one was different.
\nThe weird thing is that it completes on my notebook session with no errors or messages. It happened in the submission stage.
\nHowever, I have figured it out and it is an embarrassing discovery because it was due to trivial initialization issues . It is weird that the executed notebook did not throw any errors when I ran it on my end.
", "votes": 1, "replies": [] }, { "id": 2444368, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-18T09:00:47.823000", "content": "@aidlre001 : good to know you resolved the issues!
", "votes": 0, "replies": [] }, { "id": 2444528, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-18T10:13:07.717000", "content": "Thank you Georgii
", "votes": 1, "replies": [] }, { "id": 2453200, "author_name": "", "author_url": "", "post_date": "2023-09-23T21:27:32.507000", "content": "", "votes": 0, "replies": [] }, { "id": 2453202, "author_name": "David", "author_url": "", "post_date": "2023-09-23T21:28:53.057000", "content": "Hello @aidlre001
\nCan you add some more details on your \"trivial initialization issues\"?
\nI am having a similar issue (the notebook completes but the submision fails with the exception error) and I am not sure what else to look at!
\nThank you in advance.
", "votes": 1, "replies": [] }, { "id": 2454486, "author_name": "AIDLRE001", "author_url": "", "post_date": "2023-09-24T20:36:25.633000", "content": "Hi David,
\nBreak down your notebook into simple steps and execute each step gradually. You may discover the problem because Kaggle doesn't pick the error, at least from my experience. Check the order of operations if you are calling a function or class before defining it.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2456847, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-09-26T13:03:11.327000", "content": "Valuable, substantial material handling large(big) data.
", "votes": 2, "replies": [ { "id": 2457085, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-26T15:52:48.067000", "content": "@mpwolke : thanks a lot for your encouraging feedback!
", "votes": 1, "replies": [] } ] }, { "id": 2444585, "author_name": "Vineet Kukreti", "author_url": "", "post_date": "2023-09-18T11:01:20.087000", "content": "This is great
", "votes": 2, "replies": [ { "id": 2446010, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-19T07:34:53.910000", "content": "@vineetkukreti : thanks, Vineet!
", "votes": 1, "replies": [] } ] }, { "id": 2443394, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-17T17:31:51.840000", "content": "This is the most helpful tips by far. I have no experience with Spark but will give it a try later. For now I am planning to split the dataset by the series ID.
", "votes": 2, "replies": [ { "id": 2444166, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-18T07:41:16.583000", "content": "@rimbax : Thanks, Rimba! Your idea with splitting the series data by series id could be really helpful. As for Spark/PySpark, it would be a great companion to your next projects!
", "votes": 1, "replies": [ { "id": 2444293, "author_name": "Rimba Erlangga", "author_url": "", "post_date": "2023-09-18T08:41:43.343000", "content": "Sure, thank you for your support. My idea is so that one have flexible options which series(es) they would like to use (+ adjusting to their computational power).
", "votes": 2, "replies": [] } ] } ] }, { "id": 2443141, "author_name": "Lennart Haupts", "author_url": "", "post_date": "2023-09-17T14:47:10.070000", "content": "Thanks for sharing your insights! I found that preprocessing the test file was a major challenge, but I recently switched to performing feature engineering and predictions in smaller chunks, and it's made a significant difference. Unfortunately, I've run out of submissions for today.
", "votes": 2, "replies": [ { "id": 2443365, "author_name": "Georgii Vyshnia", "author_url": "", "post_date": "2023-09-17T17:14:57.490000", "content": "@lennarthaupts : Hey Lennart, good to know you found your way through the difficulties!
", "votes": 3, "replies": [] } ] }, { "id": 2442895, "author_name": "𝔄ℌ𝔐𝔈𝔇 𝔄𝔖ℌℜ𝔄𝔉", "author_url": "", "post_date": "2023-09-17T11:30:34.197000", "content": "Thanks good post
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2442819": "# Introduction\n\nThe contest of “Child Mind Institute - Detect Sleep States” imposes us to operate under the significant resource constraints.\n- Handling the large (really BigData-scale) dataset\n- Running it as a code competition on Kaggle notebooks\n\nThe above-mentioned circumstances force us to invent solutions that are\n- Smart in terms of utilizing RAM, CPU/GPU/TPU time, and disk storage. \n- Fast in terms of the execution time (basic Kaggle kernels have the limit of a session to run up to 9 h, with visually reported execution time not exceeding 6 h).\n\nEven if you decide to convert your Kaggle notebook into a Google Collab Pro+ one (it is possible but it comes at a price: https://colab.research.google.com/signup), it would not resolve all of the resource constraint-driven problems for your codebase.\n\nTherefore, the only viable option is to get your head around the best practices of the effective tackling BigData-scale problems in the online resource-limited kernels provided by Kaggle. Some of the best practices are universal, and others are specific to the type of operations you try to accomplish (like EDA-specific or ML Engineering-specific tricks). We are going to review them in the sections below.\n\n# Universal tricks\n\n- Load your data in fast, time-efficient manner (yes, reading BigData with Pandas directly is a pain as well as the drain of the execution time of your session, even if you utilize a batch reading approach); the good example of a fast data loading strategy with *polar* is provided in https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive ; obviously, this is not the only strategy possible to implement – you should be smart to look at the appropriate options (*pyarrrow* to be one of the things to look at, besides *polar*; btw, the nice and fast *pyarrow*-based data loading strategy demonstrated by @tolgadincer in https://www.kaggle.com/code/tolgadincer/sleep-state-fast-data-access-with-parquet)\n- Optimize the in-memory size of Pandas dataframe (some of the techniques of this sort are shown in https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive and https://www.kaggle.com/code/renatoreggiani/feat-eng-ideas-60-memory-reduction-cmi , for instance)\n- Force regular garbage collection in the certain points of your notebook, using the capabilities of *gc* package\n- Store the preprocessed training data as a separate Kaggle dataset – it will save the execution of your notebooks doing EDA or/and ML experiments down the road (for example, you can refer to Carl Ellis’s [“Zzzs: Lightweight training dataset + target” dataset](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target) as a good implementation of this particular principle)\n\n# EDA-specific tricks\n\nPlotting the datasets that are bigger them a RAM memory available to your solution will crash your conventional workflows (like trying to feed a pandas dataframe to a graph lib like plotly or seaborn and then visualize something). \n\nTherefore, more elaborate approaches should be used to achieve your goal. One of the popular patterns of doing it is described below\n- Launch a memory-efficient lightweight database engine \n- Mount or import your BigData-scale files into it as tables\n- Pre-calculate the data needed for your charts (histograms, boxplots, bar charts etc.) with SQL queries\n- Visualize your pre-calculated data with a graph library of your choice (with minimal client-side calculations)\n\nThe good case of implementing such a strategy for this contest is provided in [Plotting Big Data: Sleeping Series](https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series) where DuckDB and Plotly were used together to cook the noodles.\n\nSQLLite can be also leveraged in such a situation (see [here](https://plotly.com/python/v3/ipython-notebooks/big-data-analytics-with-pandas-and-sqlite/) as an example).\n\n# Specific ML modelling tricks\n\nExercising every general best practice described in the respective section above is essential for the time-efficient ML pipeline built on Kaggle kernels. However, there is one more trick that can help you. \n\nYou can\n- Train your model out of the Kaggle premise\n- Save (serialize) it as a binary file (either pickle or a native TensorFlow model storage format could be used, depending on what a model you train) – in such a way, you get a pre-trained model in your disposal\n- Upload the pre-trained model to a new Kaggle dataset\n- Import such a dataset into your notebook intended to submit predictions to load the pre-trained model from the binary file\n\nOther thing to consider is to unitilize the batch prediction strategy where your model predicts against the subsamples of the test set rather then on the entire test set at once.\n\n# Final notes\n\nIf we were not limited to Kaggle kernels in this competition, I would go further to share one more recommendation. It is about taking advantage of **lazy evaluation**. Lazy evaluation refers to the strategy which delays evaluation of an expression until the value is actually needed. Lazy evaluation is an important concept (used especially in functional programming), and if you want to read more about its different usages in Python you can start [here](https://towardsdatascience.com/what-is-lazy-evaluation-in-python-9efb1d3bfed0).\n\nMany industrial platforms (like Spark/PySpark, Dask, Ray etc.) facilitate it for you out of the box. For example, I stick to Spark/PySpark as a go-to choice to build BigData-scale solutions in the industrial projects. \n\nHowever, launching the infrastructure needed for the platforms like Spark/PySpark or Dask requires some resources (RAM, CPU) to be consumed, and it could make it impossible to use on Kaggle kernels with the datasets of a size compared with the data of “Child Mind Institute - Detect Sleep States” contest.\n", "2454090": "Hi again @gvyshnya \n\nI think you should have featured @tolgadincer 's [Sleep State: Fast Data Access with Parquet\n](https://www.kaggle.com/code/tolgadincer/sleep-state-fast-data-access-with-parquet) in this post. The trick is crazy fast for me.\n\nI have adapted this to [my notebook](https://www.kaggle.com/code/rimbax/detection-plots-of-all-277-series-with-flags?scriptVersionId=144052433&cellId=6) as well, boosted with caching system so no need to reload the data when it is cached. My pleasure if you want to feature my notebook in this discussion.\n\nMany thanks!", "2488757": "I was struggling on utilizing out-trained model, you nailed the point in \"Specific ML modeling tricks\" section. Thanks for the great article. :)", "2488599": "Thanks for the article. The question is when I upload data to submit predictions so that my model passes the evaluation. Can I use already processed data and trained models from another program ? Or is it necessary to use only the data that is given in the conditions of the task and send the model for evaluation only with them?", "2467313": "Thanks for this great resource Georgii. I ran into memory issues almost immediately so it is nice to know some options in how to address it. ", "2465505": "I guess we can just use pytorch data loaders to create efficient batches", "2450481": "There are many helpful tips here. Thanks.", "2450182": "Hi, I'm new here and haven't made any submissions on this challenge yet, but why are people talking so much about the dataset's size? I picked this one (in contrast to for example \"RSNA 2023 Abdominal Trauma Detection\" with 460 GB of data) due to the small data size.\n\nIt is just 1 GB on disk, and loading it to pandas dataframe brings the memory consumption to just 4.5 GB (along with 22 GB of used virtual memory). I tried loading on a computer with only about 14 GB of free RAM, and it could be done without problems in about 30 seconds. Well oddly this time the python process uses 7.7 GB of RAM, but still. It is not like tens or hundreds of gigabytes.\n\nWhat am I missing here, is the Kaggle's notebook environment really constrained on resources? The dataset's shape is only 127946340 x 5, but I guess the repeated strings may take lots of space if no compression (or de-duplication) is used?", "2447801": "First of all, thank you for taking time putting it everything together. This information is very helpful for people like me. Learnt quite a few things from this post. ", "2446994": "It may be helpful to break the data along `Series_ids` and then use `TensorFlow Dataset` to read the entire data in chunks and process them. I have been able to plot the data associated with a single series without too much trouble, provided feature-engineered columns do not grow wild.\n\nYou can find a framework for doing this [here](https://www.kaggle.com/code/biswajitsahoo1111/reading-multiple-csv-files-in-tensorflow-2)", "2445807": "Good tips @gvyshnya. Thanks for sharing!", "2444357": "Based on the observations, it is clear that the right number of right features with a streamlined efficient classifier model and avoiding memory traps is the key to this. If one can just get the data engineering right, then it becomes just testing the different classifiers to see which gives best results. It looks like LSTMs and 1D CNNs may win this with gradient boosting methods behind (I think).", "2444133": "Great tips for optimizing BigData-scale tasks in Kaggle notebooks! Efficient data loading, memory optimization, and pre-trained model usage are key to success. Thanks for sharing @gvyshnya \n\n**\"How do you handle extremely large datasets when using SQLite for data visualization in Kaggle notebooks? Any specific tips or resources you recommend for that?\"**", "2443730": "Very insightful post as always @gvyshnya \n\nWill try out the zzz dataset later", "2443540": "Hello.\n\nI actually have trained the models and also loading the training data in batches with no errors and getting an output. But now get an error when doing submission to kaggle, after some time:\n\n\"Notebook Threw Exception: While rerunning your code, your notebook hit an unhandled error. Note that the hidden dataset can be larger/smaller/different than the public dataset.\"\n\nI have checked the error help on Kaggle but still stumped.\n\nThe notebooks run successfully prior to submission but fails when scoring. Any tips?", "2456847": "Valuable, substantial material handling large(big) data.", "2444585": "This is great", "2443394": "This is the most helpful tips by far. I have no experience with Spark but will give it a try later. For now I am planning to split the dataset by the series ID.", "2443141": "Thanks for sharing your insights! I found that preprocessing the test file was a major challenge, but I recently switched to performing feature engineering and predictions in smaller chunks, and it's made a significant difference. Unfortunately, I've run out of submissions for today.", "2442895": "Thanks good post " } }