{ "id": 447593, "title": "Tips for Efficient Fast Inference", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/447593", "author_name": "Manu", "post_date": "2023-10-16T14:07:23.499000", "votes": 22, "comment_count": 18, "views": 0, "content": "

I'm going to discuss an approach for quick inference while using minimal memory. In particular, I'll showcase a submission that took 18 minutes to complete and achieved a score of 0.638.
\n\"\"

\n", "messages": [ { "id": 2484467, "postDate": "2023-10-16T14:07:23.500Z", "content": "

I'm going to discuss an approach for quick inference while using minimal memory. In particular, I'll showcase a submission that took 18 minutes to complete and achieved a score of 0.638.
\n\"\"

\n", "rawMarkdown": "\nI'm going to discuss an approach for quick inference while using minimal memory. In particular, I'll showcase a submission that took 18 minutes to complete and achieved a score of 0.638.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13599107%2Ff9e8fe1a897936d59a15f923e7f30572%2FCaptura1.PNG?generation=1697465197337503&alt=media)\n- Split the series into separate files by series_id.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13599107%2F44099f7db01b876efe45299a15ceebb1%2FCaptura.PNG?generation=1697464954620204&alt=media)\n- Read the series one by one, do minimal feature engineering (use nn aproach insted of gradient boosting), and make inference one by one using models previously trained in another notebook.\nIn this way, we use very little memory and generate predictions very quickly, allowing us to create an ensemble of multiple models, use different models depending on the type of series, and so on.", "votes": 22 }, { "id": 2484541, "postDate": "2023-10-16T14:58:51.357Z", "content": "

Good callout, could you dive further into your recommendation of using NN over gradient boosting. Not that I disagree but with a submission time of 18 minutes I would think you could do both and clear memory in between batches/predictions?

", "rawMarkdown": "Good callout, could you dive further into your recommendation of using NN over gradient boosting. Not that I disagree but with a submission time of 18 minutes I would think you could do both and clear memory in between batches/predictions?", "votes": 1, "replies": [ { "id": 2484562, "postDate": "2023-10-16T15:12:39.173Z", "content": "

Yes, of course, with this submission time you can use both.
\nOn one hand, by loading the series one by one, you completely eliminate the memory issue. You could use boosting, create multiple features, and clear memory each time.
\nIf you introduce boosting, the problem you might encounter will probably be a time issue due to the need for feature engineering on all the test series, depending on the type and amount of features you compute. This issue is eliminated when using a neural network which requires minimal additional features. Nonetheless, considering it only takes 18 minutes, it can indeed be combined with boosting models.

", "rawMarkdown": "Yes, of course, with this submission time you can use both.\nOn one hand, by loading the series one by one, you completely eliminate the memory issue. You could use boosting, create multiple features, and clear memory each time.\nIf you introduce boosting, the problem you might encounter will probably be a time issue due to the need for feature engineering on all the test series, depending on the type and amount of features you compute. This issue is eliminated when using a neural network which requires minimal additional features. Nonetheless, considering it only takes 18 minutes, it can indeed be combined with boosting models.", "votes": 2, "replies": [ { "id": 2484565, "postDate": "2023-10-16T15:14:28.203Z", "content": "

Ok I follow you! You never know what someone else may have come up with so I wanted to make sure haha

", "rawMarkdown": "Ok I follow you! You never know what someone else may have come up with so I wanted to make sure haha ", "votes": 1 } ] } ] }, { "id": 2487665, "postDate": "2023-10-18T17:48:54.223Z", "content": "

Interesting topic @mpp1998, RAM issue was a bottleneck, using nn I believe it’ll get removed. I get a question to raise: how did you model this problem, as segmentation or just time-series forecasting, or others? Thx for replying

", "rawMarkdown": "Interesting topic @mpp1998, RAM issue was a bottleneck, using nn I believe it’ll get removed. I get a question to raise: how did you model this problem, as segmentation or just time-series forecasting, or others? Thx for replying", "votes": 2, "replies": [ { "id": 2488382, "postDate": "2023-10-19T08:31:43.177Z", "content": "

Im modeling it as a segmentation problem. I divide the series into chunks and attempt to segment each chunk into sleep/awake periods. Each chunk is an entry to the neural network, and chunk duration is a parameter: one day, several days etc

", "rawMarkdown": "Im modeling it as a segmentation problem. I divide the series into chunks and attempt to segment each chunk into sleep/awake periods. Each chunk is an entry to the neural network, and chunk duration is a parameter: one day, several days etc", "votes": 4, "replies": [ { "id": 2488469, "postDate": "2023-10-19T09:19:03.973Z", "content": "

Agree with you, thanks~

", "rawMarkdown": "Agree with you, thanks~" }, { "id": 2506538, "postDate": "2023-10-31T11:16:42.653Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2506544, "postDate": "2023-10-31T11:20:55.217Z", "content": "

When infer, can the chunk duration only be one day?

", "rawMarkdown": "When infer, can the chunk duration only be one day?" }, { "id": 2506567, "postDate": "2023-10-31T11:50:21.953Z", "content": "

For each chunk, do you predict each point as sleep/awake?

", "rawMarkdown": "For each chunk, do you predict each point as sleep/awake?" }, { "id": 2506802, "postDate": "2023-10-31T14:54:51.447Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2506803, "postDate": "2023-10-31T14:55:13.467Z", "content": "

Yes, it can be one day.

", "rawMarkdown": "Yes, it can be one day.", "votes": 1 }, { "id": 2506806, "postDate": "2023-10-31T14:56:10.367Z", "content": "

Yes, but you can do it also predicting other targets like in critic points regression

", "rawMarkdown": "Yes, but you can do it also predicting other targets like in critic points regression" } ] } ] }, { "id": 2484559, "postDate": "2023-10-16T15:09:32.067Z", "content": "

Hey, I've been following a similar approach, initially loading the entire parquet file using Pandas and then performing a group-by operation to split it into smaller DataFrames, which worked well on CPU due to having 30 GB of RAM. However, when I attempted to do the same on GPU, the memory just explodes. I suspect that the grouping operation on the entire dataset is consuming a lot of memory. So, my question is whether you're using this code snippet on a GPU and if it's not causing any memory issue. (I guess it is the read_row_groups that did the trick in saving memory) Thank you in advance!

", "rawMarkdown": "Hey, I've been following a similar approach, initially loading the entire parquet file using Pandas and then performing a group-by operation to split it into smaller DataFrames, which worked well on CPU due to having 30 GB of RAM. However, when I attempted to do the same on GPU, the memory just explodes. I suspect that the grouping operation on the entire dataset is consuming a lot of memory. So, my question is whether you're using this code snippet on a GPU and if it's not causing any memory issue. (I guess it is the read_row_groups that did the trick in saving memory) Thank you in advance!", "votes": 2, "replies": [ { "id": 2484596, "postDate": "2023-10-16T15:30:17.303Z", "content": "

I'm running that snippet of code on the CPU (13 GB RAM since it's a notebook with a GPU). For the hidden test set, it's enough and I haven't had to run it on the GPU. I'm not sure how much memory it will ultimately use.
\nAnd yes, I agree with you, the key difference was using read_row_groups. When I tried reading the entire parquet file, I encountered a memory error.

", "rawMarkdown": "\nI'm running that snippet of code on the CPU (13 GB RAM since it's a notebook with a GPU). For the hidden test set, it's enough and I haven't had to run it on the GPU. I'm not sure how much memory it will ultimately use. \nAnd yes, I agree with you, the key difference was using read_row_groups. When I tried reading the entire parquet file, I encountered a memory error.", "votes": 1, "replies": [ { "id": 2484856, "postDate": "2023-10-16T18:07:38.813Z", "content": "

Oops, my bad! I didn't quite make my question clear earlier. I was actually wondering if you've tested this code with the GPU enabled, which is the 13GB of memory on CPU. But anyway, you've pretty much covered my question. I'll give your code a shot on my setup, and hopefully it lets me do my inference on GPU. Thank you !!

", "rawMarkdown": "Oops, my bad! I didn't quite make my question clear earlier. I was actually wondering if you've tested this code with the GPU enabled, which is the 13GB of memory on CPU. But anyway, you've pretty much covered my question. I'll give your code a shot on my setup, and hopefully it lets me do my inference on GPU. Thank you !!", "votes": 1 } ] } ] }, { "id": 2484728, "postDate": "2023-10-16T16:29:05.527Z", "content": "

@mpp1998 This is a very interesting endeavor, but what is the motivation?

", "rawMarkdown": "@mpp1998 This is a very interesting endeavor, but what is the motivation?", "replies": [ { "id": 2484957, "postDate": "2023-10-16T19:34:59.087Z", "content": "

Just to give an idea of how to do inference without memory issues and in a short time. If you spend 1 or 2 hours per model you wont be able to ensemble different models.
\n I've seen comments where people mention that by adding many features for boosting algorithms, they're facing memory problems or consuming the submission time.
\nMoreover, if you don't make this inference series by series and try to load all the data from the hidden test set at once, you can't use GPU because in GPU-enabled notebook you only get 13 GB of RAM and you get memory error when processing the hidden test set.

", "rawMarkdown": "Just to give an idea of how to do inference without memory issues and in a short time. If you spend 1 or 2 hours per model you wont be able to ensemble different models.\n I've seen comments where people mention that by adding many features for boosting algorithms, they're facing memory problems or consuming the submission time. \nMoreover, if you don't make this inference series by series and try to load all the data from the hidden test set at once, you can't use GPU because in GPU-enabled notebook you only get 13 GB of RAM and you get memory error when processing the hidden test set.", "votes": 3, "replies": [ { "id": 2485108, "postDate": "2023-10-17T00:24:31.770Z", "content": "

Thanks @mpp1998, I understood it well. In order to shorten the processing time, I will try implementing it using the above code as a reference.

", "rawMarkdown": "Thanks @mpp1998, I understood it well. In order to shorten the processing time, I will try implementing it using the above code as a reference." } ] } ] } ], "comments": [ { "id": 2484541, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-10-16T14:58:51.357000", "content": "

Good callout, could you dive further into your recommendation of using NN over gradient boosting. Not that I disagree but with a submission time of 18 minutes I would think you could do both and clear memory in between batches/predictions?

", "votes": 1, "replies": [ { "id": 2484562, "author_name": "Manu", "author_url": "", "post_date": "2023-10-16T15:12:39.173000", "content": "

Yes, of course, with this submission time you can use both.
\nOn one hand, by loading the series one by one, you completely eliminate the memory issue. You could use boosting, create multiple features, and clear memory each time.
\nIf you introduce boosting, the problem you might encounter will probably be a time issue due to the need for feature engineering on all the test series, depending on the type and amount of features you compute. This issue is eliminated when using a neural network which requires minimal additional features. Nonetheless, considering it only takes 18 minutes, it can indeed be combined with boosting models.

", "votes": 2, "replies": [ { "id": 2484565, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-10-16T15:14:28.203000", "content": "

Ok I follow you! You never know what someone else may have come up with so I wanted to make sure haha

", "votes": 1, "replies": [] } ] } ] }, { "id": 2487665, "author_name": "Lamal", "author_url": "", "post_date": "2023-10-18T17:48:54.223000", "content": "

Interesting topic @mpp1998, RAM issue was a bottleneck, using nn I believe it’ll get removed. I get a question to raise: how did you model this problem, as segmentation or just time-series forecasting, or others? Thx for replying

", "votes": 2, "replies": [ { "id": 2488382, "author_name": "Manu", "author_url": "", "post_date": "2023-10-19T08:31:43.177000", "content": "

Im modeling it as a segmentation problem. I divide the series into chunks and attempt to segment each chunk into sleep/awake periods. Each chunk is an entry to the neural network, and chunk duration is a parameter: one day, several days etc

", "votes": 4, "replies": [ { "id": 2488469, "author_name": "Lamal", "author_url": "", "post_date": "2023-10-19T09:19:03.973000", "content": "

Agree with you, thanks~

", "votes": 0, "replies": [] }, { "id": 2506538, "author_name": "", "author_url": "", "post_date": "2023-10-31T11:16:42.653000", "content": "", "votes": 0, "replies": [] }, { "id": 2506544, "author_name": "answer-qzd2", "author_url": "", "post_date": "2023-10-31T11:20:55.217000", "content": "

When infer, can the chunk duration only be one day?

", "votes": 0, "replies": [] }, { "id": 2506567, "author_name": "Andreu Arderiu", "author_url": "", "post_date": "2023-10-31T11:50:21.953000", "content": "

For each chunk, do you predict each point as sleep/awake?

", "votes": 0, "replies": [] }, { "id": 2506802, "author_name": "", "author_url": "", "post_date": "2023-10-31T14:54:51.447000", "content": "", "votes": 0, "replies": [] }, { "id": 2506803, "author_name": "Manu", "author_url": "", "post_date": "2023-10-31T14:55:13.467000", "content": "

Yes, it can be one day.

", "votes": 1, "replies": [] }, { "id": 2506806, "author_name": "Manu", "author_url": "", "post_date": "2023-10-31T14:56:10.367000", "content": "

Yes, but you can do it also predicting other targets like in critic points regression

", "votes": 0, "replies": [] } ] } ] }, { "id": 2484559, "author_name": "Yuchen20", "author_url": "", "post_date": "2023-10-16T15:09:32.067000", "content": "

Hey, I've been following a similar approach, initially loading the entire parquet file using Pandas and then performing a group-by operation to split it into smaller DataFrames, which worked well on CPU due to having 30 GB of RAM. However, when I attempted to do the same on GPU, the memory just explodes. I suspect that the grouping operation on the entire dataset is consuming a lot of memory. So, my question is whether you're using this code snippet on a GPU and if it's not causing any memory issue. (I guess it is the read_row_groups that did the trick in saving memory) Thank you in advance!

", "votes": 2, "replies": [ { "id": 2484596, "author_name": "Manu", "author_url": "", "post_date": "2023-10-16T15:30:17.303000", "content": "

I'm running that snippet of code on the CPU (13 GB RAM since it's a notebook with a GPU). For the hidden test set, it's enough and I haven't had to run it on the GPU. I'm not sure how much memory it will ultimately use.
\nAnd yes, I agree with you, the key difference was using read_row_groups. When I tried reading the entire parquet file, I encountered a memory error.

", "votes": 1, "replies": [ { "id": 2484856, "author_name": "Yuchen20", "author_url": "", "post_date": "2023-10-16T18:07:38.813000", "content": "

Oops, my bad! I didn't quite make my question clear earlier. I was actually wondering if you've tested this code with the GPU enabled, which is the 13GB of memory on CPU. But anyway, you've pretty much covered my question. I'll give your code a shot on my setup, and hopefully it lets me do my inference on GPU. Thank you !!

", "votes": 1, "replies": [] } ] } ] }, { "id": 2484728, "author_name": "mh", "author_url": "", "post_date": "2023-10-16T16:29:05.527000", "content": "

@mpp1998 This is a very interesting endeavor, but what is the motivation?

", "votes": 0, "replies": [ { "id": 2484957, "author_name": "Manu", "author_url": "", "post_date": "2023-10-16T19:34:59.087000", "content": "

Just to give an idea of how to do inference without memory issues and in a short time. If you spend 1 or 2 hours per model you wont be able to ensemble different models.
\n I've seen comments where people mention that by adding many features for boosting algorithms, they're facing memory problems or consuming the submission time.
\nMoreover, if you don't make this inference series by series and try to load all the data from the hidden test set at once, you can't use GPU because in GPU-enabled notebook you only get 13 GB of RAM and you get memory error when processing the hidden test set.

", "votes": 3, "replies": [ { "id": 2485108, "author_name": "mh", "author_url": "", "post_date": "2023-10-17T00:24:31.770000", "content": "

Thanks @mpp1998, I understood it well. In order to shorten the processing time, I will try implementing it using the above code as a reference.

", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2484467": "\nI'm going to discuss an approach for quick inference while using minimal memory. In particular, I'll showcase a submission that took 18 minutes to complete and achieved a score of 0.638.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13599107%2Ff9e8fe1a897936d59a15f923e7f30572%2FCaptura1.PNG?generation=1697465197337503&alt=media)\n- Split the series into separate files by series_id.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13599107%2F44099f7db01b876efe45299a15ceebb1%2FCaptura.PNG?generation=1697464954620204&alt=media)\n- Read the series one by one, do minimal feature engineering (use nn aproach insted of gradient boosting), and make inference one by one using models previously trained in another notebook.\nIn this way, we use very little memory and generate predictions very quickly, allowing us to create an ensemble of multiple models, use different models depending on the type of series, and so on.", "2484541": "Good callout, could you dive further into your recommendation of using NN over gradient boosting. Not that I disagree but with a submission time of 18 minutes I would think you could do both and clear memory in between batches/predictions?", "2487665": "Interesting topic @mpp1998, RAM issue was a bottleneck, using nn I believe it’ll get removed. I get a question to raise: how did you model this problem, as segmentation or just time-series forecasting, or others? Thx for replying", "2484559": "Hey, I've been following a similar approach, initially loading the entire parquet file using Pandas and then performing a group-by operation to split it into smaller DataFrames, which worked well on CPU due to having 30 GB of RAM. However, when I attempted to do the same on GPU, the memory just explodes. I suspect that the grouping operation on the entire dataset is consuming a lot of memory. So, my question is whether you're using this code snippet on a GPU and if it's not causing any memory issue. (I guess it is the read_row_groups that did the trick in saving memory) Thank you in advance!", "2484728": "@mpp1998 This is a very interesting endeavor, but what is the motivation?" } }