{ "id": 442004, "title": "Batch training for tree-based models", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/442004", "author_name": "", "post_date": "2023-09-21T00:22:50.785830900Z", "votes": 6, "comment_count": 12, "views": 0, "content": "

Anyone have experience training tree-based model batch by batch?

\n

I am planning to train my LGBM model by one series ID per step, but do not know whether it is possible to do. I have read some of the public discussions about batch inference, but found none for the batch training.

\n

I suspect that doing thing like

\n
model = ()\n series_id  train_series():\n  sample = train_series()\n  model(sample)\n
\n

will only reset the model each time it is fed with new sample.

", "messages": [ { "id": "2449017", "postDate": "09/21/2023 00:22:50", "content": "

Anyone have experience training tree-based model batch by batch?

\n

I am planning to train my LGBM model by one series ID per step, but do not know whether it is possible to do. I have read some of the public discussions about batch inference, but found none for the batch training.

\n

I suspect that doing thing like

\n
model = ()\n series_id  train_series():\n  sample = train_series()\n  model(sample)\n
\n

will only reset the model each time it is fed with new sample.

", "rawMarkdown": "Anyone have experience training tree-based model batch by batch?\n\nI am planning to train my LGBM model by one series ID per step, but do not know whether it is possible to do. I have read some of the public discussions about batch inference, but found none for the batch training.\n\nI suspect that doing thing like\n```\nmodel = LGBM()\nfor series_id in train_series.series_id.unique():\n sample = train_series.query(\"series_id == @series_id\")\n model.fit(sample)\n```\nwill only reset the model each time it is fed with new sample.", "votes": null }, { "id": "2449726", "postDate": "09/21/2023 11:41:09", "content": "

I was thinking about the same thing, and yesterday I found that the fit function of LGBMClassifier have a \"init_model\" parameter, which is defined as

\n
init_model : , pathlib.Path, Booster, LGBMModel  , optional (default=)\nFilename of LightGBM model, Booster instance  LGBMModel instance used   training.\n
\n

I think it allows us to init the model with previously trained model, and update it with the new data. Haven’t have time to try this for now, but I think it is worth taking a shot. If it is working, please let me know.

", "rawMarkdown": "I was thinking about the same thing, and yesterday I found that the fit function of LGBMClassifier have a [\"init_model\" ](https://lightgbm.readthedocs.io/en/latest/_modules/lightgbm/sklearn.html#LGBMClassifier.fit) parameter, which is defined as\n\n```python\ninit_model : str, pathlib.Path, Booster, LGBMModel or None, optional (default=None)\nFilename of LightGBM model, Booster instance or LGBMModel instance used for continue training.\n```\n\nI think it allows us to init the model with previously trained model, and update it with the new data. Haven’t have time to try this for now, but I think it is worth taking a shot. If it is working, please let me know.", "votes": null }, { "id": "2449791", "postDate": "09/21/2023 12:45:02", "content": "

Hi @yuchen2066 , nice catch

\n

I wonder if I set n_estimators=10 in the continuous training, will the second fit do another 10 iterations or what (so if we have 200 series will it be 2000 estimators?). I will try it and keep you posted on this. Thanks!

", "rawMarkdown": "Hi @yuchen2066 , nice catch\n\nI wonder if I set `n_estimators=10` in the continuous training, will the second fit do another 10 iterations or what (so if we have 200 series will it be 2000 estimators?). I will try it and keep you posted on this. Thanks!", "votes": null }, { "id": "2450044", "postDate": "09/21/2023 15:23:17", "content": "

Good question @rimbax ! I think that based on how decision trees work I'm not sure if you can actually do this. Decision trees need the whole features domain in order to build the partitions based on a certain criteria (Gini index, entropy…). So in my opinion is not possible to train a lgb model in batches.

\n

Again, this is my opinion and if I'm wrong please correct me :)

", "rawMarkdown": "Good question @rimbax ! I think that based on how decision trees work I'm not sure if you can actually do this. Decision trees need the whole features domain in order to build the partitions based on a certain criteria (Gini index, entropy...). So in my opinion is not possible to train a lgb model in batches.\n\nAgain, this is my opinion and if I'm wrong please correct me :)", "votes": null }, { "id": "2450106", "postDate": "09/21/2023 16:02:48", "content": "

Hi @jbowski , thank you. Based on my shallow knowledge, it is true if we consider a single decision tree.

\n

But in essence, LGBM (or XGBoost or gradient boosting in general) uses multiple trees sequentially, so it is theoretically possible to do batch training with it. The remaining thing that matter is the implementation. I haven't done any research on it so yea, will try it sometimes.

", "rawMarkdown": "Hi @jbowski , thank you. Based on my shallow knowledge, it is true if we consider a single decision tree.\n\nBut in essence, LGBM (or XGBoost or gradient boosting in general) uses multiple trees sequentially, so it is theoretically possible to do batch training with it. The remaining thing that matter is the implementation. I haven't done any research on it so yea, will try it sometimes.", "votes": null }, { "id": "2451147", "postDate": "09/22/2023 10:41:16", "content": "

Hey @rimbax ! I was having a look on what you and @yuchen2066 mentioned and I think one possibility to train lgb in batches could be the following one:

\n
\nmodel = lgb.LGBMClassifier\n\n\nfor X_, y_ in batches:\n\n    \n    model.fit\n
\n

And based on what I read, after the first iteration you will have 10 estimators, then the second one 20 and so on. With this you can increase model complexity gradually.

\n

Keep in mind that this approach assumes that your data can be divided into batches without significant loss of information.

\n

As always, correct me if I'm wrong or if I misunderstood something :)

", "rawMarkdown": "Hey @rimbax ! I was having a look on what you and @yuchen2066 mentioned and I think one possibility to train lgb in batches could be the following one:\n\n```\n# Initialize the model\nmodel = lgb.LGBMClassifier(n_estimators=10)\n\n# Sequential training\nfor X_batch, y_batch in batches:\n \n # Update the model with the current batch\n model.fit(X_batch, y_batch, init_model=model)\n```\n\nAnd based on what I read, after the first iteration you will have 10 estimators, then the second one 20 and so on. With this you can increase model complexity gradually.\n\nKeep in mind that this approach assumes that your data can be divided into batches without significant loss of information.\n\nAs always, correct me if I'm wrong or if I misunderstood something :)", "votes": null }, { "id": "2453632", "postDate": "09/24/2023 07:59:55", "content": "
\n

I am planning to train my LGBM model by one series ID per step

\n
\n

Please let me know as I don't quite understand. Is this the intention to create a model for each series ID? I was concerned because the series ID is different between the train and test data.

", "rawMarkdown": "> I am planning to train my LGBM model by one series ID per step\n\nPlease let me know as I don't quite understand. Is this the intention to create a model for each series ID? I was concerned because the series ID is different between the train and test data.", "votes": null }, { "id": "2453649", "postDate": "09/24/2023 08:27:55", "content": "

Hi @mitsuyasuhoshino

\n

Based on previous comments in this discussion, instead of training one model for each series ID, basically what I want to do is something like this instead

\n
\nmodel = lgb.LGBMClassifier(n_estimators=)\n\n\n series_id  unique_series_ids:\n    data = train_series[train_series.series_id == series_id]\n\n    \n    model.fit(data[features], data.event, init_model=model)\n
\n

Note that this will only apply when we are training the model, so it has nothing to do with hidden test data. We can straight forward do inference on the test set.

", "rawMarkdown": "Hi @mitsuyasuhoshino \n\nBased on previous comments in this discussion, instead of training one model for each series ID, basically what I want to do is something like this instead\n\n```python\n# Initialize the model\nmodel = lgb.LGBMClassifier(n_estimators=10)\n\n# Sequential training\nfor series_id in unique_series_ids:\n data = train_series[train_series.series_id == series_id]\n\n # Update the model with the current batch\n model.fit(data[features], data.event, init_model=model)\n```\n\nNote that this will only apply when we are training the model, so it has nothing to do with hidden test data. We can straight forward do inference on the test set.", "votes": null }, { "id": "2453673", "postDate": "09/24/2023 08:46:25", "content": "

Thanks @rimbax.

\n
\n

Note that this will only apply when we are training the model, so it has nothing to do with hidden test data.

\n
\n

I understood.

\n
\n

We can straight forward do inference on the test set.

\n
\n

Is it okay to apply a model trained only on one specific series ID to a different series ID of the test data?

", "rawMarkdown": "Thanks @rimbax.\n\n> Note that this will only apply when we are training the model, so it has nothing to do with hidden test data.\n\nI understood.\n\n> We can straight forward do inference on the test set.\n\nIs it okay to apply a model trained only on one specific series ID to a different series ID of the test data?", "votes": null }, { "id": "2453676", "postDate": "09/24/2023 08:47:51", "content": "

I think it's not. Generally the more data we used to train (the more series ID you used in train set), the better model we will get in terms of generalisability.

", "rawMarkdown": "I think it's not. Generally the more data we used to train (the more series ID you used in train set), the better model we will get in terms of generalisability.", "votes": null }, { "id": "2453682", "postDate": "09/24/2023 08:53:17", "content": "

If so, what is the intention of doing this batch learning?

", "rawMarkdown": "If so, what is the intention of doing this batch learning?", "votes": null }, { "id": "2453869", "postDate": "09/24/2023 11:36:36", "content": "

To involve more pattern across different series ID. This actually adapted concept from deep learning where one can continue their model training at some point, CMIIW.

", "rawMarkdown": "To involve more pattern across different series ID. This actually adapted concept from deep learning where one can continue their model training at some point, CMIIW.", "votes": null }, { "id": "2454075", "postDate": "09/24/2023 15:01:20", "content": "

Ah I see, I will clarify this once I start doing this batch training. Many thanks!

", "rawMarkdown": "Ah I see, I will clarify this once I start doing this batch training. Many thanks!", "votes": null } ], "comments": [ { "id": 2449726, "author_name": "yuchen2066", "author_url": "", "post_date": "09/21/2023 11:41:09", "content": "

I was thinking about the same thing, and yesterday I found that the fit function of LGBMClassifier have a \"init_model\" parameter, which is defined as

\n
init_model : , pathlib.Path, Booster, LGBMModel  , optional (default=)\nFilename of LightGBM model, Booster instance  LGBMModel instance used   training.\n
\n

I think it allows us to init the model with previously trained model, and update it with the new data. Haven’t have time to try this for now, but I think it is worth taking a shot. If it is working, please let me know.

", "votes": null, "replies": [ { "id": 2449791, "author_name": "rimbax", "author_url": "", "post_date": "09/21/2023 12:45:02", "content": "

Hi @yuchen2066 , nice catch

\n

I wonder if I set n_estimators=10 in the continuous training, will the second fit do another 10 iterations or what (so if we have 200 series will it be 2000 estimators?). I will try it and keep you posted on this. Thanks!

", "votes": null, "replies": [] } ] }, { "id": 2450044, "author_name": "jbowski", "author_url": "", "post_date": "09/21/2023 15:23:17", "content": "

Good question @rimbax ! I think that based on how decision trees work I'm not sure if you can actually do this. Decision trees need the whole features domain in order to build the partitions based on a certain criteria (Gini index, entropy…). So in my opinion is not possible to train a lgb model in batches.

\n

Again, this is my opinion and if I'm wrong please correct me :)

", "votes": null, "replies": [ { "id": 2450106, "author_name": "rimbax", "author_url": "", "post_date": "09/21/2023 16:02:48", "content": "

Hi @jbowski , thank you. Based on my shallow knowledge, it is true if we consider a single decision tree.

\n

But in essence, LGBM (or XGBoost or gradient boosting in general) uses multiple trees sequentially, so it is theoretically possible to do batch training with it. The remaining thing that matter is the implementation. I haven't done any research on it so yea, will try it sometimes.

", "votes": null, "replies": [ { "id": 2451147, "author_name": "jbowski", "author_url": "", "post_date": "09/22/2023 10:41:16", "content": "

Hey @rimbax ! I was having a look on what you and @yuchen2066 mentioned and I think one possibility to train lgb in batches could be the following one:

\n
\nmodel = lgb.LGBMClassifier\n\n\nfor X_, y_ in batches:\n\n    \n    model.fit\n
\n

And based on what I read, after the first iteration you will have 10 estimators, then the second one 20 and so on. With this you can increase model complexity gradually.

\n

Keep in mind that this approach assumes that your data can be divided into batches without significant loss of information.

\n

As always, correct me if I'm wrong or if I misunderstood something :)

", "votes": null, "replies": [ { "id": 2454075, "author_name": "rimbax", "author_url": "", "post_date": "09/24/2023 15:01:20", "content": "

Ah I see, I will clarify this once I start doing this batch training. Many thanks!

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2453632, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/24/2023 07:59:55", "content": "
\n

I am planning to train my LGBM model by one series ID per step

\n
\n

Please let me know as I don't quite understand. Is this the intention to create a model for each series ID? I was concerned because the series ID is different between the train and test data.

", "votes": null, "replies": [ { "id": 2453649, "author_name": "rimbax", "author_url": "", "post_date": "09/24/2023 08:27:55", "content": "

Hi @mitsuyasuhoshino

\n

Based on previous comments in this discussion, instead of training one model for each series ID, basically what I want to do is something like this instead

\n
\nmodel = lgb.LGBMClassifier(n_estimators=)\n\n\n series_id  unique_series_ids:\n    data = train_series[train_series.series_id == series_id]\n\n    \n    model.fit(data[features], data.event, init_model=model)\n
\n

Note that this will only apply when we are training the model, so it has nothing to do with hidden test data. We can straight forward do inference on the test set.

", "votes": null, "replies": [ { "id": 2453673, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/24/2023 08:46:25", "content": "

Thanks @rimbax.

\n
\n

Note that this will only apply when we are training the model, so it has nothing to do with hidden test data.

\n
\n

I understood.

\n
\n

We can straight forward do inference on the test set.

\n
\n

Is it okay to apply a model trained only on one specific series ID to a different series ID of the test data?

", "votes": null, "replies": [ { "id": 2453676, "author_name": "rimbax", "author_url": "", "post_date": "09/24/2023 08:47:51", "content": "

I think it's not. Generally the more data we used to train (the more series ID you used in train set), the better model we will get in terms of generalisability.

", "votes": null, "replies": [ { "id": 2453682, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/24/2023 08:53:17", "content": "

If so, what is the intention of doing this batch learning?

", "votes": null, "replies": [ { "id": 2453869, "author_name": "rimbax", "author_url": "", "post_date": "09/24/2023 11:36:36", "content": "

To involve more pattern across different series ID. This actually adapted concept from deep learning where one can continue their model training at some point, CMIIW.

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2449017": "Anyone have experience training tree-based model batch by batch?\n\nI am planning to train my LGBM model by one series ID per step, but do not know whether it is possible to do. I have read some of the public discussions about batch inference, but found none for the batch training.\n\nI suspect that doing thing like\n```\nmodel = LGBM()\nfor series_id in train_series.series_id.unique():\n sample = train_series.query(\"series_id == @series_id\")\n model.fit(sample)\n```\nwill only reset the model each time it is fed with new sample.", "2449726": "I was thinking about the same thing, and yesterday I found that the fit function of LGBMClassifier have a [\"init_model\" ](https://lightgbm.readthedocs.io/en/latest/_modules/lightgbm/sklearn.html#LGBMClassifier.fit) parameter, which is defined as\n\n```python\ninit_model : str, pathlib.Path, Booster, LGBMModel or None, optional (default=None)\nFilename of LightGBM model, Booster instance or LGBMModel instance used for continue training.\n```\n\nI think it allows us to init the model with previously trained model, and update it with the new data. Haven’t have time to try this for now, but I think it is worth taking a shot. If it is working, please let me know.", "2449791": "Hi @yuchen2066 , nice catch\n\nI wonder if I set `n_estimators=10` in the continuous training, will the second fit do another 10 iterations or what (so if we have 200 series will it be 2000 estimators?). I will try it and keep you posted on this. Thanks!", "2450044": "Good question @rimbax ! I think that based on how decision trees work I'm not sure if you can actually do this. Decision trees need the whole features domain in order to build the partitions based on a certain criteria (Gini index, entropy...). So in my opinion is not possible to train a lgb model in batches.\n\nAgain, this is my opinion and if I'm wrong please correct me :)", "2450106": "Hi @jbowski , thank you. Based on my shallow knowledge, it is true if we consider a single decision tree.\n\nBut in essence, LGBM (or XGBoost or gradient boosting in general) uses multiple trees sequentially, so it is theoretically possible to do batch training with it. The remaining thing that matter is the implementation. I haven't done any research on it so yea, will try it sometimes.", "2451147": "Hey @rimbax ! I was having a look on what you and @yuchen2066 mentioned and I think one possibility to train lgb in batches could be the following one:\n\n```\n# Initialize the model\nmodel = lgb.LGBMClassifier(n_estimators=10)\n\n# Sequential training\nfor X_batch, y_batch in batches:\n \n # Update the model with the current batch\n model.fit(X_batch, y_batch, init_model=model)\n```\n\nAnd based on what I read, after the first iteration you will have 10 estimators, then the second one 20 and so on. With this you can increase model complexity gradually.\n\nKeep in mind that this approach assumes that your data can be divided into batches without significant loss of information.\n\nAs always, correct me if I'm wrong or if I misunderstood something :)", "2453632": "> I am planning to train my LGBM model by one series ID per step\n\nPlease let me know as I don't quite understand. Is this the intention to create a model for each series ID? I was concerned because the series ID is different between the train and test data.", "2453649": "Hi @mitsuyasuhoshino \n\nBased on previous comments in this discussion, instead of training one model for each series ID, basically what I want to do is something like this instead\n\n```python\n# Initialize the model\nmodel = lgb.LGBMClassifier(n_estimators=10)\n\n# Sequential training\nfor series_id in unique_series_ids:\n data = train_series[train_series.series_id == series_id]\n\n # Update the model with the current batch\n model.fit(data[features], data.event, init_model=model)\n```\n\nNote that this will only apply when we are training the model, so it has nothing to do with hidden test data. We can straight forward do inference on the test set.", "2453673": "Thanks @rimbax.\n\n> Note that this will only apply when we are training the model, so it has nothing to do with hidden test data.\n\nI understood.\n\n> We can straight forward do inference on the test set.\n\nIs it okay to apply a model trained only on one specific series ID to a different series ID of the test data?", "2453676": "I think it's not. Generally the more data we used to train (the more series ID you used in train set), the better model we will get in terms of generalisability.", "2453682": "If so, what is the intention of doing this batch learning?", "2453869": "To involve more pattern across different series ID. This actually adapted concept from deep learning where one can continue their model training at some point, CMIIW.", "2454075": "Ah I see, I will clarify this once I start doing this batch training. Many thanks!" }, "source": "meta" }