I've recently started to question the effectiveness of gradient boosting. It feels like I'm stuck in a repetitive cycle of adding more rolling features -> fine-tuning hyperparameters-> submitting -> and seeing little to no improvement. With the extensive number of features I'm utilizing (around 400), I've now constantly encountering memory issues, even though I've implemented batch processing and sampling. And the feature generation is taking around 6 hour with polars. As a result, I'm contemplating trying out some neural network approaches. I'm curious to know how many others are still dedicated to gradient boosting or are exploring alternative methods.
\nIf you could, please vote for the approach you're currently using and, if possible, share your latest CV and LB results.
\nThank you all for your insights!
", "messages": [ { "id": "2479673", "postDate": "10/12/2023 19:17:07", "content": "I've recently started to question the effectiveness of gradient boosting. It feels like I'm stuck in a repetitive cycle of adding more rolling features -> fine-tuning hyperparameters-> submitting -> and seeing little to no improvement. With the extensive number of features I'm utilizing (around 400), I've now constantly encountering memory issues, even though I've implemented batch processing and sampling. And the feature generation is taking around 6 hour with polars. As a result, I'm contemplating trying out some neural network approaches. I'm curious to know how many others are still dedicated to gradient boosting or are exploring alternative methods.
\nIf you could, please vote for the approach you're currently using and, if possible, share your latest CV and LB results.
\nThank you all for your insights!
", "rawMarkdown": "I've recently started to question the effectiveness of gradient boosting. It feels like I'm stuck in a repetitive cycle of adding more rolling features -> fine-tuning hyperparameters-> submitting -> and seeing little to no improvement. With the extensive number of features I'm utilizing (around 400), I've now constantly encountering memory issues, even though I've implemented batch processing and sampling. And the feature generation is taking around 6 hour with polars. As a result, I'm contemplating trying out some neural network approaches. I'm curious to know how many others are still dedicated to gradient boosting or are exploring alternative methods.\n\n**If you could, please vote for the approach you're currently using and, if possible, share your latest CV and LB results.**\n\nThank you all for your insights!", "votes": null }, { "id": "2479674", "postDate": "10/12/2023 19:17:19", "content": "Gradient Boosting (including random forest, xgboost, lightgbm ..)
", "rawMarkdown": "**Gradient Boosting** (including random forest, xgboost, lightgbm ..)", "votes": null }, { "id": "2479675", "postDate": "10/12/2023 19:17:27", "content": "Neural Network
", "rawMarkdown": "**Neural Network**", "votes": null }, { "id": "2479740", "postDate": "10/12/2023 20:33:21", "content": "No machine learning at all CV: 0.41 LB:0.44
", "rawMarkdown": "No machine learning at all CV: 0.41 LB:0.44", "votes": null }, { "id": "2479756", "postDate": "10/12/2023 20:45:48", "content": "Interesting! I presume you've combined some algorithms from those published paper to achieve this result. It would be intriguing to incorporate what you've accomplished into a model and observe if it leads to any enhancements.
", "rawMarkdown": "Interesting! I presume you've combined some algorithms from those published paper to achieve this result. It would be intriguing to incorporate what you've accomplished into a model and observe if it leads to any enhancements.", "votes": null }, { "id": "2479944", "postDate": "10/13/2023 02:44:51", "content": "I guess it is not only decided by which model we used but also how you model this question(i.e. how to define the target for modeling). As we can see from the public notebook, the critical point modeling uses Gru and gets 0.567 which is the highest PB for now. Other GBDT notebooks cannot beat it. However, they used different way to model this question. For most public GBDT model, they try to distinguish 2 statuses: sleep and awake and then get the exact onset/wakeup time. For the GRU notebook, it tries to find the onset/wakeup time directly. I have not explored too much for now. Maybe the way how you model this question will make some difference.
", "rawMarkdown": "I guess it is not only decided by which model we used but also how you model this question(i.e. how to define the target for modeling). As we can see from the public notebook, the critical point modeling uses Gru and gets 0.567 which is the highest PB for now. Other GBDT notebooks cannot beat it. However, they used different way to model this question. For most public GBDT model, they try to distinguish 2 statuses: sleep and awake and then get the exact onset/wakeup time. For the GRU notebook, it tries to find the onset/wakeup time directly. I have not explored too much for now. Maybe the way how you model this question will make some difference.", "votes": null }, { "id": "2479973", "postDate": "10/13/2023 03:31:14", "content": "Ensemble Gradient boosting (XGBoost, Lightgbm, Randomforest): CV 0.556 LB: 0.540, we are using sampled data with 80 features.
\nMy single lightgbm model gives LB:0.524, with the CV:0.531. Similar results have been obtained using the RF and XGBoost. However, upon ensembling, I got LB: 0.540.
\nCurrently, our team is looking for other approaches such as neural networks, etc.
", "rawMarkdown": "Ensemble Gradient boosting (XGBoost, Lightgbm, Randomforest): CV 0.556 LB: 0.540, we are using sampled data with 80 features.\n\nMy single lightgbm model gives LB:0.524, with the CV:0.531. Similar results have been obtained using the RF and XGBoost. However, upon ensembling, I got LB: 0.540.\n\nCurrently, our team is looking for other approaches such as neural networks, etc.", "votes": null }, { "id": "2479982", "postDate": "10/13/2023 03:41:33", "content": "Agreed.
\nHere I am sharing some of my observations.
\nTo my understanding, the other main thing is the post-processing and dealing with the missing timestamps for the target, which could enhance the model performance.
\nMy conjecture is that we have time-based features in our training dataset, and for some series, we have quite a periodic behavior in the target, and for others, this behaviour could exist but we do not know the exact labels due to missing timestamp. These periodic behaviors tend to guide the model to predict the periodicity for those series for this might not exist in the test dataset, and this will be counted as a false positive. Dealing with such false positives in the predictions could improve the model performance significantly.
", "rawMarkdown": "Agreed.\n\nHere I am sharing some of my observations.\n\nTo my understanding, the other main thing is the post-processing and dealing with the missing timestamps for the target, which could enhance the model performance.\n\nMy conjecture is that we have time-based features in our training dataset, and for some series, we have quite a periodic behavior in the target, and for others, this behaviour could exist but we do not know the exact labels due to missing timestamp. These periodic behaviors tend to guide the model to predict the periodicity for those series for this might not exist in the test dataset, and this will be counted as a false positive. Dealing with such false positives in the predictions could improve the model performance significantly.", "votes": null }, { "id": "2480029", "postDate": "10/13/2023 05:09:34", "content": "Totally agreed. Post-processing is an essential part.
", "rawMarkdown": "Totally agreed. Post-processing is an essential part.", "votes": null }, { "id": "2484152", "postDate": "10/16/2023 09:13:00", "content": "I have not really read any papers so I don't know if any of them use the same approach. The main problem I have is that with no ML approach I do not have a way of generating scores so I just set the score to 1 which I think greatly reduces the performance. To overcome this I will still have to use ML so I think yes, things can be improved farther by incorporating both approaches and coming back to your original question I think neural network is the way to go.
", "rawMarkdown": "I have not really read any papers so I don't know if any of them use the same approach. The main problem I have is that with no ML approach I do not have a way of generating scores so I just set the score to 1 which I think greatly reduces the performance. To overcome this I will still have to use ML so I think yes, things can be improved farther by incorporating both approaches and coming back to your original question I think neural network is the way to go.", "votes": null }, { "id": "2484327", "postDate": "10/16/2023 12:03:18", "content": "RNN, best result: LB: 0.627 / CV: 0.537
", "rawMarkdown": "RNN, best result: LB: 0.627 / CV: 0.537", "votes": null }, { "id": "2484921", "postDate": "10/16/2023 19:01:14", "content": "Hi @g0lski . I don't recall seeing a direcxt comparison of with-vs-without score LB values, so I'm not sure how large an effect it is. But in any case, you don't need ML to make score values: the values are arbitrary with higher values indicating events that you think are more accurately predicted. So, if you have a non-ML way to flag events that your non-ML method usually has trouble with, then you can assign those events a lower score than the other events (say 0.5 vs 1.0; but the actual values don't matter, just their order.) Of course, this is just a simple start to having ML look at the errors and the features around the events and then learn which events are more or less trustworthy and assign a score to them 🙂
I agree with you @aeyeee , for GBDT is essential to have a good post-processing
", "rawMarkdown": "I agree with you @aeyeee , for GBDT is essential to have a good post-processing", "votes": null }, { "id": "2485117", "postDate": "10/17/2023 00:30:24", "content": "Neural net with post-processing.
\nLB:
Thanks for your insight! Indeed scores don't matter as long as predictions that fall in the same tolerance brackets are scored similarly but unfortunately my no-ml method has no way of flagging events that it has trouble predicting.
", "rawMarkdown": "Thanks for your insight! Indeed scores don't matter as long as predictions that fall in the same tolerance brackets are scored similarly but unfortunately my no-ml method has no way of flagging events that it has trouble predicting.", "votes": null }, { "id": "2486216", "postDate": "10/17/2023 18:56:30", "content": "Since it seems there are easier and harder to predict series, you could assign the score on a series basis: if you have some metric that signals \"problem series\" then all events in that series could have decreased scores compared to \"good\" series.
\n\nif you have some metric that signals \"problem series\" then all events in that series could have decreased scores compared to \"good\" series.
\n
Would this make a difference to the overall Average Precision (AP) score? I mean, for the same series and with the same predictions, is AP different based on whether the value of score is high or low?
Are you trying to fit a Gaussian to the onset/wake features, then treat it as a cv task?
", "rawMarkdown": "Are you trying to fit a Gaussian to the onset/wake features, then treat it as a cv task?", "votes": null }, { "id": "3014895", "postDate": "10/11/2024 17:27:03", "content": "Hey Yuchen, I was wondering if you would be interested in exchanging ideas about ARC 2024 competition.
", "rawMarkdown": "Hey Yuchen, I was wondering if you would be interested in exchanging ideas about ARC 2024 competition.", "votes": null } ], "comments": [ { "id": 2479674, "author_name": "yuchen2066", "author_url": "", "post_date": "10/12/2023 19:17:19", "content": "Gradient Boosting (including random forest, xgboost, lightgbm ..)
", "votes": null, "replies": [] }, { "id": 2479675, "author_name": "yuchen2066", "author_url": "", "post_date": "10/12/2023 19:17:27", "content": "Neural Network
", "votes": null, "replies": [] }, { "id": 2479740, "author_name": "g0lski", "author_url": "", "post_date": "10/12/2023 20:33:21", "content": "No machine learning at all CV: 0.41 LB:0.44
", "votes": null, "replies": [ { "id": 2479756, "author_name": "yuchen2066", "author_url": "", "post_date": "10/12/2023 20:45:48", "content": "Interesting! I presume you've combined some algorithms from those published paper to achieve this result. It would be intriguing to incorporate what you've accomplished into a model and observe if it leads to any enhancements.
", "votes": null, "replies": [ { "id": 2484152, "author_name": "g0lski", "author_url": "", "post_date": "10/16/2023 09:13:00", "content": "I have not really read any papers so I don't know if any of them use the same approach. The main problem I have is that with no ML approach I do not have a way of generating scores so I just set the score to 1 which I think greatly reduces the performance. To overcome this I will still have to use ML so I think yes, things can be improved farther by incorporating both approaches and coming back to your original question I think neural network is the way to go.
", "votes": null, "replies": [ { "id": 2484921, "author_name": "dan3dewey", "author_url": "", "post_date": "10/16/2023 19:01:14", "content": "Hi @g0lski . I don't recall seeing a direcxt comparison of with-vs-without score LB values, so I'm not sure how large an effect it is. But in any case, you don't need ML to make score values: the values are arbitrary with higher values indicating events that you think are more accurately predicted. So, if you have a non-ML way to flag events that your non-ML method usually has trouble with, then you can assign those events a lower score than the other events (say 0.5 vs 1.0; but the actual values don't matter, just their order.) Of course, this is just a simple start to having ML look at the errors and the features around the events and then learn which events are more or less trustworthy and assign a score to them 🙂
Thanks for your insight! Indeed scores don't matter as long as predictions that fall in the same tolerance brackets are scored similarly but unfortunately my no-ml method has no way of flagging events that it has trouble predicting.
", "votes": null, "replies": [ { "id": 2486216, "author_name": "dan3dewey", "author_url": "", "post_date": "10/17/2023 18:56:30", "content": "Since it seems there are easier and harder to predict series, you could assign the score on a series basis: if you have some metric that signals \"problem series\" then all events in that series could have decreased scores compared to \"good\" series.
\n\nif you have some metric that signals \"problem series\" then all events in that series could have decreased scores compared to \"good\" series.
\n
Would this make a difference to the overall Average Precision (AP) score? I mean, for the same series and with the same predictions, is AP different based on whether the value of score is high or low?
I guess it is not only decided by which model we used but also how you model this question(i.e. how to define the target for modeling). As we can see from the public notebook, the critical point modeling uses Gru and gets 0.567 which is the highest PB for now. Other GBDT notebooks cannot beat it. However, they used different way to model this question. For most public GBDT model, they try to distinguish 2 statuses: sleep and awake and then get the exact onset/wakeup time. For the GRU notebook, it tries to find the onset/wakeup time directly. I have not explored too much for now. Maybe the way how you model this question will make some difference.
", "votes": null, "replies": [ { "id": 2479982, "author_name": "aeyeee", "author_url": "", "post_date": "10/13/2023 03:41:33", "content": "Agreed.
\nHere I am sharing some of my observations.
\nTo my understanding, the other main thing is the post-processing and dealing with the missing timestamps for the target, which could enhance the model performance.
\nMy conjecture is that we have time-based features in our training dataset, and for some series, we have quite a periodic behavior in the target, and for others, this behaviour could exist but we do not know the exact labels due to missing timestamp. These periodic behaviors tend to guide the model to predict the periodicity for those series for this might not exist in the test dataset, and this will be counted as a false positive. Dealing with such false positives in the predictions could improve the model performance significantly.
", "votes": null, "replies": [ { "id": 2480029, "author_name": "hookman", "author_url": "", "post_date": "10/13/2023 05:09:34", "content": "Totally agreed. Post-processing is an essential part.
", "votes": null, "replies": [] }, { "id": 2485099, "author_name": "renatoreggiani", "author_url": "", "post_date": "10/17/2023 00:16:17", "content": "I agree with you @aeyeee , for GBDT is essential to have a good post-processing
", "votes": null, "replies": [] } ] } ] }, { "id": 2479973, "author_name": "aeyeee", "author_url": "", "post_date": "10/13/2023 03:31:14", "content": "Ensemble Gradient boosting (XGBoost, Lightgbm, Randomforest): CV 0.556 LB: 0.540, we are using sampled data with 80 features.
\nMy single lightgbm model gives LB:0.524, with the CV:0.531. Similar results have been obtained using the RF and XGBoost. However, upon ensembling, I got LB: 0.540.
\nCurrently, our team is looking for other approaches such as neural networks, etc.
", "votes": null, "replies": [] }, { "id": 2484327, "author_name": "atamazian", "author_url": "", "post_date": "10/16/2023 12:03:18", "content": "RNN, best result: LB: 0.627 / CV: 0.537
", "votes": null, "replies": [ { "id": 2488195, "author_name": "liushundawang", "author_url": "", "post_date": "10/19/2023 05:44:44", "content": "Are you trying to fit a Gaussian to the onset/wake features, then treat it as a cv task?
", "votes": null, "replies": [] } ] }, { "id": 2485117, "author_name": "nghihuynh", "author_url": "", "post_date": "10/17/2023 00:30:24", "content": "Neural net with post-processing.
\nLB:
Hey Yuchen, I was wondering if you would be interested in exchanging ideas about ARC 2024 competition.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2479673": "I've recently started to question the effectiveness of gradient boosting. It feels like I'm stuck in a repetitive cycle of adding more rolling features -> fine-tuning hyperparameters-> submitting -> and seeing little to no improvement. With the extensive number of features I'm utilizing (around 400), I've now constantly encountering memory issues, even though I've implemented batch processing and sampling. And the feature generation is taking around 6 hour with polars. As a result, I'm contemplating trying out some neural network approaches. I'm curious to know how many others are still dedicated to gradient boosting or are exploring alternative methods.\n\n**If you could, please vote for the approach you're currently using and, if possible, share your latest CV and LB results.**\n\nThank you all for your insights!", "2479674": "**Gradient Boosting** (including random forest, xgboost, lightgbm ..)", "2479675": "**Neural Network**", "2479740": "No machine learning at all CV: 0.41 LB:0.44", "2479756": "Interesting! I presume you've combined some algorithms from those published paper to achieve this result. It would be intriguing to incorporate what you've accomplished into a model and observe if it leads to any enhancements.", "2479944": "I guess it is not only decided by which model we used but also how you model this question(i.e. how to define the target for modeling). As we can see from the public notebook, the critical point modeling uses Gru and gets 0.567 which is the highest PB for now. Other GBDT notebooks cannot beat it. However, they used different way to model this question. For most public GBDT model, they try to distinguish 2 statuses: sleep and awake and then get the exact onset/wakeup time. For the GRU notebook, it tries to find the onset/wakeup time directly. I have not explored too much for now. Maybe the way how you model this question will make some difference.", "2479973": "Ensemble Gradient boosting (XGBoost, Lightgbm, Randomforest): CV 0.556 LB: 0.540, we are using sampled data with 80 features.\n\nMy single lightgbm model gives LB:0.524, with the CV:0.531. Similar results have been obtained using the RF and XGBoost. However, upon ensembling, I got LB: 0.540.\n\nCurrently, our team is looking for other approaches such as neural networks, etc.", "2479982": "Agreed.\n\nHere I am sharing some of my observations.\n\nTo my understanding, the other main thing is the post-processing and dealing with the missing timestamps for the target, which could enhance the model performance.\n\nMy conjecture is that we have time-based features in our training dataset, and for some series, we have quite a periodic behavior in the target, and for others, this behaviour could exist but we do not know the exact labels due to missing timestamp. These periodic behaviors tend to guide the model to predict the periodicity for those series for this might not exist in the test dataset, and this will be counted as a false positive. Dealing with such false positives in the predictions could improve the model performance significantly.", "2480029": "Totally agreed. Post-processing is an essential part.", "2484152": "I have not really read any papers so I don't know if any of them use the same approach. The main problem I have is that with no ML approach I do not have a way of generating scores so I just set the score to 1 which I think greatly reduces the performance. To overcome this I will still have to use ML so I think yes, things can be improved farther by incorporating both approaches and coming back to your original question I think neural network is the way to go.", "2484327": "RNN, best result: LB: 0.627 / CV: 0.537", "2484921": "Hi @g0lski . I don't recall seeing a direcxt comparison of with-vs-without `score` LB values, so I'm not sure how large an effect it is. But in any case, you don't _need_ ML to make `score` values: the values are arbitrary with higher values indicating events that you think are more accurately predicted. So, if you have a non-ML way to flag events that your non-ML method usually has trouble with, then you can assign those events a lower `score` than the other events (say 0.5 vs 1.0; but the actual values don't matter, just their order.) Of course, this is just a simple start to having ML look at the errors and the features around the events and then learn which events are more or less trustworthy and assign a score to them 🙂", "2485099": "I agree with you @aeyeee , for GBDT is essential to have a good post-processing", "2485117": "Neural net with post-processing. \nLB: \n* Without post-processing: 0.383\n* With post-processing: 0.455", "2485643": "Thanks for your insight! Indeed scores don't matter as long as predictions that fall in the same tolerance brackets are scored similarly but unfortunately my no-ml method has no way of flagging events that it has trouble predicting.", "2486216": "Since it seems there are easier and harder to predict series, you could assign the `score` on a series basis: if you have some metric that signals \"problem series\" then all events in that series could have decreased scores compared to \"good\" series.", "2488173": "> if you have some metric that signals \"problem series\" then all events in that series could have decreased scores compared to \"good\" series.\n\nWould this make a difference to the overall Average Precision (AP) score? I mean, for the same series and with the same predictions, is AP different based on whether the value of `score` is high or low?", "2488195": "Are you trying to fit a Gaussian to the onset/wake features, then treat it as a cv task?", "3014895": "Hey Yuchen, I was wondering if you would be interested in exchanging ideas about ARC 2024 competition." }, "source": "meta" }