When I joined this competition, my first hypothesis of a good predictor was to have one model per building. Theoretically, this seems to make sense since each building has a different behavior. However, when trying in practice, I got very bad LB scores (higher than 2). The score is worse than simply predicting the average per building (1.64). Did anyone else try as well? Any idea why it turns out to be a bad idea?
", "messages": [ { "id": 690722, "postDate": "2019-12-09T02:20:10.930Z", "content": "When I joined this competition, my first hypothesis of a good predictor was to have one model per building. Theoretically, this seems to make sense since each building has a different behavior. However, when trying in practice, I got very bad LB scores (higher than 2). The score is worse than simply predicting the average per building (1.64). Did anyone else try as well? Any idea why it turns out to be a bad idea?
", "rawMarkdown": "When I joined this competition, my first hypothesis of a good predictor was to have one model per building. Theoretically, this seems to make sense since each building has a different behavior. However, when trying in practice, I got very bad LB scores (higher than 2). The score is worse than simply predicting the average per building (1.64). Did anyone else try as well? Any idea why it turns out to be a bad idea?", "votes": 5 }, { "id": 693204, "postDate": "2019-12-12T06:17:24.510Z", "content": "I've manage to score 1.07 (public LB) with per-building models. More precisely, and this might be part of your problem, I used per-building and per-meter (2380 models instead of 1449). I used GBDTs, and did use different hyperparameters than I would use for a single global model.
", "rawMarkdown": "I've manage to score 1.07 (public LB) with per-building models. More precisely, and this might be part of your problem, I used per-building *and* per-meter (2380 models instead of 1449). I used GBDTs, and *did* use different hyperparameters than I would use for a single global model.\n", "votes": 6, "replies": [ { "id": 693683, "postDate": "2019-12-12T17:09:30.813Z", "content": "Wow, congrats! That will bring some hope for me to keep trying. I also tried both per bldg and per meter models with no success. The problem seems to be overfitting.
", "rawMarkdown": "Wow, congrats! That will bring some hope for me to keep trying. I also tried both per bldg and per meter models with no success. The problem seems to be overfitting." } ] }, { "id": 690871, "postDate": "2019-12-09T09:18:33.220Z", "content": "You're doing something wrong. For me it works good.
", "rawMarkdown": "You're doing something wrong. For me it works good.", "votes": 3, "replies": [ { "id": 692943, "postDate": "2019-12-11T21:40:05.570Z", "content": "Thanks for the comment @oleg90777 ! Did you use NNs?
", "rawMarkdown": "Thanks for the comment @oleg90777 ! Did you use NNs?" }, { "id": 693255, "postDate": "2019-12-12T07:27:47.043Z", "content": "At the moment only lgb
", "rawMarkdown": "At the moment only lgb" } ] }, { "id": 692437, "postDate": "2019-12-11T08:57:09.020Z", "content": "If you are doing a linear model with a lot of features and no regularization, chances are you are overfitting a lot and may have very large coefficients which will result in crazy predictions on the test set. You should try plotting your predictions to get a feel of what is going on. They should at least bear some resemblance to the data from 2016.
", "rawMarkdown": "If you are doing a linear model with a lot of features and no regularization, chances are you are overfitting a lot and may have very large coefficients which will result in crazy predictions on the test set. You should try plotting your predictions to get a feel of what is going on. They should at least bear some resemblance to the data from 2016.", "votes": 1 }, { "id": 691143, "postDate": "2019-12-09T17:39:14.160Z", "content": "Are you using the same model as you would use when training on all the data? Then the model may have some serious overfitting because you are training on less data. Adding more regularization will probably improve the score.
", "rawMarkdown": "Are you using the same model as you would use when training on all the data? Then the model may have some serious overfitting because you are training on less data. Adding more regularization will probably improve the score.", "votes": 1 }, { "id": 690858, "postDate": "2019-12-09T08:55:40.747Z", "content": "the problem is that, a single building does not have enough of observations to gerenalize well! For example, there are buildings which don't have some of the observations for some intervals, however, these interval exist in the test set.
", "rawMarkdown": "the problem is that, a single building does not have enough of observations to gerenalize well! For example, there are buildings which don't have some of the observations for some intervals, however, these interval exist in the test set.", "votes": 1 }, { "id": 697756, "postDate": "2019-12-18T11:04:06.777Z", "content": "Does sklearn allow train several models per type? And automatically apply them during prediction? Or I should manually create and evaluate each model?
", "rawMarkdown": "Does sklearn allow train several models per type? And automatically apply them during prediction? Or I should manually create and evaluate each model?" }, { "id": 692937, "postDate": "2019-12-11T21:32:30.847Z", "content": "Thanks for the comments guys! Good to know that there are some competitors having success on that road. My last failed attempt (quite frustrating because it took me some time) was to first create a generic NN trained on all buildings (LB of 1.12). Later, I used transfer learning to fine-tune the very same model for each building. But for some reason, the results are getting worse. Even when using higher dropout rates, model checkpoint and freezing the top layers.
", "rawMarkdown": "Thanks for the comments guys! Good to know that there are some competitors having success on that road. My last failed attempt (quite frustrating because it took me some time) was to first create a generic NN trained on all buildings (LB of 1.12). Later, I used transfer learning to fine-tune the very same model for each building. But for some reason, the results are getting worse. Even when using higher dropout rates, model checkpoint and freezing the top layers." }, { "id": 695242, "postDate": "2019-12-14T20:35:15.783Z", "rawMarkdown": "", "isDeleted": true }, { "id": 691327, "postDate": "2019-12-10T01:59:09.910Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 692942, "postDate": "2019-12-11T21:38:32.583Z", "content": "Thanks for the hint @davide0burba ! If you don't mind answering, how did you extract Fourier terms from the training set? To my knowledge, that's only possible in signals (which is the target in your case).
", "rawMarkdown": "Thanks for the hint @davide0burba ! If you don't mind answering, how did you extract Fourier terms from the training set? To my knowledge, that's only possible in signals (which is the target in your case). " }, { "id": 693104, "postDate": "2019-12-12T03:11:27.830Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ] }, { "id": 690740, "postDate": "2019-12-09T03:12:07.580Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 693204, "author_name": "Robert Stockton", "author_url": "", "post_date": "2019-12-12T06:17:24.510000", "content": "I've manage to score 1.07 (public LB) with per-building models. More precisely, and this might be part of your problem, I used per-building and per-meter (2380 models instead of 1449). I used GBDTs, and did use different hyperparameters than I would use for a single global model.
", "votes": 6, "replies": [ { "id": 693683, "author_name": "Fernando Wittmann", "author_url": "", "post_date": "2019-12-12T17:09:30.813000", "content": "Wow, congrats! That will bring some hope for me to keep trying. I also tried both per bldg and per meter models with no success. The problem seems to be overfitting.
", "votes": 0, "replies": [] } ] }, { "id": 690871, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-12-09T09:18:33.220000", "content": "You're doing something wrong. For me it works good.
", "votes": 3, "replies": [ { "id": 692943, "author_name": "Fernando Wittmann", "author_url": "", "post_date": "2019-12-11T21:40:05.570000", "content": "Thanks for the comment @oleg90777 ! Did you use NNs?
", "votes": 0, "replies": [] }, { "id": 693255, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-12-12T07:27:47.043000", "content": "At the moment only lgb
", "votes": 0, "replies": [] } ] }, { "id": 692437, "author_name": "Larry D.", "author_url": "", "post_date": "2019-12-11T08:57:09.020000", "content": "If you are doing a linear model with a lot of features and no regularization, chances are you are overfitting a lot and may have very large coefficients which will result in crazy predictions on the test set. You should try plotting your predictions to get a feel of what is going on. They should at least bear some resemblance to the data from 2016.
", "votes": 1, "replies": [] }, { "id": 691143, "author_name": "Carlo", "author_url": "", "post_date": "2019-12-09T17:39:14.160000", "content": "Are you using the same model as you would use when training on all the data? Then the model may have some serious overfitting because you are training on less data. Adding more regularization will probably improve the score.
", "votes": 1, "replies": [] }, { "id": 690858, "author_name": "BearInMind", "author_url": "", "post_date": "2019-12-09T08:55:40.747000", "content": "the problem is that, a single building does not have enough of observations to gerenalize well! For example, there are buildings which don't have some of the observations for some intervals, however, these interval exist in the test set.
", "votes": 1, "replies": [] }, { "id": 697756, "author_name": "Nazar", "author_url": "", "post_date": "2019-12-18T11:04:06.777000", "content": "Does sklearn allow train several models per type? And automatically apply them during prediction? Or I should manually create and evaluate each model?
", "votes": 0, "replies": [] }, { "id": 692937, "author_name": "Fernando Wittmann", "author_url": "", "post_date": "2019-12-11T21:32:30.847000", "content": "Thanks for the comments guys! Good to know that there are some competitors having success on that road. My last failed attempt (quite frustrating because it took me some time) was to first create a generic NN trained on all buildings (LB of 1.12). Later, I used transfer learning to fine-tune the very same model for each building. But for some reason, the results are getting worse. Even when using higher dropout rates, model checkpoint and freezing the top layers.
", "votes": 0, "replies": [] }, { "id": 695242, "author_name": "", "author_url": "", "post_date": "2019-12-14T20:35:15.783000", "content": "", "votes": 0, "replies": [] }, { "id": 691327, "author_name": "", "author_url": "", "post_date": "2019-12-10T01:59:09.910000", "content": "", "votes": 0, "replies": [ { "id": 692942, "author_name": "Fernando Wittmann", "author_url": "", "post_date": "2019-12-11T21:38:32.583000", "content": "Thanks for the hint @davide0burba ! If you don't mind answering, how did you extract Fourier terms from the training set? To my knowledge, that's only possible in signals (which is the target in your case).
", "votes": 0, "replies": [] }, { "id": 693104, "author_name": "", "author_url": "", "post_date": "2019-12-12T03:11:27.830000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 690740, "author_name": "", "author_url": "", "post_date": "2019-12-09T03:12:07.580000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "690722": "When I joined this competition, my first hypothesis of a good predictor was to have one model per building. Theoretically, this seems to make sense since each building has a different behavior. However, when trying in practice, I got very bad LB scores (higher than 2). The score is worse than simply predicting the average per building (1.64). Did anyone else try as well? Any idea why it turns out to be a bad idea?", "693204": "I've manage to score 1.07 (public LB) with per-building models. More precisely, and this might be part of your problem, I used per-building *and* per-meter (2380 models instead of 1449). I used GBDTs, and *did* use different hyperparameters than I would use for a single global model.\n", "690871": "You're doing something wrong. For me it works good.", "692437": "If you are doing a linear model with a lot of features and no regularization, chances are you are overfitting a lot and may have very large coefficients which will result in crazy predictions on the test set. You should try plotting your predictions to get a feel of what is going on. They should at least bear some resemblance to the data from 2016.", "691143": "Are you using the same model as you would use when training on all the data? Then the model may have some serious overfitting because you are training on less data. Adding more regularization will probably improve the score.", "690858": "the problem is that, a single building does not have enough of observations to gerenalize well! For example, there are buildings which don't have some of the observations for some intervals, however, these interval exist in the test set.", "697756": "Does sklearn allow train several models per type? And automatically apply them during prediction? Or I should manually create and evaluate each model?", "692937": "Thanks for the comments guys! Good to know that there are some competitors having success on that road. My last failed attempt (quite frustrating because it took me some time) was to first create a generic NN trained on all buildings (LB of 1.12). Later, I used transfer learning to fine-tune the very same model for each building. But for some reason, the results are getting worse. Even when using higher dropout rates, model checkpoint and freezing the top layers.", "695242": "", "691327": "", "690740": "" } }