Hey guys, I'm quite new to all of this so I have a few questions.
\n\nReading different kernels and discussion threads, it seems to me that a lot of people are using building_id as a categorical feature and in many model evaluations, it's a feature with quite high importance.
\n\nI might be wrong, but it makes sense to me that features that act as identifiers (id's or names, etc) can contribute a lot to overfitting. My way of thinking is that they could sort of \"map\" the target value to each building, or does the fact that each building can have more than 1 meter_reading prevent that? Is any of that correct or am I just being pedantic?
\n\nIs there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?
", "messages": [ { "id": "671600", "postDate": "11/12/2019 23:50:22", "content": "Hey guys, I'm quite new to all of this so I have a few questions.
\n\nReading different kernels and discussion threads, it seems to me that a lot of people are using building_id as a categorical feature and in many model evaluations, it's a feature with quite high importance.
\n\nI might be wrong, but it makes sense to me that features that act as identifiers (id's or names, etc) can contribute a lot to overfitting. My way of thinking is that they could sort of \"map\" the target value to each building, or does the fact that each building can have more than 1 meter_reading prevent that? Is any of that correct or am I just being pedantic?
\n\nIs there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?
", "rawMarkdown": "Hey guys, I'm quite new to all of this so I have a few questions.\n\nReading different kernels and discussion threads, it seems to me that a lot of people are using building_id as a categorical feature and in many model evaluations, it's a feature with quite high importance.\n\nI might be wrong, but it makes sense to me that features that act as identifiers (id's or names, etc) can contribute a lot to overfitting. My way of thinking is that they could sort of \"map\" the target value to each building, or does the fact that each building can have more than 1 meter_reading prevent that? Is any of that correct or am I just being pedantic? \n\nIs there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?", "votes": null }, { "id": "671815", "postDate": "11/13/2019 07:25:14", "content": "\n\n\nIs there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?
\n
Answer - Yes \nIt's a completely valid question , since we have been provided with a static train / test data , which is not going to change over time .
", "rawMarkdown": "> Is there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?\n\nAnswer - Yes \nIt's a completely valid question , since we have been provided with a static train / test data , which is not going to change over time .", "votes": null }, { "id": "671992", "postDate": "11/13/2019 12:35:43", "content": "Alright so if I understood correctly, the public and private sets are necessarily taken from the same place? If that's the case, isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?
", "rawMarkdown": "Alright so if I understood correctly, the public and private sets are necessarily taken from the same place? If that's the case, isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?", "votes": null }, { "id": "672213", "postDate": "11/13/2019 16:27:35", "content": "If I interpret the goals of the competition correctly, the building ids should be in both the public and private datasets. We are asked to produce a model that predicts how much a specific building will consume in the future based on its consumption patterns in the past. Not how much an unknown building will consume based on the past consumption patterns of other buildings. I would therefore be surprised if there is a trick in the train/public test/prive test split.
\n\nIt also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.
", "rawMarkdown": "If I interpret the goals of the competition correctly, the building ids should be in both the public and private datasets. We are asked to produce a model that predicts how much a specific building will consume in the future based on its consumption patterns in the past. Not how much an unknown building will consume based on the past consumption patterns of other buildings. I would therefore be surprised if there is a trick in the train/public test/prive test split.\n\nIt also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.", "votes": null }, { "id": "672281", "postDate": "11/13/2019 18:00:22", "content": "Wow amazing answer, thanks a lot!
", "rawMarkdown": "Wow amazing answer, thanks a lot!", "votes": null }, { "id": "674576", "postDate": "11/16/2019 18:17:14", "content": "\n\n\nIt also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.
\n
It does not make sense, however, from a generalizability point of view. It would be more useful IMO to create more robust models that are independent of building-specific data since it's plausible that not all buildings that require this model to forecast a counterfactual have historical data available for them.
", "rawMarkdown": "> It also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.\n\nIt does not make sense, however, from a generalizability point of view. It would be more useful IMO to create more robust models that are independent of building-specific data since it's plausible that not all buildings that require this model to forecast a counterfactual have historical data available for them.", "votes": null }, { "id": "674577", "postDate": "11/16/2019 18:18:45", "content": "What are your thoughts, then, on what is going to be different in the private leaderboard set?
", "rawMarkdown": "What are your thoughts, then, on what is going to be different in the private leaderboard set?", "votes": null }, { "id": "674609", "postDate": "11/16/2019 19:17:39", "content": "It’s quite obvious that site id an building Id are very important features, treated as categorical.
", "rawMarkdown": "It’s quite obvious that site id an building Id are very important features, treated as categorical.", "votes": null }, { "id": "674617", "postDate": "11/16/2019 19:25:23", "content": "You are right, it doesn´t help for the generalisation. We would need more details on the buildings (materials, insulation, type/ size of windows and a long list of other factors) to be able to do so. But the competition has a clear focus on predicting a specific building´s consumption to serve as a counterfactual for the impact of mofifications in the buidling or its users´ behaviour.
", "rawMarkdown": "You are right, it doesn´t help for the generalisation. We would need more details on the buildings (materials, insulation, type/ size of windows and a long list of other factors) to be able to do so. But the competition has a clear focus on predicting a specific building´s consumption to serve as a counterfactual for the impact of mofifications in the buidling or its users´ behaviour.", "votes": null }, { "id": "675274", "postDate": "11/17/2019 20:39:18", "content": "If let's say, private set's id's were completely different from the public set's id's. But I think I get it now, you guys helped me clear out my thoughts, thanks!
", "rawMarkdown": "If let's say, private set's id's were completely different from the public set's id's. But I think I get it now, you guys helped me clear out my thoughts, thanks!", "votes": null }, { "id": "675970", "postDate": "11/18/2019 20:36:03", "content": "Sorry - this question was meant for Aditya. @avikrams, could you comment?
", "rawMarkdown": "Sorry - this question was meant for Aditya. @avikrams, could you comment?", "votes": null }, { "id": "676366", "postDate": "11/19/2019 05:48:45", "content": "> Alright so if I understood correctly, the public and private sets are necessarily taken from the same place?
\n\nYes they are taken from the same source
\n\n> isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?
\n\nIt is possible , but in this case you have complete access to the private set (i.e test.csv) so you can check whether all the building present in the Test set are a part of the train set or not , fortunately for this problem we have exactly same buildings present in both test / train sets, therefore it won't be a problem if you use them directly.
\n\nHowever if you don't want to take it as a feature then you can focus on forming surrogate features which can describe a building , which in this case is - square-feet , year of build ,floors , usage type , etc . \nIn real life I think it's better to have a data preparation script in place , which will ensure that catergorical variables have same levels , and any previously unseen levels are moved to a default level - 'others'
", "rawMarkdown": "> Alright so if I understood correctly, the public and private sets are necessarily taken from the same place?\n\nYes they are taken from the same source \n\n> isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?\n\nIt is possible , but in this case you have complete access to the private set (i.e test.csv) so you can check whether all the building present in the Test set are a part of the train set or not , fortunately for this problem we have exactly same buildings present in both test / train sets, therefore it won't be a problem if you use them directly. \n\nHowever if you don't want to take it as a feature then you can focus on forming surrogate features which can describe a building , which in this case is - square-feet , year of build ,floors , usage type , etc . \nIn real life I think it's better to have a data preparation script in place , which will ensure that catergorical variables have same levels , and any previously unseen levels are moved to a default level - 'others'", "votes": null } ], "comments": [ { "id": 671815, "author_name": "avikrams", "author_url": "", "post_date": "11/13/2019 07:25:14", "content": "\n\n\nIs there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?
\n
Answer - Yes \nIt's a completely valid question , since we have been provided with a static train / test data , which is not going to change over time .
", "votes": null, "replies": [ { "id": 671992, "author_name": "prschdt", "author_url": "", "post_date": "11/13/2019 12:35:43", "content": "Alright so if I understood correctly, the public and private sets are necessarily taken from the same place? If that's the case, isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?
", "votes": null, "replies": [] }, { "id": 674577, "author_name": "calvdee", "author_url": "", "post_date": "11/16/2019 18:18:45", "content": "What are your thoughts, then, on what is going to be different in the private leaderboard set?
", "votes": null, "replies": [] }, { "id": 675274, "author_name": "prschdt", "author_url": "", "post_date": "11/17/2019 20:39:18", "content": "If let's say, private set's id's were completely different from the public set's id's. But I think I get it now, you guys helped me clear out my thoughts, thanks!
", "votes": null, "replies": [] }, { "id": 675970, "author_name": "calvdee", "author_url": "", "post_date": "11/18/2019 20:36:03", "content": "Sorry - this question was meant for Aditya. @avikrams, could you comment?
", "votes": null, "replies": [] }, { "id": 676366, "author_name": "avikrams", "author_url": "", "post_date": "11/19/2019 05:48:45", "content": "> Alright so if I understood correctly, the public and private sets are necessarily taken from the same place?
\n\nYes they are taken from the same source
\n\n> isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?
\n\nIt is possible , but in this case you have complete access to the private set (i.e test.csv) so you can check whether all the building present in the Test set are a part of the train set or not , fortunately for this problem we have exactly same buildings present in both test / train sets, therefore it won't be a problem if you use them directly.
\n\nHowever if you don't want to take it as a feature then you can focus on forming surrogate features which can describe a building , which in this case is - square-feet , year of build ,floors , usage type , etc . \nIn real life I think it's better to have a data preparation script in place , which will ensure that catergorical variables have same levels , and any previously unseen levels are moved to a default level - 'others'
", "votes": null, "replies": [] } ] }, { "id": 672213, "author_name": "panosc", "author_url": "", "post_date": "11/13/2019 16:27:35", "content": "If I interpret the goals of the competition correctly, the building ids should be in both the public and private datasets. We are asked to produce a model that predicts how much a specific building will consume in the future based on its consumption patterns in the past. Not how much an unknown building will consume based on the past consumption patterns of other buildings. I would therefore be surprised if there is a trick in the train/public test/prive test split.
\n\nIt also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.
", "votes": null, "replies": [ { "id": 672281, "author_name": "prschdt", "author_url": "", "post_date": "11/13/2019 18:00:22", "content": "Wow amazing answer, thanks a lot!
", "votes": null, "replies": [] }, { "id": 674576, "author_name": "calvdee", "author_url": "", "post_date": "11/16/2019 18:17:14", "content": "\n\n\nIt also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.
\n
It does not make sense, however, from a generalizability point of view. It would be more useful IMO to create more robust models that are independent of building-specific data since it's plausible that not all buildings that require this model to forecast a counterfactual have historical data available for them.
", "votes": null, "replies": [] }, { "id": 674617, "author_name": "panosc", "author_url": "", "post_date": "11/16/2019 19:25:23", "content": "You are right, it doesn´t help for the generalisation. We would need more details on the buildings (materials, insulation, type/ size of windows and a long list of other factors) to be able to do so. But the competition has a clear focus on predicting a specific building´s consumption to serve as a counterfactual for the impact of mofifications in the buidling or its users´ behaviour.
", "votes": null, "replies": [] } ] }, { "id": 674609, "author_name": "pfichou", "author_url": "", "post_date": "11/16/2019 19:17:39", "content": "It’s quite obvious that site id an building Id are very important features, treated as categorical.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "671600": "Hey guys, I'm quite new to all of this so I have a few questions.\n\nReading different kernels and discussion threads, it seems to me that a lot of people are using building_id as a categorical feature and in many model evaluations, it's a feature with quite high importance.\n\nI might be wrong, but it makes sense to me that features that act as identifiers (id's or names, etc) can contribute a lot to overfitting. My way of thinking is that they could sort of \"map\" the target value to each building, or does the fact that each building can have more than 1 meter_reading prevent that? Is any of that correct or am I just being pedantic? \n\nIs there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?", "671815": "> Is there any guarantee that for instance, building_ids in the private set will be the same as those in the given training/test set?\n\nAnswer - Yes \nIt's a completely valid question , since we have been provided with a static train / test data , which is not going to change over time .", "671992": "Alright so if I understood correctly, the public and private sets are necessarily taken from the same place? If that's the case, isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?", "672213": "If I interpret the goals of the competition correctly, the building ids should be in both the public and private datasets. We are asked to produce a model that predicts how much a specific building will consume in the future based on its consumption patterns in the past. Not how much an unknown building will consume based on the past consumption patterns of other buildings. I would therefore be surprised if there is a trick in the train/public test/prive test split.\n\nIt also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.", "672281": "Wow amazing answer, thanks a lot!", "674576": "> It also makes sense from the physical interpretation point of view to use building_id in a model, there are most probably building specific characteristics -apart from the ones we have access to- which can be proxied by building id.\n\nIt does not make sense, however, from a generalizability point of view. It would be more useful IMO to create more robust models that are independent of building-specific data since it's plausible that not all buildings that require this model to forecast a counterfactual have historical data available for them.", "674577": "What are your thoughts, then, on what is going to be different in the private leaderboard set?", "674609": "It’s quite obvious that site id an building Id are very important features, treated as categorical.", "674617": "You are right, it doesn´t help for the generalisation. We would need more details on the buildings (materials, insulation, type/ size of windows and a long list of other factors) to be able to do so. But the competition has a clear focus on predicting a specific building´s consumption to serve as a counterfactual for the impact of mofifications in the buidling or its users´ behaviour.", "675274": "If let's say, private set's id's were completely different from the public set's id's. But I think I get it now, you guys helped me clear out my thoughts, thanks!", "675970": "Sorry - this question was meant for Aditya. @avikrams, could you comment?", "676366": "> Alright so if I understood correctly, the public and private sets are necessarily taken from the same place?\n\nYes they are taken from the same source \n\n> isn't it still possible that the way the public set was chosen could only provide information about a set of id's and leave the others to the private set?\n\nIt is possible , but in this case you have complete access to the private set (i.e test.csv) so you can check whether all the building present in the Test set are a part of the train set or not , fortunately for this problem we have exactly same buildings present in both test / train sets, therefore it won't be a problem if you use them directly. \n\nHowever if you don't want to take it as a feature then you can focus on forming surrogate features which can describe a building , which in this case is - square-feet , year of build ,floors , usage type , etc . \nIn real life I think it's better to have a data preparation script in place , which will ensure that catergorical variables have same levels , and any previously unseen levels are moved to a default level - 'others'" }, "source": "meta" }