I've been running some basic ARIMA models and it appears that several of the time series in the train set, as indicated by building IDs, are not continuous.
\nFor example the building_id=914 has two just in one '2016-10-17'. And there is non-negligible number of others.
\nI'm fine with resampling them, but it can be a pain if they are also in the test data.
", "messages": [ { "id": "1797098", "postDate": "05/21/2022 14:30:20", "content": "I've been running some basic ARIMA models and it appears that several of the time series in the train set, as indicated by building IDs, are not continuous.
\nFor example the building_id=914 has two just in one '2016-10-17'. And there is non-negligible number of others.
\nI'm fine with resampling them, but it can be a pain if they are also in the test data.
", "rawMarkdown": "I've been running some basic ARIMA models and it appears that several of the time series in the train set, as indicated by building IDs, are not continuous. \n\nFor example the building_id=914 has two just in one '2016-10-17'. And there is non-negligible number of others.\n\nI'm fine with resampling them, but it can be a pain if they are also in the test data.", "votes": null }, { "id": "1798029", "postDate": "05/22/2022 16:32:33", "content": "The competition host mentioned that some of the records have been removed due to erroneous values. This might contribute to the issue - @samy101
\nBesides, the building ids in train data are not present in test data and vice versa, so even if the model learns any building specific features, they may not be useful for predictions on the test dataset. In which case, we should be okay with having missing records for certain buildings, since the learning will happen at a more aggregated level.
That depends on how we define anomalous observation. Not all anomalies are apparent when you take a global view of all observations across individuals and use them as if they were objects. That only gives you global outliers, but not necessarily the local outliers, which are objective here.
\nIn order to identify those one needs context. And if there are missing observations in that window … the context changes.
\nInterpolating the values doesn't have to be hard, it isn't for 'meter_reading' column. But it's something that should be mentioned nonetheless. And values created by upsampling have to be dropped before submission.
", "rawMarkdown": "That depends on how we define anomalous observation. Not all anomalies are apparent when you take a global view of all observations across individuals and use them as if they were objects. That only gives you global outliers, but not necessarily the local outliers, which are objective here.\n\nIn order to identify those one needs context. And if there are missing observations in that window ... the context changes.\n\nInterpolating the values doesn't have to be hard, it isn't for 'meter_reading' column. But it's something that should be mentioned nonetheless. And values created by upsampling have to be dropped before submission.", "votes": null } ], "comments": [ { "id": 1798029, "author_name": "rohannanaware", "author_url": "", "post_date": "05/22/2022 16:32:33", "content": "The competition host mentioned that some of the records have been removed due to erroneous values. This might contribute to the issue - @samy101
\nBesides, the building ids in train data are not present in test data and vice versa, so even if the model learns any building specific features, they may not be useful for predictions on the test dataset. In which case, we should be okay with having missing records for certain buildings, since the learning will happen at a more aggregated level.
That depends on how we define anomalous observation. Not all anomalies are apparent when you take a global view of all observations across individuals and use them as if they were objects. That only gives you global outliers, but not necessarily the local outliers, which are objective here.
\nIn order to identify those one needs context. And if there are missing observations in that window … the context changes.
\nInterpolating the values doesn't have to be hard, it isn't for 'meter_reading' column. But it's something that should be mentioned nonetheless. And values created by upsampling have to be dropped before submission.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1797098": "I've been running some basic ARIMA models and it appears that several of the time series in the train set, as indicated by building IDs, are not continuous. \n\nFor example the building_id=914 has two just in one '2016-10-17'. And there is non-negligible number of others.\n\nI'm fine with resampling them, but it can be a pain if they are also in the test data.", "1798029": "The competition host mentioned that some of the records have been removed due to erroneous values. This might contribute to the issue - @samy101\n**Besides, the building ids in train data are not present in test data and vice versa**, so even if the model learns any building specific features, they may not be useful for predictions on the test dataset. In which case, we should be okay with having missing records for certain buildings, since the learning will happen at a more aggregated level.", "1798522": "That depends on how we define anomalous observation. Not all anomalies are apparent when you take a global view of all observations across individuals and use them as if they were objects. That only gives you global outliers, but not necessarily the local outliers, which are objective here.\n\nIn order to identify those one needs context. And if there are missing observations in that window ... the context changes.\n\nInterpolating the values doesn't have to be hard, it isn't for 'meter_reading' column. But it's something that should be mentioned nonetheless. And values created by upsampling have to be dropped before submission." }, "source": "meta" }