![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F653205%2F41620b050b7ad37a076c04d9766b5591%2Fbuilding_id_218_Public%20services.png?generation=1571723578818309&alt=media\")
In 2016 (training) data, some time series have maintained the exact same value for some time.\nOr some have maintained the zero value.
\n\nIs it instrument failure? And test data have this?
\n\nIf 2017-2018 (test) data have time series like this, we have to predict this failure and it is difficult problem.
\n\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F653205%2Fa59e6848b459342135be1f14c65ac577%2Fbuilding_id_192_Education.png?generation=1571724061451499&alt=media\")
In 2016 (training) data, some time series have maintained the exact same value for some time.\nOr some have maintained the zero value.
\n\nIs it instrument failure? And test data have this?
\n\nIf 2017-2018 (test) data have time series like this, we have to predict this failure and it is difficult problem.
\n\n
I made a detection code.\n```python\nimport numpy as np\nimport pandas as pd
\n\ntrain_df = pd.read_csv(\"../input/train.csv\")
\n\nfor building_id in range(1449):\n train_gb = train_df[train_df['building_id'] == building_id].groupby(\"meter\")
\n\nfor meter, tmp_df in train_gb:\n print(\"building_id: {}, meter: {}\".format(building_id, meter))\n data = tmp_df['meter_reading'].values\n splited_value = np.split(data, np.where(data[1:] != data[:-1])[0] + 1)\n splited_date = np.split(tmp_df.timestamp.values, np.where(data[1:] != data[:-1])[0] + 1)\n for i, x in enumerate(splited_date):\n if len(x) > 24:\n print(\"length: {},\\t{}-{},\\tvalue: {}\".format(len(x), x[0], x[-1], splited_value[i][0]))\n print()\n```
", "rawMarkdown": "I made a detection code.\n```python\nimport numpy as np\nimport pandas as pd\n\ntrain_df = pd.read_csv(\"../input/train.csv\")\n\nfor building_id in range(1449):\n train_gb = train_df[train_df['building_id'] == building_id].groupby(\"meter\")\n \n for meter, tmp_df in train_gb:\n print(\"building_id: {}, meter: {}\".format(building_id, meter))\n data = tmp_df['meter_reading'].values\n splited_value = np.split(data, np.where(data[1:] != data[:-1])[0] + 1)\n splited_date = np.split(tmp_df.timestamp.values, np.where(data[1:] != data[:-1])[0] + 1)\n for i, x in enumerate(splited_date):\n if len(x) > 24:\n print(\"length: {},\\t{}-{},\\tvalue: {}\".format(len(x), x[0], x[-1], splited_value[i][0]))\n print()\n```", "votes": null }, { "id": "655011", "postDate": "10/22/2019 15:36:25", "content": "> marshi wrote:\n> If 2017-2018 (test) data have time series like this, we have to predict this failure and it is difficult problem.\n> \n>
\n\nWhat if a failure is solved? Do you want to predict also when a failure will be solved?
", "rawMarkdown": "> **marshi wrote:**\n> If 2017-2018 (test) data have time series like this, we have to predict this failure and it is difficult problem.\n> \n> \n\nWhat if a failure is solved? Do you want to predict also when a failure will be solved?", "votes": null }, { "id": "658880", "postDate": "10/26/2019 17:14:28", "content": "Instrument failure ?
\n\nMaybe or Kaggle noise just for us as a gift....
", "rawMarkdown": "Instrument failure ?\n\nMaybe or Kaggle noise just for us as a gift....", "votes": null }, { "id": "658900", "postDate": "10/26/2019 18:03:32", "content": "Looks like potential Null values... \nFrom the official data description: \"There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\"
\n\nLooks like we have some work to do cleaning the data!
", "rawMarkdown": "Looks like potential Null values... \nFrom the official data description: \"There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\"\n\nLooks like we have some work to do cleaning the data!", "votes": null }, { "id": "662969", "postDate": "11/01/2019 09:23:04", "content": "For real life, we usually meet the same problem, missing data due to some connection error if the data is collected through network, or forget to record the data if the data is collected by people. etc.
", "rawMarkdown": "For real life, we usually meet the same problem, missing data due to some connection error if the data is collected through network, or forget to record the data if the data is collected by people. etc.", "votes": null }, { "id": "664125", "postDate": "11/03/2019 07:36:16", "content": "Have you made any progress on this , i noticed the same thing today ? . Any help will be appreciated
", "rawMarkdown": "Have you made any progress on this , i noticed the same thing today ? . Any help will be appreciated", "votes": null }, { "id": "666253", "postDate": "11/05/2019 23:53:52", "content": "I am not worried about the flat regions, they say they won't score them and I believe they will. But in my plots I see a lot of weird stuff going on (like random offsets suddenly added for a few weeks, not-looking random oscillations around the average value..), which look strange, but almost undetectable. That worries me, not sure I want to spend time on such a competition.
", "rawMarkdown": "I am not worried about the flat regions, they say they won't score them and I believe they will. But in my plots I see a lot of weird stuff going on (like random offsets suddenly added for a few weeks, not-looking random oscillations around the average value..), which look strange, but almost undetectable. That worries me, not sure I want to spend time on such a competition.", "votes": null } ], "comments": [ { "id": 654909, "author_name": "marshi", "author_url": "", "post_date": "10/22/2019 13:26:33", "content": "I made a detection code.\n```python\nimport numpy as np\nimport pandas as pd
\n\ntrain_df = pd.read_csv(\"../input/train.csv\")
\n\nfor building_id in range(1449):\n train_gb = train_df[train_df['building_id'] == building_id].groupby(\"meter\")
\n\nfor meter, tmp_df in train_gb:\n print(\"building_id: {}, meter: {}\".format(building_id, meter))\n data = tmp_df['meter_reading'].values\n splited_value = np.split(data, np.where(data[1:] != data[:-1])[0] + 1)\n splited_date = np.split(tmp_df.timestamp.values, np.where(data[1:] != data[:-1])[0] + 1)\n for i, x in enumerate(splited_date):\n if len(x) > 24:\n print(\"length: {},\\t{}-{},\\tvalue: {}\".format(len(x), x[0], x[-1], splited_value[i][0]))\n print()\n```
", "votes": null, "replies": [ { "id": 664125, "author_name": "avikrams", "author_url": "", "post_date": "11/03/2019 07:36:16", "content": "Have you made any progress on this , i noticed the same thing today ? . Any help will be appreciated
", "votes": null, "replies": [] } ] }, { "id": 655011, "author_name": "stenford23", "author_url": "", "post_date": "10/22/2019 15:36:25", "content": "> marshi wrote:\n> If 2017-2018 (test) data have time series like this, we have to predict this failure and it is difficult problem.\n> \n>
\n\nWhat if a failure is solved? Do you want to predict also when a failure will be solved?
", "votes": null, "replies": [] }, { "id": 658880, "author_name": "pfichou", "author_url": "", "post_date": "10/26/2019 17:14:28", "content": "Instrument failure ?
\n\nMaybe or Kaggle noise just for us as a gift....
", "votes": null, "replies": [] }, { "id": 658900, "author_name": "sdoria", "author_url": "", "post_date": "10/26/2019 18:03:32", "content": "Looks like potential Null values... \nFrom the official data description: \"There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\"
\n\nLooks like we have some work to do cleaning the data!
", "votes": null, "replies": [] }, { "id": 662969, "author_name": "longyin2", "author_url": "", "post_date": "11/01/2019 09:23:04", "content": "For real life, we usually meet the same problem, missing data due to some connection error if the data is collected through network, or forget to record the data if the data is collected by people. etc.
", "votes": null, "replies": [] }, { "id": 666253, "author_name": "andreacazzaniga86", "author_url": "", "post_date": "11/05/2019 23:53:52", "content": "I am not worried about the flat regions, they say they won't score them and I believe they will. But in my plots I see a lot of weird stuff going on (like random offsets suddenly added for a few weeks, not-looking random oscillations around the average value..), which look strange, but almost undetectable. That worries me, not sure I want to spend time on such a competition.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "654640": "In 2016 (training) data, some time series have maintained the exact same value for some time.\nOr some have maintained the zero value.\n\nIs it instrument failure? And test data have this?\n\nIf 2017-2018 (test) data have time series like this, we have to predict this failure and it is difficult problem.\n\n\n\n", "654909": "I made a detection code.\n```python\nimport numpy as np\nimport pandas as pd\n\ntrain_df = pd.read_csv(\"../input/train.csv\")\n\nfor building_id in range(1449):\n train_gb = train_df[train_df['building_id'] == building_id].groupby(\"meter\")\n \n for meter, tmp_df in train_gb:\n print(\"building_id: {}, meter: {}\".format(building_id, meter))\n data = tmp_df['meter_reading'].values\n splited_value = np.split(data, np.where(data[1:] != data[:-1])[0] + 1)\n splited_date = np.split(tmp_df.timestamp.values, np.where(data[1:] != data[:-1])[0] + 1)\n for i, x in enumerate(splited_date):\n if len(x) > 24:\n print(\"length: {},\\t{}-{},\\tvalue: {}\".format(len(x), x[0], x[-1], splited_value[i][0]))\n print()\n```", "655011": "> **marshi wrote:**\n> If 2017-2018 (test) data have time series like this, we have to predict this failure and it is difficult problem.\n> \n> \n\nWhat if a failure is solved? Do you want to predict also when a failure will be solved?", "658880": "Instrument failure ?\n\nMaybe or Kaggle noise just for us as a gift....", "658900": "Looks like potential Null values... \nFrom the official data description: \"There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\"\n\nLooks like we have some work to do cleaning the data!", "662969": "For real life, we usually meet the same problem, missing data due to some connection error if the data is collected through network, or forget to record the data if the data is collected by people. etc.", "664125": "Have you made any progress on this , i noticed the same thing today ? . Any help will be appreciated", "666253": "I am not worried about the flat regions, they say they won't score them and I believe they will. But in my plots I see a lot of weird stuff going on (like random offsets suddenly added for a few weeks, not-looking random oscillations around the average value..), which look strange, but almost undetectable. That worries me, not sure I want to spend time on such a competition." }, "source": "meta" }