![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F539648%2F19bc2775d773695735defcd23f9bab57%2Foutliers.png?generation=1571322078659836&alt=media\")
Does the test data contain real, messy data with sensor failures and mistakes like the train set? I mean in regard to the target variable. If yes, should we treat these outlier values with pre-processing in train and with post-processing in the final prediction? Or will that impact our predictions negatively -by definition- because such bad values are meant to exist in the test set as well?
\n\nE.g for building_id==0 and meter==0 the plot of meter_reading is as below:\n
Should we be treating such situations (red circles) in train (but also and maybe most importantly in our test file) or are we biasing our predictions unnecessarily?
", "messages": [ { "id": "651476", "postDate": "10/17/2019 14:22:41", "content": "Does the test data contain real, messy data with sensor failures and mistakes like the train set? I mean in regard to the target variable. If yes, should we treat these outlier values with pre-processing in train and with post-processing in the final prediction? Or will that impact our predictions negatively -by definition- because such bad values are meant to exist in the test set as well?
\n\nE.g for building_id==0 and meter==0 the plot of meter_reading is as below:\n
Should we be treating such situations (red circles) in train (but also and maybe most importantly in our test file) or are we biasing our predictions unnecessarily?
", "rawMarkdown": "Does the test data contain real, messy data with sensor failures and mistakes like the train set? I mean in regard to the target variable. If yes, should we treat these outlier values with pre-processing in train and with post-processing in the final prediction? Or will that impact our predictions negatively -*by definition*- because such bad values are meant to exist in the test set as well? \n\nE.g for building_id==0 and meter==0 the plot of meter_reading is as below:\n\n\nShould we be treating such situations (red circles) in train (but also and maybe most importantly in our test file) or are we biasing our predictions unnecessarily?", "votes": null }, { "id": "651749", "postDate": "10/17/2019 22:22:46", "content": "I don't think anyone except the organizers knows the answer to this. Which means, sans them telling us, one submission trained w/o outliers / using your imputation scheme, and one sub biased with the messy data. Then pray for the best.
", "rawMarkdown": "I don't think anyone except the organizers knows the answer to this. Which means, sans them telling us, one submission trained w/o outliers / using your imputation scheme, and one sub biased with the messy data. Then pray for the best.", "votes": null }, { "id": "651953", "postDate": "10/18/2019 06:22:30", "content": "In my view, it should be one of the basic things to clarify. But in the case they don't, I guess we'll go your way somehow...
", "rawMarkdown": "In my view, it should be one of the basic things to clarify. But in the case they don't, I guess we'll go your way somehow...", "votes": null } ], "comments": [ { "id": 651749, "author_name": "authman", "author_url": "", "post_date": "10/17/2019 22:22:46", "content": "I don't think anyone except the organizers knows the answer to this. Which means, sans them telling us, one submission trained w/o outliers / using your imputation scheme, and one sub biased with the messy data. Then pray for the best.
", "votes": null, "replies": [ { "id": 651953, "author_name": "georsara1", "author_url": "", "post_date": "10/18/2019 06:22:30", "content": "In my view, it should be one of the basic things to clarify. But in the case they don't, I guess we'll go your way somehow...
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "651476": "Does the test data contain real, messy data with sensor failures and mistakes like the train set? I mean in regard to the target variable. If yes, should we treat these outlier values with pre-processing in train and with post-processing in the final prediction? Or will that impact our predictions negatively -*by definition*- because such bad values are meant to exist in the test set as well? \n\nE.g for building_id==0 and meter==0 the plot of meter_reading is as below:\n\n\nShould we be treating such situations (red circles) in train (but also and maybe most importantly in our test file) or are we biasing our predictions unnecessarily?", "651749": "I don't think anyone except the organizers knows the answer to this. Which means, sans them telling us, one submission trained w/o outliers / using your imputation scheme, and one sub biased with the messy data. Then pray for the best.", "651953": "In my view, it should be one of the basic things to clarify. But in the case they don't, I guess we'll go your way somehow..." }, "source": "meta" }