There can be different views and approaches on how to fill null values. The simplest being the mean, in kernel I often see the mean of train is imputed on the test data itself, but since the test data is 2 years ahead and the train data is only of 1 year. Will fitting on test and transforming on test still gives data leakage or some other issues?
", "messages": [ { "id": 654062, "postDate": "2019-10-21T11:36:19.603Z", "content": "The best way to fill the meteorological would be filling it with the previous hour/day value rather than mean.
", "rawMarkdown": "The best way to fill the meteorological would be filling it with the previous hour/day value rather than mean. ", "votes": 1 }, { "id": 655252, "postDate": "2019-10-22T21:13:20.767Z", "content": "To streamline this though process it is useful to know the 3 categories in which missing data can be classified into:\nMissing Completely at Random (MCAR)\nMissing at Random (MAR)\nMissing Not at Random (MNAR)
\n\ndeeper into each one of them and look for potential treatments that can help us circumvent the problems that missing data creates.
\n\nSome ideas:\nTime series imputation\n - Non-time-series specific method\n- mean imputation\nmedian - imputation\nmode - imputation\n- calcucate the appropriate measure and replace NAs with the values. Appropriate for stationary time series, for example, white noise data\n- Random sample imputation\nreplace missing values with observations randomly selected from the remaining (either of it or just some section of it)
\n\nTime-Series specific method\n- Last observation carried forward (LOCF)\n- Next observation carried backward (NOCB)
\n\nThis article can be helpfull: https://www.linkedin.com/pulse/data-cleaning-guide-dealing-na-values-karan-rajwanshi/
", "rawMarkdown": "To streamline this though process it is useful to know the 3 categories in which missing data can be classified into:\nMissing Completely at Random (MCAR)\nMissing at Random (MAR)\nMissing Not at Random (MNAR)\n\ndeeper into each one of them and look for potential treatments that can help us circumvent the problems that missing data creates. \n\nSome ideas:\nTime series imputation\n - Non-time-series specific method\n- mean imputation\nmedian - imputation\nmode - imputation\n- calcucate the appropriate measure and replace NAs with the values. Appropriate for stationary time series, for example, white noise data\n- Random sample imputation\nreplace missing values with observations randomly selected from the remaining (either of it or just some section of it)\n\nTime-Series specific method\n- Last observation carried forward (LOCF)\n- Next observation carried backward (NOCB)\n\nThis article can be helpfull: https://www.linkedin.com/pulse/data-cleaning-guide-dealing-na-values-karan-rajwanshi/", "votes": 1 }, { "id": 654296, "postDate": "2019-10-21T17:13:56.473Z", "content": "the test data reflect the actual live examples so anyway we need to put in place something to handle Null values.\nTry to use something like df.interpolate() for now and maybe there will be better suggestions in the future.
", "rawMarkdown": "the test data reflect the actual live examples so anyway we need to put in place something to handle Null values.\nTry to use something like df.interpolate() for now and maybe there will be better suggestions in the future.", "votes": 1 }, { "id": 654022, "postDate": "2019-10-21T10:19:29.907Z", "content": "There can be different views and approaches on how to fill null values. The simplest being the mean, in kernel I often see the mean of train is imputed on the test data itself, but since the test data is 2 years ahead and the train data is only of 1 year. Will fitting on test and transforming on test still gives data leakage or some other issues?
", "rawMarkdown": "There can be different views and approaches on how to fill null values. The simplest being the mean, in kernel I often see the mean of train is imputed on the test data itself, but since the test data is 2 years ahead and the train data is only of 1 year. Will fitting on test and transforming on test still gives data leakage or some other issues? ", "votes": 2 }, { "id": 654592, "postDate": "2019-10-22T04:15:22.130Z", "content": "I have not yet made my first submission, but I would say, it depends on the size of the \"hole\" (duration of missingness) in the data. If the it's just one or two hours of missing data for a building, even a simple, ffill() will work. If the \"hole\" is little larger, may be interpolation could be a better approach. But, if the size of the 'hole' is too large, either we have to ignore the data or pick up data from another \"similar\" building. I am not sure if that is at all possible or not. Will expect to hear back from others.
", "rawMarkdown": "I have not yet made my first submission, but I would say, it depends on the size of the \"hole\" (duration of missingness) in the data. If the it's just one or two hours of missing data for a building, even a simple, ffill() will work. If the \"hole\" is little larger, may be interpolation could be a better approach. But, if the size of the 'hole' is too large, either we have to ignore the data or pick up data from another \"similar\" building. I am not sure if that is at all possible or not. Will expect to hear back from others. " }, { "id": 654274, "postDate": "2019-10-21T16:38:22.107Z", "content": "@themonologue I did as you suggested, but there are no data for some columns in some site_id. \nIn this case, I filled with mean of others site_id's.
", "rawMarkdown": "@themonologue I did as you suggested, but there are no data for some columns in some site_id. \nIn this case, I filled with mean of others site_id's." }, { "id": 654209, "postDate": "2019-10-21T15:37:07.233Z", "content": "Try setting timestamp as the index of yor dataframe and use df.interpolate(). Think method=linear would be most accurate, but if the model is to be used in practice, then I guess one would have to use \"ffill\". This has to be done for each site_id
", "rawMarkdown": "Try setting timestamp as the index of yor dataframe and use df.interpolate(). Think method=linear would be most accurate, but if the model is to be used in practice, then I guess one would have to use \"ffill\". This has to be done for each site_id\n" }, { "id": 654642, "postDate": "2019-10-22T06:05:59.327Z", "rawMarkdown": "", "isDeleted": true }, { "id": 654208, "postDate": "2019-10-21T15:36:36.500Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 654062, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-10-21T11:36:19.603000", "content": "The best way to fill the meteorological would be filling it with the previous hour/day value rather than mean.
", "votes": 1, "replies": [] }, { "id": 655252, "author_name": "Caesar Lupum", "author_url": "", "post_date": "2019-10-22T21:13:20.767000", "content": "To streamline this though process it is useful to know the 3 categories in which missing data can be classified into:\nMissing Completely at Random (MCAR)\nMissing at Random (MAR)\nMissing Not at Random (MNAR)
\n\ndeeper into each one of them and look for potential treatments that can help us circumvent the problems that missing data creates.
\n\nSome ideas:\nTime series imputation\n - Non-time-series specific method\n- mean imputation\nmedian - imputation\nmode - imputation\n- calcucate the appropriate measure and replace NAs with the values. Appropriate for stationary time series, for example, white noise data\n- Random sample imputation\nreplace missing values with observations randomly selected from the remaining (either of it or just some section of it)
\n\nTime-Series specific method\n- Last observation carried forward (LOCF)\n- Next observation carried backward (NOCB)
\n\nThis article can be helpfull: https://www.linkedin.com/pulse/data-cleaning-guide-dealing-na-values-karan-rajwanshi/
", "votes": 1, "replies": [] }, { "id": 654296, "author_name": "Adrian Zinovei", "author_url": "", "post_date": "2019-10-21T17:13:56.473000", "content": "the test data reflect the actual live examples so anyway we need to put in place something to handle Null values.\nTry to use something like df.interpolate() for now and maybe there will be better suggestions in the future.
", "votes": 1, "replies": [] }, { "id": 654592, "author_name": "arnab", "author_url": "", "post_date": "2019-10-22T04:15:22.130000", "content": "I have not yet made my first submission, but I would say, it depends on the size of the \"hole\" (duration of missingness) in the data. If the it's just one or two hours of missing data for a building, even a simple, ffill() will work. If the \"hole\" is little larger, may be interpolation could be a better approach. But, if the size of the 'hole' is too large, either we have to ignore the data or pick up data from another \"similar\" building. I am not sure if that is at all possible or not. Will expect to hear back from others.
", "votes": 0, "replies": [] }, { "id": 654274, "author_name": "Vitor Alcântara Batista", "author_url": "", "post_date": "2019-10-21T16:38:22.107000", "content": "@themonologue I did as you suggested, but there are no data for some columns in some site_id. \nIn this case, I filled with mean of others site_id's.
", "votes": 0, "replies": [] }, { "id": 654209, "author_name": "maka", "author_url": "", "post_date": "2019-10-21T15:37:07.233000", "content": "Try setting timestamp as the index of yor dataframe and use df.interpolate(). Think method=linear would be most accurate, but if the model is to be used in practice, then I guess one would have to use \"ffill\". This has to be done for each site_id
", "votes": 0, "replies": [] }, { "id": 654642, "author_name": "", "author_url": "", "post_date": "2019-10-22T06:05:59.327000", "content": "", "votes": 0, "replies": [] }, { "id": 654208, "author_name": "", "author_url": "", "post_date": "2019-10-21T15:36:36.500000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "654062": "The best way to fill the meteorological would be filling it with the previous hour/day value rather than mean. ", "655252": "To streamline this though process it is useful to know the 3 categories in which missing data can be classified into:\nMissing Completely at Random (MCAR)\nMissing at Random (MAR)\nMissing Not at Random (MNAR)\n\ndeeper into each one of them and look for potential treatments that can help us circumvent the problems that missing data creates. \n\nSome ideas:\nTime series imputation\n - Non-time-series specific method\n- mean imputation\nmedian - imputation\nmode - imputation\n- calcucate the appropriate measure and replace NAs with the values. Appropriate for stationary time series, for example, white noise data\n- Random sample imputation\nreplace missing values with observations randomly selected from the remaining (either of it or just some section of it)\n\nTime-Series specific method\n- Last observation carried forward (LOCF)\n- Next observation carried backward (NOCB)\n\nThis article can be helpfull: https://www.linkedin.com/pulse/data-cleaning-guide-dealing-na-values-karan-rajwanshi/", "654296": "the test data reflect the actual live examples so anyway we need to put in place something to handle Null values.\nTry to use something like df.interpolate() for now and maybe there will be better suggestions in the future.", "654022": "There can be different views and approaches on how to fill null values. The simplest being the mean, in kernel I often see the mean of train is imputed on the test data itself, but since the test data is 2 years ahead and the train data is only of 1 year. Will fitting on test and transforming on test still gives data leakage or some other issues? ", "654592": "I have not yet made my first submission, but I would say, it depends on the size of the \"hole\" (duration of missingness) in the data. If the it's just one or two hours of missing data for a building, even a simple, ffill() will work. If the \"hole\" is little larger, may be interpolation could be a better approach. But, if the size of the 'hole' is too large, either we have to ignore the data or pick up data from another \"similar\" building. I am not sure if that is at all possible or not. Will expect to hear back from others. ", "654274": "@themonologue I did as you suggested, but there are no data for some columns in some site_id. \nIn this case, I filled with mean of others site_id's.", "654209": "Try setting timestamp as the index of yor dataframe and use df.interpolate(). Think method=linear would be most accurate, but if the model is to be used in practice, then I guess one would have to use \"ffill\". This has to be done for each site_id\n", "654642": "", "654208": "" } }