{
  "id": 122471,
  "title": "Outlier Detection - ASHRAE Way",
  "url": "/competitions/ashrae-energy-prediction/discussion/122471",
  "author_name": "Gunes Evitan",
  "post_date": "2019-12-20T11:38:23.794000",
  "votes": 82,
  "comment_count": 22,
  "views": 0,
  "content": "<p>The fact that outlier removal giving the largest boost to models was not a secret since the early stages. It was possible to reach <strong>1.05</strong> in this competition with only outlier removal. However most of the teams did it only for site 0.</p>\n\n<p>Traditional outlier detection is 1 dimensional, but outliers in this data are dependent to several external factors such as seasonality, meter type, outliers' date frequency within site, outliers' length frequency within site and etc. That's why it had to be done for every building separately even though it was very inefficient. I tried to make it a little bit more efficient with this detection function. It is the modified version of @marshi  code <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#654909\">here</a>.</p>\n\n<p>This function basically returns the indices of constant series of the given building and meter type if the length of series are greater than <code>min_length</code>.  <code>min_length</code> parameter had to be tuned for every building differently since the length of series were always different. I did it for every building in <a href=\"https://www.kaggle.com/gunesevitan/ashrae-eda-and-preprocessing\">this</a> kernel.</p>\n\n<p>```\ndef filter2(building_id, meter, min_length, plot=False, verbose=False):\n    if verbose:\n        print(\"building_id: {}, meter: {}\".format(building_id, meter))\n    temp_df = df_train[(df_train['building_id'] == building_id) &amp; (df_train['meter'] == meter)] <br>\n    target = temp_df['meter_reading'].values</p>\n\n<pre><code>splitted_target = np.split(target, np.where(target[1:] != target[:-1])[0] + 1)\nsplitted_date = np.split(temp_df['timestamp'].values, np.where(target[1:] != target[:-1])[0] + 1)\n\nbuilding_idx = []\nfor i, x in enumerate(splitted_date):\n    if len(x) &gt; min_length:\n        start = x[0]\n        end = x[-1]\n        value = splitted_target[i][0]\n        idx = df_train.query('(@start &lt;= timestamp &lt;= @end) and meter_reading == @value and meter == @meter and building_id == @building_id', engine='python').index.tolist()\n        building_idx.extend(idx)\n        if verbose:\n            print('Length: {},\\t{}  -  {},\\tvalue: {}'.format(len(x), start, end, value))\n\nbuilding_idx = pd.Int64Index(building_idx)\nif plot:\n    fig, axes = plt.subplots(nrows=2, figsize=(16, 18), dpi=100)\n\n    temp_df.set_index('timestamp')['meter_reading'].plot(ax=axes[0])     \n    temp_df.drop(building_idx, axis=0).set_index('timestamp')['meter_reading'].plot(ax=axes[1])\n\n    axes[0].set_title(f'Building {building_id} raw meter readings')\n    axes[1].set_title(f'Building {building_id} filtered meter readings')\n\n    plt.show()\n\nreturn building_idx\n</code></pre>\n\n<p>```</p>\n\n<h2>What to drop?</h2>\n\n<ul>\n<li><strong>Electricity meter readings on another high or low level at the start of 2016.</strong> This was the only edge case this function was not used. I had to do it manually. Most of the public kernels only dropped zeros at the start of 2016, but that approach didn't cover buildings like this.\n<img src=\"https://i.ibb.co/b3xnFKV/site0.jpg\" alt=\"s0m0\"></li>\n<li><strong>Meter readings on another high or low level due to API errors. I thought those were API errors because they exist on multiple buildings in the same sites at the same dates.  That was the time when I started considering outlier's date frequency within site for detection.</strong>\n<img src=\"https://i.ibb.co/SJGMzrr/site0m1.jpg\" alt=\"s0m1\"></li>\n<li><strong>Meter readings on mean level at random dates.</strong> Removing them gave the least improvement compared to other outliers, but it still helped.\n<img src=\"https://i.ibb.co/BfLYbyr/s1m0.jpg\" alt=\"s1m0\"></li>\n<li><strong>Chilled Water, Steam and Hot Water meter readings that doesn't fit their seasons.</strong> Outliers at those dates are greater than outliers at other dates.\n<img src=\"https://i.ibb.co/yS7pdvV/b784m2.jpg\" alt=\"b784\"></li>\n</ul>\n\n<h2>What not to drop?</h2>\n\n<ul>\n<li><p><strong>Constant meter readings at the end of 2016.</strong> I didn't drop them because some of the test set have those values.\n<img src=\"https://i.ibb.co/XDCZ4Zt/b1264-0.jpg\" alt=\"b1264\"></p></li>\n<li><p><strong>Constant Chilled Water, Steam and Hot Water meter readings happening due to seasonality.</strong> If the model couldn't learn the effect of seasons, the result was disastrous.\n<img src=\"https://i.ibb.co/dtdLC3X/b1028m1.jpg\" alt=\"b1028\">\n<img src=\"https://i.ibb.co/tPd1cgF/b1012m3.jpg\" alt=\"b1012\"></p></li>\n<li><p><strong>Outliers that are too common. Dropping them hurts the model</strong>\n<img src=\"https://i.ibb.co/Y7KK7xX/b610m0.jpg\" alt=\"b610\"></p></li>\n</ul>\n\n<p>After doing this operation for every buildings' every meter type. I dropped <strong>789682</strong> data points and break <strong>1.04</strong> with LightGBM in this kernel. Kaggle still evaluates the previous submissions of the kernel so it is showing old score right now.</p>\n\n<p><a href=\"https://www.kaggle.com/gunesevitan/ashrae-lightgbm-1-048-no-leak\">https://www.kaggle.com/gunesevitan/ashrae-lightgbm-1-048-no-leak</a></p>",
  "messages": [
    {
      "id": 699392,
      "postDate": "2019-12-20T11:38:23.793Z",
      "content": "<p>The fact that outlier removal giving the largest boost to models was not a secret since the early stages. It was possible to reach <strong>1.05</strong> in this competition with only outlier removal. However most of the teams did it only for site 0.</p>\n\n<p>Traditional outlier detection is 1 dimensional, but outliers in this data are dependent to several external factors such as seasonality, meter type, outliers' date frequency within site, outliers' length frequency within site and etc. That's why it had to be done for every building separately even though it was very inefficient. I tried to make it a little bit more efficient with this detection function. It is the modified version of @marshi  code <a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#654909\">here</a>.</p>\n\n<p>This function basically returns the indices of constant series of the given building and meter type if the length of series are greater than <code>min_length</code>.  <code>min_length</code> parameter had to be tuned for every building differently since the length of series were always different. I did it for every building in <a href=\"https://www.kaggle.com/gunesevitan/ashrae-eda-and-preprocessing\">this</a> kernel.</p>\n\n<p>```\ndef filter2(building_id, meter, min_length, plot=False, verbose=False):\n    if verbose:\n        print(\"building_id: {}, meter: {}\".format(building_id, meter))\n    temp_df = df_train[(df_train['building_id'] == building_id) &amp; (df_train['meter'] == meter)] <br>\n    target = temp_df['meter_reading'].values</p>\n\n<pre><code>splitted_target = np.split(target, np.where(target[1:] != target[:-1])[0] + 1)\nsplitted_date = np.split(temp_df['timestamp'].values, np.where(target[1:] != target[:-1])[0] + 1)\n\nbuilding_idx = []\nfor i, x in enumerate(splitted_date):\n    if len(x) &gt; min_length:\n        start = x[0]\n        end = x[-1]\n        value = splitted_target[i][0]\n        idx = df_train.query('(@start &lt;= timestamp &lt;= @end) and meter_reading == @value and meter == @meter and building_id == @building_id', engine='python').index.tolist()\n        building_idx.extend(idx)\n        if verbose:\n            print('Length: {},\\t{}  -  {},\\tvalue: {}'.format(len(x), start, end, value))\n\nbuilding_idx = pd.Int64Index(building_idx)\nif plot:\n    fig, axes = plt.subplots(nrows=2, figsize=(16, 18), dpi=100)\n\n    temp_df.set_index('timestamp')['meter_reading'].plot(ax=axes[0])     \n    temp_df.drop(building_idx, axis=0).set_index('timestamp')['meter_reading'].plot(ax=axes[1])\n\n    axes[0].set_title(f'Building {building_id} raw meter readings')\n    axes[1].set_title(f'Building {building_id} filtered meter readings')\n\n    plt.show()\n\nreturn building_idx\n</code></pre>\n\n<p>```</p>\n\n<h2>What to drop?</h2>\n\n<ul>\n<li><strong>Electricity meter readings on another high or low level at the start of 2016.</strong> This was the only edge case this function was not used. I had to do it manually. Most of the public kernels only dropped zeros at the start of 2016, but that approach didn't cover buildings like this.\n<img src=\"https://i.ibb.co/b3xnFKV/site0.jpg\" alt=\"s0m0\"></li>\n<li><strong>Meter readings on another high or low level due to API errors. I thought those were API errors because they exist on multiple buildings in the same sites at the same dates.  That was the time when I started considering outlier's date frequency within site for detection.</strong>\n<img src=\"https://i.ibb.co/SJGMzrr/site0m1.jpg\" alt=\"s0m1\"></li>\n<li><strong>Meter readings on mean level at random dates.</strong> Removing them gave the least improvement compared to other outliers, but it still helped.\n<img src=\"https://i.ibb.co/BfLYbyr/s1m0.jpg\" alt=\"s1m0\"></li>\n<li><strong>Chilled Water, Steam and Hot Water meter readings that doesn't fit their seasons.</strong> Outliers at those dates are greater than outliers at other dates.\n<img src=\"https://i.ibb.co/yS7pdvV/b784m2.jpg\" alt=\"b784\"></li>\n</ul>\n\n<h2>What not to drop?</h2>\n\n<ul>\n<li><p><strong>Constant meter readings at the end of 2016.</strong> I didn't drop them because some of the test set have those values.\n<img src=\"https://i.ibb.co/XDCZ4Zt/b1264-0.jpg\" alt=\"b1264\"></p></li>\n<li><p><strong>Constant Chilled Water, Steam and Hot Water meter readings happening due to seasonality.</strong> If the model couldn't learn the effect of seasons, the result was disastrous.\n<img src=\"https://i.ibb.co/dtdLC3X/b1028m1.jpg\" alt=\"b1028\">\n<img src=\"https://i.ibb.co/tPd1cgF/b1012m3.jpg\" alt=\"b1012\"></p></li>\n<li><p><strong>Outliers that are too common. Dropping them hurts the model</strong>\n<img src=\"https://i.ibb.co/Y7KK7xX/b610m0.jpg\" alt=\"b610\"></p></li>\n</ul>\n\n<p>After doing this operation for every buildings' every meter type. I dropped <strong>789682</strong> data points and break <strong>1.04</strong> with LightGBM in this kernel. Kaggle still evaluates the previous submissions of the kernel so it is showing old score right now.</p>\n\n<p><a href=\"https://www.kaggle.com/gunesevitan/ashrae-lightgbm-1-048-no-leak\">https://www.kaggle.com/gunesevitan/ashrae-lightgbm-1-048-no-leak</a></p>",
      "rawMarkdown": "The fact that outlier removal giving the largest boost to models was not a secret since the early stages. It was possible to reach **1.05** in this competition with only outlier removal. However most of the teams did it only for site 0.\n\nTraditional outlier detection is 1 dimensional, but outliers in this data are dependent to several external factors such as seasonality, meter type, outliers' date frequency within site, outliers' length frequency within site and etc. That's why it had to be done for every building separately even though it was very inefficient. I tried to make it a little bit more efficient with this detection function. It is the modified version of @marshi  code [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#654909).\n\nThis function basically returns the indices of constant series of the given building and meter type if the length of series are greater than `min_length`.  `min_length` parameter had to be tuned for every building differently since the length of series were always different. I did it for every building in [this](https://www.kaggle.com/gunesevitan/ashrae-eda-and-preprocessing) kernel.\n\n```\ndef filter2(building_id, meter, min_length, plot=False, verbose=False):\n    if verbose:\n        print(\"building_id: {}, meter: {}\".format(building_id, meter))\n    temp_df = df_train[(df_train['building_id'] == building_id) &amp; (df_train['meter'] == meter)]        \n    target = temp_df['meter_reading'].values\n    \n    splitted_target = np.split(target, np.where(target[1:] != target[:-1])[0] + 1)\n    splitted_date = np.split(temp_df['timestamp'].values, np.where(target[1:] != target[:-1])[0] + 1)\n\n    building_idx = []\n    for i, x in enumerate(splitted_date):\n        if len(x) &gt; min_length:\n            start = x[0]\n            end = x[-1]\n            value = splitted_target[i][0]\n            idx = df_train.query('(@start &lt;= timestamp &lt;= @end) and meter_reading == @value and meter == @meter and building_id == @building_id', engine='python').index.tolist()\n            building_idx.extend(idx)\n            if verbose:\n                print('Length: {},\\t{}  -  {},\\tvalue: {}'.format(len(x), start, end, value))\n                \n    building_idx = pd.Int64Index(building_idx)\n    if plot:\n        fig, axes = plt.subplots(nrows=2, figsize=(16, 18), dpi=100)\n        \n        temp_df.set_index('timestamp')['meter_reading'].plot(ax=axes[0])     \n        temp_df.drop(building_idx, axis=0).set_index('timestamp')['meter_reading'].plot(ax=axes[1])\n        \n        axes[0].set_title(f'Building {building_id} raw meter readings')\n        axes[1].set_title(f'Building {building_id} filtered meter readings')\n        \n        plt.show()\n        \n    return building_idx\n```\n\n## What to drop?\n* **Electricity meter readings on another high or low level at the start of 2016.** This was the only edge case this function was not used. I had to do it manually. Most of the public kernels only dropped zeros at the start of 2016, but that approach didn't cover buildings like this.\n![s0m0](https://i.ibb.co/b3xnFKV/site0.jpg)\n* **Meter readings on another high or low level due to API errors. I thought those were API errors because they exist on multiple buildings in the same sites at the same dates.  That was the time when I started considering outlier's date frequency within site for detection.**\n![s0m1](https://i.ibb.co/SJGMzrr/site0m1.jpg)\n* **Meter readings on mean level at random dates.** Removing them gave the least improvement compared to other outliers, but it still helped.\n![s1m0](https://i.ibb.co/BfLYbyr/s1m0.jpg)\n* **Chilled Water, Steam and Hot Water meter readings that doesn't fit their seasons.** Outliers at those dates are greater than outliers at other dates.\n![b784](https://i.ibb.co/yS7pdvV/b784m2.jpg)\n\n## What not to drop?\n* **Constant meter readings at the end of 2016.** I didn't drop them because some of the test set have those values.\n![b1264](https://i.ibb.co/XDCZ4Zt/b1264-0.jpg)\n\n* **Constant Chilled Water, Steam and Hot Water meter readings happening due to seasonality.** If the model couldn't learn the effect of seasons, the result was disastrous.\n![b1028](https://i.ibb.co/dtdLC3X/b1028m1.jpg)\n![b1012](https://i.ibb.co/tPd1cgF/b1012m3.jpg)\n\n* **Outliers that are too common. Dropping them hurts the model**\n![b610](https://i.ibb.co/Y7KK7xX/b610m0.jpg)\n\nAfter doing this operation for every buildings' every meter type. I dropped **789682** data points and break **1.04** with LightGBM in this kernel. Kaggle still evaluates the previous submissions of the kernel so it is showing old score right now.\n\nhttps://www.kaggle.com/gunesevitan/ashrae-lightgbm-1-048-no-leak",
      "votes": 82
    },
    {
      "id": 699465,
      "postDate": "2019-12-20T13:41:11.937Z",
      "content": "<p>Nice <a href=\"/gunesevitan\">@gunesevitan</a> !\nI was experimenting with sigma clipping on the last days but was too late to get anything out of it. Here's the code anyways:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F451025%2F9e26e217ca0ebdfd50f33920a1f31d7e%2FWhatsApp%20Image%202019-12-18%20at%2011.05.28.jpeg?generation=1576849115018279&amp;alt=media\" alt=\"sigma clipping\">\n<code>\ndef sigma_filter(df, tolerance=3):\n    ''' sigma clipping '''\n    df['meter_reading_ln'] = np.log1p(df.meter_reading)\n    stats = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter'])\\\n                    .rolling(24*7, min_periods=2, center=True).meter_reading_ln.agg(['median'])\n    std = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter']).meter_reading_ln.std()\n    stats['max'] = np.expm1(stats['median'] + tolerance*std)\n    stats['min'] = np.expm1(stats['median'] - tolerance*std)\n    stats['median'] = np.expm1(stats['median'])\n    df = df.merge(stats[['median', 'min', 'max']], left_on=['building_id', 'meter', 'timestamp'], right_index=True)\n    return df[(df.meter_reading &lt;= df['max']) &amp; (df.meter_reading &gt;= df['min'])]\n</code></p>",
      "rawMarkdown": "Nice @gunesevitan !\nI was experimenting with sigma clipping on the last days but was too late to get anything out of it. Here's the code anyways:\n![sigma clipping](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F451025%2F9e26e217ca0ebdfd50f33920a1f31d7e%2FWhatsApp%20Image%202019-12-18%20at%2011.05.28.jpeg?generation=1576849115018279&amp;alt=media)\n```\ndef sigma_filter(df, tolerance=3):\n    ''' sigma clipping '''\n    df['meter_reading_ln'] = np.log1p(df.meter_reading)\n    stats = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter'])\\\n                    .rolling(24*7, min_periods=2, center=True).meter_reading_ln.agg(['median'])\n    std = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter']).meter_reading_ln.std()\n    stats['max'] = np.expm1(stats['median'] + tolerance*std)\n    stats['min'] = np.expm1(stats['median'] - tolerance*std)\n    stats['median'] = np.expm1(stats['median'])\n    df = df.merge(stats[['median', 'min', 'max']], left_on=['building_id', 'meter', 'timestamp'], right_index=True)\n    return df[(df.meter_reading &lt;= df['max']) &amp; (df.meter_reading &gt;= df['min'])]\n```\n",
      "votes": 7,
      "replies": [
        {
          "id": 699677,
          "postDate": "2019-12-20T18:40:11.107Z",
          "content": "<p>Very neat approach... If only you had time to try it :/</p>",
          "rawMarkdown": "Very neat approach... If only you had time to try it :/",
          "votes": 1
        },
        {
          "id": 701578,
          "postDate": "2019-12-23T16:22:26.080Z",
          "content": "<p>how did you choose tolerance=3 ?</p>",
          "rawMarkdown": "how did you choose tolerance=3 ?",
          "votes": 1
        },
        {
          "id": 704586,
          "postDate": "2019-12-27T16:54:30.237Z",
          "content": "<p><a href=\"/dhacker\">@dhacker</a> to be clear, the plot above is actually using tolerance/sigma=0.5 , but it cuts too much stuff out in some other cases, so 3 seemed like a safe value (in the building 1251, as above, it only eliminates the zeros) \nHowever, to be able to use larger sigma values in this data, it's important to use log scale!</p>",
          "rawMarkdown": "@dhacker to be clear, the plot above is actually using tolerance/sigma=0.5 , but it cuts too much stuff out in some other cases, so 3 seemed like a safe value (in the building 1251, as above, it only eliminates the zeros) \nHowever, to be able to use larger sigma values in this data, it's important to use log scale!",
          "votes": 3
        }
      ]
    },
    {
      "id": 700050,
      "postDate": "2019-12-21T11:07:21.243Z",
      "content": "<p>nice work <a href=\"/gunesevitan\">@gunesevitan</a> . I also did analysis of all buildings, but i guess i did some aggressive cleaning of data that's why didn't get much help. The only thing which helped me apart from public cleaning of site 0,was:</p>\n\n<p><code>train_df = train_df[train_df.groupby(['building_id','meter'])['meter_reading'].cumsum().gt(0)]</code></p>\n\n<p>removed trailing zeros from other buildings and it gave additional boost of around .02</p>",
      "rawMarkdown": "nice work @gunesevitan . I also did analysis of all buildings, but i guess i did some aggressive cleaning of data that's why didn't get much help. The only thing which helped me apart from public cleaning of site 0,was:\n\n`train_df = train_df[train_df.groupby(['building_id','meter'])['meter_reading'].cumsum().gt(0)]`\n\nremoved trailing zeros from other buildings and it gave additional boost of around .02",
      "votes": 3,
      "replies": [
        {
          "id": 700068,
          "postDate": "2019-12-21T12:05:48.923Z",
          "content": "<p>Nice approach, can it handle the edge cases in site 0? My cleaning code for site 0 is a mess. I found the last zero for every building before <code>2016-09-01</code> then dropped everything before that date.\n```\nleading_zeros = defaultdict(list)\nfor building_id in range(105):\n    leading_zeros_last_date = df_train.query('building_id == <a href=\"/building\">@building</a>_id and meter_reading == 0 and timestamp &lt; \"2016-09-01 01:00:00\"', engine='python')['timestamp'].max()\n    leading_zeros[leading_zeros_last_date].append(building_id)</p>\n\n<p>for timestamp in leading_zeros.keys():\n    building_ids = leading_zeros[pd.Timestamp(timestamp)]\n    filtered_idx = df_train[df_train['building_id'].isin(building_ids)].query('meter == 0 and timestamp &lt;= @timestamp').index\n    df_train.loc[filtered_idx, 'IsFiltered'] = 1\n```</p>",
          "rawMarkdown": "Nice approach, can it handle the edge cases in site 0? My cleaning code for site 0 is a mess. I found the last zero for every building before `2016-09-01` then dropped everything before that date.\n```\nleading_zeros = defaultdict(list)\nfor building_id in range(105):\n    leading_zeros_last_date = df_train.query('building_id == @building_id and meter_reading == 0 and timestamp &lt; \"2016-09-01 01:00:00\"', engine='python')['timestamp'].max()\n    leading_zeros[leading_zeros_last_date].append(building_id)\n\nfor timestamp in leading_zeros.keys():\n    building_ids = leading_zeros[pd.Timestamp(timestamp)]\n    filtered_idx = df_train[df_train['building_id'].isin(building_ids)].query('meter == 0 and timestamp &lt;= @timestamp').index\n    df_train.loc[filtered_idx, 'IsFiltered'] = 1\n```"
        },
        {
          "id": 701072,
          "postDate": "2019-12-23T03:40:01.077Z",
          "content": "<p><a href=\"/gunesevitan\">@gunesevitan</a> it didn't take care of edge cases.</p>",
          "rawMarkdown": "@gunesevitan it didn't take care of edge cases."
        }
      ]
    },
    {
      "id": 700075,
      "postDate": "2019-12-21T12:24:47.547Z",
      "content": "<p>Great work. Data cleaning was the key to get a good score. </p>",
      "rawMarkdown": "Great work. Data cleaning was the key to get a good score. ",
      "votes": 2
    },
    {
      "id": 700942,
      "postDate": "2019-12-22T21:06:12.807Z",
      "content": "<p>Very interesting. Did you find a way to locally validate the improvements from data cleaning, or did you have to submit predictions each time you modified your cleaning process and look at LB score?</p>",
      "rawMarkdown": "Very interesting. Did you find a way to locally validate the improvements from data cleaning, or did you have to submit predictions each time you modified your cleaning process and look at LB score?",
      "votes": 1,
      "replies": [
        {
          "id": 701087,
          "postDate": "2019-12-23T04:09:55.633Z",
          "content": "<p>I had to submit each time.</p>",
          "rawMarkdown": "I had to submit each time.",
          "votes": 2
        },
        {
          "id": 702334,
          "postDate": "2019-12-24T15:06:43.747Z",
          "content": "<p>I think local validation wasn't very useful to optimize LB score for this competition.\nValidating on LB or on the leaked data was better.</p>",
          "rawMarkdown": "I think local validation wasn't very useful to optimize LB score for this competition.\nValidating on LB or on the leaked data was better.",
          "votes": 1
        },
        {
          "id": 702389,
          "postDate": "2019-12-24T16:13:34.053Z",
          "content": "<p>Yes, totally agree. That's why I thought low submission novice teams were cheating. I used more than 20 submissions only for verifying outlier removal. How come they are using less than 10 submissions and have the same score as me?</p>",
          "rawMarkdown": "Yes, totally agree. That's why I thought low submission novice teams were cheating. I used more than 20 submissions only for verifying outlier removal. How come they are using less than 10 submissions and have the same score as me?",
          "votes": 1
        }
      ]
    },
    {
      "id": 700684,
      "postDate": "2019-12-22T12:36:04.327Z",
      "content": "<p>amazing work !</p>",
      "rawMarkdown": "amazing work !",
      "votes": 1
    },
    {
      "id": 3018978,
      "postDate": "2024-10-16T07:21:23.267Z",
      "content": "<p>Hi! Is there an automatic method to remove these outliers?</p>",
      "rawMarkdown": "Hi! Is there an automatic method to remove these outliers?"
    },
    {
      "id": 750719,
      "postDate": "2020-02-19T16:26:14.617Z",
      "content": "<p>Great approach buddy ! Keep working ! Thanks <a href=\"/gunesevitan\">@gunesevitan</a> !</p>",
      "rawMarkdown": "Great approach buddy ! Keep working ! Thanks @gunesevitan !"
    },
    {
      "id": 702435,
      "postDate": "2019-12-24T17:10:06.493Z",
      "content": "<p>Great work. </p>",
      "rawMarkdown": "Great work. "
    },
    {
      "id": 702076,
      "postDate": "2019-12-24T08:24:16.867Z",
      "content": "<p>Awesome job, well done</p>",
      "rawMarkdown": "Awesome job, well done"
    },
    {
      "id": 700557,
      "postDate": "2019-12-22T08:00:51.223Z",
      "content": "<p>Great work done.</p>",
      "rawMarkdown": "Great work done."
    },
    {
      "id": 700376,
      "postDate": "2019-12-21T22:28:40.270Z",
      "content": "<p>Nice insights and explanation! Our team had a strategy for deleting 'near zero' observations but we got into some of the pitfalls you mentioned. Happy to see this so we can learn from it!</p>",
      "rawMarkdown": "Nice insights and explanation! Our team had a strategy for deleting 'near zero' observations but we got into some of the pitfalls you mentioned. Happy to see this so we can learn from it!"
    },
    {
      "id": 699741,
      "postDate": "2019-12-20T20:43:15.880Z",
      "content": "<p>This is awesome work. Thank you for sharing :) </p>",
      "rawMarkdown": "This is awesome work. Thank you for sharing :) "
    },
    {
      "id": 699395,
      "postDate": "2019-12-20T11:42:04.767Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 699438,
      "postDate": "2019-12-20T13:08:38.330Z",
      "content": "<p>Thanks for sharing.</p>",
      "rawMarkdown": "Thanks for sharing."
    }
  ],
  "comments": [
    {
      "id": 699465,
      "author_name": "Henrique Mendonça",
      "author_url": "",
      "post_date": "2019-12-20T13:41:11.937000",
      "content": "<p>Nice <a href=\"/gunesevitan\">@gunesevitan</a> !\nI was experimenting with sigma clipping on the last days but was too late to get anything out of it. Here's the code anyways:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F451025%2F9e26e217ca0ebdfd50f33920a1f31d7e%2FWhatsApp%20Image%202019-12-18%20at%2011.05.28.jpeg?generation=1576849115018279&amp;alt=media\" alt=\"sigma clipping\">\n<code>\ndef sigma_filter(df, tolerance=3):\n    ''' sigma clipping '''\n    df['meter_reading_ln'] = np.log1p(df.meter_reading)\n    stats = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter'])\\\n                    .rolling(24*7, min_periods=2, center=True).meter_reading_ln.agg(['median'])\n    std = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter']).meter_reading_ln.std()\n    stats['max'] = np.expm1(stats['median'] + tolerance*std)\n    stats['min'] = np.expm1(stats['median'] - tolerance*std)\n    stats['median'] = np.expm1(stats['median'])\n    df = df.merge(stats[['median', 'min', 'max']], left_on=['building_id', 'meter', 'timestamp'], right_index=True)\n    return df[(df.meter_reading &lt;= df['max']) &amp; (df.meter_reading &gt;= df['min'])]\n</code></p>",
      "votes": 7,
      "replies": [
        {
          "id": 699677,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2019-12-20T18:40:11.107000",
          "content": "<p>Very neat approach... If only you had time to try it :/</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 701578,
          "author_name": "Kim Wilson",
          "author_url": "",
          "post_date": "2019-12-23T16:22:26.080000",
          "content": "<p>how did you choose tolerance=3 ?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 704586,
          "author_name": "Henrique Mendonça",
          "author_url": "",
          "post_date": "2019-12-27T16:54:30.237000",
          "content": "<p><a href=\"/dhacker\">@dhacker</a> to be clear, the plot above is actually using tolerance/sigma=0.5 , but it cuts too much stuff out in some other cases, so 3 seemed like a safe value (in the building 1251, as above, it only eliminates the zeros) \nHowever, to be able to use larger sigma values in this data, it's important to use log scale!</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 700050,
      "author_name": "HarshitMehta",
      "author_url": "",
      "post_date": "2019-12-21T11:07:21.243000",
      "content": "<p>nice work <a href=\"/gunesevitan\">@gunesevitan</a> . I also did analysis of all buildings, but i guess i did some aggressive cleaning of data that's why didn't get much help. The only thing which helped me apart from public cleaning of site 0,was:</p>\n\n<p><code>train_df = train_df[train_df.groupby(['building_id','meter'])['meter_reading'].cumsum().gt(0)]</code></p>\n\n<p>removed trailing zeros from other buildings and it gave additional boost of around .02</p>",
      "votes": 3,
      "replies": [
        {
          "id": 700068,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2019-12-21T12:05:48.923000",
          "content": "<p>Nice approach, can it handle the edge cases in site 0? My cleaning code for site 0 is a mess. I found the last zero for every building before <code>2016-09-01</code> then dropped everything before that date.\n```\nleading_zeros = defaultdict(list)\nfor building_id in range(105):\n    leading_zeros_last_date = df_train.query('building_id == <a href=\"/building\">@building</a>_id and meter_reading == 0 and timestamp &lt; \"2016-09-01 01:00:00\"', engine='python')['timestamp'].max()\n    leading_zeros[leading_zeros_last_date].append(building_id)</p>\n\n<p>for timestamp in leading_zeros.keys():\n    building_ids = leading_zeros[pd.Timestamp(timestamp)]\n    filtered_idx = df_train[df_train['building_id'].isin(building_ids)].query('meter == 0 and timestamp &lt;= @timestamp').index\n    df_train.loc[filtered_idx, 'IsFiltered'] = 1\n```</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 701072,
          "author_name": "HarshitMehta",
          "author_url": "",
          "post_date": "2019-12-23T03:40:01.077000",
          "content": "<p><a href=\"/gunesevitan\">@gunesevitan</a> it didn't take care of edge cases.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 700075,
      "author_name": "Oleg Knaub",
      "author_url": "",
      "post_date": "2019-12-21T12:24:47.547000",
      "content": "<p>Great work. Data cleaning was the key to get a good score. </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 700942,
      "author_name": "Larry D.",
      "author_url": "",
      "post_date": "2019-12-22T21:06:12.807000",
      "content": "<p>Very interesting. Did you find a way to locally validate the improvements from data cleaning, or did you have to submit predictions each time you modified your cleaning process and look at LB score?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 701087,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2019-12-23T04:09:55.633000",
          "content": "<p>I had to submit each time.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 702334,
          "author_name": "Vopani",
          "author_url": "",
          "post_date": "2019-12-24T15:06:43.747000",
          "content": "<p>I think local validation wasn't very useful to optimize LB score for this competition.\nValidating on LB or on the leaked data was better.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 702389,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2019-12-24T16:13:34.053000",
          "content": "<p>Yes, totally agree. That's why I thought low submission novice teams were cheating. I used more than 20 submissions only for verifying outlier removal. How come they are using less than 10 submissions and have the same score as me?</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 700684,
      "author_name": "ravi tanwar",
      "author_url": "",
      "post_date": "2019-12-22T12:36:04.327000",
      "content": "<p>amazing work !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3018978,
      "author_name": "Gio Almonte",
      "author_url": "",
      "post_date": "2024-10-16T07:21:23.267000",
      "content": "<p>Hi! Is there an automatic method to remove these outliers?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 750719,
      "author_name": "Caesar Lupum",
      "author_url": "",
      "post_date": "2020-02-19T16:26:14.617000",
      "content": "<p>Great approach buddy ! Keep working ! Thanks <a href=\"/gunesevitan\">@gunesevitan</a> !</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 702435,
      "author_name": "remesh ck ",
      "author_url": "",
      "post_date": "2019-12-24T17:10:06.493000",
      "content": "<p>Great work. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 702076,
      "author_name": "Jonathan Mallia",
      "author_url": "",
      "post_date": "2019-12-24T08:24:16.867000",
      "content": "<p>Awesome job, well done</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 700557,
      "author_name": "Waqas Nawaz",
      "author_url": "",
      "post_date": "2019-12-22T08:00:51.223000",
      "content": "<p>Great work done.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 700376,
      "author_name": "Carlo",
      "author_url": "",
      "post_date": "2019-12-21T22:28:40.270000",
      "content": "<p>Nice insights and explanation! Our team had a strategy for deleting 'near zero' observations but we got into some of the pitfalls you mentioned. Happy to see this so we can learn from it!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 699741,
      "author_name": "Vishal Sharma",
      "author_url": "",
      "post_date": "2019-12-20T20:43:15.880000",
      "content": "<p>This is awesome work. Thank you for sharing :) </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 699395,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-12-20T11:42:04.767000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 699438,
      "author_name": "Dean",
      "author_url": "",
      "post_date": "2019-12-20T13:08:38.330000",
      "content": "<p>Thanks for sharing.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "699392": "The fact that outlier removal giving the largest boost to models was not a secret since the early stages. It was possible to reach **1.05** in this competition with only outlier removal. However most of the teams did it only for site 0.\n\nTraditional outlier detection is 1 dimensional, but outliers in this data are dependent to several external factors such as seasonality, meter type, outliers' date frequency within site, outliers' length frequency within site and etc. That's why it had to be done for every building separately even though it was very inefficient. I tried to make it a little bit more efficient with this detection function. It is the modified version of @marshi  code [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#654909).\n\nThis function basically returns the indices of constant series of the given building and meter type if the length of series are greater than `min_length`.  `min_length` parameter had to be tuned for every building differently since the length of series were always different. I did it for every building in [this](https://www.kaggle.com/gunesevitan/ashrae-eda-and-preprocessing) kernel.\n\n```\ndef filter2(building_id, meter, min_length, plot=False, verbose=False):\n    if verbose:\n        print(\"building_id: {}, meter: {}\".format(building_id, meter))\n    temp_df = df_train[(df_train['building_id'] == building_id) &amp; (df_train['meter'] == meter)]        \n    target = temp_df['meter_reading'].values\n    \n    splitted_target = np.split(target, np.where(target[1:] != target[:-1])[0] + 1)\n    splitted_date = np.split(temp_df['timestamp'].values, np.where(target[1:] != target[:-1])[0] + 1)\n\n    building_idx = []\n    for i, x in enumerate(splitted_date):\n        if len(x) &gt; min_length:\n            start = x[0]\n            end = x[-1]\n            value = splitted_target[i][0]\n            idx = df_train.query('(@start &lt;= timestamp &lt;= @end) and meter_reading == @value and meter == @meter and building_id == @building_id', engine='python').index.tolist()\n            building_idx.extend(idx)\n            if verbose:\n                print('Length: {},\\t{}  -  {},\\tvalue: {}'.format(len(x), start, end, value))\n                \n    building_idx = pd.Int64Index(building_idx)\n    if plot:\n        fig, axes = plt.subplots(nrows=2, figsize=(16, 18), dpi=100)\n        \n        temp_df.set_index('timestamp')['meter_reading'].plot(ax=axes[0])     \n        temp_df.drop(building_idx, axis=0).set_index('timestamp')['meter_reading'].plot(ax=axes[1])\n        \n        axes[0].set_title(f'Building {building_id} raw meter readings')\n        axes[1].set_title(f'Building {building_id} filtered meter readings')\n        \n        plt.show()\n        \n    return building_idx\n```\n\n## What to drop?\n* **Electricity meter readings on another high or low level at the start of 2016.** This was the only edge case this function was not used. I had to do it manually. Most of the public kernels only dropped zeros at the start of 2016, but that approach didn't cover buildings like this.\n![s0m0](https://i.ibb.co/b3xnFKV/site0.jpg)\n* **Meter readings on another high or low level due to API errors. I thought those were API errors because they exist on multiple buildings in the same sites at the same dates.  That was the time when I started considering outlier's date frequency within site for detection.**\n![s0m1](https://i.ibb.co/SJGMzrr/site0m1.jpg)\n* **Meter readings on mean level at random dates.** Removing them gave the least improvement compared to other outliers, but it still helped.\n![s1m0](https://i.ibb.co/BfLYbyr/s1m0.jpg)\n* **Chilled Water, Steam and Hot Water meter readings that doesn't fit their seasons.** Outliers at those dates are greater than outliers at other dates.\n![b784](https://i.ibb.co/yS7pdvV/b784m2.jpg)\n\n## What not to drop?\n* **Constant meter readings at the end of 2016.** I didn't drop them because some of the test set have those values.\n![b1264](https://i.ibb.co/XDCZ4Zt/b1264-0.jpg)\n\n* **Constant Chilled Water, Steam and Hot Water meter readings happening due to seasonality.** If the model couldn't learn the effect of seasons, the result was disastrous.\n![b1028](https://i.ibb.co/dtdLC3X/b1028m1.jpg)\n![b1012](https://i.ibb.co/tPd1cgF/b1012m3.jpg)\n\n* **Outliers that are too common. Dropping them hurts the model**\n![b610](https://i.ibb.co/Y7KK7xX/b610m0.jpg)\n\nAfter doing this operation for every buildings' every meter type. I dropped **789682** data points and break **1.04** with LightGBM in this kernel. Kaggle still evaluates the previous submissions of the kernel so it is showing old score right now.\n\nhttps://www.kaggle.com/gunesevitan/ashrae-lightgbm-1-048-no-leak",
    "699465": "Nice @gunesevitan !\nI was experimenting with sigma clipping on the last days but was too late to get anything out of it. Here's the code anyways:\n![sigma clipping](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F451025%2F9e26e217ca0ebdfd50f33920a1f31d7e%2FWhatsApp%20Image%202019-12-18%20at%2011.05.28.jpeg?generation=1576849115018279&amp;alt=media)\n```\ndef sigma_filter(df, tolerance=3):\n    ''' sigma clipping '''\n    df['meter_reading_ln'] = np.log1p(df.meter_reading)\n    stats = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter'])\\\n                    .rolling(24*7, min_periods=2, center=True).meter_reading_ln.agg(['median'])\n    std = df.reset_index().set_index('timestamp').groupby(['building_id', 'meter']).meter_reading_ln.std()\n    stats['max'] = np.expm1(stats['median'] + tolerance*std)\n    stats['min'] = np.expm1(stats['median'] - tolerance*std)\n    stats['median'] = np.expm1(stats['median'])\n    df = df.merge(stats[['median', 'min', 'max']], left_on=['building_id', 'meter', 'timestamp'], right_index=True)\n    return df[(df.meter_reading &lt;= df['max']) &amp; (df.meter_reading &gt;= df['min'])]\n```\n",
    "700050": "nice work @gunesevitan . I also did analysis of all buildings, but i guess i did some aggressive cleaning of data that's why didn't get much help. The only thing which helped me apart from public cleaning of site 0,was:\n\n`train_df = train_df[train_df.groupby(['building_id','meter'])['meter_reading'].cumsum().gt(0)]`\n\nremoved trailing zeros from other buildings and it gave additional boost of around .02",
    "700075": "Great work. Data cleaning was the key to get a good score. ",
    "700942": "Very interesting. Did you find a way to locally validate the improvements from data cleaning, or did you have to submit predictions each time you modified your cleaning process and look at LB score?",
    "700684": "amazing work !",
    "3018978": "Hi! Is there an automatic method to remove these outliers?",
    "750719": "Great approach buddy ! Keep working ! Thanks @gunesevitan !",
    "702435": "Great work. ",
    "702076": "Awesome job, well done",
    "700557": "Great work done.",
    "700376": "Nice insights and explanation! Our team had a strategy for deleting 'near zero' observations but we got into some of the pitfalls you mentioned. Happy to see this so we can learn from it!",
    "699741": "This is awesome work. Thank you for sharing :) ",
    "699395": "",
    "699438": "Thanks for sharing."
  }
}