{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Prophet's Prophecy\n![Prophet.png](attachment:Prophet.png)\n\nThis competition is pretty much ruined with all the scraping opportunities to hunt down the test data labels. A lot of us are here to learn true Data Science. This notebook is for such people.\n\nProphet is a time series model open sourced by Facebook in 2017:  https://facebook.github.io/prophet/\n\nThe notebook shows the building blocks for using Prophet to forecast the energy consumption. We will be building a Prophet model for each **building x meter** combination.","attachments":{"Prophet.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAeIAAACKCAIAAAD5W953AAAXlUlEQVR42uzdeXhU5b0H8DPLmS2Tmez7npAFkpCEgAkgINJiQRQr1EYQab1et7p0ub3a2ntbn9va1bpURatUkSKiAiLKIiBrNkL2jZCFLJN1kpnMJLPPnPuU3OujQOYckjNnziTfz+Nf4XXmzHvOfOcs7/t7xUuKthEAAMBXQnQBAABiGgAAENMAAIhpAABATAMAAGIaAAAxDQAAiGkAAMQ0AAAgpgEAADENAICYBgAAxDQAAGIaAAAQ0wAAgJgGAEBMAwAAYhoAADENAAB8I0YXAP9lJIVuuiNncW68hBQ1tQ/tPVp/6HQLugVmCVFc1u3oBeCzm7Jj//ZfdyREB4pE/7r4Cw30W5afKBaLLjRo0DkwG+CmB/DdU1uXXPvHLXfm5mZEoXMAMQ3gZQvmRcdGqK/7TysWJaJ/ADEN4GVhwX6T/VNkmD/6BxDTAF42Mmqe7J+GRkzoH0BMA3jZ+dqeEf314/jshcvoH0BMA3iZi6Jeeq/42r/vP9ZYUt2F/oHZAAPygO86enSNbUORof7hwUqCILS68Xf2Vb7xQTl6BmYJwZKibegF8Alqf5mEFA2NjKMrYFbBLETwGaNGCzoBZiHcmwYAQEwDAABiGgAAMQ0AAFzDI0TgjkJG5l2p0UGSwmGdqbFtqKNnBN0CgJgG71MqJD/ckL9xdaZAIPj636ub+t47UFVW040uApgMpreAx2Ukh734i7UF8+OuymiCICJC/VcvneNyEdXNfegoAMQ0eEFUmOrlX94edmUC4WTy5kVZrI66lgF0F8C18AgRPOtHmwsD1XLaZo/eWzAnPhjdBYCYBk5lp0Usy09g2HjjbVnoMQDENHBq6YIE5o1vLUxGjwEgpoFT6YmhzBtLJeKk2CB0GgBiGrij8pfeUHt/Pyk6DQAxDdwZN9luqL3JbEOnASCmgTutXcM31P5yrx6dBoCYBu4UV97AOlgny9vtdic6DQAxDdwpq+2ubmI6vfDjow3oMQDENHDt1V2lTJrtOlhT1diL7gJATAPXmtoGn/7LYYfD5abN/mONrzFLc4BZCEvWznYSUpSeFCqXkRJS5HS6YiMDhnUmq93hclHjZpvF6mhuH5r+uyTGBD64ceGyhYlX/X1AO/bOvgufftmMHQHAfkxnJIVuWZ93bc0z5gxjln7tmMVq1wwYNIOGiXQwme3cfPKs1Ij8zGi2Xk03atYMGrr7R202p93hNI5b+bzXg9TyuSnhC7NislLDRUKhRCKKjVBft2WnRm9zOPuGjBfqNeeqOge0YxRFTSOsg/Izo+Mi1SQp0upMja0DxVU38IwxIzl007ocsUg05Q2w2R0dPTqCIC51Dg9ojcOjZpvNMWbiaBRgTLj60U0Fwsm/Mg6n61cvHZ1yB0tI0S8fvkUqEbt5/T2Hamsv9rP4oR7bVBAbEcDl0et0uf55oLqxbXDKr5AcG/TDu/NFIu/cS3BR1Ovvl3b3jTL/X6Zebzo2MuDm/AQWt35Yb6pr6a9p7q9p7mu5rPV0ZwWoZFvW55JiEbsva7bYLzRo2nt0F+o1Fxo0vEpnoUCwbGHiqsLkxJig+GhGX62JZnPig5flJ/zg7gWXLmv3H2s8db5jahvQ0TMynXUAAlXyxbnxEpK1XWazOyvqNZUNmsa2wfpLAy4X5dH+Dw9Ruq9wYrM7SbHINtXhLkKhIDstIjTIz02b0+c72I3p7NSIeXPCuTyMKYr64PPa6byCSCxcuiDeWzFNEMS+Lxo4immKYPmYDg5QrFiUtGJR0tDIeE//6M4D1WW1nq0Wz3pGEwQhl5FLFyQsXZCw7pb0UaNl3xcNn59uMVvs3g1oqURcmBu3/taM3LnRIuEUL4AC/GV+ckl7t9fWWxEIBCxm9MTp5+LcuMW5cbpRc1v3yIeH64qrOinKc9tPH0DTuTwlCMJiddBGObsfyubwyTGUlE9tLR9XbwkN8gsN8kuJDz5W3LrzQPXA8JgvHgeBKnmgSv7jrUs33pb1t50lZys7vbUluRlRj20qSIkPFk/v9MHhdP39w/Pd/aPEjBOolueroxdkRp8oadvxSWVbF5b+AsQ0A/5+0ru+NS89OeytD8/79CJMMRHqZx5ecaq849VdpeMmTidDq/ykRbfPX7U4JTLUf5ov5XJRv37l+Pm6nhn8ZRBcqdKXlhjyys6Sc977WQW4+hqI59uXkRT6h5/etrLAt0tcqpWyO1ZmPP3g8ohpxyVzUWGq55781n135kay8aY7P60+W3l5NnwlYiLUzz3xrY23ZcmlWCkUENMMT/jFwqfuX7wkL97X+/qWm5IeLbpJ7S/j4L2iw1V/+9U6toayXGjQ7DxQ5X7s80wilYh+tLlw1eIUBAQgppkKUise31w4AxZhWlmQfN+duZ5+l+y0iJefXed++UHmLnUOP/PCEc4GSvKESCh48v4lLA7ZBJjhMT1xKfpIUYFK6fP1iG9fkV6YE+e5109PCv3N46vCWcrooZHxP/z91GzL6AkyifjRewtCAhWICUBMM7UoO6bAkwHHDaVC8sCGfDdzEKYjPFj53BOr3I+cZW5weOyFf5xlZRaij0pNCLl25iQAxzz4kMThdJ0q77DZndeOBKUoIjLMXy4l5VKx0k8axGDl6Ql3rZp75vxls5WLkzuT2X6irM39IDaVUhaklktIkVIhCQ3yYzjoNSEmMDcjspTt4SsqpfTpf18eFaa6oc84MDxWUt3V2Dqg1ZtEV7afIgini4oM8SdJ4ZkLPvbY8FR5h8lqv+5uUPvLQgIVElKskJHMf8k2rs769ESz3YEKq1N0WaNrbBsUClgbr213uEzTm4ggIAi9wawzWJhsk1AoCAv2UyrcXcdbrA6tbpx20PoEqVR8oxNfPRjTVqvj7Y8quvr07rsgJz1yUXbsyoIkJvmSlRoRG6Vu6dBycHh1949u212mN1iYNE6JC543J/zWwuS8uVFMrqZzMqJYj+mVBckLs2KYtz9e0nbwZHNFfc91J3Q0XBrwxVDYd6yhol7j/iuakRKWFBN0x60Zc5PDaF8wNlK9dnna/uONCNypqajXvPjuOV5t0sUO7YYndzGcxyQUCJ7csnj9qrlu2gyNjD/86/0Mw1dw5RSWLzFtdzhpt9vloiobeysbez88XPfU/Utuzk+gnYKxcXXWb7d9yc3ulElIgmAU061dw61dw8dKWm+7OfXfNuTTruk3NyUsQCVj+BvA8PJ887ocho2b2gb/sbeyvLb7Rg8X/pOQNIc0RRCNrYONrYOlNV33rMkuWjuf9jXXrkBMzzTMhy05CcIwRlOiZ9xsGzVaPLe1fLk3Paw3/W7bSSZzCjK5LSBwQ8ZNto+P1L+7v5J2dnju3KgrvwHskErEW9bnMRmUTVHEsZLWn/z+8+KqzpmX0TdEqzNte7/s9ffLaFsqFdLkuGBE2+wl8PL78+gRotlq/+s757Q6k/tmchmZwu+Rebs/q61r6afd7xEhSrbeMSpMxWRcOUUQB082//6NU9wU8BMI+P7tc7qoQ6cv0j4jjY1Uh7H0VBbAt2N64pz6WEkr3amN5IaeknnFgRP0BZQDGD84pVW0NpsU0+/KU+XtL+04Z7E5OAjoBfOif7x1Kf+/ACOj5kOnW2ibcTl9FOAq/JoOS1FUE10ZWZlUHMr7U5u6ln4XRbl/uh0exM7ZdFxkQGYq/Y2g1s7hF989x/Bh9LR++YWCorXzt96VJxAKDp9umU5dYG5c6tSOGi3uZ4cGB2D0NOBs+v8xuRMvFvF9uLfLRdHeWGBr1nhKfHBcJH3x6A8O1dLeUJo+Uiz8zeOrHvzeQrmMlEnEBTmx/P8OXOzQ0lbjZHKxAjArzqYnnnFNv+qu93/9GFT11QwaWHmvby+hLz1xrrLzi3Otnv7Uc5PDHr23YH56xFe7b058iEJGmiy8nsRo9fxdIIAZFdNikdDXM3piuI+fXOK+zbiZhaKmYrEwOy3SfRuTxX7k7CVPj+soyIl7csviqxbrKsiJFfH+0kcmEYuEOFkGxDRjASr6WwE23p/+xEUF0N6ZMbER0wXZsbRn7na7s9iT1ZOlEvG6W9I33ZETGuh3zb0C0fz0yLP8nsqYkRJGe0+Dm/V3JKQoKkzF5CtwLZeLUvlJ1f68K3ojl5FSidjpmu5ZgtNJURSFmOaF+ek054bDetPA8Dif+5QUC7+/Jtt9G4ogunpZWAYlPESpVNCctlc29npudIdSIXm46Kb1t046RyslLojnMT0vJYx2OtIgJ4ecQCDY8ceNgpkVMUty42Ij1k5/nUmjyfrSu+f6tWOIaS9LTQi5KZvmoRNFEZWNGj73aWFOHO2js+rGXlbKRAQH0A968dz6XkFqxQvPrElxO/UjKTaIzzsrLFj5nZvT3LcZ0ZtGRk3cbI9gxkVMgEoeoGJh7OnIqPlKYQ3EtFdJJeL/fHB5ON2kD73RzMGosin2pli4KCvmiS1LaAvgdWh0w3rTtE/bRUzKbPax9KzyWiqllDaF+TyUTUKKHrpnEe0i60aTrc43K5wAYppNgWr58z9ZnRAdSNvyeEkbD/tRKhFHhvp/7ztZq5fOoc1oi9VRUtU1/TeVScURIfTTLqZ/vTn5Rfq/Lm7cnwHy83aiQCBQK6U/e2DZ8oUJtI3bu0c4XsQSgKOYFotFMrrV5MJDlMFqRWFu3JrlaUwq2TudrmPFrdx0TUigIi5KPW+OuyJqcikZEqjwk0tSE0NyMqJEzFbX7+kfrW7qZSUl5XKawiAOh8vuscWxHE7KZLbR3tjlkpPBgJa0xJDcuVEbVmcy+ZGjCOJEaRuSAmZmTMul4mcfuWVEb5rsbEtKigPU8oSoALmMaRGi4yVtLFaVo71af+Hptay/rMtFvfXReTNX920cTpfnxjeKRQI7zxZI3PrdvHUr0900CA3yC1IrmC/j29Y1jFXGYcbGtEgkzE6LYPEFtTrTh0fquFkTwHMOnmxm5Y4HQ1KpWCj0VE67XJRCRvKqe7NSI9h9wd2f1drsWBMAvEnoWwHX1ObbCz5VNvS+s6/SydLNYpeLoh3PK7hS19xTR49QICFFdL/WPjx44cjZFpxKA2KaqbKa7u0fV/h0X/drjS/tODc4zNqIIqvNeblHR9uMleFQk/1O0J6qe+4Bpqf1Dhp27K/ipugrgM/HdE1z3/NvnvTdLzxBEKU13U/99rO27hEWX9PucDIZz8tk3a+poZ2LRBBEd9+oL+6vS53Dz774RWevHhnBmz2i9fUbnlMm5v8mltf2/Hbbl9MfZewtJrPtwInmfx6s1o2aWX/xoRH62XFL8uLf3FPuicHmKxYl0bZp7Rr2uV3W2av/01unWy5rOX5fiiAefHbv1EonUi4qKEDx1P1LaGelcuzouUt7jzZMv7SL3mDWDBgQ07yjM5hLq7t99zzabLW3dY28/n5ZTXOfh96iXztmGLeq3A6JUymluRlRJdUsP7eMCPVPjKEf517V1Odbe+1EadvL75VodV4oSGCzOTp6dFOu2CeTiu9fn8e3mDaMWesxOWimxnRFvWb7xxV1F/t99E5H7cX+N/eUX+zQerRqT2Vj77jJ5j6mtTrTwDD7U2wXZsbQrs8wNDI+oDX6yi67rNHtPFBdXNVlGLMgGgAxPSnjuLVvyPjmnvPVTb28nRTORJBaPqAd83RlNYqizlV2blidOVmDweGxP28/087qPfGJafHrV2Uw+K3t4X815zGTrX/IePBk86EzLZhtCIjpSdW3DAyPmhpbB+ta+msv9vNhkyiK2nOo7rrFr/3k5NIFCe7vIcZEqB/fXPibV094OqdKqrsmi2mKorbvvcD6LRehUPCDuxakJoTQdSBRWtPNt/kvXxkdszRcGmjv1lU2aspre5AFMEtj+tT5DpP5OqeTVpujq09vtjoEVx6p2x1Ok9neO8iv5wPjJtvuz2sne0ZXtFb/2KYC96+wND9h3S3pHx2p9+h2dvfp27pHkq8pgeR0Ue8frPHEoi1pCSHfW5NNu3pDv9ZYUc91LcOS6q7rzlM1Wex6g3lYb6Ioqm/IOGq0WGwO47iVs0mtAHyM6TGT7e2PKrp69ddOFqdclJP/TwUFAjeLenx0pH5+esTSBe4K9wgFgvvuzL3Yoa1r8eD1Qe+g8ejZS48U3XTV38tqut/+qIKVcqlflxgT9Meff0cupT9y9n3RwGRlS3a990lVU9sQRVDXnto7nS584cEXeXDctMPh1BssDqfL4bj6P6fL51dhsDuc2z++QLtQVnCA4gffXUBbM2+a9h9rvNjxjfmZ7d0jL757jvWMjosM+I8Hbg5kMF+mb8jolfl7SoXU7nBe55BDRgNiehZq7x559Z+ltLeeF2XHbL0rz6NbMm62HTx58at1jPoGjc+9doLdm0gCgrhpfuzvf7qaYZ2WT79swtwQAMS0lzmcrsNnWtq66AdRbFiduSQv3qMbc/Bkc0uHdmJy9va9Fa2d9JNKkt2uuvJ1MRHqnz2w7LnHV8VFBTBpX9Pc5+k78gCzhxhdMB02u/Pnfz701v/cHeF20Rm5jNyyPq+hdcBzD6zsducbH5T/4qEVB040HT7TQtu+aO38NcvTLDbHwS+bjxW3UhRl+ubYQZIUkWLh/PTIWwuSM+eEx3xzyXB3W+Jw7TpYc91HxzALCQhCpZSxWKXRdeVYdThciGlgSm+wHDjeuPnOXPclPeelhN2/Pu+VnSWem1FZUa95fXdZaXW3+wVTosJURbfPv31FGikWEQSRkRT6xH2FVU19/UNGm92pM5hlUrHKTxqkVmSmhk9hiazXdpWiqhx85e7Vmd/99jw2D3qKOHy25XfbTiKm4Qbs+KQqLSl0+cJE983uWJlR1zLg0aVAaIffpcQFP35f4YJ50V//o1QiLpgfy8oGFFd1Hjl7CYcEfOOEWsBqLV0BQYpEs6oDcW+aHW/uKe/ppykFJ5WIH7pnUTyz27usI8WirNTw3/3k21dlNIu+LGt//o1TmGkNgJjmo06NfvveC7TNosNV967L8dxyKpPuZoFg+cLEl59dFxWm8tBbnCht+9Pbp3UGMw4GAMQ0T5XVdJ+uuEy7kPba5Wmrl87h+Dz6R5sLf/Hwiomb0Z5w6HTLX/5x1jCGCvoA7MO9adaMGi2v7CzOSg2nnf3x2KbCtq4RbsoZq/1lzz2xKjcjykOn8Bar47VdpSdK27ifcAiAs2m4YX2Dxjd2l1ttNHP/AvxlW+/KU8i5WOw1WK0wW+weyujy2p5f/vXo3i8a9MhoAMS0rzh8pqW0hr4A/7KFietuyeBge9p7Rv77lePPv3mK3TmBnRrdSzuKn3nhSFltN3Y6gEfhpgfLHE7Xu/sq0xJDIkL83bfcvC7nYvtQdbPHFzex2hyfnWyubNBkp0XcsyY7LjJAJp3ifjdb7bpR855DdSVVXZpBA3Y3AK9j2m53DQyP2e2TXuAHBSh4vvi/+4WxewcMUysV3XJZu/uz2i3r80xuCzNJJOKHvr/okV9/ws2H7Rsy9g0ZT5S2L8yKzsmIKsyJY7JE1lfqWvqb24cq6jUl1V3eWvPMbLGPGi1ulvpW+klNFv7W9TdZ7EO6cevki12o/WU2+9SrZQkFgqAAuZuBoVKJmPXHvGqljHYoKrvkMtJi43SOq1gk1AwYqMmHB0xt+UrmBEuKtk2nv9xsulAo4POM4ehwVUJ0ICm+/m0f6kq96emUS3bfOf/X+wKBp5d3mXTzpGSASpafGZ2RHBYerCRJkZQUJV0pWj06ZtGOmOwOp8Pp0gwYLmt0xVVdOoPZW5v6dbTXATxf8UchI12THxUioZC25iLN68tJ9z+irPePhBRxP8DU6aRYr/7o/nsqlYi8eOBNK6ZhJkmOC5oouEq5KK3exGTNcgDg9U0PmGGYlPoDAO5hpAcAAGIaAAAQ0wAAiGkAAEBMAwAAYhoAADENAACIaQAAxDQAACCmAQAAMQ0AgJgGAADENAAAYhoAABDTAACAmAYAQEwDAABiGgAAMQ0AAIhpAABATAMAIKYBAAAxDQCAmEYXAAAgpgEAADENAICYBgAAxDQAACCmAQAQ0wAAgJgGAEBMAwCA9/xvAAAA//+zhRxSOKI4MAAAAABJRU5ErkJggg=="}}},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"## importing packages\nimport itertools\nimport gc\n\nimport numpy as np\nimport pandas as pd\n\nfrom fbprophet import Prophet\nfrom matplotlib import pyplot as plt\nfrom sklearn.metrics import mean_absolute_error, mean_squared_error\n\n## defining paths\npath_data = \"/kaggle/input/ashrae-energy-prediction/\"\npath_train = path_data + \"train.csv\"\npath_test = path_data + \"test.csv\"\npath_building = path_data + \"building_metadata.csv\"\npath_weather_train = path_data + \"weather_train.csv\"\npath_weather_test = path_data + \"weather_test.csv\"\n\n## setting seed\nmyfavouritenumber = 13\nseed = myfavouritenumber\nnp.random.seed(seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since Prophet is built for every building-meter combination, it exceeds the 9-hr runtime limit of Kaggle Notebook. Hence we will run it separately for each site_id. Note that you can run these in parallel on Kaggle.\n\nNotebook versions for each site run -   \nSite 0: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024737   \nSite 1: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024752   \nSite 2: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024767   \nSite 3: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024791   \nSite 4: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024811   \nSite 5: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024832   \nSite 6: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024853   \nSite 7: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024873   \nSite 8: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024895   \nSite 9: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24024904   \nSite 10: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24039606   \nSite 11: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24039629   \nSite 12: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24039651   \nSite 13: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24039661   \nSite 14: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24039680   \nSite 15: https://www.kaggle.com/rohanrao/ashrae-prophet-s-prophecy?scriptVersionId=24039699   "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"## site can be changed to run the notebook for different sites (choosing 11 here since it runs the fastest)\nsite = 11","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preparing data\nPreparing the train and test data with weather features and in the format required by Prophet."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"## reading train and test data\ndf_train = pd.read_csv(path_train)\ndf_test = pd.read_csv(path_test)\n\n## reading building metadata\nbuilding = pd.read_csv(path_building)\ndf_train = df_train.merge(building, on=\"building_id\")\ndf_test = df_test.merge(building, on=\"building_id\")\n\n## reading weather data\nweather_train = pd.read_csv(path_weather_train)\nweather_test = pd.read_csv(path_weather_test)\n\n## expanding weather data to all missing values\nweather_train = pd.merge(pd.DataFrame(list(itertools.product(df_train.site_id.unique(), df_train.timestamp.unique())), columns=[\"site_id\", \"timestamp\"]),\n                         weather_train, on = [\"site_id\", \"timestamp\"], how=\"left\")\nweather_test = pd.merge(pd.DataFrame(list(itertools.product(df_test.site_id.unique(), df_test.timestamp.unique())), columns=[\"site_id\", \"timestamp\"]),\n                         weather_test, on = [\"site_id\", \"timestamp\"], how=\"left\")\n\n## filling missing values with closest available values\nweather_train = weather_train.groupby(\"site_id\").apply(lambda group: group.interpolate(limit_direction=\"both\"))\nweather_test = weather_test.groupby(\"site_id\").apply(lambda group: group.interpolate(limit_direction=\"both\"))\n\n## preparing final train data\ndf_train = df_train.merge(weather_train, on=[\"site_id\", \"timestamp\"], how=\"left\")\ndf_train = df_train[~((df_train.site_id==0) & (df_train.meter==0) & (df_train.building_id <= 104) & (df_train.timestamp < \"2016-05-21\"))]\ndf_train = df_train[df_train.site_id == site]\ndf_train.reset_index(drop=True, inplace=True)\ndf_train[\"ds\"] = pd.to_datetime(df_train.timestamp, format='%Y-%m-%d %H:%M:%S')\ndf_train[\"y\"] = np.log1p(df_train.meter_reading)\n\n## preparing final test data\ndf_test = df_test.merge(weather_test, on=[\"site_id\", \"timestamp\"], how=\"left\")\ndf_test = df_test[df_test.site_id == site]\ndf_test.reset_index(drop=True, inplace=True)\ndf_test[\"ds\"] = pd.to_datetime(df_test.timestamp, format='%Y-%m-%d %H:%M:%S')\n\n## deleting unneeded dataset\ndel building, weather_train, weather_test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prophet model\nBuilding a Prophet model for each building x meter combination."},{"metadata":{"trusted":true},"cell_type":"code","source":"## external features\nexogenous_features = [\"air_temperature\", \"cloud_coverage\", \"dew_temperature\",\n                      \"precip_depth_1_hr\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\"]\n\n## list of predictions\ndf_preds = []\n\n## iterating over buildings\nfor building in df_train.building_id.unique():\n    \n    ## subsetting building data\n    df_train_building = df_train[df_train.building_id == building]\n    df_test_building = df_test[df_test.building_id == building]\n    \n    ## iterating over meters\n    for meter in df_train_building.meter.unique():\n        \n        ## subsetting meter data\n        df_train_building_meter = df_train_building[df_train_building.meter == meter]\n        df_test_building_meter = df_test_building[df_test_building.meter == meter]\n        \n        ## drop features with all NaNs\n        df_train_building_meter.dropna(axis=1, how=\"all\", inplace=True)\n        \n        print(\"Building Prophet model for building\", building, \"and meter\", meter)\n        \n        ## initializing model\n        model_prophet = Prophet()\n        \n        ## adding regressors\n        remove_features = []\n        \n        for feature in exogenous_features:\n            if feature in df_train_building_meter.columns:\n                model_prophet.add_regressor(feature)\n            else:\n                remove_features.append(feature)\n\n        for feature in remove_features:\n            exogenous_features.remove(feature)\n        \n        ## building model\n        model_prophet.fit(df_train_building_meter[[\"ds\", \"y\"] + exogenous_features])\n        \n        ## forecasting predictions\n        forecast = model_prophet.predict(df_test_building_meter[[\"ds\"] + exogenous_features])\n        \n        ## creating predictions dataframe\n        df_pred = pd.DataFrame({\"row_id\": df_test_building_meter.row_id.values, \"meter_reading\": np.expm1(forecast.yhat.values)})\n        df_preds.append(df_pred)\n\n        print(\"Prophet model completed for building\", building, \"and meter\", meter, \"\\n\")\n        \n        ## cleanup\n        gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## visualizing last model built\nmodel_prophet.plot_components(forecast)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Saving Predictions\nSaving predictions for the site. The final submission file will be combining the predictions of all sites together."},{"metadata":{"trusted":true},"cell_type":"code","source":"## saving predictions\ndf_site = pd.concat(df_preds).reset_index(drop=True)\ndf_site.to_csv(\"prophet_pred_site_\" + str(site) + \".csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}