{"cells":[{"metadata":{"_uuid":"e4d0cfb4eb8bb5617022a3d2118b22db1af1b757"},"cell_type":"markdown","source":"In this tutorial, we'll try to forecast transactions data and poke around open source [Prophet](https://facebook.github.io/prophet/) library released by Facebook. We'll learn  how to integrate holiday periods and optimize the model by playing parameters.\n\nProphet is a procedure for forecasting time series data. It is based on an additive model where non-linear trends are fit with yearly and weekly seasonality, plus holidays. It works best with daily periodicity data with at least one year of historical data. Prophet is robust to missing data, shifts in the trend, and large outliers. You can learn more about from the [docs](https://facebook.github.io/prophet/docs/quick_start.html).\n\nLet's start loading libraries and data."},{"metadata":{"_uuid":"00e60bd44272ef95cb68aab3fac252880a8a4d28","_cell_guid":"8622f914-bf69-49bb-9cc2-4d2fa4d26a32","trusted":true},"cell_type":"code","source":"# Load libraries\nimport numpy as np\nRANDOM_SEED = 42\nnp.random.seed(RANDOM_SEED)\nimport pandas as pd\npd.set_option('display.max_rows', 10)\npd.set_option('display.max_columns', 70)\nimport plotly.offline as py\nimport plotly.graph_objs as go\npy.init_notebook_mode()\n\nfrom fbprophet import Prophet\n\n%time df_transactions = pd.read_csv('../input/transactions.csv')\n%time df_holidays_events = pd.read_csv('../input/holidays_events.csv')\n\nprint('Data and libraries are loaded.')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"183e4e88fe6f6417afb6bb698f971d0fa2f757e7","trusted":true},"cell_type":"code","source":"df_transactions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"597b13277e444b9f5845fcc54eb6e11ff58871e7"},"cell_type":"markdown","source":"If we look at the transactions data, transactions are grouped by store no. We'll simplify this for now and group them by date. We can clearly see seasonality and holiday effect on total transactions."},{"metadata":{"_uuid":"b99d099182c6222a73f448327e6f1aefc3a1949d","trusted":true},"cell_type":"code","source":"transactions = df_transactions.groupby('date')['transactions'].sum()\npy.iplot([go.Scatter(\n    x=transactions.index,\n    y=transactions\n)])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59fd7d2435d186745a114b2426677d0d09937673"},"cell_type":"markdown","source":"Now, let's try the prophet library and see how well it predicts. But before that, we should prepare the data. According to docs:\n\n> Prophet follows the sklearn model API. We create an instance of the Prophet class and then call its fit and predict methods.\n> The input to Prophet is always a dataframe with two columns: **ds** and **y**. The ds (datestamp) column must contain a date or datetime (either is fine). The **y** column must be numeric, and represents the measurement we wish to forecast.\n"},{"metadata":{"_uuid":"fa528687f03d48b00434650eabbd00e3e0cab39d","trusted":true},"cell_type":"code","source":"transactions = pd.DataFrame(transactions).reset_index()\ntransactions.columns = ['ds', 'y']\ntransactions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"90a586cdc93c35086645b7395810339d46ed3bf5","trusted":true},"cell_type":"code","source":"m = Prophet()\nm.fit(transactions)\nfuture = m.make_future_dataframe(periods=365)\nforecast = m.predict(future)\nforecast","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"46c4053fd5f38c221215df1001a365cd803a02e1","trusted":true},"cell_type":"code","source":"py.iplot([\n    go.Scatter(x=transactions['ds'], y=transactions['y'], name='y'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat'], name='yhat'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_upper'], fill='tonexty', mode='none', name='upper'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_lower'], fill='tonexty', mode='none', name='lower'),\n    go.Scatter(x=forecast['ds'], y=forecast['trend'], name='Trend')\n])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"745617e2f9a187a4f44c459a00eb07cc83ec0d48","trusted":true},"cell_type":"code","source":"# Calculate root mean squared error.\nprint('RMSE: %f' % np.sqrt(np.mean((forecast.loc[:1682, 'yhat']-transactions['y'])**2)) )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a81b93721c994a58f270874689facb47f4bed0c6"},"cell_type":"markdown","source":"As we see in the graph above, prediction is fairly well and aligns with the data's up and downs. You can zoom in the graph by selecting a zoom area with mouse.\n\nBut the trend is fairly rigid, it misses the sub trends in mid-years. The trend is rising at first half of the year and a little bit slowing down after that. Let's make the trend a little bit flexible. If the trend changes are being overfit (too much flexibility) or underfit (not enough flexibility), you can adjust the strength of the sparse prior using the input argument **changepoint_prior_scale**. By default, this parameter is set to 0.05. Increasing it will make the trend more flexible. (https://facebook.github.io/prophet/docs/trend_changepoints.html)"},{"metadata":{"_uuid":"76080396704a6e9fd6fb3d6fc613496f32f4cc6d","trusted":true},"cell_type":"code","source":"m = Prophet(changepoint_prior_scale=2.5)\nm.fit(transactions)\nfuture = m.make_future_dataframe(periods=365)\nforecast = m.predict(future)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"084aa59bd902c67bd8ba943d150906abe120d274","trusted":true},"cell_type":"code","source":"# Calculate root mean squared error.\nprint('RMSE: %f' % np.sqrt(np.mean((forecast.loc[:1682, 'yhat']-transactions['y'])**2)) )\npy.iplot([\n    go.Scatter(x=transactions['ds'], y=transactions['y'], name='y'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat'], name='yhat'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_upper'], fill='tonexty', mode='none', name='upper'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_lower'], fill='tonexty', mode='none', name='lower'),\n    go.Scatter(x=forecast['ds'], y=forecast['trend'], name='Trend')\n])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d24ed132351681c4e1f09fa8184a95701a86c58a"},"cell_type":"markdown","source":"Now we'll add more seasonality to the model. As we see, prophet calculates weekly and yearly seasonality. We don't need daily seasonality, because we don't have intra-day data for this tutorial. Just adding monthly seasonality should be enough."},{"metadata":{"_uuid":"5031bfad09f46add6695d0ca9f5db04ea5ece246","trusted":true},"cell_type":"code","source":"m = Prophet(changepoint_prior_scale=2.5)\nm.add_seasonality(name='monthly', period=30.5, fourier_order=5)\nm.fit(transactions)\nfuture = m.make_future_dataframe(periods=365)\nforecast = m.predict(future)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"973bd0acd5dad116f32079379f3a96fa8e6f6ec3","trusted":true},"cell_type":"code","source":"# Calculate root mean squared error.\nprint('RMSE: %f' % np.sqrt(np.mean((forecast.loc[:1682, 'yhat']-transactions['y'])**2)) )\npy.iplot([\n    go.Scatter(x=transactions['ds'], y=transactions['y'], name='y'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat'], name='yhat'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_upper'], fill='tonexty', mode='none', name='upper'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_lower'], fill='tonexty', mode='none', name='lower'),\n    go.Scatter(x=forecast['ds'], y=forecast['trend'], name='Trend')\n])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8735bdaad581aadc77627c2b6e2a65fa2ba26e28"},"cell_type":"markdown","source":"Now it's time to add effects of holiday events into our model. We need to adjust data format first. Prophet needs two columns (holiday and ds) and a row for each occurrence of the holiday. We could also include columns lower_window and upper_window which extend the holiday out to `[lower_window, upper_window]` days around the date. But, I think data is arranged well and we don't have to pay so much attention to that."},{"metadata":{"_uuid":"c553de9d82916e761c33e4e8dc7fe1789737fd68","trusted":true},"cell_type":"code","source":"df_holidays_events","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c389e6f21fed17c142162efbd2d8031b05d48cd5","trusted":true},"cell_type":"code","source":"holidays = df_holidays_events[df_holidays_events['transferred'] == False][['description', 'date']]\nholidays.columns = ['holiday', 'ds']\n#holidays['lower_window'] = 0\n#holidays['upper_window'] = 0\nholidays","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a809e8048d678a9ba3c7abed45b358aa69146a4","trusted":true},"cell_type":"code","source":"m = Prophet(changepoint_prior_scale=2.5, holidays=holidays)\nm.add_seasonality(name='monthly', period=30.5, fourier_order=5)\nm.fit(transactions)\nfuture = m.make_future_dataframe(periods=365)\nforecast = m.predict(future)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"ef20347745305b4fc7c9c1ea25ba73dd802c9b4d","trusted":true},"cell_type":"code","source":"# Calculate root mean squared error.\nprint('RMSE: %f' % np.sqrt(np.mean((forecast.loc[:1682, 'yhat']-transactions['y'])**2)) )\npy.iplot([\n    go.Scatter(x=transactions['ds'], y=transactions['y'], name='y'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat'], name='yhat'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_upper'], fill='tonexty', mode='none', name='upper'),\n    go.Scatter(x=forecast['ds'], y=forecast['yhat_lower'], fill='tonexty', mode='none', name='lower'),\n    go.Scatter(x=forecast['ds'], y=forecast['trend'], name='Trend')\n])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0fbe4e892a3f7fbacddddc71b1eebecb8696226"},"cell_type":"markdown","source":"We managed to predict the spikes for new-year period. The model couldn't catch the downward spike for Jan 4th 2016, so it could not predict Jan 1st 2017 successfully. That's very reasonable, as there is no holiday for Jan 4th of 2016. But the model predicts the sales on the 24th Decembers nicely. And also forecasted period after Aug 15th 2017 looks good.\n\nAnd, as a non-ecuadorian person, I can say from the graph that \"Dia de la Madre\" is an important event, and it boosts sales previous day.\n\nAnd, that's it. Prophet is a fairly easy to use library to forecast time-series data, which only uses previous data and holidays for that. There are more features and parameters like saturating forecasts, uncertanity intervals etc. which we didn't cover here. You can read more from their paper https://peerj.com/preprints/3190.pdf.\n\nI think it's not a full-featured regression or forecasting tool, as we cannot use other data to use correlations or effects of another variable etc. But, It could be a replacement for time-series models like ARMA, ARIMA etc. Please comment below if you think you know more ways to improve this model. Thank you!"}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}