{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# How to Detect Random Walk and White Noise in Time Series Forecasting\n## Find out if the target is worth predicting!\n\n\n \n Photo by \n Tobi\n on \n Pexels\n \n","metadata":{}},{"cell_type":"markdown","source":"## Setup","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"import warnings\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\nplt.style.use(\"ggplot\")\n\nwarnings.filterwarnings(\"ignore\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-16T09:16:15.195537Z","iopub.execute_input":"2021-07-16T09:16:15.196329Z","iopub.status.idle":"2021-07-16T09:16:16.596088Z","shell.execute_reply.started":"2021-07-16T09:16:15.196186Z","shell.execute_reply":"2021-07-16T09:16:16.594895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tps_july = pd.read_csv(\n \"../input/tabular-playground-series-jul-2021/train.csv\",\n parse_dates=[\"date_time\"],\n index_col=\"date_time\",\n)","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:16.597469Z","iopub.execute_input":"2021-07-16T09:16:16.597788Z","iopub.status.idle":"2021-07-16T09:16:16.660978Z","shell.execute_reply.started":"2021-07-16T09:16:16.597758Z","shell.execute_reply":"2021-07-16T09:16:16.659521Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Introduction ","metadata":{}},{"cell_type":"markdown","source":"No matter how powerful, machine learning cannot predict everything. A well-known area where it can become pretty helpless is related to time series forecasting.\n\nDespite the availability of a large suite of autoregressive models and many other algorithms for time series, you cannot predict the target distribution if it is **white noise** or follows a **random walk**.\n\nSo, you must detect such distributions before you make further efforts.\n\nIn this article, you will learn what white noise and random walk are and explore proven statistical techniques to detect them.","metadata":{}},{"cell_type":"markdown","source":"## Table of Contents ","metadata":{}},{"cell_type":"markdown","source":"#### [1. Brief notes on Autocorrelation](#1)\n#### [2. What is white noise?](#2)\n#### [3. Importance of White Noise in Forecasting](#3)\n#### [4. Random Walks](#4)\n#### [5. Random Walks with drift](#5)\n#### [6. Detecting random walks statistically](#6)\n#### [7. Summary](#7)","metadata":{}},{"cell_type":"markdown","source":"> You can read the Medium article of this notebook [here](https://towardsdatascience.com/how-to-detect-random-walk-and-white-noise-in-time-series-forecasting-bdb5bbd4ef81).","metadata":{}},{"cell_type":"markdown","source":"## Before we start...","metadata":{}},{"cell_type":"markdown","source":"This is my third article on the time series forecasting series (you can check out the whole series from this [list](https://ibexorigin.medium.com/list/time-series-forecast-from-scratch-c72ccf7a9229), a new Medium feature).\n\nWhile the first one was about [every single Pandas function to manipulate TS data](https://medium.com/r/?url=https%3A%2F%2Ftowardsdatascience.com%2Fevery-pandas-function-you-can-should-use-to-manipulate-time-series-711cb0c5c749%3Fsource%3Dyour_stories_page-------------------------------------), the second was about [time series decomposition and autocorrelation](https://www.kaggle.com/bextuychiev/advanced-time-series-analysis-decomposition/comments).\n\nTo get the most out of this post, you need to understand at least what autocorrelation is. Here, I will give a brief explanation, but check out my last article if you want to go deeper.","metadata":{}},{"cell_type":"markdown","source":"## 1. Brief notes on Autocorrelation ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝\n\nAutocorrelation involves finding the correlation between a time series and a lagged version of itself. Consider this distribution:","metadata":{}},{"cell_type":"code","source":"deg_C = tps_july[\"deg_C\"].to_frame(\"temperature\")\n\ndeg_C.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:16.663061Z","iopub.execute_input":"2021-07-16T09:16:16.663423Z","iopub.status.idle":"2021-07-16T09:16:16.694745Z","shell.execute_reply.started":"2021-07-16T09:16:16.66339Z","shell.execute_reply":"2021-07-16T09:16:16.693587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lagging a time series means shifting it 1 or more periods backward:","metadata":{}},{"cell_type":"code","source":"deg_C[\"lag_1\"] = deg_C[\"temperature\"].shift(periods=1)\ndeg_C[\"lag_2\"] = deg_C[\"temperature\"].shift(periods=2)\ndeg_C[\"lag_3\"] = deg_C[\"temperature\"].shift(periods=3)\n\ndeg_C.head(6)","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:16.696158Z","iopub.execute_input":"2021-07-16T09:16:16.696443Z","iopub.status.idle":"2021-07-16T09:16:16.724385Z","shell.execute_reply.started":"2021-07-16T09:16:16.696415Z","shell.execute_reply":"2021-07-16T09:16:16.723137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Autocorrelation Function (ACF) finds the correlation coefficient between a time series and its lagged version at each lag *k*. You can plot it using the `plot_acf` function from `statsmodels`. Here is what it looks like:","metadata":{}},{"cell_type":"code","source":"from matplotlib import rcParams\nfrom statsmodels.graphics.tsaplots import plot_acf\n\nrcParams[\"figure.figsize\"] = 9, 4\n# ACF function up to 50 lags\nfig = plot_acf(deg_C[\"temperature\"], lags=50)\n\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:16.726334Z","iopub.execute_input":"2021-07-16T09:16:16.726685Z","iopub.status.idle":"2021-07-16T09:16:17.31006Z","shell.execute_reply.started":"2021-07-16T09:16:16.726651Z","shell.execute_reply":"2021-07-16T09:16:17.308911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The XAxis is the lag *k*, and the YAxis is the Pearson's correlation coefficient at each lag. The red shaded region is a confidence interval. If the height of the bars is outside this region, it means the correlation is statistically significant.","metadata":{}},{"cell_type":"markdown","source":"## 2. What is white noise? ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝\n\nIn short, white noise distribution is any distribution that has:\n\n- Zero mean\n- A constant variance/standard deviation (does not change over time)\n- Zero autocorrelation at all lags\n\nEssentially, it is a series of random numbers, and by definition, no algorithm can reasonably model its behavior.\n\nThere are special types of white noise. If the noise is normal (follows a [normal distribution](https://medium.com/r/?url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-use-normal-distribution-like-you-know-what-you-are-doing-1cf4c55241e3%3Fsource%3Dyour_stories_page-------------------------------------)), it is called Gaussian white noise. Let's see an example of this visually:","metadata":{}},{"cell_type":"code","source":"# Generate Gaussian white noise dist with mean 0 and 0.5 std\nnoise = np.random.normal(loc=0, scale=0.5, size=1000)\n\nplt.figure(figsize=(12, 4))\nplt.plot(noise);","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:17.311401Z","iopub.execute_input":"2021-07-16T09:16:17.311722Z","iopub.status.idle":"2021-07-16T09:16:17.550321Z","shell.execute_reply.started":"2021-07-16T09:16:17.311693Z","shell.execute_reply":"2021-07-16T09:16:17.549438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Gaussian white noise distribution with a standard deviation of 0.5\n\nEven though there are occasional spikes, there are no discernible patterns visible, i.e., the distribution is completely random.\n\nThe best way you can validate this is to create the ACF plot:","metadata":{}},{"cell_type":"code","source":"fig = plot_acf(noise, lags=40)\n\nplt.title(\"Autocorrelation of a White Noise Series\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:17.551352Z","iopub.execute_input":"2021-07-16T09:16:17.551677Z","iopub.status.idle":"2021-07-16T09:16:17.820828Z","shell.execute_reply.started":"2021-07-16T09:16:17.551637Z","shell.execute_reply":"2021-07-16T09:16:17.819501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> White noise distributions have approximately 0 autocorrelation at all lags.\n\nThere are also \"strict\" white noise distributions - these have strictly 0 serial correlation. This is different from [brown/pink](https://medium.com/r/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FBrownian_noise) noise or other natural random phenomena where there is a weak serial correlation but still remain memory-free.","metadata":{}},{"cell_type":"markdown","source":"## 3. Importance of White Noise in Forecasting ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝\n\nEven though white noise distributions are considered dead ends, they can be quite useful in other contexts.\n\nFor example, in time series forecasting, if the differences between predictions and actual values represent a white noise distribution, you can pat yourself on the back for a job well done.\n\nWhen the residual errors show any pattern, whether seasonal or trending or have a non-zero mean, this suggests there is still room for improvement. In contrast, if the residuals are purely white noise, you maxed out the abilities of the chosen model.\n\nIn other words, the algorithm managed to capture all the important signals and properties of the target. What's left are the random fluctuations and inconsistent data points that could not be attributed to anything.\n\nFor example, we will predict the amount of carbon monoxide in the air using the July Kaggle playground competition. We will leave the inputs \"as-is\" - we won't perform any feature engineering, and we will choose a baseline model with default parameters:","metadata":{}},{"cell_type":"code","source":"X = tps_july.drop(\n [\"target_carbon_monoxide\", \"target_benzene\", \"target_nitrogen_oxides\"], axis=1\n)\ny = tps_july[\"target_carbon_monoxide\"].values.reshape(-1, 1)","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:17.823633Z","iopub.execute_input":"2021-07-16T09:16:17.824141Z","iopub.status.idle":"2021-07-16T09:16:17.83523Z","shell.execute_reply.started":"2021-07-16T09:16:17.824092Z","shell.execute_reply":"2021-07-16T09:16:17.832367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nfrom sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(\n X, y, random_state=1121218, test_size=0.1\n)\n\nforest = RandomForestRegressor()\nforest.fit(X_train, y_train)\n\npreds = forest.predict(X_test)\nresiduals = y_test.flatten() - preds\n\nplt.plot(residuals)\nplt.title(\"Plot of the Error residuals\");","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:17.837347Z","iopub.execute_input":"2021-07-16T09:16:17.837756Z","iopub.status.idle":"2021-07-16T09:16:23.356517Z","shell.execute_reply.started":"2021-07-16T09:16:17.83772Z","shell.execute_reply":"2021-07-16T09:16:23.355073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plot_acf(residuals, lags=50)\n\nplt.title(\"Autocorrelation Plot of the Error residuals\")\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:23.358737Z","iopub.execute_input":"2021-07-16T09:16:23.359287Z","iopub.status.idle":"2021-07-16T09:16:23.58536Z","shell.execute_reply.started":"2021-07-16T09:16:23.359227Z","shell.execute_reply":"2021-07-16T09:16:23.584009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is some pattern in the ACF plot, but they are within the confidence interval. These two plots suggest that Random Forests could capture almost all the important signals from the training data even with default parameters.","metadata":{}},{"cell_type":"markdown","source":"## 4. Random Walks ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝\n\nA more challenging but equally unpredictable distribution in time series forecasting is a random walk. Unlike white noise, it has non-zero mean, non-constant std/variance, and when plotted, looks a lot like a regular distribution:\n\n","metadata":{}},{"cell_type":"markdown","source":"> Doesn't it resemble a plot of stocks from Yahoo Finance?","metadata":{}},{"cell_type":"markdown","source":"Random walk series are always cleverly disguised in this manner, but still, they are unpredictable as ever. The best guess for today's value is yesterday's.\n\nA common confusion among beginners is thinking of a random walk as a simple sequence of random numbers. This is not the case because, in a random walk, each step is dependent on the previous step. \n\nFor this reason, the Autocorrelation function of random walks does return non-zero correlations.\nThe formula of a random walk is simple:\n\n","metadata":{}},{"cell_type":"markdown","source":"Whatever the previous data point is, add some random value to it and continue for as long as you like. Let's generate this in Python with a starting value of, let's say, 99:","metadata":{}},{"cell_type":"code","source":"walk = [99]\n\nfor i in range(1000):\n # Create random noise\n noise = -1 if np.random.random() < 0.5 else 1\n walk.append(walk[-1] + noise)\n\nrcParams[\"figure.figsize\"] = 14, 4\nplt.plot(walk);","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:23.587415Z","iopub.execute_input":"2021-07-16T09:16:23.588026Z","iopub.status.idle":"2021-07-16T09:16:23.791153Z","shell.execute_reply.started":"2021-07-16T09:16:23.58797Z","shell.execute_reply":"2021-07-16T09:16:23.789888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's also plot the ACF:","metadata":{}},{"cell_type":"code","source":"fig = plot_acf(walk, lags=50)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:23.792539Z","iopub.execute_input":"2021-07-16T09:16:23.792969Z","iopub.status.idle":"2021-07-16T09:16:24.040272Z","shell.execute_reply.started":"2021-07-16T09:16:23.792937Z","shell.execute_reply":"2021-07-16T09:16:24.038224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the first ~40 lags yield statistically significant correlations.\n\nSo, how do we detect a random walk when a visualization is not an option?\n\nBecause of how they are created, differencing the time series should isolate the random addition of each step. Taking the first-order difference is done by lagging the series by 1 and subtracting it from the original. Pandas has a convenient `diff` function to do this:","metadata":{}},{"cell_type":"code","source":"walk_diff = pd.Series(walk).diff()\n\nplt.plot(walk_diff);","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:24.042425Z","iopub.execute_input":"2021-07-16T09:16:24.043035Z","iopub.status.idle":"2021-07-16T09:16:24.328317Z","shell.execute_reply.started":"2021-07-16T09:16:24.042972Z","shell.execute_reply":"2021-07-16T09:16:24.327264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you plot the first-order difference of a time series and the result is white noise, then it is a random walk.","metadata":{}},{"cell_type":"markdown","source":"## 5. Random Walks with drift ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝\n\nA slight modification to regular random walks is adding a constant value called a drift at random step:\n\n","metadata":{}},{"cell_type":"markdown","source":"Drift is often denoted with μ, and in terms of values changing over time, drift means gradually changing into something.\n\nFor example, even though stocks fluctuate constantly, they might have a positive drift, i.e., gain an overall gradual increase over time.\n\nNow, let's see how to simulate this in Python. We will first create the regular random walk with a start value of 25:","metadata":{}},{"cell_type":"code","source":"walk = [25]\n\nfor i in range(1000):\n # Create random noise\n noise = -1 if np.random.random() < 0.5 else 1\n # Add the drift too\n walk.append(walk[-1] + noise)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-07-16T09:16:24.329574Z","iopub.execute_input":"2021-07-16T09:16:24.329892Z","iopub.status.idle":"2021-07-16T09:16:24.33798Z","shell.execute_reply.started":"2021-07-16T09:16:24.329864Z","shell.execute_reply":"2021-07-16T09:16:24.33672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above formula, we see that we need to add the desired drift at each step. Let's add a drift of 5 and look at the plot:","metadata":{}},{"cell_type":"code","source":"drift = 5\ndrifty_walk = pd.Series(walk) + 5\n\ndrifty_walk.plot(title=\"A Random Walk with Drift\");","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:24.339563Z","iopub.execute_input":"2021-07-16T09:16:24.339956Z","iopub.status.idle":"2021-07-16T09:16:24.588309Z","shell.execute_reply.started":"2021-07-16T09:16:24.339922Z","shell.execute_reply":"2021-07-16T09:16:24.587166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Despite the wild fluctuations, the series has a discernible upward drift. If we perform differencing, we will see that the series is still a random walk:","metadata":{}},{"cell_type":"code","source":"drifty_walk.diff().plot();","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:24.590351Z","iopub.execute_input":"2021-07-16T09:16:24.591282Z","iopub.status.idle":"2021-07-16T09:16:24.861492Z","shell.execute_reply.started":"2021-07-16T09:16:24.591209Z","shell.execute_reply":"2021-07-16T09:16:24.859932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6. Detecting random walks statistically ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝\n\nYou might ask if there are better methods of identifying random walks than just \"eyeballing\" them from plots.\n\nAs an answer, there is a hypothesis test outlined in 1979 by Dicker D. A. and Fuller W. A., and it is called the augmented Dickey-Fuller test.\n\nEssentially, it tries to test the null hypothesis that a series follows a random walk. Under the hood, it regresses the difference in prices on the lagged price.\n\n\n\nIf the found slope (β) is equal to 0, the series is a random walk. If the slope is significantly different from 0, we reject the null hypothesis that the series follows a random walk.\n\nFortunately, you don't have to worry about the math because the test is already implemented in Python. \n\nWe import the `adfuller` function from `statsmodels` and use it on the drifty random walk created in the last section:","metadata":{}},{"cell_type":"code","source":"from statsmodels.tsa.stattools import adfuller\n\nresults = adfuller(drifty_walk)\n\nprint(f\"ADF Statistic: {results[0]}\")\nprint(f\"p-value: {results[1]}\")\nprint(\"Critical Values:\")\nfor key, value in results[4].items():\n print(\"\\t%s: %.3f\" % (key, value))","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:24.863184Z","iopub.execute_input":"2021-07-16T09:16:24.863743Z","iopub.status.idle":"2021-07-16T09:16:24.941868Z","shell.execute_reply.started":"2021-07-16T09:16:24.863687Z","shell.execute_reply":"2021-07-16T09:16:24.940433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We look at the p-value, which is ~0.26. Since 0.05 is the significance threshold, we fail to reject the null hypothesis that `drifty_walk` is a random walk, i.e., it is a random walk.\n\nLet's perform another test on a distribution we know isn't a random walk. We will use the carbon monoxide target from the TPS July Kaggle playground competition:","metadata":{}},{"cell_type":"code","source":"results = adfuller(tps_july[\"target_carbon_monoxide\"])\n\nprint(f\"ADF Statistic: {results[0]}\")\nprint(f\"p-value: {results[1]}\")\nprint(\"Critical Values:\")\nfor key, value in results[4].items():\n print(\"\\t%s: %.3f\" % (key, value))","metadata":{"execution":{"iopub.status.busy":"2021-07-16T09:16:24.943719Z","iopub.execute_input":"2021-07-16T09:16:24.944459Z","iopub.status.idle":"2021-07-16T09:16:25.480705Z","shell.execute_reply.started":"2021-07-16T09:16:24.944401Z","shell.execute_reply":"2021-07-16T09:16:25.479401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The p-value is extremely small, suggesting we can easily reject the null hypothesis that `target_carbon_monoxide` follows a random walk.","metadata":{}},{"cell_type":"markdown","source":"## Summary ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝\n\nWe only finished the third part of this Time Series \"series,\" and you already know a ton.\n\nFrom here on, things are only going to get more and more interesting as we draw closer to the actual \"forecasting\" part in the series. There are some interesting articles planned on key time series topics such as stationarity and time-series cross-validation.\n\nBesides, I will dedicate a post solely on feature engineer specific to time series - this is something to be excited about! [Stay tuned](https://ibexorigin.medium.com/)!","metadata":{}},{"cell_type":"markdown","source":"## My other kernels you might be interested...\n- [Comprehensive Guide to Multiclass Classification](https://www.kaggle.com/bextuychiev/comprehensive-guide-to-mutliclass-classification)\n- [Advanced Time Series Analysis in Python: Decomposition, Autocorrelation](https://www.kaggle.com/bextuychiev/advanced-time-series-analysis-decomposition)\n- [Master the Subtle Art of Train/Test Generation](https://www.kaggle.com/bextuychiev/master-the-subtle-art-of-train-test-set-generation)","metadata":{}}]}