{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Advanced Time Series Analysis in Python: Seasonality and Trend Analysis (Decomposition), Autocorrelation \n## and other advanced techniques to find relationship between multiple time series.\n![](https://cdn-images-1.medium.com/max/1500/1*Vkes2H7Soxgdngi969oAEQ.jpeg)\n<figcaption style=\"text-align: center;\">\n    <strong>\n        Photo by \n        <a href='https://www.pexels.com/@bentonphotocinema?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels'>Jordan Benton</a>\n        on \n        <a href='https://www.pexels.com/photo/shallow-focus-photography-of-hourglass-1095602/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels'>Pexels</a>\n    </strong>\n</figcaption>","metadata":{}},{"cell_type":"markdown","source":"## Setup","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"import warnings\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import rcParams\n\nplt.style.use(\"ggplot\")\nrcParams[\"figure.figsize\"] = [12, 9]\nrcParams[\"xtick.labelsize\"] = 15\nrcParams[\"ytick.labelsize\"] = 15\nrcParams[\"font.size\"] = 15\n\nwarnings.filterwarnings(\"ignore\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-13T11:35:27.988168Z","iopub.execute_input":"2021-07-13T11:35:27.988584Z","iopub.status.idle":"2021-07-13T11:35:27.997288Z","shell.execute_reply.started":"2021-07-13T11:35:27.988549Z","shell.execute_reply":"2021-07-13T11:35:27.996435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Introduction","metadata":{}},{"cell_type":"markdown","source":"Following my very [well-received post](https://medium.com/r?url=https%3A%2F%2Ftowardsdatascience.com%2Fevery-pandas-function-you-can-should-use-to-manipulate-time-series-711cb0c5c749%3Fsource%3Dyour_stories_page-------------------------------------) and [Kaggle notebook](https://medium.com/r/?url=https%3A%2F%2Fwww.kaggle.com%2Fbextuychiev%2Fevery-pandas-function-to-manipulate-time-series%2Fcomments) on every single Pandas function to manipulate time series, it is time to take the trajectory of this TS project to visualization.\n\nThis post is about the core processes that make up an in-depth time series analysis. Specifically, we will talk about:\n\n- Decomposition of time series - seasonality and trend analysis\n- Analyzing and comparing multiple time series simultaneously\n- Calculating autocorrelation and partial autocorrelation and what they represent\n\nand if seasonality or trends among multiple series affect each other.\n\nMost importantly, we will build some very cool visualizations, and this image should be a preview of what you will be learning.\n\n![](https://cdn-images-1.medium.com/max/800/1*ToUsutofdinrqAann6druA.png)\n\nI hope you are as excited about learning these things as I when writing this article. Let's begin.","metadata":{}},{"cell_type":"markdown","source":"## 1. Decomposition of Time Series ","metadata":{}},{"cell_type":"markdown","source":"Any time series distribution has 3 core components:\n\n1. **Seasonality** - does the data have a clear cyclical/periodic pattern?\n2. **Trend** - does the data represent a general upward or downward slope?\n3. **Noise** - what are the outliers or missing values that are not consistent with the rest of the data?\n\nDeconstructing a time series into these components is called **decomposition**, and we will explore each one in detail.","metadata":{}},{"cell_type":"markdown","source":"### 1.1 Seasonality analysis","metadata":{}},{"cell_type":"markdown","source":"Consider this TPS July Kaggle playground dataset:","metadata":{}},{"cell_type":"code","source":"tps = pd.read_csv(\n    \"../input/tabular-playground-series-jul-2021/train.csv\",\n    parse_dates=[\"date_time\"], index_col=\"date_time\"\n)\n\ntps[\"deg_C\"].plot(figsize=(16, 6), fontsize=15)\nplt.xlabel(\"Date\")\nplt.ylabel(\"Degrees Celcius\");","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:27.998561Z","iopub.execute_input":"2021-07-13T11:35:27.99894Z","iopub.status.idle":"2021-07-13T11:35:28.518309Z","shell.execute_reply.started":"2021-07-13T11:35:27.998912Z","shell.execute_reply":"2021-07-13T11:35:28.517236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Obviously, summer months have higher temperatures, and we would expect this behavior to repeat every year. However, the human eye and its ability to detect patterns can only go so far.\n\nFor example, it might be harder to find seasonal patterns from plots such as these:","metadata":{}},{"cell_type":"code","source":"# Load meat production dataset in the USA\nmeat = pd.read_csv(\n    \"https://s3.amazonaws.com/assets.datacamp.com/production/course_5226/datasets/ch4_meat.csv\", \n    parse_dates=[\"date\"], index_col=\"date\")\n\nmeat.plot(figsize=(16, 10))\nplt.legend(fontsize=\"large\")\nplt.xlabel(\"Date\")\nplt.ylabel(\"Meat Production (t)\");","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:28.519754Z","iopub.execute_input":"2021-07-13T11:35:28.520106Z","iopub.status.idle":"2021-07-13T11:35:29.498885Z","shell.execute_reply.started":"2021-07-13T11:35:28.520052Z","shell.execute_reply":"2021-07-13T11:35:29.497836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To find hidden seasonal patterns from time series like above, we will use the `seasonal_decompose` function from `statsmodels`:","metadata":{}},{"cell_type":"code","source":"import statsmodels.api as sm\nfrom matplotlib import rcParams\n\ndecomposition = sm.tsa.seasonal_decompose(meat[\"beef\"])\n\nrcParams[\"figure.figsize\"] = 16, 4\ndecomposition.seasonal.plot();","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:29.500831Z","iopub.execute_input":"2021-07-13T11:35:29.501264Z","iopub.status.idle":"2021-07-13T11:35:29.742745Z","shell.execute_reply.started":"2021-07-13T11:35:29.501222Z","shell.execute_reply":"2021-07-13T11:35:29.741853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using `sm.tsa.seasonal_decompose` on 'beef' time-series returns a `DecomposeResult` object with attributes like seasonal, trend and resid (more on the last two later).\n\nAbove, we are plotting the seasonality, but the plot is not useful since it has too much noise. Let's choose an interval to give the line some room to breathe:","metadata":{}},{"cell_type":"code","source":"decomposition.seasonal[\"1999\":\"2005\"].plot();","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:29.744222Z","iopub.execute_input":"2021-07-13T11:35:29.744618Z","iopub.status.idle":"2021-07-13T11:35:30.021993Z","shell.execute_reply.started":"2021-07-13T11:35:29.744575Z","shell.execute_reply":"2021-07-13T11:35:30.020919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This plot shows that beef production really goes down at the beginning of each year, but it reaches its peak towards the end.\n\n> Note on `seasonal_decompose` function: it produces small figures by default. You have to control its aspects on your own and the `plot` function does not accept most of the regular Matplotlib parameters.\n\nNow, let's plot the seasonality of all types of meat over a 5-year interval:","metadata":{}},{"cell_type":"code","source":"# Init empty dict to store decompose results\nseasonality_dict = {}\n\nfor ts in meat.columns:\n    decompositions = sm.tsa.seasonal_decompose(meat[ts].dropna())\n    # Store the results back\n    seasonality_dict[ts] = decompositions.seasonal\n\n# Plot\nrcParams[\"figure.figsize\"] = 16, 14\npd.DataFrame(seasonality_dict)[\"2000\":\"2005\"].plot(\n    subplots=True, layout=(4, 2), linewidth=3\n);","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:30.023629Z","iopub.execute_input":"2021-07-13T11:35:30.024056Z","iopub.status.idle":"2021-07-13T11:35:31.737786Z","shell.execute_reply.started":"2021-07-13T11:35:30.024014Z","shell.execute_reply":"2021-07-13T11:35:31.736711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, each meat types have rather different seasonality patterns. Now, let's explore trends.","metadata":{}},{"cell_type":"markdown","source":"### 1.2 Trend Analysis","metadata":{}},{"cell_type":"markdown","source":"Once again, the overall trend of a time series shows whether it increased, decreased, or stayed constant (flat) over a time period. The above `DecomposeResult` object contains values that show the overall slope of a time series under the `trend` attribute.\n\nLet's plot them for the meat production dataset:","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:24:41.883453Z","iopub.execute_input":"2021-07-13T11:24:41.883821Z","iopub.status.idle":"2021-07-13T11:24:41.891113Z","shell.execute_reply.started":"2021-07-13T11:24:41.883791Z","shell.execute_reply":"2021-07-13T11:24:41.889444Z"}}},{"cell_type":"code","source":"trend_dict = {}\n\nfor ts in meat.columns:\n    decomposition = sm.tsa.seasonal_decompose(meat[ts].dropna())\n    # Store back the results\n    trend_dict[ts] = decomposition.trend\n\npd.DataFrame(trend_dict).plot(subplots=True, layout=(4, 3), linewidth=3);","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:31.740888Z","iopub.execute_input":"2021-07-13T11:35:31.74136Z","iopub.status.idle":"2021-07-13T11:35:33.364276Z","shell.execute_reply.started":"2021-07-13T11:35:31.741315Z","shell.execute_reply":"2021-07-13T11:35:33.363218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This plot is massively insightful compared to the simple line plot we saw in the beginning. Indeed, we now see that meat from lambs and veal production has decreased dramatically since the 1940s.\n\nThis might be in part caused by the double or triple production increases in beef, broilers, and turkey. We are performing informed guesses now, but we will explore some powerful methods to validate them in later sections.","metadata":{}},{"cell_type":"markdown","source":"### 1.3 Noise","metadata":{}},{"cell_type":"markdown","source":"The third component of time series is noise. There is nothing fancy about it like the other two components. It only shows random and irregular data points that could not be attributed to either seasonality or noise.\n\nYou can plot them using the `resid` attribute from the `DecomposeResult` object:","metadata":{}},{"cell_type":"code","source":"resid_dict = {}\n\nfor ts in meat.columns:\n    decomposition = sm.tsa.seasonal_decompose(meat[ts].dropna())\n    resid_dict[ts] = decomposition.resid\n\npd.DataFrame(resid_dict).plot(subplots=True, layout=(3, 3), linewidth=1);","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:33.366194Z","iopub.execute_input":"2021-07-13T11:35:33.366549Z","iopub.status.idle":"2021-07-13T11:35:35.012472Z","shell.execute_reply.started":"2021-07-13T11:35:33.366518Z","shell.execute_reply":"2021-07-13T11:35:35.011189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Calling `plot` on the whole `DecomposeResult` object will produce a plot with all components displayed on the same axes.\n\nDecomposing your times series helps you think of them in a structured manner. Instead of imagining a series as a value changing over time, you can think of it as a distribution with a particular seasonality signal or a feature with a particular slope. This level of data understanding can be a key factor during feature engineering and modeling.","metadata":{}},{"cell_type":"markdown","source":"## 2. Working With Multiple Time Series","metadata":{}},{"cell_type":"markdown","source":"### 2.1 Generals","metadata":{}},{"cell_type":"markdown","source":"Working with multiple time series presents certain challenges. One example, as we saw, is the different scales each distribution comes in:","metadata":{}},{"cell_type":"code","source":"meat.plot(figsize=(16, 8), title=\"Meat Production in Tons in the USA\")\nplt.xlabel(\"Date\");","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:35.013783Z","iopub.execute_input":"2021-07-13T11:35:35.014179Z","iopub.status.idle":"2021-07-13T11:35:35.423537Z","shell.execute_reply.started":"2021-07-13T11:35:35.014132Z","shell.execute_reply":"2021-07-13T11:35:35.422309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When features with larger scales squish others to a flat line (lamb and veal), it is impossible to compare their growth. One solution is using normalization.\n\nWhen normalizing time series, you divide every data point in the distribution by the first sample. This has the effect of representing every single data point as the percentage increase relative to the first sample:","metadata":{}},{"cell_type":"code","source":"meat.div(meat.iloc[0]).head()","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:35.424908Z","iopub.execute_input":"2021-07-13T11:35:35.425342Z","iopub.status.idle":"2021-07-13T11:35:35.443927Z","shell.execute_reply.started":"2021-07-13T11:35:35.425279Z","shell.execute_reply":"2021-07-13T11:35:35.442866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The best part is now, each distribution has the same scale. Let's plot the meat production data by performing normalization:","metadata":{}},{"cell_type":"code","source":"normalized = meat.div(meat.iloc[0]).mul(100)  # Multiply by 100 to get percentages\n\nnormalized.plot(figsize=(16, 8), title=\"Growth of Production in Different Meat Sectors\")\nplt.xlabel(\"Date\")\nplt.ylabel(\"Increase (%)\");","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:35.445345Z","iopub.execute_input":"2021-07-13T11:35:35.445669Z","iopub.status.idle":"2021-07-13T11:35:35.926777Z","shell.execute_reply.started":"2021-07-13T11:35:35.445636Z","shell.execute_reply":"2021-07-13T11:35:35.925776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Beef and pork saw the highest percentage increases while veal and lamb meat production plummeted over the given time period.","metadata":{}},{"cell_type":"markdown","source":"### 2.1 Relationships between time series with correlation","metadata":{}},{"cell_type":"markdown","source":"I know what you are saying: \"Correlation? Really? What's new….\"\n\nBut bear with me. A simple correlation heatmap can indeed tell a lot about the linear relationships between variables:","metadata":{}},{"cell_type":"code","source":"# Create a custom palette\ncmap = sns.diverging_palette(250, 15, s=75, l=40, n=9, center=\"light\", as_cmap=True)\n\n# Compute corr matrix\nmatrix = meat.corr(method=\"pearson\")\n# Create a mask\nmask = np.triu(np.ones_like(matrix, dtype=bool))\n\nfig, ax = plt.subplots(figsize=(7, 7))\nsns.heatmap(matrix, mask=mask, cmap=cmap, square=True, annot=True, fmt=\".2f\", ax=ax)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:35.928129Z","iopub.execute_input":"2021-07-13T11:35:35.92865Z","iopub.status.idle":"2021-07-13T11:35:36.306839Z","shell.execute_reply.started":"2021-07-13T11:35:35.92861Z","shell.execute_reply":"2021-07-13T11:35:36.305854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What we are more interested in is how underlying components of time series affect each other. For example, let's see how the seasonality of each time series influences others:","metadata":{}},{"cell_type":"code","source":"seasonality_dict = {\n    ts: sm.tsa.seasonal_decompose(meat[ts].dropna()).seasonal for ts in meat.columns\n}\n\n# Compute corr matrix\nseasonality_corr = pd.DataFrame(seasonality_dict).corr()\n\nsns.clustermap(seasonality_corr, annot=True, square=True)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:36.308355Z","iopub.execute_input":"2021-07-13T11:35:36.308935Z","iopub.status.idle":"2021-07-13T11:35:37.018594Z","shell.execute_reply.started":"2021-07-13T11:35:36.308891Z","shell.execute_reply":"2021-07-13T11:35:37.017637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This time, we are using a ClusterMap rather than a heatmap to see closely correlated groups with the help of dendrograms immediately.\n\nThe plot tells us that the seasonality of beef, broilers, and other chicken meats are heavily correlated. The same is true with pork and lamb/mutton meats. This positive correlation can be indicative of close seasonality matches. \n\nFor example, it is possible that increase/decrease patterns of beef, broilers, and other chicken meats often matched over the given period.\n\nLet's do the same for trends, which I think should be more interesting:","metadata":{}},{"cell_type":"code","source":"trend_dict = {\n    ts: sm.tsa.seasonal_decompose(meat[ts].dropna()).trend for ts in meat.columns\n}\n\n# Compute corr matrix\ntrend_dict = pd.DataFrame(trend_dict).corr()\n\nsns.clustermap(trend_dict, annot=True, square=True)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:37.019892Z","iopub.execute_input":"2021-07-13T11:35:37.020481Z","iopub.status.idle":"2021-07-13T11:35:37.74846Z","shell.execute_reply.started":"2021-07-13T11:35:37.020438Z","shell.execute_reply":"2021-07-13T11:35:37.747498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above plot is awesome because it helps us validate our assumptions in the trend analysis section. Let's look at the normalized plot of meat production once again:","metadata":{}},{"cell_type":"code","source":"normalized = meat.div(meat.iloc[0]).mul(100)\n\nnormalized.plot(figsize=(16, 8), title=\"Growth of Production in Different Meat Sectors\")\nplt.xlabel(\"Date\")\nplt.ylabel(\"Increase (%)\");","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:37.749828Z","iopub.execute_input":"2021-07-13T11:35:37.750108Z","iopub.status.idle":"2021-07-13T11:35:38.23493Z","shell.execute_reply.started":"2021-07-13T11:35:37.750081Z","shell.execute_reply":"2021-07-13T11:35:38.23412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Can you match the patterns in the cluster map to the line plot? For example, beef has strong negative correlations with lamb/mutton and veal. This is matched by the fact that beef production tripled in amount while the production of the other two decreased by ~75% (seen from the line plot).\n\nThe same observations can be made between pork and veal, lamb/mutton.","metadata":{}},{"cell_type":"markdown","source":"### 2.2 Correlation does not mean causation","metadata":{}},{"cell_type":"markdown","source":"I want you to tread carefully when making assumptions about correlated features. Always remember that correlation does not mean causation. When two features are heavily correlated, it does not mean an increase in one causes an increase in another.\n\nAn example I like to use is that even though the number of storks in a town can correlate with the number of newborn babies, it does not mean that storks deliver the babies.\n\nIt might take a while to draw the line between correlation and causation clearly, so why don't you take a look at my [other article](https://towardsdev.com/how-to-not-misunderstand-correlation-75ce9b0289e?source=your_stories_page-------------------------------------) on the topic.","metadata":{}},{"cell_type":"markdown","source":"## 3. Autocorrelation and partial autocorrelation","metadata":{}},{"cell_type":"markdown","source":"### 3.1 Autocorrelation","metadata":{}},{"cell_type":"markdown","source":"Autocorrelation is a powerful analysis tool for modeling time series data. As the name suggests, it involves computing the correlation coefficient. But here, rather than computing it between two features, correlation of a time series is found with a lagging version of itself.\n\nLet's first look at an example plot and explain further:","metadata":{}},{"cell_type":"code","source":"from statsmodels.graphics import tsaplots\n\nrcParams[\"figure.figsize\"] = 10, 6\n\n# Stands for Time Series Analysis Plots (TSA Plots)\nfig = tsaplots.plot_acf(tps[\"deg_C\"], lags=60)\n\nplt.xlabel(\"Lag at k\")\nplt.ylabel(\"Correlation coefficient\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:38.236161Z","iopub.execute_input":"2021-07-13T11:35:38.236429Z","iopub.status.idle":"2021-07-13T11:35:38.478532Z","shell.execute_reply.started":"2021-07-13T11:35:38.236403Z","shell.execute_reply":"2021-07-13T11:35:38.476582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The XAxis of an autocorrelation function plot (ACF) is the lag number k. For example, when k=1, the correlation is found by shifting the series by 1. This is the same as using the `shift` function of Pandas:","metadata":{}},{"cell_type":"code","source":"beef = meat[\"beef\"].to_frame(name=\"beef\")\n\nbeef[\"lag_1\"] = beef[\"beef\"].shift(periods=1)\nbeef[\"lag_2\"] = beef[\"beef\"].shift(periods=2)\nbeef[\"lag_3\"] = beef[\"beef\"].shift(periods=3)\n\nbeef.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:38.479596Z","iopub.execute_input":"2021-07-13T11:35:38.479883Z","iopub.status.idle":"2021-07-13T11:35:38.499269Z","shell.execute_reply.started":"2021-07-13T11:35:38.479854Z","shell.execute_reply":"2021-07-13T11:35:38.498366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The YAXis is the amount of correlation at each lag k. The shaded red region is a confidence interval - if the height of the bars is outside this region, it means the correlation is *statistically significant*.\n\nPlease pause and think of what you can learn from an ACF plot.\n\nThey offer an alternative way of detecting patterns and seasonality. For example, the ACF plot of temperature in Celcius shows that the correlation at every 15 lags decreases or every 25 lags increases.\n\nWhen a clear trend exists in a time series, the autocorrelation tends to be high at small lags like 1 or 2. When seasonality exists, the autocorrelation goes up periodically at larger lags.\n\nLet's look at another example:","metadata":{}},{"cell_type":"code","source":"fig = tsaplots.plot_acf(tps[\"target_carbon_monoxide\"], lags=70)\n\nplt.title(\"Autocorrelation of Carbon Monoxide\")\nplt.xlabel(\"Lag at k\")\nplt.ylabel(\"Correlation Coefficient\")\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:38.500631Z","iopub.execute_input":"2021-07-13T11:35:38.500954Z","iopub.status.idle":"2021-07-13T11:35:38.746907Z","shell.execute_reply.started":"2021-07-13T11:35:38.500913Z","shell.execute_reply":"2021-07-13T11:35:38.745956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The ACF of carbon monoxide confirms that small lags tend to have high correlations. It also shows that every 25 lags, the correlation increases significantly but quickly drops down to the negative. But most of the downward bars are inside the shaded area, suggesting that they are *not statistically significant*.\n\nThis ability to compare the relationship between past and present data points present a unique advantage. If you can associate the present value to points k periods before, this also means you can find a link to values that come after k periods.\n\nBesides, understanding autocorrelation is key to modeling time series with ARIMA models (a topic for another article).","metadata":{}},{"cell_type":"markdown","source":"### 3.2 Partial autocorrelation\n\nEven though discussing partial autocorrelation means we are getting way ahead of things, I will give you the gist.\n\nIt is similar to autocorrelation - it is calculated using the series and its lagged version at k:","metadata":{}},{"cell_type":"code","source":"# Mind the extra 'p' in plot_pacf\nfig = tsaplots.plot_pacf(tps[\"target_benzene\"], lags=70)\n\nplt.title(\"Partial Autocorrelation of Benzene\")\nplt.xlabel(\"Lag at k\")\nplt.ylabel(\"Correlation Coefficient\")\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-13T11:35:38.749453Z","iopub.execute_input":"2021-07-13T11:35:38.749761Z","iopub.status.idle":"2021-07-13T11:35:39.27729Z","shell.execute_reply.started":"2021-07-13T11:35:38.74973Z","shell.execute_reply":"2021-07-13T11:35:39.276198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The only difference is that this method tries to account for the effect the intervening lags have. For example, at lag 3, partial autocorrelation removes the effect lags 1 and 2 have on computing the correlation.\n\nWhile autocorrelation is useful for analyzing a time series's properties and choosing what type of ARIMA model to use, partial autocorrelation tells what order of autoregressive model to fit.\n\nAgain, this topic will be discussed in-depth when we talk about forecasting. So, [stay tuned](https://medium.com/r/?url=https%3A%2F%2Fibexorigin.medium.com%2F)!","metadata":{}},{"cell_type":"markdown","source":"## Summary\n\nCongratulations!\n\nBy reading this post, you learned powerful techniques to dissect any time series and derive meaningful insights. Most importantly, you now have the ability to apply these techniques to multiple time series and critically evaluate the relationships between them.\n\nThank you for reading, and I will see you in the next one!","metadata":{}},{"cell_type":"markdown","source":"### You might also be interested...\n- [Every Pandas Function You Can (Should) Use to Manipulate Time Series](https://towardsdatascience.com/every-pandas-function-you-can-should-use-to-manipulate-time-series-711cb0c5c749?source=your_stories_page-------------------------------------)\n- [Comprehensive Guide on Multiclass Classification Metrics](https://towardsdatascience.com/comprehensive-guide-on-multiclass-classification-metrics-af94cfb83fbd?source=your_stories_page-------------------------------------)\n- [Practical Sklearn Feature Selection in 3 stages](https://towardsdatascience.com/4-part-practical-study-guide-to-sklearn-feature-selection-b959a28d9ef7?source=your_stories_page-------------------------------------)","metadata":{}}]}