{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Time Series Forecasting: Is Simplicity the Key?

\n\n
\n
\n
\n \n---\n\n
\n
\n
\n\n

Abstract

\n\n
\n  Time series forecasting is a rich and interesting field of research. In recent years, a surge in transformers-based deep learning models has made considerable progress in time series forecasting tasks. However, more classically-based models and gradient boosting trees seem to still prove strong competition to current state of the art deep learning methods. This paper dives deeper into these topics through the lense of both Kaggle and some of the most recent state of the art deep learning research papers. Ultimately, discerning possible directions for future time series research.\n
\n
\n
\n
\n \n---","metadata":{}},{"cell_type":"markdown","source":"## 1. Introduction: A Short History of Time Series Analysis\n\n
\n  With the advent of the internet of things, smart devices, and microelectronics has come an exponential growth in high quality, real-time, and reliable datasets and data sources. This explosion in the amount of available, high quality data to the world has brought about increasing opportunity to take advantage of time series analysis to both diagnose past behavior as well as predict future behavior (Neilsen, 2023). Whether being utilized to predict population growth/ decay of endangered species, the strength and trajectory of a hurricane, or prices in the stock market, time series analysis can provide huge benefits to society as a whole by providing more accurate insight into probable future events or by helping fix mistakes made in the past. Further, advancements in accurately forecasting time series data will have a trickle down benefit to a massive amount of applicable use cases, and should be actively researched now and in the future.\n
\n
\n
\n  Though the origins of time series analysis can be tracked back to the creation of statistics, and was utilized as early as 800 BC to predict sunspots in ancient China, the formal discipline of time series analysis is suprisingly young (Neilsen, 2023). According to Neilson, the first application of autoregressive models only occurred when a Cambridge University professor, Udny Yule, applied an autoregressive model to sunspot data in the 1920s. And in 1969, \"The Combination of Forecasts\" first purposed the idea of ensemble methods, and over the late 1970s to early 1980s multiple developments in time series analysis brought about the early beginnings of machine learning. Since then, the advancements in time series analysis, machine learning, and deep learning have increased exponentially along with the advancements in computing technology.\n
\n
\n
\n  The following discussion will present the most recent advancements in time series forecasting and contemplate what might be the future direction for further advancement. First, the components of time series data will be broken down and defined, and a brief introduction to basic time series forecasting methods will be presented. Then, an analysis of the latest state of the art (SOTA) time series forecasting models found on Papers with Code (www.paperswithcode.com) will be performed. Finally, findings from the Kaggle commmunity competitions will be discussed, and the combined findings from within Kaggle and current state of the art deep learning models will be utilized to conclude on possible future directions for time series forecasting research.\n
\n
\n
\n
\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 2. Basics of Time Series Forecasting\n\n### 2.1- Structure\n
\n  In order to develop state of the art time series forecasting models, it is first important to understand the basic structures time series data can take. Time series data can be broken down into four main characteristics: stationarity, trend, seasonality, and cyclicality. Additionally, there can be unexpected variation in time series data that can not be accounted for in any of the previously mentioned characteristics, which can be referred to as irregularities. Figure 1 depicts an idea of these basic characteristics, excluding stationarity:\n
\n
\n\n
\n\n
\n
Figure 1: Characteristics of Time Series Data (Shukla, 2023)
\n\n
\n
\n
\n\n
\nThe following is a brief description of the four main characteristics of time series data:\n
\n\n#### 2.1.1- Stationarity\n
\n  According to Nam, \"A strictly stationary time series is one for which the probabilistic behavior of every collection of values is identical to that of the time shifted set\". In other words, a stationary time series dataset is one in which the mean and variance of the dataset does not change over time (Nam, 2020). This definition of stationarity is known as strictly stationary, while a looser definition of stationarity, known as weakly stationary, defines stationarity as follows:\n
\n
\n
\n\n\n
\n\"A weakly stationary time series, x_t , is a finite variance process such that:\n
\n  (i) the mean value function, μ_t , is constant and does not depend on time t\n
\n  (ii) the autocovariance function, γ(s, t), depends on s and t only through their difference |s − t|.\" (Nam, 2020)\n
\n
\n
\n\n
\nAs a counter example, commonly known non-stationary time series data comes from stock market data where natural volatility and uncertainty in the markets cause prices to swing in a hard to predict and statistically random manner. Figure 2 illustrates an example of both stationary and non-stationary time series datasets: \n
\n
\n
\n\n
\n\n
\n
Figure 2: (top) A stationary time series dataset, (bottom) A non-stationary time series dataset (Halder & Ozdemir, 2023)
\n\n
\n
\n
\n\n#### 2.1.2- Trend\n\n
\n  Trend refers to the general upward, downward, or unchanging pattern over time within time series datasets. In datasets where a clear trend is present, capturing this pattern may allow for a model to predict the general direction in which the time series is going (Nam, 2020). Therefore, it is common that data scientists will determine the trend and decompose the dataset into trend and remainder terms to analyze separately. Figure 3 depicts a decomposed trendline from a time series dataset:\n
\n
\n\n
\n\n
\n
Figure 3: A decomposed trendline from time series data (Nam, 2020)
\n\n
\n
\n
\n\n#### 2.1.3- Seasonality\n\n
\n  Seasonality refers to the regular pattern within a time interval (Nam, 2020). As an example, seasonality refers to patterns in consumer sales year-over-year or the seasonal trend airlines see in ticket purchases closer to holidays. Figure 4 illustrates seasonality:\n
\n
\n
\n\n
\n\n
\n
Figure 4: Seasonality, seen as the recurring peaks in the time series dataset (Radecic, 2021)
\n\n
\n
\n
\n\n#### 2.1.4- Cyclicality\n
\n  Cyclicality, not to be confused with seasonality, refers to data that exhibits rises and falls that are not of a fixed frequency (Hyndman & Athanasopoulos, 2018). Typically, cycles are longer and magnitudes more variable than seasonal patterns (Hyndman & Athanasopoulos, 2018). Looking closer at (b) and (c) in Figure 1, cyclicality exhibits greater variations in amplitude and frequency as compared to seasonality.\n
\n\n
\n
\n
\n\n### 2.2- Origins of Time Series Forecasting\n\n#### 2.2.1- Autoregression\n
\n  The basis of time series forecasting goes back to the development of autoregressive models. The simplest and most well known form of time series forecasting is linear regression. Linear regression optimizes a linear equation (yhat = b0 + b1*x) for optimal parameters (a and b) such that a time input, x, the linear equation outputs a prediction, yhat (Brownlee, 2021). Further, a value at time (t+1) can be predicted using the two previous time steps (t and t-1) by utlizing the following equation (Brownlee, 2021):\n
\n\n
\n
\n
\n\n
\n \n*x(t+1) = b0 + b1*x(t-1) + b2*x(t-2)*\n
\n
\n
\n\n
\nConsequently, the term autoregression stems from the idea that the model utilizes data from the same input variable at previous time steps to create the prediction (Brownlee, 2021).\n
\n\n#### 2.2.2- The Moving Average, ARMA, ARIMA, and Box-Jenkins Models\n
\n  After the introduction of autoregression, the moving average was introduced to remove periodic fluctuations due to seasonality and cyclicality (Stockholm UNiveristy, 2022). For reference, the simple moving average equation is provided below:\n
\n\n
\n
\n
\n\n
\n\n
\n
Figure 5: Equation for the Simple Moving Average (Fernando, 2023)
\n\n
\n
\n
\n\n
\nThis led to the development of the AutoRegressive Moving Average (ARMA) models (Stockholm University, 2022). AutoRegressive Integrated Moving Average (ARIMA) models further extend ARMA models by adding integration factor to the model. Finally, in 1970 G.E.P. Box and G.M. Jenkins came out with the book \"Time Series Analysis\" which contained the full modelling procedure for individual time series: specification, estimation, diagnositics, and forecasting. This was the first time that time series forecasting utilized a likelihood function and enabled maximum likelihood estimation (Stockholm University, 2022).\n
\n\n#### 2.2.3- ARCH and GARCH Models\n
\n  The AutoRegressive Conditional Heteroscedasticity (ARCH) and Generalized AutoRegressive Conditional Heteroscedasticity (GARCH) models were a further development based on the Box-Jenkins models (Stockholm University, 2022). These models were developed to allow parameterization and prediction of non-constant variance models. In other words, these were some of the first models to attempt to deal with predicting non-stationary time series.\n
\n\n
\n
\n
\n\n
\n  ARMA, ARIMA, ARCH, and Box-Jenkins models are still widely used today, and are the basis for the time series models discussed in the following sections.\n
\n
\n
\n
\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 3. A Study on Current Deep Learning-based SOTA Models\n\n### 3.1- Introduction to Papers With Code\n
  Papers with Code is a well known source for the latest SOTA research in machine learning and deep learning. \"The mission of Papers with Code is to create a free and open resource with Machine Learning papers, code, datasets, methods and evaluation tables.\" (Papers with Code, 2023). The choice to study current SOTA models on Papers with Code was two fold:\n
\n
\n
\n\n
\n1. Papers with Code provides high quality research papers coupled with relevant code for models and standardized evaluation metrics, available through their evaluation tables.\n
\n2. With 1000's of research papers on time series forecasting available, Papers with Code's evaluation tables make it much easier to filter through models for the highest performing models. This makes it possible to comparatively learn similarities and differences between top performing models, and helps inform on possible directions for future research.\n
\n
\n
\n \n### 3.2- Foundations of Current SOTA Models\n\n
\n  As deep learning was first developing, time series forecasting (TSF) methods did not make much progress past traditional regression methods such as ARIMA or machine learning-based gradient boosting trees (GBT). However, the introduction of recurrent neural networks (RNNs) and long-short term memory networks (LSTMs) showed some promise. Most recently, after the publication of \"Attention Is All You Need\" in 2017, the transformer wave has quickly made a significant impact on the SOTA for deep learning-based models for TSF. The following sections will further dive into some of the most recent developments in deep learning-based TSF models.\n
\n
\n \n#### 3.2.1- Transformers\n\n
\n  To fully understand the most recent developments in TSF models, a brief summary of the basics of a transformer is necessary. The novel concept that was presented in \"Attention Is All You Need\" was to replace recurrent or convolutional layers with multi-headed self-attention blocks within the encoder-decoder of a deep learning model (Vaswani et al., 2017). Figures 6 and 7 present the original transformer architecture along with the multi-headed self-attention block:\n
\n
\n
\n\n
\n\n
\n
Figure 6: The Transformer - model architecture (Vaswani et al., 2017)
\n\n
\n
\n
\n \n
\n\n
\n
Figure 7: (left) Scaled Dot-Product Attention, (right) Multi-Head Attention consists of several\nattention layers running in parallel (Vaswani et al., 2017)
\n\n
\n
\n
\n\n
\nSelf-attention provided main advantages compared to recurrent and convolutional layers: decreased computational complexity per layer, increased amount of computation that can be parallelized, and decreased path length between long-range dependencies in the network (Vaswani et al., 2017). This made learning long-range dependencies easier, and made training of the model significantly faster than recurrent and convolutional models. However, the Transformer suffers from time and memory complexity of O(N^2), where N is the input time series length. This made predicting long-term time series tasks both time consuming and memory hungry.\n
\n
\n
\n \n### 3.3- Current SOTA Models\n\n#### 3.3.1- Transformers-Based Models\n\n
\n  Since the introduction of the Transformer in 2017, many ammended transformer architectures have been proposed that apply well to TSF tasks. In 2019, LogTrans first attempted to deal with the Transformer's time and memory complexity bottle neck (Wen at el., 2023). Then, in 2021, Informer first specifically dealt with adapting the Transformer to TSF by proposing to encode timestamps as additional positional encoding by using learnable embedding layers (Wen et al., 2023). Further, it introduced ProbSparse, which replaced canonical self-attention and further decreased the time and memory complexity to O(LlogL) (Zhou et al., 2021). Figure 8 illustrates the Informer model:\n
\n
\n
\n \n
\n\n
\n
Figure 8: The Informer Model, originally introduced by (Zhou et al., 2021)
\n\n
\n
\n
\n\n
\n  \nAfter Informer, Autoformer (2021) further explored the problem of applying transformers to long-term forecasting problems. Autoformer tackled the idea of decomposing time series data into potential trend, cyclical, and seasonal components (Wu et al., 2022). This was achieved by decomposing and eliminating trend and cyclical components to isolate seasonal components in the encoder, and accumulating the trend progressively through the decoder. This was accomplished through the introduction of the Auto-Correlation block in addition to a series decomposition block (Wu et al., 2022). Figures 9 and 10 further describe the Autoformer architecture and its Auto-Correlation block:\n
\n
\n
\n \n
\n\n
\n
Figure 9: The Autoformer Model, originally introduced by (Wu et al., 2022)
\n\n
\n
\n
\n\n
\n\n
\n
Figure 10: (left) Auto-Correlation, (right) Time Delay Aggregation, originally introduced by (Wu et al., 2022)
\n\n
\n
\n
\n\n
\n  \nTime series, through the structure of its four main characteristics described previously, exhibits both long-term and short-term temporal dependencies which has been a main point of weakness in transformer models. Pyraformer (2022) further explored this problem by introducing the Pyramidal Attention Module (PAM) (Liu et al., 2022). This is achieved through a Coarser-Scale Construction Module (CSCM) that constructs a C-ary tree, and the PAM passes messages in both the inter-scale and the intra-scale fashion (Liu et al., 2022). Figures 11 and 12 illustrate these concepts:\n
\n
\n
\n\n
\n\n
\n
Figure 11: The Pyraformer Model, originally introduced by (Liu et al., 2022)
\n\n
\n
\n
\n\n
\n\n
\n
Figure 12: Coarser-Scale Construction Module, originally introduced by (Liu et al., 2022)
\n\n
\n
\n
\n\n
\n  Following Autoformer, FEDformer (2022) introduced a similar architecture to Autoformer but with one key difference- it introduced the Frequency Enhanced Block (FEB) and Frequency Enhanced Attention (FEA) (Zhou at el., 2022). FEB and FEA utilize fourier transforms to analyze time series in the frequency domain. Consequently, FEDformer brought the time and memory complexity to O(L) (Zhou at el., 2022). The following depict FEB and FEA as well as the FEDformer architecture:\n
\n
\n
\n\n
\n\n
\n
Figure 13: (top) The Frequency Enhanced Block, (middle) Frequency Enhanced Attention, (bottom) The FEDformer model (Zhou at el., 2022)
\n\n
\n
\n
\n\n
\n  \nMost recently, PatchTST (2023) was introduced partly as a response to a growing question: are transformers effective for time series forecasting? Though this has been challenged naturally through performances of other models, it was brought to the forefront of the TSF world by the paper \"Are Transformers Effective for Time Series Forecasting?\" (2022). For continuity PatchTST is first discussed here, and further elaboration on other models and this question will be discussed in the following section.\n
\n
\n
\n\n
\n  PatchTST brought two concepts to TSF. First, it segmented time series into subseries-lovel patches which serve as input tokens to a transformer architecture (Nie et al., 2023). Second, it created channel independence where each channel contains a single univariate time series. These concepts reduce time and space complexity, better capture local semantic information, and benefit from longer look-back windows (Nie et al., 2023). The following illustrates these concepts:\n
\n
\n
\n \n
\n\n
\n
Figure 14: (top) The PatchTST model, (bottom-left) Supervised Transformer Backbone, (bottom-right) Self-supervised Transformer Backbone (Nie et al., 2023)
\n\n
\n
\n
\n\n#### 3.3.2- Other Architectures\n\n##### 3.3.2.1- FiLM\n\n
\n  The Frequency improved Legendre Memory (FiLM) model was introduced in late 2022. FiLM \"applies Legendre Polynomials projections to approximate historical information, uses Fourier projection to remove noise, and\nadds a low-rank approximation to speed up computation\" (Zhou et al., 2022). To do this, it redesigns the Legendre Projection Unit (LPU) and proposes Frequency Enhanced Layers (FEL) (Zhou et al., 2022). Though not a transformer architecture, it outperforms most of the previously mentioned transformer-based models in multiple evaluation tables. The main architecture for FiLM is presented below:\n
\n
\n
\n\n
\n\n
\n
Figure 15: (top) The FiLM model, (middle) Redesigned LPU for FiLM, (bottom) The Frequency Enhanced Layer (Zhou et al., 2022)
\n\n
\n
\n
\n\n##### 3.3.2.2- SCINet\n \n
\n  SCINet is a recursive downsample-convolve-interact architecture that proposes a SCI-Block which splits inputs into two sub-features (Liu et al., 2022). SCINet, like FiLM, also outperforms most transformer-based solutions. Its architecture is presented below:\n
\n
\n
\n\n
\n\n
\n
Figure 16: The SCINet model (Liu et al., 2022)
\n\n
\n
\n
\n\n##### 3.3.2.3- DLinear and NLinear\n\n
\n  DLinear was first proposed in 2022 and later appended to include NLinear. These models specifically challenged transformers through their simplicity and SOTA performance at the time of publishing. DLinear implements linear layers in combination with a decomposition block that separates time series data into trend and remainder (Zeng et al., 2022). Then, NLinear adds normalization to account for distribution shifts over long-term time series (Zeng et al., 2022). The architecture for DLinear, taken from an original version of the paper, is presented below:\n
\n
\n
\n\n
\n\n
\n
Figure 17: The DLinear model (Zeng et al., 2022)
\n\n
\n
\n
\n\n##### 3.3.2.4- ATD\n\n
\n  Lastly, Alternative Telescopic Displacement (ATD) was just recently proposed in June 2023. ATD proposes the ATD block which performs feature alignment on multi-modal information, and allows for robust combinations of forecasting methods (Qin et al., 2023). ATD utilizes its ATD block to combine Res-Net and LSTM model predictions which are fed differing data types- Res-Net is given image data, while the LSTM is given text data (Qin et al., 2023). By formatting data in differing ways the ATD hopes to extract features from both modals that most accurately predict the final result, find the architecture below:\n
\n
\n
\n\n
\n\n
\n
Figure 18: (top) The ATD model, (bottom) The ATD Block (Qin et al., 2023)
\n\n
\n
\n
\n\n
\nThough still largely unverified, ATD claims to have achieved SOTA performance in some of the key metrics provided in Papers with Codes evaluation tables.\n
\n
\n
\n\n#### 3.3.3- Trends and Conclusions from SOTA Models\n\nThe following figures show the most recent updates to some of the evaluation tables for TSF, taken directly from a few of the research papers discussed:\n\n
\n\n
\n
Figure 19: Performance of various models, taken from PatchTST paper (*Note NLinear is not included in this table) (Nie et al., 2023)
\n\n
\n
\n
\n \n
\n\n
\n
Figure 20: ATD performance in Comparison to the ETT datset (Qin et al., 2023)
\n\n
\n
\n
\n\n
\n  From these results, it seems that the question if transformers are effective for TSF should not be a question. Both transformer-based models, such as PatchTST, and non-transformer-based models, such as DLinear and ATD, have clearly proven competitive in SOTA performance. Consequently, the true question should be what makes these models so effective. Taking an overarching look at the trends among all of these SOTA models, a few clear patterns emerge:\n
\n
\n
\n\n
\n1. The best performing models contain some form of decomposition block.\n
\n2. Most models attempt to take in account both short- and long-term dependencies, though through various methodologies.\n
\n3. Tree-based architectures, or architectures in which data is split into separate patches, seem to perform well on time series data.\n
\n4. Combined architectures, such as in ATD, (though not strictly an ensemble method) shows promise in capturing differing features within time series data.\n
\n
\n
\n
\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 4. A Study on Kaggle Time Series Competition Winners\n\n### 4.1- Overview of Historical Winning Models\n\n
\n  Figure 21, made possible by all the helpful Kagglers who have posted previous winning solutions, depicts some of the most recent winning solutions for TSF-based Kaggle competitions:\n
\n
\n
\n\n
\n\n
\n
Figure 21: Past Kaggle TSF Winning Models
\n\n
\n
\n
\n\n
\nClearly, light gradient-boosting machines (LGBMs) make up the bulk of the models used in winning solutions, though other models have started to make their way into top solutions over the most recent competitions. This brings up the question: are deep learning models appropriate for time series forecasting? Further, ensembling methods prove quite promising for performance. And lastly, Figure 20 makes another clear observation- feature engineering is of great importance to most TSF tasks. Though feature engineering differs between most models, a clear trend is present in winning models and careful selected features. The next sections will further discuss these two findings in greater detail.\n
\n
\n
\n \n#### 4.1.1- A Look at Gradient Boosting Models\n\n
\n  Gradient boosting machines (GBMs) are decision trees. Each node in a decision tree represents what may be thought of as a question and the answers to these questions decide the direction in which the model movea down the layers of the tree. Practically, GBM models do not ask questions, but optimize numerical nodes to best find the output based on the input data. Figure 22 shows some popular GBMs:\n
\n
\n
\n \n
\n\n
\n
Figure 22: Popular GBM models (Nahon, 2019)
\n\n
\n
\n
\n\n
\nSo, why have GBM-based models, specifically lightGBM, been so successful on Kaggle? There is no clear answer to this question, but a few theories come to mind:\n
\n
\n
\n\n
\n1. GBM models may be able to more accurately capture long- and short-term dependencies by extracting these features in separate branches of the tree.\n
\n2. GBM models may be able to decompose the model into more accurate representations of trend, cyclical, and seasonal components through separation of feature extraction in its branches.\n
\n3. Non-stationarity may, in some not fully understood way, be mitigated through GBMs tree structure.\n
\n4. Relative simplicity of GBMs as compared to other SOTA methods may eliminate some of the issues with long path dependencies.\n
\n5. GBMs may, in their own way, act somewhat like an ensembled model with different groups of branches behaving in slightly differing ways.\n
\n5. Familiarity and ease of use of GBMs may mean that a larger proportion of models submitted to Kaggle competitions are GBM-based, which would statistically make it more likely for GBM models to be within winning solutions. \n
\n6. SOTA deep learning models, due to their more recent publication, may not be as easy and accessible to experiment with during a competition. Most of these models only exist within Github projects, and have not been made accessible through popular tools such as Pytorch, Hugging Face, etc. This may lead to a much smaller subset of Kagglers willing to experiment with the new models.\n
\n
\n
\n \n#### 4.1.2- The Importance of Feature Engineering\n\n
\n  To further explore feature engineering, a brief discussion about the 1st and 4th place solutions of the Optiver Realized Volatility Prediction competition will be taken. What is most interesting about these solutions is that both solutions go to great lengths to utilize nearest neighbors feature engineering. Thinking critically about the consequence of nearest neighbors on a volatility prediction, nearest neighbors may be quite good at capturing short-term dependencies over a long-term lookback window. To clarify, nearest neighbors may be good at capturing repetitive features within volatility spikes by looking at similar patterns over a long lookback window. What is the importance of this? Nearest neighbors may be the most application specific feature engineering technique for tracking volatility in time series data. Therefore, it may produce excellent comparative results regardless of model complexity. Figure 23 illustrates how nearest neighbors groups closely related features together. However, within time series nearest neighbors selects lookback windows of fixed time length and finds similar features among the differing lookback windows.\n
\n
\n
\n\n
\n\n
\n
Figure 23: An idea of nearest neighbors classification (Harrison, 2018)
\n\n
\n
\n
\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"## 5. Conclusion\n\n
\n  This paper has broken down time series data into its basic forms, discussed current SOTA deep learning methods for TSF, and explored the most successful methods of TSF among Kaggle competitions. Ultimately, it has brought up multiple questions on the future of TSF methods. However, the variety of successful use cases of differing methods may be the answer to all of these questions. There is no incorrect or correct method of TSF, and pursuing further research into the effectiveness of both deep learning and more classical methods for TSF should continue in the future. Key findings of this paper highlight a few trends that may be worth further pursuing:\n
\n
\n
\n\n
\n1. Investigate further ways of more accurately decomposing time series into its main characteristics: trend, seasonality, and cyclicality, with the remainder accounting for non-stationarity and irregularities.\n
\n2. Research and classify standardized methods of feature engineering that provide superior results for specific TSF characteristics, such as in the case of nearest neighbors and its performance with volatility prediction.\n
\n3. Investigate different combinations of ensemble methods that combine both deep learning and classically-based models.\n
\n4. Explore the possibilities of combining classical concepts directly inside of deep learning models to possibly get a best of both worlds scenario.\n
\n
\n
\n\n
\nUltimately, TSF is a rich and exciting field of study and there is much to learn moving forward.\n
\n
\n
\n
\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 6. Future Recommendations\n\n
\nThe findings in this paper bring up a few recommendations for both Kaggle and the general data science community:\n
\n
\n
\n\n
\n1. In order to facilitate more exploration into SOTA models on Kaggle, it would be useful to develop some method of standardizing SOTA models for use within Kaggle as they are proposed. The method in which this could be done is to be determined. However, to create a sort of \"SOTA sandbox\" where Kagglers can pick and choose more recent SOTA models from the models tab, and ensemble/edit them to their fancy, would make playing with new models dramatically easier. Hopefully, leading to more exploration with TSF.\n
\n2. In addition to a \"SOTA sandbox\", it would be interesting to have a \"SOTA block sandbox\" where different blocks/modules from within the SOTA models are standardized for use. This would allow for creation of experimental new models relatively easily. For example, one could combine a PatchTST architecture with DLinear decomposition blocks and SCI-blocks to test ways in which different combinations may help with feature extraction. Ultimately, this may lead to new SOTA models coming directly out of Kaggle. Further, this could be extended to feature engineering blocks to simplify the model creation process.\n
\n3. Currently, there is a lack of comparative results between the SOTA evaluation tables and winning Kaggle GBM models. It would be worthwhile to run the evaluation table datasets on high performing GBM-based models to further compare deep learning and GBM-based models. This would help provide further insight in to the question of which models perform best in TSF, and could provide further insight into methods worth further researching.\n
\n
\n
\n
\n\n---","metadata":{}},{"cell_type":"markdown","source":"## References\n\nBrownlee, J. (2021, September 7). Autoregression Models for Time Series Forecasting With Python. Machine Learning Mastery. https://machinelearningmastery.com/autoregression-models-time-series-forecasting-python/#:~:text=Autoregression%20is%20a%20time%20series,range%20of%20time%20series%20problems.. Accessed July 3 2023\n
\n
\nFernando, J. (2023, March 31). Moving Average (MA): Purpose, Uses, Formula, and Examples. Investopedia. https://www.investopedia.com/terms/m/movingaverage.asp. Accessed July 3 2023\n
\n
\nHalder, S., & Ozdemir, S. (2023, July 3). Stationarity of a time series models. O'Reilly. https://www.oreilly.com/library/view/hands-on-machine-learning/9781788992282/15c9cc40-bea2-4b75-902f-2e9739fec4ae.xhtml. Accessed July 3 2023\n
\n
\nHarrison, O. (2018, September 10). Machine Learning Basics with the K-Nearest Neighbors Algorithm. Towards Data Science. https://towardsdatascience.com/machine-learning-basics-with-the-k-nearest-neighbors-algorithm-6a6e71d01761. Accessed July 3 2023\n
\n
\nHyndman, R.J., & Athanasopoulos, G. (2018) Forecasting: principles and practice, 2nd edition, OTexts: Melbourne, Australia. OTexts.com/fpp2. Accessed July 3 2023\n
\n
\nLiu, M., Zeng, A., Chen, M., Xu, Z., Lai, Z., Ma, Q., & Xu, Q. (2022, October 13). SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction. Arxiv. https://doi.org/10.48550/arXiv.2106.09305.\n
\n
\nLiu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A., & Dustdar, S. (2022). Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting. Papers with Code. https://paperswithcode.com/paper/pyraformer-low-complexity-pyramidal-attention.\n
\n
\nNahon, A. (2019, December 30). XGBoost, LightGBM or CatBoost — which boosting algorithm should I use?. Medium. https://medium.com/riskified-technology/xgboost-lightgbm-or-catboost-which-boosting-algorithm-should-i-use-e7fda7bb36bc. Accessed July 3 2023\n
\n
\nNam, T. (2020, September 26). Time Series Data Characteristics. Medium. https://medium.com/@namnguyenthe/time-series-data-characteristics-994e43c470c6. Accessed July 3 2023\n
\n
\nNie, Y., Nguyen, N., Sinthong, P., & Kalagnanam, J. (2023, March 5). A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. Arxiv. https://doi.org/10.48550/arXiv.2211.14730.\n
\n
\nNielsen, A. (2023, July 3). Practical Time Series Analysis. O'Reilly. https://www.oreilly.com/library/view/practical-time-series/9781492041641/ch01.html. Accessed 3 July 2023\n
\n
\nPapers with Code. (2023). Our Mission. Papers with Code. https://paperswithcode.com/about. Accessed July 3 2023\n
\n
\nQin, J., Xu, Y., Luo, Z., Liu, C., Lu, Z., & Zhang, X. (2023, June 29). Alternative Telescopic Displacement: An Efficient Multimodal Alignment Method. Arxiv. https://doi.org/10.48550/arXiv.2306.16950.\n
\n
\nRadecic, D. (2021, July 15). Time Series From Scratch — Decomposing Time Series Data. Medium. https://towardsdatascience.com/time-series-from-scratch-decomposing-time-series-data-7b7ad0c30fe7. Accessed July 3 2023\n
\n
\nShukla, S. (2023, February 20). Various Techniques to Detect and Isolate Time Series Components Using Python. Analytics Vidhya. https://www.analyticsvidhya.com/blog/2023/02/various-techniques-to-detect-and-isolate-time-series-components-using-python/. Accessed 3 July 2023\n
\n
\nStockholm University (2022, May 13). A brief history of time series analysis. Stockholm University. https://www.su.se/english/research/research-subjects/statistics/statistical-models-in-the-social-sciences/a-brief-history-of-time-series-analysis-1.612367. Accessed July 3 2023\n
\n
\nVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., & Polosukhin, I. (2017). Attention is All You Need. Arxiv. https://doi.org/10.48550/arXiv.1706.03762.\n
\n
\nWen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2023, May 11). Transformers in Time Series: A Survey. Arxiv. https://doi.org/10.48550/arXiv.2202.07125.\n
\n
\nWu, H., Xu, J., Wang, J., & Long, M. (2022, January 7). Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. Arxiv. https://doi.org/10.48550/arXiv.2106.13008.\n
\n
\nZeng, A., Chen, M., Zhang, L., & Xu, Q. (2022, August 17). Are Transformers Effective for Time Series Forecasting?. Arxiv. https://doi.org/10.48550/arXiv.2205.13504.\n
\n
\nZhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021, March 28). Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. Arxiv. https://doi.org/10.48550/arXiv.2012.07436.\n
\n
\nZhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., & Jin, R. (2022, June 16). FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. Arxiv. https://doi.org/10.48550/arXiv.2201.12740.\n
\n
\nZhou, T., Ma, Z., Wang, X., Wen, Q., Sun, L., Yao, T., Yin, W., & Jin, R. (2022, September 16). FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting. Arxiv. https://doi.org/10.48550/arXiv.2205.08897.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2023-07-05T07:29:34.226704Z","iopub.execute_input":"2023-07-05T07:29:34.228714Z","iopub.status.idle":"2023-07-05T07:29:34.291866Z","shell.execute_reply.started":"2023-07-05T07:29:34.228576Z","shell.execute_reply":"2023-07-05T07:29:34.290186Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-05T09:22:12.633634Z","iopub.execute_input":"2023-07-05T09:22:12.634009Z","iopub.status.idle":"2023-07-05T09:22:12.702037Z","shell.execute_reply.started":"2023-07-05T09:22:12.633979Z","shell.execute_reply":"2023-07-05T09:22:12.700712Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val = [\"'Tabular and/or time series data'\", \"'https://www.kaggle.com/code/subsailor/time-series-forecasting-is-simplicity-the-key'\",\n \"'https://www.kaggle.com/code/umuttoygargoz/time-series-forecasting/comments#2335843'\",\n \"'https://www.kaggle.com/code/laalinibhgoadi/kaggle-ai-report-for-tabular-and-time-series-data/comments#2335850'\",\n \"'https://www.kaggle.com/code/kavisekar/latest-development-in-time-series-analysis-report/comments#2335854'\"]\nsubmission_df.value = val\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-05T09:24:21.174596Z","iopub.execute_input":"2023-07-05T09:24:21.174979Z","iopub.status.idle":"2023-07-05T09:24:21.190794Z","shell.execute_reply.started":"2023-07-05T09:24:21.174948Z","shell.execute_reply":"2023-07-05T09:24:21.189528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-05T09:24:22.831032Z","iopub.execute_input":"2023-07-05T09:24:22.831650Z","iopub.status.idle":"2023-07-05T09:24:22.841344Z","shell.execute_reply.started":"2023-07-05T09:24:22.831614Z","shell.execute_reply":"2023-07-05T09:24:22.839738Z"},"trusted":true},"execution_count":null,"outputs":[]}]}