{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction :\n\nIn the past century, the academic publishing world has changed drastically in volume and velocity. The volume of the paper has increased sharply from <1 million papers published in 1980 to >7 million papers published in 2014. Furthermore, the speed at which researchers share and publish their studies has increased significantly. AI research is on the rise, across the board. The total number of AI publications has more than doubled since 2010. The specific AI topics that continue to dominate research include pattern recognition, machine learning, and computer vision\n\n![AI publications Trends](https://aiindex.stanford.edu/wp-content/uploads/2023/03/fig_1.1.1_num_AI_pub_world-scaled.jpg)\n*[source :2023 AI index Report-HAI-Human Centered Artificial Intelligence-Stanford]* \n\n### INDUSTRY RACES AHEAD OF ACADEMIA\n\nHigher education has become increasingly monitored and managed by national states over the past half century [Amaral, Meek, et al., 2003](https://doi.org/10.1007/978-94-010-0072-7). As part of this, countries typically have competitive systems in place to fund academic research. In addition to project-based funding, some nations also now directly reward research quality through systematic procedures to assess this at the departmental level. Until 2014, the most significant machine learning models were released by academia. Since then, the industry has taken over. In 2022, there were 32 significant industry-produced machine learning models compared to just three produced by academia. Building state-of-the-art AI systems increasingly requires large amounts of data, computer power, and money—resources that industry actors inherently possess in greater amounts compared to nonprofits and academia.\n\n![ML system by sector](https://aiindex.stanford.edu/wp-content/uploads/2023/03/fig_1.2.2_num_models_by_ind_year-1-scaled.jpg)\n[source:2023 AI index Report-HAI-Human Centered Artificial Intelligence-Stanford]  \n\nThe Gap in External research funding for private versus public American CS Departments continues to widen. In 2011, the median amount of total expenditure from external sources for computing research was roughly the same for private and public CS departments in the United States. Since then, the gap has widened, with private U.S. CS departments receiving millions more in additional funding than public universities. In 2021, the median expenditure for private universities was $9.7 million, compared to $5.7 million for public universities.\n\n![private research vs public research](https://aiindex.stanford.edu/wp-content/uploads/2023/04/fig_5.1.20_median_total_expenditure_external_sources_computing_research_US_CS_dep-scaled.jpg)\n*[source :2022 AI index Report-HAI-Human Centered Artificial Intelligence-Stanford]*\n\nThe number of AI research collaborations between the United States and China increased roughly 4 times since 2010 and was 2.5 times greater than the collaboration totals of the next nearest country pair, the United Kingdom and China. However, the total number of U.S.-China collaborations only increased by 2.1% from 2020 to 2021, the smallest year-over-year growth rate since 2010.\n\n![U.S and china collaboration](https://aiindex.stanford.edu/wp-content/uploads/2023/03/fig_1.1.6_num_AI_pub_collabs_us_cn-1-scaled.jpg)\n*[source :2023 AI index Report-HAI-Human Centered Artificial Intelligence-Stanford]*\n\nIn the \"Age of big data\", we are witnessing a rapid increase in quantity and quality of the massive production of data. The data come from various industries such as continuous monitoring networks and remote sensing data, industrial processes, weather, Agriculture sensor data, financials, medical, research, customer monitoring, Recommendation system...etc. On each day we produce about 2.5 quintillion (2.5 X 1018) bytes of data and out of that 90% possibly have been created in the last two-three years. The present era is undoubtedly the era of Big Data and it is a prime need of the hour  to make use of these large time series data using powerful time series analysis techniques, either classical/statistical or machine learning-based techniques, which will further increase.\n\n![AI publication by field](https://aiindex.stanford.edu/wp-content/uploads/2022/03/1.1.3.png)\n[source :2022 AI index Report-HAI-Human Centered Artificial Intelligence-Stanford] \n\nA time series is an ordered sequence of data indexed by time usually obtained at equally spaced points in time likely to be collected yearly, monthly, weekly, or hourly. Time series analysis is the attempt of extracting meaningful characteristics and statistical information from data organized in chronological order. \n\nTime series analysis is one of the dynamic research areas there has been an explosion of interest in this subject and hence it has a fairly stable proportion of papers published over time. It is a daunting task to analyze thousands of papers published over the years and it is impossible to do full justice for papers of this length in the development of this field. This report provides a selective guide on time series analysis, covering the period 2021-2023 and summarizing about 939 papers published under the “time series” out of a total of over 217K papers only under the CS category.\n\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport calendar\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom plotly.offline import init_notebook_mode, iplot \nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objects as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nfrom IPython.display import HTML\nfrom ast import literal_eval\nfrom wordcloud import WordCloud\nimport os\nfrom IPython.display import Image\n\nplt.style.use('dark_background')\npd.set_option('display.max_rows', 500)\npd.set_option('display.max_columns', 500)\npd.set_option('display.width', 1000)\n\n# Graphics in retina format \n%config InlineBackend.figure_format = 'retina' \n\n#Load data\narxiv_df = pd.read_csv('/kaggle/input/arxive-paper-structuring/arxiv_papers.csv',low_memory=False)\ncs_arxivdf = pd.read_csv('/kaggle/input/arxive-paper-structuring/cs_arxiv.csv',low_memory=False)\nTSA_arxivdf = pd.read_csv('/kaggle/input/arxive-paper-structuring/time_series_arxivdf.csv',low_memory=False)\nkaggle_comp = pd.read_csv('/kaggle/input/medals-every-kaggle-competition-solution-writeup/kaggle_ai_report_data.csv')\n\ndef human_format(num):\n    num = float('{:.3g}'.format(num))\n    magnitude = 0\n    while abs(num) >= 1000:\n        magnitude += 1\n        num /= 1000.0\n    return '{}{}'.format('{:f}'.format(num).rstrip('0').rstrip('.'), ['', 'K', 'M', 'B', 'T'][magnitude])\n\nyear_dict = {\n    0:2000,  \n    1:2001,  \n    2:2002,  \n    3:2003,  \n    4:2004,  \n    5:2005,  \n    6:2006,  \n    7:2007,  \n    8:2008,  \n    9:2009, \n    10:2010, \n    11:2011, \n    12:2012, \n    13:2013, \n    14:2014, \n    15:2015, \n    16:2016,\n    17:2017, \n    18:2018, \n    19:2019, \n    20:2020, \n    21:2021, \n    22:2022, \n    23:2023, \n    91:1991, \n    92:1992, \n    93:1993, \n    94:1994, \n    95:1995, \n    96:1996, \n    97:1997, \n    98:1998, \n    99:1999    \n}\n\n\n\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T21:36:35.320050Z","iopub.execute_input":"2023-07-16T21:36:35.320488Z","iopub.status.idle":"2023-07-16T21:36:57.066648Z","shell.execute_reply.started":"2023-07-16T21:36:35.320447Z","shell.execute_reply":"2023-07-16T21:36:57.065438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"arxiv_df[\"year\"].replace(year_dict, inplace=True)\npaper_count = arxiv_df.groupby('year').agg({'title':'count'}).reset_index(names=\"year\")\npaper_count['count_string'] = [human_format(x) for x in paper_count['title']]\ncol = ['#e63946']\n\nx = paper_count['year'].values\ny = paper_count['title'].values\n\nfig = go.Figure()\n\nfig.add_trace(\n    go.Scatter(\n        x=x,\n        y=y,\n        name=\"line\",\n        line=dict(color='#e63946')\n    \n    ))\n\nfig.add_trace(\n    go.Bar(\n        x=x,\n        y=y,\n        text=paper_count['count_string'],\n        width=1,\n        textposition='auto',\n        name=\"bar\",\n        marker= dict(color='black')\n    ))\n\nfig.data[0].marker.line.width = 0.2\nfig.data[0].marker.line.color = \"black\"\nfig.update_layout(yaxis=dict(title='Number of Papers'),\n                  title='arXiv-Total number of Paper Published over the years',template=\"plotly_dark\",\n                  xaxis=dict(title='Years',tickmode = 'array',tickvals = list(paper_count['year']),ticktext = list(paper_count['year'])))\n\nfig.show(renderer=\"notebook\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:36:57.068003Z","iopub.execute_input":"2023-07-16T21:36:57.068361Z","iopub.status.idle":"2023-07-16T21:36:58.649272Z","shell.execute_reply.started":"2023-07-16T21:36:57.068310Z","shell.execute_reply":"2023-07-16T21:36:58.648268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"arXiv is a curated research-sharing platform open to anyone. As a pioneer in digital open access, arXiv.org hosts more than two million scholarly articles in eight subject areas. It offers researchers a broad range of services: article submission, compilation, production, retrieval, search and discovery, web distribution for human readers, and API access for machines, together with content curation and preservation. The impact of this is proven by the exponential growth shown by the paper published over the year, we can see the sturdy increase in the number of papers published and it's still increasing tremendously. As of July 14, 2023, there are 2,292,060  new submissions received during each month since August 1991 (after 32.0 years).","metadata":{}},{"cell_type":"markdown","source":"## arXiv Totalt submission by discipline from 1991 - 2023","metadata":{}},{"cell_type":"code","source":"arxiv_df[\"discipline\"] = arxiv_df.discipline.str[1:-1].str.split(\",\\s\").map(set).apply(list)\narxiv_df[\"discipline\"] = arxiv_df[\"discipline\"].apply(lambda x: ', '.join([i for i in x if isinstance(i, str)]))#arxiv_df[\"discipline\"].apply(tuple)\n\nviz_discipline = arxiv_df.groupby(['discipline']).agg({'title':'count'}).sort_values('title',ascending=False).reset_index()\ndiscipline_list = list(viz_discipline['discipline'])\n\narxiv_discipline = arxiv_df.groupby(['discipline','year']).agg({'title':'count'}).reset_index()\narxiv_discipline['count_string'] = [human_format(x) for x in arxiv_discipline['title']]\narxiv_discipline = arxiv_discipline.sort_values('title',ascending=False)#.head(15)\ntemp =arxiv_discipline[arxiv_discipline['discipline'].isin(discipline_list[:10])].sort_values('year',ascending=False)\n\n# Pivot the DataFrame to have subjects as columns and years as rows\npivot_df = temp.pivot(index='year', columns='discipline', values='title').sort_values('year',ascending=False)\n\n# Create a heatmap\nplt.figure(figsize=(13, 6))\nsns.heatmap(pivot_df, cmap='crest', annot=False, cbar=True,vmin=0, linewidth=1)\nplt.title('Paper Counts by discipline Over Years',size=20)\nplt.xticks(horizontalalignment='right',rotation=75,wrap=True,size=9)\nplt.xlabel('Discipline',size=12)\nplt.ylabel('Year',size=12)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:36:58.651005Z","iopub.execute_input":"2023-07-16T21:36:58.651414Z","iopub.status.idle":"2023-07-16T21:37:14.336122Z","shell.execute_reply.started":"2023-07-16T21:36:58.651378Z","shell.execute_reply":"2023-07-16T21:37:14.334713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In arXive the number of  papers published is dominated by \"Physics\" and then \"mathematics\" as we observe closely, In the recent year from 2016 computer science is catching up and publishing more papers. Some inter-discipline also publish a considerable amount of papers.","metadata":{}},{"cell_type":"markdown","source":"# A brief history:\n\nThe theoretical advances in time series analysis started early at the beginning of the last century with new developments in the field of stochastic processes. The first actual application of autoregressive models to time series can be identified in the work of Yule (1927) and Walker (1931).\n\nDuring this time the **moving average(MA)** was introduced to remove periodic fluctuations in the time series, for example, fluctuations due to seasonality. Herman Wold introduced **ARMA (AutoRegressive Moving Average)** models for stationary series. But it is since the pioneering book “Time Series Analysis” by Box and Jenkins in the 1970s (Box and Jenkins, 1970), containing the full modeling procedure for individual series: specification, estimation, diagnostics, and forecasting this leads to many lines of study in time series analysis.\n\nThe first generalization was to accept multivariate ARMA models, among which especially VAR models (Vector AutoRegressive) have become popular. These techniques, however, are only applicable to stationary time series.\n\nARCH and GARCH models\nAnother line of development in time series, originating from Box-Jenkins models, is the non-linear generalizations, mainly ARCH (AutoRegressive Conditional Heteroscedasticity) - and GARCH- (G = Generalized) models. These models allow parameterization and prediction of non-constant variance. These models have thus proved very useful for financial time series. The invention of them and the launch of the error correction model gave C. W. J Granger and R. F. Engle the Nobel Memorial Prize in Economic Sciences in 2003.\n\n\n# The Origins of Machine Learning Time Series Analysis\n\nEarly machine learning in time series analysis dates back many decades. An oft-cited paper from 1969, “The Combination of Forecasts,” analyzed the idea of combining forecasts rather than choosing the “best one” as a way to improve forecast performance.\n\nMore recently, practical uses for time series analysis and machine learning emerged as early as the 1980s(Nielsen, 2019), to deal with the increasing amount of available data in an automated way.\n\nThe traditional time-series forecasting models such as ARIMA have been prominent for a long time and many researchers still rely on such models as they can predict efficiently and provide interpretability. However, advances in machine learning research concluded that Deep Learning(DL) and neural networks can be more powerful models as they can give higher accuracy \nalong with ML and DL models AutoML provided solutions to build and validate machine learning pipelines minimizing user intervention.\n\n\nTime series analysis and forecasting are in their golden period, and, to date, time series analysis researchers use both traditional statistical methods as well as machine learning, Deep Learning techniques. However, AutoML for time series is still in the evolution phase and requires efforts from researchers to become an answer to the problem of modeling time series. We are still waiting for a great leap forward in predicting the future.\n\n# Time series Data Analysis Types\nNowadays, there are numerous types of data analysis approaches available for time series which are suitable for different purposes: \n* diagnosing past behavior, \n* prediction and forecasting, \n* curve fitting, \n* interpolation and extrapolation, \n* classification and clustering,\n* segmentation and decomposition, \n* frequency characterization, etc.\n\n\nBeyond this goal, the pervasiveness of time series has generated an increasing demand for performing various tasks on time series data (visualization, discovery of recurrent patterns, correlation discovery, classification, clustering, outlier detection, segmentation,\nforecasting, data simulation).\n\nTemporal visualization is one of the simplest and quickest ways to represent time series data. More importantly, graphical analysis is a crucial step of time series analysis, whatever the downstream data processing line. Notably, it should be the first step before starting with time series modeling.","metadata":{}},{"cell_type":"code","source":"cs_arxivdf[\"category_name\"] = cs_arxivdf.category_name.str[1:-1].str.split(\",\\s\").map(set).apply(list)\ncs_arxivdf[\"category_name\"] = cs_arxivdf[\"category_name\"].apply(lambda x: ', '.join([i for i in x if isinstance(i, str)]))\n\nviz_category = cs_arxivdf.groupby(['category_name']).agg({'title':'count'}).sort_values('title',ascending=False).reset_index()\ncategory_list = list(viz_category['category_name'])\n\ncs_category = cs_arxivdf.groupby(['category_name','year']).agg({'title':'count'}).reset_index()\ncs_category['count_string'] = [human_format(x) for x in cs_category['title']]\ncs_category = cs_category.sort_values('title',ascending=False)#.head(15)\ntemp_cat =cs_category[cs_category['category_name'].isin(category_list[:10])].sort_values('year',ascending=False)\n#category_name\n# Pivot the DataFrame to have subjects as columns and years as rows\npivot_cat = temp_cat.pivot(index='year', columns='category_name', values='title').sort_values('year',ascending=False)\n\n# Create a heatmap\nplt.figure(figsize=(35,1.5))\nsns.heatmap(pivot_cat, cmap='rocket_r', annot=False, cbar=True,cbar_kws={'shrink': 3},vmin=0, linewidth=10)\nplt.title('Major CS paper category for 2021-2023',fontsize=50)\nplt.xticks(horizontalalignment='right',rotation=70,wrap=True,size=16)\nplt.yticks(size=16)\nplt.xlabel('category_name',fontsize=22)\nplt.ylabel('Year',fontsize=32)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:14.339659Z","iopub.execute_input":"2023-07-16T21:37:14.340455Z","iopub.status.idle":"2023-07-16T21:37:16.471543Z","shell.execute_reply.started":"2023-07-16T21:37:14.340411Z","shell.execute_reply":"2023-07-16T21:37:16.470652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data used for this Report\n\nMore than a hundred AI papers are published every day, making it exceedingly hard to keep up with current innovations. To centralize and summarize the rapid advancements in AI from the past two years. Kaggle has provided a collection of writeups from the Kaggle competition and a collection of scholarly articles from arXiv data in the official competition dataset.\n\nIn this Report, I used both the Arxiv scholar paper and Kaggle writeup data both preprocessed, and added a few features for further extension of the Analysis. From the given arXiv dataset there are 2.25M papers for the year range from 2007 to 2023.\n\nThen Narrowed it down to the computer science category which has 217K papers and further narrowed it down to the Time series Analysis paper which has 939 papers in total for the past 2 years\n","metadata":{}},{"cell_type":"code","source":"cspaper_count = cs_arxivdf.groupby('category_name').agg({'title':'count'}).reset_index(names=\"categories\")\ncspaper_count['count_string'] = [human_format(x) for x in cspaper_count['title']]\ncspaper_count = cspaper_count.sort_values('title',ascending=False).head(15)\ncol = ['#e63946']\n\nx = cspaper_count['title'].values\ny = cspaper_count['categories'].values\n\nfig = go.Figure()\n'''\nfig.add_trace(\n    go.Scatter(\n        x=x,\n        y=y,\n        name=\"line\",\n        line=dict(color='black')\n    \n    ))\n'''\nfig.add_trace(\n    go.Bar(\n        x=x,\n        y=y,\n        text=cspaper_count['count_string'],\n        width=0.9,\n        textposition= 'auto',\n        name=\"bar\",\n        orientation='h',\n        marker= dict(color=col)\n    ))\n\nfig.data[0].marker.line.width = 1\nfig.data[0].marker.line.color = \"black\"\nfig.update_layout(yaxis=dict(title='Categories'),\n                  title='Top computer science categeories paper published 2021-2023',template=\"plotly_dark\",\n                  xaxis=dict(title='Number of Papers',tickmode = 'array',tickvals = list(cspaper_count['count_string']),ticktext = list(cspaper_count['count_string'])))\n\nfig.show(renderer=\"notebook\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:16.472916Z","iopub.execute_input":"2023-07-16T21:37:16.473478Z","iopub.status.idle":"2023-07-16T21:37:16.665722Z","shell.execute_reply.started":"2023-07-16T21:37:16.473441Z","shell.execute_reply":"2023-07-16T21:37:16.664487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TSA_arxivdf[\"year\"].replace(year_dict, inplace=True)\nTSApaper_count = TSA_arxivdf.groupby('year').agg({'title':'count'}).reset_index(names=\"year\")\nTSApaper_count.sort_values('title',ascending=False)\ncol = ['#e63946']\n\nx = TSApaper_count['year'].values\ny = TSApaper_count['title'].values\n\nfig = go.Figure()\n\nfig.add_trace(\n    go.Scatter(\n        x=x,\n        y=y,\n        name=\"line\",\n        line=dict(color='black')\n    \n    ))\n\nfig.add_trace(\n    go.Bar(\n        x=x,\n        y=y,\n        text=y,\n        width=0.4,\n        textposition='auto',\n        name=\"bar\",\n        marker= dict(color=col)\n    ))\n\nfig.data[0].marker.line.width = 1\nfig.data[0].marker.line.color = \"black\"\nfig.update_layout(yaxis=dict(title='Number of Papers'),\n                  title='Time Series Papers Published over the year',template=\"plotly_dark\",\n                  xaxis=dict(title='Years',tickmode = 'array',tickvals = list(TSApaper_count['year']),ticktext = list(TSApaper_count['year'])))\n\nfig.show(renderer=\"notebook\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:16.667277Z","iopub.execute_input":"2023-07-16T21:37:16.667631Z","iopub.status.idle":"2023-07-16T21:37:16.767926Z","shell.execute_reply.started":"2023-07-16T21:37:16.667601Z","shell.execute_reply":"2023-07-16T21:37:16.766715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Methods used for Analysis\nArxiv Dataset is filtered for duplicate data, As there are several versions of the same paper available, Hence keeping only the **date of the paper published** extracted from the unique id of the arXiv paper which is then used for Analysis. For Getting the keywords from the Abstract **keyBERT** is used to extract the models and or methods as keywords. For further depth analysis of papers over distinct research areas, disciplines, categories, and subtasks are extracted using arXiv taxonomy with given category for individual papers, this helps to recognize a pattern, and trends and increase the paper count of upcoming research areas like change point detection, dynamic time warping, energy trading..etc. The latest models in the time series are discussed and mentioned in the reference papers for further readings, also state of art models are discussed.\n\nNext in-depth analysis of the time series Kaggle competitions is discussed along with the top 5 places solutions and key takeaway from each competition and the impact of other models over the stable model LGBM which is used in all the competitions are compared and discussed in detail with solution writeups and their references.","metadata":{}},{"cell_type":"markdown","source":"## Subtasks in Time series:\nTime series subtasks match the trend extracted over keywords extracted from the abstract of the papers and it is useful to analyze the research area in Time series, the common subtasks in traditional time series are Time series forecasting, time series analysis, time series classification, Anomaly Detection. Along with the traditional methods other emerging fields also show an increase in a paper published such as energy trading, dynamic time warping, and change point detection.","metadata":{}},{"cell_type":"code","source":"subtask_count = TSA_arxivdf.groupby('TS_subtasks').agg({'title':'count'}).reset_index(names=\"TS_subtasks\")\nsubtask_count['count_string'] = [human_format(x) for x in subtask_count['title']]\nsubtask_count = subtask_count.sort_values('title',ascending=False).head(10)\ncol = ['#e63946']\n\nx = subtask_count['TS_subtasks'].values\ny = subtask_count['title'].values\n\nfig = go.Figure()\n'''\nfig.add_trace(\n    go.Scatter(\n        x=x,\n        y=y,\n        name=\"line\",\n        line=dict(color='black')\n    \n    ))\n'''\nfig.add_trace(\n    go.Bar(\n        x=y,\n        y=x,\n        text=cspaper_count['count_string'],\n        width=0.9,\n        textposition='auto',\n        name=\"bar\",\n        orientation='h',\n        marker= dict(color=col)\n    ))\n\nfig.data[0].marker.line.width = 1\nfig.data[0].marker.line.color = \"black\"\nfig.update_layout(yaxis=dict(title='TSA SUBtasks'),\n                  title='Top Time series subtask with papers published from 2021-2023',template=\"plotly_dark\",\n                  xaxis=dict(title='Number of Papers',tickmode = 'array',tickvals = list(subtask_count['TS_subtasks']),ticktext = list(subtask_count['TS_subtasks'])))\n\nfig.show(renderer=\"notebook\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:16.769545Z","iopub.execute_input":"2023-07-16T21:37:16.769965Z","iopub.status.idle":"2023-07-16T21:37:16.887818Z","shell.execute_reply.started":"2023-07-16T21:37:16.769929Z","shell.execute_reply":"2023-07-16T21:37:16.887080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Common tags Time series papers\nTags provide a concise representation research area associated with the paper, By analyzing the tags in time series papers we get a list of tags that represent the area of interest. The top tags are ML, AI and ML, signal processing, pattern recognition, and computer vision. Computer vision tags are mentioned where these two disciplines  intersect in scenarios such as video-based time series analysis, sensor data with images..etc. It's used where temporal and visual data are combined.","metadata":{}},{"cell_type":"code","source":"TSA_arxivdf[\"category_name\"] = TSA_arxivdf.category_name.str[1:-1].str.split(\",\\s\").map(set).apply(list)\nTSA_arxivdf[\"category_name\"] = TSA_arxivdf[\"category_name\"].apply(lambda x: ', '.join([i for i in x if isinstance(i, str)]))\n\nTSA_cat = TSA_arxivdf.groupby('category_name').agg({'title':'count'}).reset_index(names=\"categories\")\nTSA_cat['count_string'] = [human_format(x) for x in TSA_cat['title']]\nTSA_cat = TSA_cat.sort_values('title',ascending=False).head(15)\ncol = ['#e63946']\n\nx = TSA_cat['categories'].values\ny = TSA_cat['title'].values\n\nfig = go.Figure()\n'''\nfig.add_trace(\n    go.Scatter(\n        x=x,\n        y=y,\n        name=\"line\",\n        line=dict(color='black')\n    \n    ))\n'''\nfig.add_trace(\n    go.Bar(\n        x=y,\n        y=x,\n        text=TSA_cat['count_string'],\n        width=0.9,\n        textposition='auto',\n        name=\"bar\",\n        orientation='h',\n        marker= dict(color=col)\n    ))\n\nfig.data[0].marker.line.width = 1\nfig.data[0].marker.line.color = \"black\"\nfig.update_layout(yaxis=dict(title='TSA Tags'),\n                  title='Top Tags in Time series paper published 2021-2023',template=\"plotly_dark\",\n                  xaxis=dict(title='Number of Papers',tickmode = 'array',tickvals = list(TSA_cat['categories']),ticktext = list(TSA_cat['categories'])))\n\nfig.show(renderer=\"notebook\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:16.888966Z","iopub.execute_input":"2023-07-16T21:37:16.889440Z","iopub.status.idle":"2023-07-16T21:37:16.993876Z","shell.execute_reply.started":"2023-07-16T21:37:16.889410Z","shell.execute_reply":"2023-07-16T21:37:16.992740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Keyword Frequency in Time series\nThe pervasiveness of time series has generated an increasing demand for performing various tasks on time series data (visualization, discovery of recurrent patterns, correlation discovery, classification, clustering, outlier detection, segmentation,\nforecasting, and data simulation).By using BERT for keyword Extraction. It gives the various task in time series.","metadata":{}},{"cell_type":"code","source":"def get_keyword_value_counts(df):\n    \n    keywords_count = pd.Series(df['keywords'].explode()).value_counts().reset_index()\n    keywords_count.columns = ['keyword', 'count']\n\n    return keywords_count\n\nTSA_arxivdf['keywords'] = TSA_arxivdf['keywords'].apply(literal_eval)\nTSA_keyword_count = get_keyword_value_counts(TSA_arxivdf)\n#TSA_keyword_count = TSA_keyword_count.sort_values('count',ascending=False,inplace=True).head(15)\nTSA_keyword_15 =TSA_keyword_count.head(15)\ncol = ['#e63946']\n\nx = TSA_keyword_15['keyword'].values\ny = TSA_keyword_15['count'].values\n\nfig = go.Figure()\n\nfig.add_trace(\n    go.Scatter(\n        x=x,\n        y=y,\n        name=\"line\",\n        line=dict(color='black')\n    \n    ))\n\nfig.add_trace(\n    go.Bar(\n        x=x,\n        y=y,\n        text=y,\n        width=0.5,\n        textposition='auto',\n        name=\"bar\",\n        marker= dict(color=col)\n    ))\n\nfig.data[0].marker.line.width = 1\nfig.data[0].marker.line.color = \"black\"\nfig.update_layout(yaxis=dict(title='Number of Keywords'),\n                  title='Top Keywords in Abstract of Time series paper published',template=\"plotly_dark\",\n                  xaxis=dict(title='Keywords',tickmode = 'array',tickvals = list(TSA_keyword_15['keyword']),ticktext = list(TSA_keyword_15['keyword'])))\n\nfig.show(renderer=\"notebook\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:16.995202Z","iopub.execute_input":"2023-07-16T21:37:16.995585Z","iopub.status.idle":"2023-07-16T21:37:17.121540Z","shell.execute_reply.started":"2023-07-16T21:37:16.995552Z","shell.execute_reply":"2023-07-16T21:37:17.120754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Keyword Frequency:\nIt shows the importance of different keywords in time series papers. We can analyze patterns, trends, and key focus areas in the latest academic papers. Popular key areas are time series classification, dynamic time warping, and anomaly detection. Some less frequent areas are energy trading, networks, energy management, and synthetic time series.","metadata":{}},{"cell_type":"code","source":"d = {}\nfor a, x in TSA_keyword_count.values:\n    d[a] = x\n\n#key_mask = np.array(Image.open('/kaggle/input/notebook-asthetics/key.png'))\nplt.subplots(figsize = (15,10))\n\nwordcloud = WordCloud (\n                    background_color = 'black',#mask=key_mask,\n                    width = 810,\n                    height = 450\n                        ).generate_from_frequencies(frequencies=d)\nplt.imshow(wordcloud,interpolation=\"bilinear\") # image show\nplt.axis('off') # to off the axis of x and y\n#plt.savefig('Plotly-World_Cloud.png')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:17.122563Z","iopub.execute_input":"2023-07-16T21:37:17.123464Z","iopub.status.idle":"2023-07-16T21:37:19.175872Z","shell.execute_reply.started":"2023-07-16T21:37:17.123433Z","shell.execute_reply":"2023-07-16T21:37:19.174681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are the highlights of academic paper for top keyword time series task:\n\n* Forecasting [Li et al., 2019](https://arxiv.org/abs/1907.00235); [Zhou et al., 2022](https://arxiv.org/pdf/2201.12740.pdf), \n* Anomaly detection [Xu et al., 2022](https://openreview.net/pdf?id=LzQQ89U1qm_); [Tuli et al., 2022](https://arxiv.org/abs/2201.07284), \n* Specifically, seasonality or periodicity is an important feature of time series [Wen et al., 2021a](https://arxiv.org/abs/2002.09535). \n* How to effectively model long-range and short-range temporal dependency and capture seasonality simultaneously remains a challenge [Wu et al., 2021](https://openreview.net/forum?id=J4gRj6d5Qm); [Wen et al., 2022](https://www.researchgate.net/publication/362690142_Robust_Time_Series_Analysis_and_Applications_An_Industrial_Perspective). \n* Surveys:\n    1.         Deep learning for time series, including forecasting [Lim and Zohren, 2021](https://royalsocietypublishing.org/doi/full/10.1098/rsta.2020.0209); [Benidis et al., 2022](https://arxiv.org/abs/2004.10240); [Torres et al., 2021](https://pubmed.ncbi.nlm.nih.gov/33275484/), \n    1.         anomaly detection [Choi et al., 2021](https://www.researchgate.net/publication/354155314_Deep_Learning_for_Anomaly_Detection_in_Time-Series_Data_Review_Analysis_and_Guidelines); [Blazquez-Garc ´ ´ıa et al., 2021](https://arxiv.org/abs/2002.04236).\n    ","metadata":{}},{"cell_type":"markdown","source":"\n# Highlights of Transformers in Time Series \n\nTransformers have achieved superior performances in many tasks in natural language processing and\ncomputer vision, which also triggered great interest in the time series community. Among multiple\nadvantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. Many variants of Transformer have been proposed to address special challenges in time series modeling and have been successfully applied to various time series tasks. It is proved to be effective in various time series classification tasks due to its prominent capability in capturing long-term dependency. Transformer for time series is an emerging subject in deep learning.\n\nThe first type of variant for module-level Transformers is to design new attention modules, which is the category with the largest proportion. Here we first describe six typical works:\nLogTrans [Li et al., 2019](https://arxiv.org/abs/1907.00235), Informer[Zhou et al., 2021](https://arxiv.org/abs/2012.07436), AST [Wu et al., 2020a](https://proceedings.neurips.cc/paper/2020/file/c6b8c8d762da15fa8dbbdfb6baf9e260-Paper.pdf), Pyraformer[Liu et al., 2022a](https://paperswithcode.com/paper/pyraformer-low-complexity-pyramidal-attention), Quatformer [Chen et al., 2022](https://www.researchgate.net/publication/362690267_Learning_to_Rotate_Quaternion_Transformer_for_Complicated_Periodical_Time_Series_Forecasting), and FEDformer [Zhou et al.,2022](https://arxiv.org/pdf/2201.12740.pdf)","metadata":{}},{"cell_type":"code","source":"Image(filename=\"/kaggle/input/notebook-asthetics/benchmark images/Transformer_Are Transformers Effective for Time Series Forecasting.JPG\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.177156Z","iopub.execute_input":"2023-07-16T21:37:19.177491Z","iopub.status.idle":"2023-07-16T21:37:19.197431Z","shell.execute_reply.started":"2023-07-16T21:37:19.177454Z","shell.execute_reply":"2023-07-16T21:37:19.196135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Image source:[Are Transformers Effective for Time Series Forecasting?](https://arxiv.org/pdf/2205.13504v3.pdf)\n\nNote that DLinear [Zeng et al., 2022](https://arxiv.org/pdf/2205.13504v3.pdf) questions the necessity of using Transformers for long-term time series forecasting, and shows that a simpler MLP-based model can\nachieve better results compared to some Transformer baselines through empirical studies. However, we notice that a recent Transformer model PatchTST [Nie et al., 2023](https://arxiv.org/abs/2211.14730) achieves a better numerical result compared to DLinear for long-term time series forecasting. Moreover, there is a thorough theoretical study [Yun et al., 2020](https://arxiv.org/abs/1912.10077) showing that the Transformer models are universal approximators of sequence-tosequence functions. \n## Other few notable advancements in transformers for time series:\n\n- GTN [Liu et al., 2021](https://arxiv.org/pdf/2103.14438.pdf) uses a two-tower Transformer with each tower respectively working on time-step-wise attention and channel-wise attention. To merge the feature of the two towers, a learnable weighted concatenation (also known as ‘gating’) is used. The proposed extension of the Transformer achieves state-of-the-art results on 13 multivariate time series classifications.\n\n- TARNet [Chowdhury et al., 2022](https://dl.acm.org/doi/pdf/10.1145/3534678.3539329)designs Transformers to learn task-aware data reconstruction that augments classification performance, which utilizes attention score for important timestamps masking and reconstruction and brings superior performance.\n\n- Pre-trained Transformers are also investigated in classification tasks.[Zerveas et al., 2021](https://arxiv.org/abs/2010.02803) introduced an unsupervised pre-trained framework and the model is pre-trained with proportionally masked data. The pre-trained models are\nthen fine-tuned in downstream tasks such as classification.\n\n- [Yang et al., 2021](https://arxiv.org/pdf/2106.09296.pdf) propose to use a large-scale pre-trained speech processing model for downstream time series classification problems and generates 19 competitive results on 30 popular time series classification datasets.\n\nIt is a overclaim to question the potential of any type of method for time series forecasting based solely on experimental results from some variant instantiations of such method, especially for Transformer models which already demonstrate the performances in most machine learning-based tasks. Therefore,Transformer-based models for time series forecasting is necessary and would benefit the whole community.\n\n","metadata":{}},{"cell_type":"markdown","source":"# LSTM in Time Series\n\nLSTMs are the most widely-used subclass of rnns, as they perform better than rnns in capturing long dependencies. Due to their ability to address gradient explosion and vanishing gradient, LSTMs are actively investigated for time series forecasting purposes. LSTMs have been used for time series prediction in several domains, such as traffic speed [Ma et al., 2015](https://www.sciencedirect.com/science/article/abs/pii/S0968090X15000935), electricity price [Peng et al., 2018](https://www.sciencedirect.com/science/article/abs/pii/S0360544218308727), electric load [Chenget al., 2017](https://www.researchgate.net/publication/312593525_Electric_Load_Forecasting_in_Smart_Grid_Using_Long-Short-Term-Memory_based_Recurrent_Neural_Network), renewable energy power [Gensler et al., 2016](https://www.researchgate.net/publication/313590613_Deep_Learning_for_solar_power_forecasting_-_An_approach_using_AutoEncoder_and_LSTM_Neural_Networks), and financial markets [Selvin et al., 2017;\nFischer and Krauss, 2018)](https://www.researchgate.net/publication/321503983_Stock_price_prediction_using_LSTM_RNN_and_CNN-sliding_window_model).\n\n","metadata":{}},{"cell_type":"code","source":"Image(filename=\"/kaggle/input/notebook-asthetics/LSTM.JPG\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:40:05.641004Z","iopub.execute_input":"2023-07-16T21:40:05.641458Z","iopub.status.idle":"2023-07-16T21:40:05.656553Z","shell.execute_reply.started":"2023-07-16T21:40:05.641426Z","shell.execute_reply":"2023-07-16T21:40:05.655472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Image source :[Machine Learning Algorithms for Time Series\nAnalysis and Forecasting](https://arxiv.org/ftp/arxiv/papers/2211/2211.14387.pdf)\n\nSome works dedicated to times series forecasting implement an optimization procedure to identify the hyperparameters, before LSTM learning. For instance, [Chung and Shin (2018)](https://www.researchgate.net/publication/328373411_Genetic_Algorithm-Optimized_Long_Short-Term_Memory_Network_for_Stock_Market_Prediction) combined LSTM learning with a genetic algorithm in an application dedicated to stock market forecasting. A solution using the differential evolution algorithm is described by [Peng et al. (2018)](https://www.sciencedirect.com/science/article/abs/pii/S0360544218308727), for energy-related time\nseries forecasting.\n\nInterestingly, [Hua et al. (2019)](https://arxiv.org/abs/1810.10161) introduced stochastic connectivity into conventional LSTM modules, a certain level of sparsity is thus obtained in the module architecture, which alleviates the computational\nburden in training.\n\n\n# Deep Learning in Time Series\n Amongst nonlinear models, artificial neural networks hold a place apart as deep learning has recently gained considerable attention. To model time series, parametric models informed by domain expertise have been mainly used, such as autoregressive models and exponential smoothing frameworks. Not only did these established statistical models gain their popularity from their high accuracy. They are also suitable for nonspecialists as they are efficient, robust and user-friendly. \n \n In times series processing, deep neural networks (dnns) hold a distinctive place. Recently, notable achievements have open up an avenue for deep learning. Deep learning is not a restricted learning approach, but it abides various procedures and topographies, to cope with a large spectrum of complex problems [Raghu and Schmidt, 2020](https://arxiv.org/abs/2003.11755). \n \n Deep learning relies on deep artificial neural networks (anns), that is anns with a high number of layers. Deep learning has become an active field of research in the next generation of time series forecasting models. dnns are particularly suitable for finding the appropriate complex nonlinear mathematical function to turn an input into an output. Therefore, deep learning provides a means to learn temporal dynamics in a purely data-driven manner.\n\nThe categories of Deep Learning models are the following:\n*  Multilayer Perceptrons(MLP),\n*  Recurrent Neural Networks(RNN),\n*  Long Short-Term Memory networks(LSTM),\n*  Convolutional Neural Networks(CNN),\n*  Transformers.\n","metadata":{}},{"cell_type":"markdown","source":"\n# History of Time series forecasting competitions:\nForecasting competition has become widespread. The earliest non-trivial study of time series forecast accuracy was probably by David Reid as part of his PhD at the University of Nottingham (1969). Building on his work, Paul Newbold and Clive Granger conducted a study of forecast accuracy involving **106-time series** (JRSSA, 1974)\n\nIn 1979 Spyros Makridakis and Michèle Hibon put together a collection of **111-time series** and compared many more forecasting methods. Then in 1982 Makridakis & Hibon followed up with a new competition involving The 1001 series and The M3 competition involved The 3003-time series in 2000. Sven Crone organized a subsequent competition Neural network competition(the NN3) was organized in 2006 involving 111 of the monthly M3 series. The blog [A brief history of time series forecasting competitions](https://perma.cc/32LJ-RFJW) has an informative and fun competition history that would suggest further reading\n\nKaggle has a rich history in time series competition dating back decades ago some memorable competitions are [Heritage Health Prize](https://www.kaggle.com/c/hhp/) and [Tourism Forecasting Part One and Part Two](https://www.kaggle.com/competitions/tourism1/) and many more. Some community-based competitions have also been held recently like [Predict My Sleep Patterns](https://www.kaggle.com/competitions/kaggle-pog-series-s01e04). Let's analyze Kaggle competition for past two years\n\n","metadata":{}},{"cell_type":"code","source":"kagg_topics = kaggle_comp.groupby(['Competition_topic']).Id_Competitions.nunique().sort_values(ascending=False)\nkagg_topics = pd.DataFrame({'Competition_topic':kagg_topics.index, 'Competition_count':kagg_topics.values})\n\ncol = ['#e63946']\n\nx = kagg_topics['Competition_topic'].values\ny = kagg_topics['Competition_count'].values\n\nfig = go.Figure()\n'''\nfig.add_trace(\n    go.Scatter(\n        x=x,\n        y=y,\n        name=\"line\",\n        line=dict(color='black')\n    \n    ))\n'''\nfig.add_trace(\n    go.Bar(\n        x=y,\n        y=x,\n        text=kagg_topics['Competition_count'],\n        width=0.9,\n        textposition='auto',\n        name=\"bar\",\n        orientation='h',\n        marker= dict(color=col)\n    ))\n\nfig.data[0].marker.line.width = 1\nfig.data[0].marker.line.color = \"white\"\nfig.update_layout(yaxis=dict(title='Competitions Topics'),\n                  title='Kaggle competitions by fields 2021-2023',template=\"plotly_dark\",\n                  xaxis=dict(title='Competitions ',tickmode = 'array',tickvals = list(kagg_topics['Competition_topic']),ticktext = list(kagg_topics['Competition_topic'])))\n\nfig.show(renderer=\"notebook\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.728105Z","iopub.status.idle":"2023-07-16T21:37:19.728663Z","shell.execute_reply.started":"2023-07-16T21:37:19.728442Z","shell.execute_reply":"2023-07-16T21:37:19.728464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Kaggle hosts a vast variety of Competitions in different fields, Computer vision competitions are quite popular and then followed by tabular and text-based competitions. There are other quite interesting fields such as recommendation systems, reinforcement learning, audio, and video, As the audience of Kaggle is a wide variety, these serve them best, apart from this there is also analytic competition, for those who love exploring the data. Fewer competitions are held in AI ethics- AI security, puzzle category but need much attention to this to create awareness and the importance of ethical consideration in AI field.","metadata":{}},{"cell_type":"markdown","source":"# Analysing Recent Kaggle-Time Series competitions\n\n## M5 Forecasting - Accuracy","metadata":{}},{"cell_type":"code","source":"Image(filename=\"/kaggle/input/notebook-asthetics/m5_forcasting_accuracy_competitions_02.png\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.730230Z","iopub.status.idle":"2023-07-16T21:37:19.730866Z","shell.execute_reply.started":"2023-07-16T21:37:19.730652Z","shell.execute_reply":"2023-07-16T21:37:19.730673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The M5 Forecasting Accuracy competition ended with 7,022 participants on 5,558 teams - 415 teams beat the highest statistical benchmark! It had 88,741 submissions from 101 countries! For 1,563 users (including 15 in the top 100!), this was their first competition. Kaggle got 28 new Masters or Experts after both competitions. \n\n***First place*** was owned by Yeonjun In with a single [LGBM model](https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/163684) and used an ensemble of non-recursive and recursive methods. Then select the final model using mean(cv's, public score) and std(cv's, public score). \n\n***Second place*** by Matthias The overall approach consists of two separate [modeling streams](https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/164599) on top and bottom of the grouping hierarchy that is aligned to get a result. The bottom level is based on the LGB models, there are 15 different bottom-level LGB solutions with different loss multipliers.\nThose bottom-level predictions were aggregated and aligned with the N-Beats predictions for levels 1-5.\nSelection of the best fit was done when the mean error between N-Beats and aggregated bottom-level predictions \"flipped\" from positive to negative or vice versa.\n\n***3rd place*** by Jeon trained modified [DeepAR](https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/164374) which consisted of multiple LSTMs with Tweedie loss and make a final prediction from the ensemble of multiple models chosen using the past 14 periods of WRMSSEs.\n\n***4th place*** by Monsaraida used single LightGBM with Tweedie and no recursive features are used. Separate models for each store and week which are composed of about [40 LGBM](https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/163216) models, no blending, no early stopping and stacking are used.\n\n***5th place*** by Alan Lahoud used [LGBM](https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/163916) model with early stopping, and with Poisson objective. Total 7 models are used for each department.\n\n### Key takeaway:\nLGBM model provides promising results for this competition, if you observe closely All use LGBM except 3rd place but the approach towards the problem is different, hence  besides model domain knowledge and understanding of the problem plays a critical role in winning the competition\n\n\n\n## Ubiquant Market Prediction\n","metadata":{}},{"cell_type":"code","source":"Image('/kaggle/input/notebook-asthetics/ubiquant_competition-02.png')","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.732440Z","iopub.status.idle":"2023-07-16T21:37:19.732835Z","shell.execute_reply.started":"2023-07-16T21:37:19.732656Z","shell.execute_reply":"2023-07-16T21:37:19.732673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Ubiquant Market Prediction competition ended with 2,949 participants on 2,435 teams. It had 4,159 submissions. \n\n***First place*** was won by Yuuniee with [LGBM, TABNET](https://www.kaggle.com/competitions/ubiquant-market-prediction/discussion/338220) model and 100 added Features along with 300. LGBM is used because it is a powerful model whose performance has been proven in many competitions. It was also the most stable (especially the consistency of CV and LB) and excellent in the experiment on the competition data. In addition, candidate models to be ensembled in LGBM were found and tested. Among them, TABNET was selected with the best ensemble effect while being relatively stable.\n\n***2nd place*** by Davide Stenner used [5 LightGBM](https://www.kaggle.com/competitions/ubiquant-market-prediction/discussion/338615) with early stopping based on CV Correlation and features added are 5 macro aggregations,100 new columns, and existing 300 features are used. \n\n***3rd place*** by Hydantess used [Transfomer](https://www.kaggle.com/competitions/ubiquant-market-prediction/discussion/338561)-6 layers transformer, max_seq_length=3500 investments along with 300 features along with augmentation and 5 seeds ensemble.\n\n***5th place*** by Ricopue used [Neural Network](https://www.kaggle.com/competitions/ubiquant-market-prediction/discussion/338400)-Simple NN model with four dense layers and no features are added only feature transform is performed\n\n***7th place*** by Wenrui Kong, this is an interesting approach it is purely domain based, The approach has no reverse engineering, no ensemble, and even the supplemental training data provided were not used. The model is a [single LGB model](https://www.kaggle.com/competitions/ubiquant-market-prediction/discussion/338293)with hand-tuned parameters. The model takes in 900+ features, which are selected from an even larger feature pool. As a control, in the second submission, I have a similar model but only used the 300 original features. That one scored 0.112 which is not even in the medal range.\n\n### Key takeaway\nTransformers and neural networks in Time series are promising and we can see the result in winning solutions. Again majority goes to LGBM we can see the difference in 7th place is completely domain-based and the other two solutions also have different approaches towards the competition using LGBM.\n\n## G-Research Crypto Forecasting","metadata":{}},{"cell_type":"code","source":"Image('/kaggle/input/notebook-asthetics/G-research crypto_02.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.734193Z","iopub.status.idle":"2023-07-16T21:37:19.734577Z","shell.execute_reply.started":"2023-07-16T21:37:19.734396Z","shell.execute_reply":"2023-07-16T21:37:19.734413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"G-Research Crypto Forecasting is Working with a massive, real-world dataset and needs to find meaningful signals amongst huge amounts of noise. The Model will be tested against live market data during a three-month evaluation phase. The competition ended with 2,398 participants on 1,946 teams. It had 3,141 submissions. \n\n***The 2nd place*** own by Nathaniel Maddux with [LGBM and GBDT] (https://www.kaggle.com/competitions/g-research-crypto-forecasting/discussion/323098). Feature generation resulted in high performance, there was no ensembling other than the gradient boosting in GBDT. Good N-fold cross-validation (CV) is essential so use 6-fold, walk-forward, grouped cross-validation. It is a competition with a large dataset, and performance was crucial. Our kernels were limited to running in 9 hours or less, using up to 16GB of RAM. As we all can attest, it's easy to bump into those limits.\n\n***3rd place*** by Sugghi used a single model of [LightGBM (7-fold CV)](https://www.kaggle.com/competitions/g-research-crypto-forecasting/discussion/323703)The model is trained for each coin using a common set of features for all the coins. The difference between the change of each currency and the change of all currencies is provided as a feature.\n\n***7th place*** by Patrick Yam used [two MLP and two Transformer encoders](https://www.kaggle.com/competitions/g-research-crypto-forecasting/discussion/323250), The only feature I added is the time of day. The Final submission is an ensemble of 4 models trained with different sequence lengths.\n\n***9th place*** by bturan19 used 3 different [LightGBM models](https://www.kaggle.com/competitions/g-research-crypto-forecasting/discussion/324180) that were trained for different market conditions. Upmarket, Downmarket, and relatively more stable market. Then I get the average of them and used feature engineering to better the result.\n\n### Key takeaway\nIn this competition, LGBM is taking the lead, and key factors are feature engineering, ensemble, and cross-validation, with single-added feature MLP and Transformer can secure top results. Both single and multiple LGBM are used with different market data.\n\n## M5 Forecasting - Uncertainty","metadata":{}},{"cell_type":"code","source":"Image('/kaggle/input/notebook-asthetics/m5_uncertainity_02.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.735711Z","iopub.status.idle":"2023-07-16T21:37:19.736421Z","shell.execute_reply.started":"2023-07-16T21:37:19.736087Z","shell.execute_reply":"2023-07-16T21:37:19.736116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"M5 Forecasting - Uncertainty competition ended with 1,103 participants on 909 teams - 202 teams beat the highest statistical benchmark. We had 10,075 submissions from 65 countries! For 74 users (including 2 in the top 10!), this was their first competition. Kaggle got 28 new Masters or Experts after both competitions.\n\n***2nd place*** by Ouranos used a [lightgbm](https://www.kaggle.com/competitions/m5-forecasting-uncertainty/discussion/166875) model per store (10 stores) and three Keras models. Ensembling was the key to the uncertainty competition.\n\n***3rd place*** by Marisaka Mozz used two [neural network models](https://github.com/marisakamozz/m5/blob/master/ModelSummary.md) with the same structure. One model predicts sales of individual items, and the other model predicts sales aggregated by all items, each category, or each department. It consists of an embedding layer that encodes categorical features, two LSTM layers that process time series data, and 7 fully-connected layers for each day of the week. Both \"Each Model\" and \"Agg Model\" have the same structure except that \"Agg Model\" does not use features of prices. The final output layer is a fully-connected layer for each day of the week. Did not do much feature engineering because I used the model structure shown above. The features I created are whether it is Christmas, whether it is a weekday or a holiday, and the price trend.\n\n***5th place*** by Rafael de Rezende used [ISSM](https://www.kaggle.com/competitions/m5-forecasting-uncertainty/discussion/163559)Innovation State Space Model. It is a model with considerable flexibility in representing commonly occurring time series patterns and underlies the exponential smoothing methods.\nThe idea behind ISSMs is to maintain a latent state vector with recent information about level, trend, and seasonality factors. The observations are then a linear combination of the components of the current state. This solution states that tailored statical models were a good fit for the problem at hand. We had a fairly stable score across the years and periods and above all full control of the solution.\n\n***6th place*** by Ka Ho used [seq2seq LSTM](https://www.kaggle.com/competitions/m5-forecasting-uncertainty/discussion/168053) to produce the mean forecast on the next 28D. One is trying to forecast on the store-item level, and another attempts to forecast on the dept-store level. The final accuracy forecast consists of a simple averaging of both models. The uncertainty estimate is built using the estimated residuals of both models while assuming the forecast residuals are i.i.d. normal. To resolve the instability issue, each seq2seq LSTM model consists of models trained at 20 different checkpoints (i.e. training epochs) and is a simple average of the 20 checkpoint models. Along with feature engineering to Re-scale each time series into [0, 1] using its max. Train the model to forecast the increment of sales value since the first day of the input window (i.e. remove trend).\n\n## Key takeaway:\nThis competition evident that both statistical and machine learning models can solve the problem, we can see that each solution has a different approach to this competition\n","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nThe research area in Time series revealed a consistent upward trend, indicating a gradual increase over time, and models are being developed daily over the benchmark data, which shows advancement in the fields. Apart from time series forecasting, analysis, and stock price prediction, other subtasks like energy management, energy trading, anomaly detection, change point detection, and dynamic time wrapping are also increasingly developed. From the competition perspective, LGBM is considered a useful tool in the community because it is a powerful model whose performance has been proven in many competitions. It was also the most stable (especially the consistency of CV and LB) and excellent in the experiment on the competition data. Neural networks and Transformers also achieve high results in competitions. Apart from using the latest or stable model, other factors also influence the winning such as feature engineering, ensemble, cross-validation, Domain-knowledge, and approach towards the problem, No single model is good for all challenges. we can see the statistical model and domain-knowledge- based solutions have taken the highest ranks in the competitions we discussed early.\n\nGiven the array of gadgets humans carry around with them as well as the timestamps they create as they shop for groceries, log in to a computer portal at work, browse the internet, check a health indicator, make a phone call, or navigate traffic with GPS, we can safely say that an average Human likely produces thousands of time series data points every year of their life which can be very useful in financial markets and e-commerce and many more applications like dynamic wrapping, extracting a pattern from corrupted data. These can be very fascinating and in the last few years we have seen a considerable amount of improvement in predicting atmospheric occurrences but still a long path to go to predict the precise time, zone, and impacts. Moreover, in recent years we can vividly watch the failure of different forecasting models to predict the impact of covid-19 globally because of the sudden shift in underlying dynamics, uncertainty, and lack of data which made it challenging for models to capture the trends. But they still provided valuable insights and guidance for decision-making. The pandemic highlighted the need for continuous model refinement, data improvement, and consideration of multiple modeling approaches. Hopefully, current research altogether will successfully coverage to that horizon in the near future.","metadata":{}},{"cell_type":"markdown","source":"# Reference:\nMost of the references are provided while discussing the paper in the report.\n\nEditorial: Advanced Time Series Analysis in Geosciences-frontiersin.org-https://www.frontiersin.org/articles/10.3389/feart.2021.666148/full\n\nJan De Gooijer and Rob Hyndman, “25 Years of Time Series Forecasting,” International Journal of Forecasting 22, no. 3 (2006): 443–73, https://perma.cc/84RG-58BU.\n\nRob Hyndman, “A Brief History of Time Series Forecasting Competitions,” Hyndsight blog, April 11, 2018, https://perma.cc/32LJ-RFJW.\n\nA brief history of time series analysis- https://www.su.se/english/research/research-subjects/statistics/statistical-models-in-the-social-sciences/a-brief-history-of-time-series-analysis-1.612367\n\nbook- Practical Time Series Analysis by Aileen Nielsen-https://www.oreilly.com/library/view/practical-time-series/9781492041641/ch01.html\n\nAlsharef, A., Aggarwal, K., Sonia et al. Review of ML and AutoML Solutions to Forecast Time-Series Data. Arch Computat Methods Eng 29, 5297–5311 (2022). https://doi.org/10.1007/s11831-022-09765-0\n\nFatoumata Dama, Christine Sinoquet. Time Series Analysis and Modeling to Forecast: a Survey\n-arXiv:2104.00164 [cs.LG]\n\nBlog-A brief history of time series forecasting competitions(https://perma.cc/32LJ-RFJW)\n\nE Siegel, Predictive analytics: The power to predict who will click, buy, lie, or die, Hoboken, New Jersey: John Wiley & Sons, 2013.","metadata":{}},{"cell_type":"markdown","source":"## Submission","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"submission_df = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\nsubmission_df","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.738002Z","iopub.status.idle":"2023-07-16T21:37:19.739072Z","shell.execute_reply.started":"2023-07-16T21:37:19.738792Z","shell.execute_reply":"2023-07-16T21:37:19.738819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.loc[0]['value'] = 'time series data'\nsubmission_df.loc[1]['value'] = 'https://www.kaggle.com/code/kavisekar/latest-development-in-time-series-analysis-report'\nsubmission_df.loc[2]['value'] = 'https://www.kaggle.com/code/saharkhazali/final-ai-report-tabular-time-series/comments#2331792'\nsubmission_df.loc[3]['value'] = 'https://www.kaggle.com/code/pinkythomas/the-state-of-time-series-in-kaggle-competitions/comments#2332982'\nsubmission_df.loc[4]['value'] = 'https://www.kaggle.com/code/subsailor/time-series-forecasting-is-simplicity-the-key/comments#2345722'\nsubmission_df","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.740259Z","iopub.status.idle":"2023-07-16T21:37:19.740994Z","shell.execute_reply.started":"2023-07-16T21:37:19.740708Z","shell.execute_reply":"2023-07-16T21:37:19.740735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv('submission.csv',index=False)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T21:37:19.742834Z","iopub.status.idle":"2023-07-16T21:37:19.743617Z","shell.execute_reply.started":"2023-07-16T21:37:19.743314Z","shell.execute_reply":"2023-07-16T21:37:19.743356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}