{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nr1 = ['essay_category', 'Tabular and/or time series data', ]\nr2 = ['essay_url', 'https://www.kaggle.com/code/vcolliym/submission' ]\nr3 = ['feedback1_url', 'https://www.kaggle.com/code/iamleonie/towards-green-ai/comments#2340744', ]\nr4 = ['feedback2_url', 'https://www.kaggle.com/code/denismunene/the-advancement-of-kaggle-competitions-over-years/comments#2341346', ]\nr5 = ['feedback3_url', 'https://www.kaggle.com/code/dhorvay/kaggle-is-the-global-data-science-classroom/comments#2341424', ]\n\nsubmission = pd.DataFrame( [r1,r2,r3,r4,r5], columns=['type','value'])\nsubmission.to_csv('submission.csv', index=False)\n\nfrom IPython.display import Image\nImage('/kaggle/input/unsplash-plastic-ocean/map_image.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-08-25T19:18:12.955725Z","iopub.execute_input":"2023-08-25T19:18:12.956189Z","iopub.status.idle":"2023-08-25T19:18:13.051692Z","shell.execute_reply.started":"2023-08-25T19:18:12.956153Z","shell.execute_reply":"2023-08-25T19:18:13.050044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Image source: DataBasin.org\n\n> First stop for this maritime plume of radioactive waste is the Pacific Ocean, and from there to all the beaches and oceans of the world to the seaweed, invertebrates, fish, and mammals of the oceans, throughout the marine food chain, and, via evaporation, to global rainfall, [as shown in the above figure from DataBasin].\" \n\nProfessor Paull John writes in his [2023 article](https://www.researchgate.net/profile/John-Paull/publication/372866749_Fukushima_Who_will_drink_Japan's_13_billion_litres_of_radioactive_water/links/64cb9b9f40a524707b94441a/Fukushima-Who-will-drink-Japans-13-billion-litres-of-radioactive-water.pdf). ","metadata":{}},{"cell_type":"markdown","source":"\n## Prelude\n\nAbout one million tonne of contaminated wastewater from Fukushima is in the process of being dumped into the open oceans, amidst concerns raised by scientists [(Nogrady 2023)](#nature) and ongoing public protests both within and outside Japan [(Associated Press, July 2023)](#korea). \n\nEnvironmental advocates protest that the ocean is owned by no single human civilization, but co-shared by all living creatures on this planet. Unfortunately, helpless sea animals cannot speak for themselves [(Xu et al, 2021)](#xu).\n\nOther protestors are concerned about the impact to fishing industries. And many are concerned about health hazards.\n\n\nAt a panel discussion, Professor Richmond, a marine biologist at the University of Hawaii at Manoa and four other scientists were convened to advise on the concerns and reported to *Nature* that there were unanswered questions about tritium and carbon-14, some of the 64 radioactive elements that pose threats to human health. \n\n> The concept of dilution as the solution to pollution has demonstrably been shown to be false. The very chemistry of dilution is undercut by the biology of the ocean.\n\nAs living organisms consume other contaminated organisms, the tritium could concentrate in the food web. \n\n> Has it been shown to public's satisfaction that the dump will be safe for ocean and human health? \n\nProfessor Richmond answers “No” to the above question [(Nogrady 2023)](#nature). German researchers also echoed [(BenDriss et al, 2022)](#crab):\n\n> The lack of transparency in reporting dose, dose rate and duration of exposure, makes it inaccurate to evaluate ionizing radiation efects in aquatic invertebrates\n\nLikewise, when discussing how the [1986 Chernobyl nuclear accident](https://www.forbes.com/sites/davidnikel/2019/06/08/chernobyl-33-years-on-radiation-still-impacts-scandinavian-farmers/?sh=79cdce24949f) still impacts Scandinavian farmers, the [Swedish Radiation Safety Agency](https://www.forbes.com/sites/davidnikel/2019/06/08/chernobyl-33-years-on-radiation-still-impacts-scandinavian-farmers/?sh=79cdce24949f) warns:\n\n> People don't have to worry unless they have a diet heavy on specific mushrooms, reindeer, game and lake fish...\n\n\nBesides water pollution, our Planet Earth has been dealing with few hundred million metric tons of carbon pollution from wars around the globe. It has also been conjectured that global warming [(Peri et al., 2019)](#global_warming), [(Abram et al., 2021)](#australia_fire) renders enormous and intense wildfires inevitable and thus becoming more frequent each year. The list goes on with dozen more examples of warming temperatures leading to Amazon deforestation, Arctic permafrost thaw, and pine beettle outbreaks...\n","metadata":{"_uuid":"f41cf740-27e3-48ba-9d07-6e36c48ee65f","_cell_guid":"0fcae728-5371-4761-860d-1f57b47df4dd","trusted":true}},{"cell_type":"code","source":"from IPython.display import Image\nImage('/kaggle/input/unsplash-plastic-ocean/naja-bertolt-jensen-unsplash.jpeg')","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Image source: freepik","metadata":{}},{"cell_type":"markdown","source":"Slowing the ticking clock for Planet Earth with climate research, machine learning and explainable AI\n\n*Keywords: pollution; climate models; historical climate data; time-series; machine learning; explainable artificial intelligence*\n \n# Introduction\n\nPollution, \"the unwanted waste of human origin released to air, land, water, and the ocean without regard for cost or consequence\", has caused 9 million premature deaths just in 2019 [(Fuller et al., 2022)](#pollution_update). \n\nAs the authors remarked in their *Lancet* update review, exposure to toxic chemicals during prenatal and early postnatal periods is shown to be associated with increased incidence rates of many types of cancer and reproductive diseases later in human life [(Fuller et al., 2022)](#pollution_update). Toxic pollutants (e.g. perfluoroalkyl acids) have also been shown to be associated with reduced antibody responses to vaccines and increased severity of COVID-19 infections.\n\nThese are only a few snapshots of the harmful impacts pollution has done to humans, and the harm done to animals is unthinkable. \n\nAs one step towards pollution control, research in pollution and climate changes are urgently needed to develop sustainable action plans. \n\nPollution and climate change control are intertwined so the two areas are often studied together [(Manisalidis et al. 2020)](#review). \n\nTo understand the human-caused drivers and their negative impact on climate change, Labe et al. [(2021)](#labe) proposed to use neural networks and visualization tools to examine regional affects and found that aerosols emitted in the past two decades have increased the temperature trends in specific parts of the world, predominantly over the North Atlantic Ocean.\n\nThis is especially concerning due to the risk of crossing the **critical climate thresholds**, the times at which large changes in the climate system will be activated, leading to irreversible consequences to our planet and all of its living creatures [(Lenton, T. M., 2021)](#tip). \n\nClimate research is best tackled with time-series data and analyses. Here, in this report, we examine how time-series analysis, **machine learning (ML)**, **explainable artificial intelligence (XAI)**, and latest advances in pollution research may slow down the climate countdown. Through the lens of climate and environmental science, this report aims to:\n- Raise reader's attention on pollution and the importance of climate actions we should take; \n- Review state-of-the-art time-series analysis methods (starting from Section 3).\n\nIn doing so, we highlight how these advances and scientific studies are benefiting the ML communities, climate science, and society as a whole.\n\nNote that:\n- While we focused on time-series analyses (and tabular data) in the context of environmental research, many of the algorithms to be highlighted originated from other disciplines such as computer vision (see [Notebook](https://www.kaggle.com/code/vcolliym/common-themes/) for more discussions on this).\n- To improve readability, we bold domain-specific terms that will be followed by an explanation in a subsequent section, if not immediately.\n\n\n\n \n
\n
Image source: freepik
\n
","metadata":{"_uuid":"777319d2-2ba7-488e-a535-6157465913e9","_cell_guid":"85717944-8806-4bc1-b7af-0e507c30228e","trusted":true}},{"cell_type":"code","source":"\nverbose=False # disable in final notebook submission\n\nimport plotly.graph_objects as go\n\nfrom collections import Counter\nimport string\nimport re\n\nimport matplotlib.pyplot as plt \n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\nimport plotly.express as px\nfrom plotly.offline import init_notebook_mode, iplot\nfrom plotly.offline import init_notebook_mode; init_notebook_mode(connected=True);\n\n\n# this will set limit of columns to 500\npd.options.display.max_columns = 500 \npd.options.display.max_rows = 500\n\nfrom IPython.core.display import HTML\n\nimport polars as pl\nfrom datetime import datetime\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\ndata_path = \"../input/meta-kaggle/\"\n\ntry:\n !mkdir /kaggle/working/figs/\nexcept:\n pass\n\n\ndef proc(A): \n A['Year'] = A.DeadlineDate.dt.year\n A['Month'] = A.DeadlineDate.dt.month\n\n # on or after July 1, 2020 \n A= A.loc[ A.DeadlineDate > '2020-07-01 00:00:00' ,:]\n \n A[\"ProhibitNewEntrantsDeadlineDate\"] = pd.to_datetime(A[\"ProhibitNewEntrantsDeadlineDate\"], format=\"%m/%d/%Y %H:%M:%S\")\n A[\"TeamMergerDeadlineDate\"] = pd.to_datetime(A[\"TeamMergerDeadlineDate\"], format=\"%m/%d/%Y %H:%M:%S\")\n A[\"EnabledDate\"] = pd.to_datetime(A[\"EnabledDate\"], format=\"%m/%d/%Y %H:%M:%S\")\n return A\n\nif ( 'competitions_df' in globals())==False:\n competitions_df = pd.read_csv(data_path + \"Competitions.csv\", parse_dates=['DeadlineDate'] )\n competitions_df = proc(competitions_df) \n contests_df_B = pd.read_csv(data_path + \"Competitions.csv\", parse_dates=['DeadlineDate'] )\n contests_df_B = proc(contests_df_B)\n \nif ('k_forum_msgs_df' in globals())==False:\n forum_topics_df = pl.read_csv(data_path + \"ForumTopics.csv\")\n \n comp_tags_df = pl.read_csv(data_path + \"CompetitionTags.csv\")\n tags_df = pd.read_csv(data_path + \"Tags.csv\", usecols=[\"Id\", \"Name\"])\n\n k_forum_msgs_df = pl.read_csv(data_path + \"ForumMessages.csv\")\n \n forum_topics_df=forum_topics_df.to_pandas()\n comp_tags_df=comp_tags_df.to_pandas()\n \n \ncompetitions_df[\"duration\"] = (competitions_df[\"TeamMergerDeadlineDate\"] - competitions_df[\"EnabledDate\"] ).dt.days\ncontests_df_B[\"duration\"] = (contests_df_B[\"TeamMergerDeadlineDate\"] - contests_df_B[\"EnabledDate\"] ).dt.days\n\nCTYPES=['Playground', 'Community','Research', 'Featured'] \nCTYPES= competitions_df['HostSegmentTitle'].unique()\n\nif verbose:\n print('Inclusion criteria A:', competitions_df.shape, '| Inclusion criteria B:', contests_df_B.shape )\n\n \ndic_num2season = {202:'2020 May-Aug', 203:'2020 Sep-Dec', 211:'2021 Jan-Apr', \n 212:'2021 May-Aug', 213:'2021 Sep-Dec', 221:'2022 Jan-Apr', \n 222:'2022 May-Aug', 223:'2022 Sep-Dec', 231:'2023 Jan-Apr', \n 232:'2023 May-Aug'}\ndic_season2num = {v: k for k, v in dic_num2season.items()}\n\n\ndef group_loss( A):\n A['contest_oracle']=\\\n A.EvaluationAlgorithmName.map( {'Benetech Mixed Data Type Matching Score':'text', \n 'SMAPE':'error', 'DiceFBeta':'dice-prrc', 'imc2023':'other', 'Mean Absolute Error':'error',\n 'sklearn_average_precision_score':'prrc', 'Root Mean Squared Error':'error',\n 'buffered_cmAP':'other', 'FathomNet 2023':'other', 'smape_plus_1':'error',\n 'MeanCosineSimilarity':'acc', 'lux_ai_s2':'other', 'MAP@{K}':'error',\n 'PostProcessorKernelDesc':'acc', 'MeanAngularError':'error',\n 'sklearn_roc_auc_score':'auc', 'Mean Squared Error':'error',\n 'Mean Squared Log Error':'error', 'F-Score (Micro)':'prrc',\n 'FScoreBeta (deprecated)':'prrc', 'Log Loss':'logloss', 'F-Score Beta (Micro)':'prrc',\n 'Matthews correlation coefficient':'cc',\n 'Area Under Receiver Operating Characteristic Curve':'auc',\n 'Probabilistic F-Score Beta (Micro)':'prrc', 'QuadraticWeightedKappa':'cc',\n 'WeightedRecall@{K}':'prrc', \"Santa's Print Shop 2022\":'text', 'SpearmanR':'cc',\n 'Categorization Accuracy':'acc', 'Lux AI 2022':'other', 'DFLEventDetectionAP':'avpr',\n 'Mean Weighted Columnwise Root Mean Squared Error':'error',\n 'SmartphoneDecimeter':'acc', 'MeanPearsonOld':'cc', 'AI4CodeKendallTau':'cc',\n 'Mean Columnwise Log Loss':'logloss-v', 'Weighted Mean Columnwise Log Loss':'logloss-v',\n 'JPXSharpe':'ratio', 'Weighted Multiclass Loss':'wloss', 'Dice':'dice', 'F-Score (Macro)':'prrc',\n 'Nvidia Defcon':'other', 'Amex Custom Gini And X% Percentage Capture':'index',\n 'Multiclass Loss':'loss', 'Adjusted Rand Index':'index', 'kore_fleets':'index',\n 'MeanPearson':'cc', 'Dice3DHausdorff':'dice', 'Jaccard':'dice',\n 'PearsonCorrelationCoefficient':'cc',\n 'Image Matching Challenge pose mAA':'acc', 'MeanBestErrorAtK':'error',\n 'Weighted Categorization Accuracy':'acc',\n 'Weighted Correlation Coefficient':'cc', 'Medical Board F-Beta':'prrc',\n 'IntersectionOverUnionObjectSegmentationBeta':'dice', 'Levenshtein Mean':'text',\n 'TextOverlapFBeta':'text', 'Mean Columnwise Root Mean Squared Error':'error',\n 'Root Mean Squared Logarithmic Error':'error', 'CSIROObjectDetectionFBeta':'prrc',\n 'Jigsaw Agreement with Annotators':'dice',\n 'Mean Columnwise Area Under Receiver Operating Characteristic Curve':'auc-v',\n 'MAPE':'error', \"Santa's Superpermutations 2021\":'text',\n 'Root Mean Square Percentage Error':'error',\n 'IntersectionOverUnionObjectSegmentation':'dice', 'Absolute Error':'error',\n 'Lux AI 2021':'other', 'NDCG@{K}':'text',\n 'R^2 score (coefficient of determination)':'cc',\n 'NFL Helmet Identification':'dice','GoogleGlobalAP':'avgpr',\n 'Mean Columnwise Mean Absolute Error':'error', 'Jane Street Trading':'finance',\n 'Mean Columnwise Root Mean Squared Logarithmic Error':'error',\n 'Hungry Geese':'reinlearn', 'OpenImagesObjectDetectionAP':'avgpr',\n 'Weighted Root Mean Squared Error':'error'} ) \n return A\n\ncompetitions_df = group_loss(competitions_df)\ncontests_df_B = group_loss(contests_df_B) \n\ndef get_yr_indices( A ):\n d=( A.DeadlineDate <= '2020-07-01 00:00:00' )\n d=4*d.values\n a=( A.DeadlineDate > '2020-07-01 00:00:00' ) & ( A.DeadlineDate < '2021-07-01 00:00:00' ) \n a=3*a.values\n b=( A.DeadlineDate >= '2021-07-01 00:00:00' ) & ( A.DeadlineDate < '2022-07-01 00:00:00' )\n b=2*b.values\n c= ( A.DeadlineDate >= '2022-07-01 00:00:00' ) & ( A.DeadlineDate < '2023-07-01 00:00:00' )\n c=c.values + b + a + d\n A['year_index']=c \n A['year_index']=A['year_index'].replace( { 3:'Aug 2020 - Jul 2021', 2:'Aug 2021 - Jul 2022', 1:'Aug 2022 - Jul 2023'})\n \n a=( A.DeadlineDate > '2020-05-01 00:00:00' ) & ( A.DeadlineDate < '2020-09-01 00:00:00' ) \n a=202*a.values\n b=( A.DeadlineDate >= '2020-09-01 00:00:00' ) & ( A.DeadlineDate < '2021-01-01 00:00:00' )\n b=203*b.values\n c= ( A.DeadlineDate >= '2021-01-01 00:00:00' ) & ( A.DeadlineDate < '2021-05-01 00:00:00' )\n c=211*c.values\n d= ( A.DeadlineDate >= '2021-05-01 00:00:00' ) & ( A.DeadlineDate < '2021-09-01 00:00:00' )\n d=212*d.values\n e= ( A.DeadlineDate >= '2021-09-01 00:00:00' ) & ( A.DeadlineDate < '2022-01-01 00:00:00' )\n e=213*e.values\n f= ( A.DeadlineDate >= '2022-01-01 00:00:00' ) & ( A.DeadlineDate < '2022-05-01 00:00:00' )\n f=221*f.values\n g= ( A.DeadlineDate >= '2022-05-01 00:00:00' ) & ( A.DeadlineDate < '2022-09-01 00:00:00' )\n g=222*g.values\n h= ( A.DeadlineDate >= '2022-09-01 00:00:00' ) & ( A.DeadlineDate < '2023-01-01 00:00:00' )\n h=223*h.values\n i= ( A.DeadlineDate >= '2023-01-01 00:00:00' ) & ( A.DeadlineDate < '2023-05-01 00:00:00' )\n i=231*i.values\n j= ( A.DeadlineDate >= '2023-05-01 00:00:00' ) & ( A.DeadlineDate < '2023-09-01 00:00:00' )\n j=232*j.values\n A['season_index2']=a+b+c+d+e+f+g+h+i+j \n A[[\"season_index2\"]]=A[[\"season_index2\"]].replace( dic_num2season ) \n \n a=(A.DeadlineDate > '2021-07-01 00:00:00' ) & (A.DeadlineDate < '2021-09-01 00:00:00' ) \n a=7*a.values\n b=(A.DeadlineDate >= '2021-09-01 00:00:00' ) & (A.DeadlineDate < '2021-11-01 00:00:00' )\n b=9*b.values\n c=(A.DeadlineDate >= '2021-11-01 00:00:00' ) & (A.DeadlineDate < '2022-01-01 00:00:00' )\n c=11*c.values\n d=(A.DeadlineDate >= '2022-01-01 00:00:00' ) & (A.DeadlineDate < '2022-03-01 00:00:00' )\n d=1*d.values\n e=(A.DeadlineDate >= '2022-03-01 00:00:00' ) & (A.DeadlineDate < '2022-05-01 00:00:00' )\n e=3*e.values\n f=(A.DeadlineDate >= '2022-05-01 00:00:00' ) & (A.DeadlineDate < '2022-07-01 00:00:00' )\n f=5*f.values\n g=(A.DeadlineDate >= '2022-07-01 00:00:00' ) & (A.DeadlineDate < '2022-09-01 00:00:00' )\n g=7*g.values\n h=(A.DeadlineDate >= '2022-09-01 00:00:00' ) & (A.DeadlineDate < '2022-11-01 00:00:00' )\n h=9*h.values\n i=(A.DeadlineDate >= '2022-11-01 00:00:00' ) & (A.DeadlineDate < '2023-01-01 00:00:00' )\n i=11*i.values\n j=(A.DeadlineDate >= '2023-01-01 00:00:00' ) & (A.DeadlineDate < '2023-03-01 00:00:00' )\n j=1*j.values\n k=(A.DeadlineDate >= '2023-03-01 00:00:00' ) & (A.DeadlineDate < '2023-05-01 00:00:00' )\n k=3*k.values\n l=(A.DeadlineDate >= '2023-05-01 00:00:00' ) & (A.DeadlineDate < '2023-07-01 00:00:00' )\n l=5*l.values\n A['bimonthly_index']=a+b+c+d+e+f+g+h+i+j+k+l \n return A\n\ncompetitions_df = get_yr_indices( competitions_df )\ncontests_df_B = get_yr_indices( contests_df_B )\n ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-19T05:03:49.212528Z","iopub.execute_input":"2023-07-19T05:03:49.213151Z","iopub.status.idle":"2023-07-19T05:04:07.400231Z","shell.execute_reply.started":"2023-07-19T05:03:49.213118Z","shell.execute_reply":"2023-07-19T05:04:07.398722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Review on time-series analyses\n\n# 1. Back to basics\n\nIn layman's terms, a time-series is simply a series of measurements taken at different points in time. Mathematically, we may use subscripts to index a particular time-instance:\n\n$X = \\{x_{1}, \\dots, x_{t} \\}$\n\nHence, above $X$ is a time-series with $t$ time-instances.\n\nOften, we use time-series to monitor changes over time as well as to make future **forecasts**, which would involve an additional stage of risk-benefit assessment by a decision-maker. \n\nMost of us have probably seen countless visualizations of stock-exchange data. What are specific example environmental time-series that you know of? \n- [x] Sound wave\n- [x] Biosignals collected by health-watch\n- [x] ...\n\nDid any above cross your mind?\n\nA single measurement that is collected at regular intervals is known as **univariate** time-series, while **bivariate** and **multivariate** have two or more variables, respectively. \n\n**Hierarchical** time-series may also be collected to capture a phenomenon at different time scales; e.g. buoy measurements on an hourly, daily, and weekly basis.\n\nBelow is a sample of a time-series that reflected daily ozone indices taken from a dataset called \"40 years of air quality index from EPA daily\" that can be found on Kaggle.","metadata":{"_uuid":"ccf8a442-db9e-4cb3-9970-43115055311e","_cell_guid":"9294b2e8-146b-406c-bb72-be8b042a0692","trusted":true}},{"cell_type":"code","source":"import plotly.express as px\nimport pandas as pd\n\n# Code mirrored from https://www.kaggle.com/code/gatandubuc/forecast-with-n-beats-interpretable-model\n\nozone_data = pd.read_csv('/kaggle/input/40-years-of-air-quality-index-from-the-epa-daily/aqi_daily_1980_to_2021.csv')\nozone_data = pd.pivot_table(ozone_data.query('`Defining Parameter`==\"Ozone\"')[['State Name','AQI', 'County Name', 'Date']], \n index='Date', columns=['State Name', 'County Name'])\n\nozone_data2 = ozone_data.loc[:, ozone_data.notna().sum().sort_values().index[-1] ].dropna()\ndata_to_plot = ozone_data2.to_frame().reset_index()\ndata_to_plot.columns = ['Date', 'AQI']\n\nfig = px.line(data_to_plot, x='Date', y='AQI',\n hover_data={\"Date\": \"|%B %d, %Y\"},\n title='Daily ozone indices from the Air Quality Index dataset (measured in USA from 1980 - May 2021)')\nfig.show()","metadata":{"_uuid":"490cc41a-b093-46d7-bb0b-0174ed000333","_cell_guid":"ee49d00e-4d8e-4b80-bf83-1e112602d6b6","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-19T05:04:07.402783Z","iopub.execute_input":"2023-07-19T05:04:07.403136Z","iopub.status.idle":"2023-07-19T05:04:40.223780Z","shell.execute_reply.started":"2023-07-19T05:04:07.403101Z","shell.execute_reply":"2023-07-19T05:04:40.222641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n## 1.1 Input data ($X$)\n\n**Forecasting** involves making prediction(s) for a future time-point when given some historical data $X$.\n\n$X$ may be a series of measurements, each of which denoted as $X_t$, which may be:\n- **A single measure**, e.g. total usage of electricity at a residence, daily measure of ozone index see example [figure](#ts_ozone)\n- **A vector that may be archived in tabular data format**, e.g. age, heart beat, oxygen saturation measured at time of hospital admission \n- **Structured data**, e.g. three-dimensional ultrasound scan \n- **$n$-dimensional digital signal**, e.g. underwater sound recordings of whales\n\nDepending on the contexts of the forecasting task, one may also lead to **prognostic questions**: \n- An example *industry application* includes fault prediction: how many hours of utility before a lightbulb goes out?\n- In *healthcare*, doctors might need to calculate the chance of infection after performing a surgery on their patients and devise treatment plans accordingly.\n\n\n\n\nImage source: [Meisenbacher et al. 2022](#ensemble)\n\n\n### Characteristics of inputs\n\nOne may describe a time-series with a dozen of terms...\n\n**Lag** which can be identified by calculating the correlation between two subsequences $X_p$, $X_q$ so that $X_p$ is a lag of $X_q$ if their correlation is higher than a given threshold.\n\n**Seasonality** trends with start and end. An example is shown in the third plot of above figure.\n\n**Stationarity** is often confused with cycles, but they are recurring patterns that do not have a clear start nor clear end. Examples are shown in plots (b) and (g) in the figure below.\n\nA type of **noise** that we often hear is white noise, i.e. the \"buzzing\" background sound that has no meaning.\n\nTo stay in scope with our objectives, we use the figure below to elaborate more on the interpretations of these descriptors. The figure is borrowed from an excellent [online textbook on forecasting by Professor Hyndman](https://otexts.com/fpp3/stationarity.html) which we refer interested readers to review for more details.\n\n
","metadata":{"_uuid":"4f96a904-a782-4f06-91f4-90afa385c55b","_cell_guid":"068037a7-3611-4ba8-99b2-06ab2b3447fc","trusted":true}},{"cell_type":"markdown","source":"## 1.2 Output / Outcome being predicted ($X_{t+1}$)\n\nThe prediction ($X_{t+1}$) may be expressed as:\n1. A point-estimate;\n2. A prediction interval (uncertainty of the prediction can be inferred);\n3. A probability distribution.\n\nAn example for each of the above is, respectively:\n- What is the afternoon temperature tomorrow?\n- What is the minimum and maximum values for the predicted temperature?\n- What is the probability of getting a range of temperature values? \n\ne.g.\n| ... | 24°C | 25°C | 26°C | ... |","metadata":{}},{"cell_type":"markdown","source":"## 1.3 Naive forecasting methods \n\nThe simplest strategy to predict weather tomorrow is based on the weather today: high chance of a sunny weather tomorrow if it has not rained today [s](#book). \n\nA natural extension is to use a **moving window**: we predict the morning temperature tomorrow by averaging the temperature over the last $w$ days, where $w$ is known as the length of the **time window**.\n\nAnother naive method is known as the **random walk** method that employs the last observation as the prediction [(Meisenbacher et al. 2022)](#ensemble).\n\n# 1.4 Common time-series analysis tasks\n\n- **Classification**: this is self-explanatory; it answers questions as:\n > Is a sound recording capturing the call of a wolf, or the call of a dog?\n- **Extrinsic regression** [(Foumani et al. 2023)](#extrinsic_reg): the values to be predicted are numeric instead of categorical. \n- **Spatial-temporal forecasting** to predict outcomes of multiple units; e.g. to predict the energy consumption of each node of an electricity grid, or to predict measurements collected at GPS-enabled buoy stations atop ocean surface.\n- **Discovery of motifs** [(Law 2019)](#stumpy): to discover a recurring subsequence that appears to be an interesting patterns. Once we have found a potential motif, we may then examine its recurrence pattern such as when and how often it reappears? Motif search may be tackled through **matrix profiling** (to be discussed later).\n- **Anomaly detection**: to detect a subsequence that captures an interesting and/or rare event (example figure below from temperature data). \n- **Survival analysis** aims to predict the **probability of an event occurring at any given time-point in the future**. Another [notebook of mine](https://www.kaggle.com/vcolliym/survival-analysis-101/) will be devoted to this subtopic.\n\nWhen the input data consists of multiple modalities, we may tackle more variants of the above tasks, as to be illustrated later when we discuss [**transformers**](#multi).\n\n\nImage source: climate.gov","metadata":{"_uuid":"682c56e0-a1a6-4656-88c4-1739b01867fd","_cell_guid":"dbba785d-35a0-449d-bf89-3bef624a3d0f","trusted":true}},{"cell_type":"markdown","source":"# 1.5 Forecasting 101 \n\nA series of open competitions known as the **M-Competitions** started 40 years ago to achieve collaborative benchmarks produced by the research communities for different forecasting tasks [(Makridakis et al. 2020)](#m5). \n- The first two competitions (M1-M2) led to evidence advocating the combination of multiple forecasting methods. \n- The M3-Competition demonstrated that Brown’s Single and Gardner’s Dampen Trend **exponential smoothing** outperformed **Auto-Regressive Integrated Moving Average (ARIMA)**. \n- The M4-Competition showcased that a hybrid approach of combining multiple networks outperformed simpler models\n- The latest competition that just finished early in 2023 (M5) asked participants to predict 42,840 **hierarchical** time-series of 12 levels, and later found out that a winning theme is to combine different forecasting methods, such that two different algorithms may be deployed to forecast at two different levels [(Makridakis et al. 2022)](#m5b). \n\nThe above recounts of developments in time-series forecasting highlight following key methods to be reviewed briefly.\n\nIn a nutshell, ARIMA explains a time-series using its own past values. \nARIMA can be used whenever a time-series:\n1. Does not exhibit **seasonality** \n2. Does not contain purely **white noise**.\n\nTo operate on ultra-long time-series, distributed ARIMA may be considered. However, to account for seasonality, one needs alternative models and may consider **Seasonality-ARIMA (SARIMA)**.\n\nAnother alternative technique is the **exponential smoothing (ES)**, which smooths a subsequence using a window function. Conceptually, by removing the noises captured by historical measurements, a forecast that takes an average of these past measurements may become more accurate predictions of future timepoints.\n\nAs explained by [Barnes et al. (2019)](#barnes), **neural networks** can separate signal from noise without:\n- Assuming stationarity \n- Needing to detrend observations\n- Relying on long control simulations to compute internal variability. \n\nThis makes neural network architectures very a popular alternative when the number of time-series is sufficiently large [Hyndman et al. (2018)](#textbook). This is evident in M4 where the winning solution combined a **residual network** with **attention dilated Long Short Term Memory (LSTM)**, stacked on top of a statistical model [Oreshkin et al. (2019)](#nbeats).\n\nInteresting, most of the methods in M5 used **Light Gradient Boosting Model (LightGBM)** which performs nonlinear regression via gradient boosted trees. The winning solution employed an equal weighted combination of various LightGBMs while the second place winning solution employed N-BEATS, which is a neural network framework developed by [Oreshkin et al. (2019)](#nbeats) (to be reviewed later) in addition to their use of LightGBMs.\n\n\n\n## 2. Getting started\n \n\n**Python** packages that you may consider:\n```\nArrow\nAutoTS\nDarts\nFlint\nGreyKite\nKats (Kits to Analyze TS)\nProphet\nPyflux\nTSFRESH\n```\n\nAnd **R** packages that you may consider:\n```\nforecast\ntsibble\nTimeTk\nZoo\n```\n\n","metadata":{"_uuid":"21bfea18-739d-4ff5-8075-d34bf1d90230","_cell_guid":"1fc81e77-0e5e-4c07-986a-34952fcf5466","trusted":true}},{"cell_type":"markdown","source":"# 3. Advances from last two years\n\n# 3.1 Data augmentation in time-series \n\n\n\nImage source: [Semenoglou et al. 2022)](#data_aug)\n\nScaling, rotating, flipping, and slicing a time-series are popular methods to augment a training set \n[(Iglesias et al. 2023;](#gan)[ Semenoglou et al. 2022)](#data_aug). Examples of these operations are illustrated in the above figure borrowed from [Semenoglou et al. 2022)](#data_aug). Other arguably more expensive approaches include the use of Generative Adversarial Networks, as done in [(Iglesias et al. 2023)](#gan).\n\nVery recently, [Siebert et al. (2023)](#badgers) presents one of the latest data augmentation Python package that tries to simulate quality deficits in time-series; example of deficits are:\n- Outliers\n- Drift\n- Datasets with heavy class-imbalance\n\nThis Python package, available on their [GitHub repository](https://github.com/Fraunhofer-IESE/badgers) also works on tabular data. Below code snippets show an example usage of generating new time-series using their ```GaussianNoiseGenerator``` function.\n","metadata":{}},{"cell_type":"code","source":"try:\n from badgers.core.pipeline import Pipeline \nexcept: \n !pip install badgers ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-19T05:04:40.225208Z","iopub.execute_input":"2023-07-19T05:04:40.225572Z","iopub.status.idle":"2023-07-19T05:04:54.705993Z","shell.execute_reply.started":"2023-07-19T05:04:40.225541Z","shell.execute_reply":"2023-07-19T05:04:54.704303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\n\nimport matplotlib.pyplot as plt\n\n# tabular\nfrom badgers.generators.tabular_data.imbalance import RandomSamplingClassesGenerator\nfrom badgers.generators.tabular_data.noise import GaussianNoiseGenerator\n\n# time-series\nfrom badgers.generators.time_series.noise import GaussianNoiseGenerator\n\nfrom badgers.core.pipeline import Pipeline \n\n# samples x t-dimensions\ninds1=np.arange( 0, 7)\ninds2=np.arange( 114, 121)\nname1=ozone_data.columns[ inds1[0]]\nname2=ozone_data.columns[ inds2[0]]\n\no1=ozone_data.iloc[ :, inds1 ].dropna()\no2=ozone_data.iloc[ :, inds2 ].dropna()\n\nd = np.hstack( (o1.iloc[:180,:], o2.iloc[:180,:] )).T\n\ngen = {'imbalance': RandomSamplingClassesGenerator( proportion_classes={0:0.6, 1:0.4}),\n 'noise': GaussianNoiseGenerator(noise_std=0.5) }\n\npipel = Pipeline(generators=gen)\nX2, y2 = pipel.generate( d, np.array([0]*7+7*[1] ).squeeze() )\nX2 = X2.squeeze()\nif y2[0]==0:\n na='Generated-'+'-'.join(name1)\nelse:\n na='Generated-'+'-'.join(name2)\n \nfig = plt.figure( figsize=(12, 3)) \nplt.plot(o1.iloc[:,0].values, alpha=.2, label='-'.join(name1) );\nplt.plot(o2.iloc[:,0].values, alpha=.2, label='-'.join(name2) ); \nplt.plot(X2[0], label= na );\nplt.legend();\nplt.xlim( [0,180] );\nplt.ylabel( 'Air Quality Index (AQI)')\nplt.xlabel( 'Day $t$')\nplt.title( '180 days of real and generated measurements from the Ozone dataset' );","metadata":{"execution":{"iopub.status.busy":"2023-07-19T05:04:54.709542Z","iopub.execute_input":"2023-07-19T05:04:54.709964Z","iopub.status.idle":"2023-07-19T05:04:55.390399Z","shell.execute_reply.started":"2023-07-19T05:04:54.709927Z","shell.execute_reply":"2023-07-19T05:04:55.389574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 3.2 Faster computations\n\n## Nowcasting via ```ROCKET``` \n\n**Convolutional kernels** capture features by producing large output values when the kernel matches a subsequence of a time-series. Accordingly, training convolutional neural networks (CNNs) entails presenting training samples such that the set of kernel weights will progressively emerge to local patterns of the training samples. \n\nMore specifically, a kernel convolves with an input time-series $X$ via a dot production operation done through a sliding window approach and typically is simply a one-dimensional vector with these parameters:\n- **Length** of kernel ($s$)\n- **Dilation** ($d$) governs the spread of a kernel; weights in a kernel are convolved with every $d$th element in $X$ \n- **Padding** appends zeros to $X$\n- **Bias** can be sampled from a uniform distribution. \n\n\nFor fast and accurate time-series classification, [Dempster et al. (2020)](#rocket) proposed a method named **RandOm Convolutional KErnel Transform (ROCKET)** which employs a random set of kernels that differ by random kernel length, dilation, and padding.\n\nIn a later work, the authors found that a smaller set of special kernels will be sufficient. The secret is using a kernel with a fixed length of $l=9$ that are random configurations of two unique numeric values, e.g. $[1., 0.2, 0.2, 0.2, 1., 0.2, 0.2, 0.2, 1.]$.\n \nFor $k$ kernels, ROCKET generates features of dimensions $2k$ and completes $k\\times n \\times l$ steps where:\n- $k$ is the number of random kernels\n- $n$ is the number of time-series to be transformed\n- $l$ is the length of each of time-series\n\nIn contrast, a Shapelet transform takes $n^2 \\times l^4$ steps. \n\n[Ahmed et al. (2020)](#rocket_weather) applied ROCKET to detect severe weather using lightning data collected from the Geostationary Lightning Mapper and Advanced Baseline Imager’s (ABI) visible image sequences (1500 $\\times$ 2500 pixel in the infrared, near-infrared, and visible spectrum that transmitted data every 5 minutes). \n\nTheir model first predicted the next video frame (i.e. $X_{t+1}$ is an image frame) using a **convolutional recurrent autoencoder**:\n- where the encoder is a 2D CNN that compressed data into lower-dimension 2D frame sequences which were then fed into...\n- recurrent unit built of three chained GRUs \n\nThe predicted image frame is then fed into time-series classification module that would predict event classes such as tornadoes, severe hail, etc. Thanks to ROCKET, their approach is efficient enough to enable *time-constrained now-casting* (predictions made within 15-30 minutes prior to a weather event).\n\n## Fast time-series labeling via ```STUMPY``` \n\n\n**Matrix profiling** is often used to characterize subsequences of a time-series. For a given subsequence denoted as $x_p$, the procedure aims to find its nearest neighbour subsequence $x_q$ in the entire time-sequence it originates from.\n\nThe calculation progress of finding nearest-neighbours can be done naively with two for-loops as the above animation suggests but such naive implementation is not scalable. To alleviate computational bottlenecks, a Python package ```STUMPY``` can be used to compute exactly (or approximate) the nearest-neighbour calculations within $l^2$ steps, where $l$ is the length of the time-series being analyzed [(Law, 2021)](#stumpy).\n\n\n\nIn health care applications [(Reed et al., 2022)](#matrix_prof), matrix profiling has enabled disease modelling. An example application in climate research is illustrated by Saha et al. [(2022)](#saha) who performed time-series labeling via matrix profiling and compared it against approaches based on LSTM and SARIMA.\n\nTheir test and evaluation datasets include:\n- Bitcoin dataset available on Kaggle was used for internal-validation \n- PM2.5 dataset (from Kolkata, India)\n- PM10 dataset (from Kolkata, India)\n- Temperature dataset from Kolkata, India (also available on Kaggle)\n\nBased on experimental findings, they conclude that their approach of using matrix profiling has improved performance by 25%.\n\n\n \n\n
\n\n## Neural Basis Expansion Analysis (N-BEATS)\n\n\n\nDeveloped by [Oreshkin et al. (2020)](#nbeats), N-BEATS is a highly efficient deep learning model that uses two residual stacks of fully connected layers, stacked on top of statistical models. For interested readers, a demo notebook was prepared elegantly by @gatandubuc [in a Kaggle notebook](https://www.kaggle.com/code/gatandubuc/forecast-with-n-beats-interpretable-model) that explores how N-BEATS can be used to examine changes in temperature and daily ozone indices. \n\nAs explained by the developers, N-BEATS is an instance of meta-learning with outer and inner learning procedures. Their method has been deployed to forecast mid-term electricity load in a later work [(Oreshkin et al., 2021)](#eload) where they showed statistically significant gain in forecast accuracy over ARIMA and exponential smoothing (ES). \n\nAnother side note is the benchmarking results they presented. They compared a hybrid approach that combined for instance: \n- $k$-nearest neighbour weighted regression to forecast yearly cycles \n- Fuzzy neighbourhood model to forecast yearly cycles \n- Exponential smoothing that would forecast yearly loads and dispersons\n\n\nBelow figures show their N-BEATS and comparative analysis results.\n
\n\n\n\n\n\n\n\n# 3.3 Data fusion with transformers \n\nAs reviewed in [another Kaggle essay](https://www.kaggle.com/code/kkiryll/generative-ai-rise-of-the-transformers), two essential components behind transformers are:\n\n1. **Self-attention** involves calculating a score that measures how subsequences are similar to each other and attributes a high attention score when two subsequences are likely to be close neighbours. It is an elegant way to provide contextual information that conveys that a subsequence $x_p$ is *more* related to $x_q$ than $x_r$. \n\n2. **Positional encoding** is used so that transformers can recognize how subsequence $B-C-D$ is different from a subsequence $D-B-C$. This is needed because the sequential relationships between subsequences are gone once they passed through self-attention calculations.\n\n[Civitarese et al. (2021)](#transformers) applied a variant of the **temporal fusion transformer model** to predict extreme precipitation that may cause flooding and crop damages. \n\nTheir multi-modal dataset consisted of a set of forest measurements: \n- Non-gridded data: temperature at 2m, geopotential height at 500 mbar, and soil moisture.\n- Daily gridded data with spatial resolution of 0.05°C (known as CHIRPSv2)\n- Gridded data with spatial resolution of 0.25°C (known as ERA5).\n\n... which corresponded to three modalities (types) of inputs that they fuse via three branches:\n- *Historical*: actual observed values; e.g. temperature or soil moisture\n- *Static*, which remains unchanged over time; e.g. month, latitude and longitude\n- *Future*: year-of-the-month and external variables such as temperature.\n\nTheir inputs were first fed into two embedding layers (L1), one for each set of the real and categorical time-series. The outputs of L1 are then fed separately into modality-specific encoders. Outputs of the encoders of the *static* branch are filtered so that time-invariant features will be fed into the next layer. To mine short temporal relationships, a sequence-to-sequence LSTM module is used. Further, a mult-head attention block is used to capture long-term dependencies. \n\n\n\n# 3.4 Approaches to hybridization \n\nAs alluded to earlier when we discussed the M-competitions, hybridization refers to combining different models [(Meisenbacher et al. 2022)](#ensemble), and the variants can be gropued into three approaches:\n- **Ensembling** fed all input data into different foecasting models in the same manner and compute the final forecast as some form of weighted average.\n- **Preprocessing-based** first decompse time-series into components and then process each component separately and finally fed into a single model afer recomposing.\n- **Sequential** first use statistical forecasting method(s), followed by deep learning followed (e.g. exponential smoothing and then recurrent neural network). \n \nStrategies to find the best hybridization include [(Meisenbacher et al. 2022)](#ensemble):\n- Heuristics composed of a fixed set of rules; example:\n - Naive forecasting approaches ([see section 1.3](#naive)) may be used when only small amounts of data is available ($n~600$)\n - Neural networks may be deployed when $n>2600$\n- Empirical experiments such as the use of cross-validation\n- Train a model that selects optimal forecasting method(s) by using, for example, out-of-sample errors derived from a decision tree.\n\nIn their meta-analysis, [Meisenbacher et al. (2022)](#ensemble)] observed that 27 articles employed statistical forecasting methods only while 24 other ones employed only machine learning methods, and 16 articles employed both. \nThey recommend neural architecture search and meta-learning for automating the search of hybrid forecasting methods, albeit large-scale benchmark is pending to evaluate their hypothesis.\n\n\n# 3.5 Explainable AI \n\nIn climate research [(Bommer et al. 2023)](#xai), the use of XAI helps to: \n- Validate a framework \n- Provide new insights about possible physical processes that reflect seasonality in precipitation and sub-seasonal drivers of climate change. \n- Compare multiple climate models\n\nSome flavours of XAI strategies include...\n\n## Gradient-based:\n\n- **Input-Gradient** assigns a high score to a variable if signal is present in a subsequence and the model network responses to the signal. \n\n- **Integrated-Gradients** computes the difference between an explanation with a signal and that without signal.\n\n- **Layer-wise Relevance Propagation (LRP)** feeds the model's prediction backwards to compute relevance scores for each variable via **back-propagation procedure** based on different rules, e.g.:\n - $\\alpha-\\beta$ rule: all positive activations of neurons in the previous layer are weighted\n - **z-rule** calculates weights using both negative and positive activations\n - These calculations are done *backwards*, starting from the output layer, and *propagate* these calculations in a *backward*, layer by layer fashion\n\n## Noise-based\n\n- **Smooth-Grad** adds multiple versions of random noise to the inputs to produce a set of model explanations that are then averaged. Noise-induced flunctuations will suppress background noise in the time-series and highlight signals in the inputs.\n\n- **Noise-Grad** perturbs model weights instead of the inputs to estimate the uncertainties induced by both the inputs and the model.\n\n- **Fusion-Grad** combines NoiseGrad and SmoothGrad.","metadata":{"_uuid":"bc9f1a06-b389-4d69-8154-030fff8b608e","_cell_guid":"68d26e7d-b8e8-4449-8e48-710924a61277","_kg_hide-output":false,"_kg_hide-input":false,"trusted":true}},{"cell_type":"markdown","source":"## Latest XAI applications in climate research\n\n\n### Attention heatmap via Layer-wise Relevance Propagation (LRP)\n\n[Borujeni et al.](#xai_pollution) extended the *Layer-wise Relevance Propagation (LRP)* technique to be deployed with a **gated recurrent unit (GRU)** network, in order to obtain heatmaps that show response levels of four air pollutants, namely, PM10, NO$_2$, NO, O$_3$. Their framework helped them identify critical temporal and meteorological factors responsible for these air pollutants. \n\n\n### Attention heatmap via gradient-based methods\n\n**Gradient-based method** approximates the local gradient of $X_{t+1}$ (the output prediction) relative to $X$ (inputs). \n\nWe illustrate one application in the context of the article by [Diffenbaugh et al. (2023)](#simulated_data) who employed spatial patterns of temperature observations to predict time-to-critical-thresholds of the tipping point (see Introduction). \n\nUsing observed patterns of annual temperature anomalies as inputs to a neural network, they sought to predict time-to-climate-thresholds for different parts of the world. \n\nTheir network has two hidden layers, each of 25 units, and outputs two values representing the mean and standard deviation of a conditional Gaussian distribution. Examination of the validation and test predictions as shown in the Appendix of their article affirms the validity of the uncertainty estimates.\n\nTheir approach is highly innovative and distinctive from prior studies in the sense that their neural networks were only trained on simulated data generated from climate models. Based on empirical results, they observed that the time-to-threshold estimations are consistent with previous studies.\n\nFurthermore, using \"gradient multiplied by input\" [(Shrikumar et al., 2017)](#heatmap) to identify geographic regions, their model projected that many regions will likely experience shorter time-to-thresholds and helped confirm that global warming is on the verge of crossing the critical threshold and that unfortunately means a high chance of high-impact climate changes happening \"over the next three decades\".\n\n\n### eXtreme Gradient Boosting (XGBoost) + SHapley Additive exPlanations (SHAP) approach \n\nTo examine the effects of cooling energy consumptions $E$ on climate change, [Chakraborty et al. (2021)](#cooling) proposed to predict $E$ using XGBoost, an ensemble method that has high model interpretability and scalability to large-scale datasets. In a nutshell, the objective of XGB is to incrementally enlarge an ensemble of trees. Similar to decision trees, each observation is passed down the tree where each leaf represents an output prediction with confidence denoted by $w$. The final prediction is calculated as a sum of the predictions made at all leaf nodes. \n\nThe authors conducted ablation studies that include comparison with other popular machine learning algorithms, e.g.:\n- Support vector machine \n- Random forest\n- LSTM\n\nFurther, they explained their predictions using SHAP, which works by examining how each variable impacts the accuracy of prediction. Their overall framework can predict climate scenarios with $R^2>0.9$.\n\n\n","metadata":{"_uuid":"90e056b6-ee8d-4d30-b57c-3272ab60bb4c","_cell_guid":"ce755063-c31b-4c01-bc88-64d5b2881b3d","trusted":true}},{"cell_type":"markdown","source":"# Discussion \n\n# - Recap \n\nWe have reviewed the basics of time-series data and their applications that include fusing them with tabular data and heterogenous data types such as the use of: \n- Multi-modal transformers operating on **hierarchical** forest measurements to predict precipitation\n- N-BEATS for probablistic forecasting of electricity loads \n- Matrix-profiling with STUMPY for efficient time-series labeling\n- ROCKET for nowcasting severe weather using image sequences.\n\nWe also examined latest climate and pollution research illustrating XAI applications in the contexts of:\n- XGBoost + SHAP for predicting energy consumptions by buildings\n- GRU + LRP for understanding air pollutants\n- MLP + Input-Gradients to generate regional maps that project time-to-critical climate threshods.\n\n\n\nCross-examining the presented techniques with discussion posts and solution write-ups found on Kaggle using the [Meta Kaggle dataset](#meta), we gauged their relative popularities with interactive plots shown in my [second notebook](https://www.kaggle.com/code/vcolliym/common-themes/). Inspecting the competition titles, we observe that XAI and data augmentation techniques were discussed for time-series data in only a handful of posts.\n\nOn a broader level, ML and climate researchers continue to explore new applications of XAI as well as developing rigourous procedures that ensure the \"add-ons\" needed by XAI will exhibit desirable properties, such as those done by [Diffenbaugh et al. (2023)](#simulated_data) and [Borujeni et al. (2023)](#xai_pollution), who, for instance, validate the correctness of heatmap calculations by conducting experiments to measure the consistency of the heatmaps computed under various settings. \n\nIn taking this further, [Bommer et al. (2023)](#xai) recently conducted a benchmark that examined desirable properties of XAI methods. Using data simulated by the [general climate model (CESM1)](https://www.cesm.ucar.edu/community-projects/lens/data-sets) and neural networks to map yearly temperature maps to categorical maps, they benchmarhed various algorithms and derived metrics that measure desirable properties such as: \n- **Complexity** that quantifies how concise the explanations are.\n- **Robustness**, to ensure that small changes in the inputs will give explanations that remain stable\n- **Faithfulness**, to ensure that \"noise\"-corrupting highly relevant variables will result in high reduction in forecast performance \n\nIn their experiments based on MLP and CNN, they found that:\n- XAI methods that involve the inputs (e.g. Input-Gradients and specific variants of LRP) achieved favourable robustness, faithfulness, and complexity but at the cost of being susceptible to randomization effects induced by decision boundaries and model parameters.\n- This was true regardless of type of model.\n- The converse is true for gradient-based methods (e.g. Smooth-Grad, Noise-Grad, Fusion-Grad). \n- Perturbation of inputs by noise did not improve robustness, faithfulness, and complexity of a given XAI method. \n\nFuture research using real datasets is needed to test on more machine learning frameworks.\n\nWe next showcase how experiences working with Kaggle datasets and competitions may push forward the attainment of sustainable development goals in the future to come.\n\n# - How did research communities benefit from each other? \n\n## Towards understanding interplay between climate, agriculture, and urban growth \n\nThe National Oceanic and Atmospheric Administration (NOAA) is a government agency based in Washington, D.C. that has hosted various data science competitions on Kaggle with long-term results (see next section). NOAA also posted a dataset called *Global Surface Summary of the Day* (GSSD), which consists of daily temperature data collected over 9000 weather stations around the globe. \n- The validity of the GSSD dataset has been examined in a recent article where Lavigne & Li [(2022)](#noaa) explained the importance of data scrutiny and cleansing. \n- After excluding stations with abnormal changes in the fog day reports, the authors observed that warming conditions around the world impact annual fog day over different regions in non-consistent ways. \n- In summary, this study on a high level has enriched the understanding of the impact of fog on crop yields, vegetation as well as management of water utilization and regional transportation safety. \n\nThe United Nations also has posted several datasets, including the \"Greenhouse Gas Inventory Dataset\" that contains attributes on anthropogenic emissions by sources, such as carbon dioxide, methane, etc. The dataset has used by Ravi et al. [(2020)](#gg_gas).\n\n\n## Towards monitoring and improving quality of drinking water\n\nThe Indian water Quality (IWQ) dataset consists of six measurements taken from 1991 water samples surveyed from rivers in various states of India during 2003-2014. Using this dataset, [Hoque et al. (2022)](#iwq) benchmarked six regression algorithms to regress water quality index from the six measurements in the IWQ dataset and found that linear regression and ridge regression outperformed more advanced algorithms such as extra tree regression, random forest, and support vector regression. \n\nAnother research group led by [Yusri et al.](#iwq2) examined binary classification of water quality instead of regression on quality indices. Using a five-fold cross-validation experiment, they found that eXtreme Gradient Boosting achieved mean accuracy score of 90% while support vector machine achieved 65% in mean accuracy.\n\n## Promoting sustainable development goals & driving analytical innovations through Kaggle competitions\n\nKaggle also stimulates analytical innovations that advance the development of a sustainable future.\n\nIn the [Environmental Insights Explorer competiton](https://www.kaggle.com/competitions/ds4g-environmental-insights-explorer/overview), participants were asked to derive methods that can compute the average emissions factor of electricity generated from a region using remote sensing data.\n- The [winning solution](https://www.kaggle.com/code/katemelianova/ds4g-spatial-panel-data-modeling/notebook) examined sources of NO2 in the atmosphere with statistical procedures that account for confounding factors.\n- [Another winning solution ](https://www.kaggle.com/niyamatalmass/1-introduction-to-the-solution-of-dsg-eie) focused on NO2 emission from a single production plant.\n\n\nIn the [Unlocking Climate Solutions](https://www.kaggle.com/competitions/cdp-unlocking-climate-solutions/overview), participants were asked to propose key performance indicators that would examine a city in terms of water security, deforestation, climate change, and social inequality. The challenge's textual dataset involved disclosures on climate change actions published by corporates and government bodies. \n\nTo illustrate, the challenge's demo notebook suggested users several directions: \n- identify key themes in the disclosures using natural language processing (via term frequency-inverse document frequency); \n- examine how government bodies perceive individual environmental issues as a concern or an opportunity using sentiment analysis and create metrics that would have identify cities who will be proactive in adopting environmental measures; \n- and water price coefficient that would quantify a corporate's social impact and environmental risks brought to its city. \n\nThe top three winning submissions included notebook visualizations that: \n1. Classified cities by novel indices to inform users how climate change will impact water potability; \n2. Identify mutually beneficial new partnerships between cities/ corporations (via graph theory and structural topic modeling); and \n3. Rank countries according to their current performance and future potentials by examining water efficiency in terms of geospatial contexts.\n\n\n","metadata":{"_uuid":"8d854d3b-cdf5-47e2-b143-e3e92f5a050a","_cell_guid":"abe0a9c1-d122-4aad-a544-e9b43d0dffef","trusted":true}},{"cell_type":"markdown","source":"# Conclusions\n\nThis report started with recent examples of massive degradations to the world’s ecosystems, from both natural or manmade disasters, all of which demand our immediate attention and global efforts to take actions. \n\nFor most ordinary people including myself, the writer, I have these suggestions to kick-start climate actions:\n1. Bike or walk to work\n2. Bring our own mugs to workshops and conferences, not just office meetings \n2. Dine-in instead of take-outs, or better, prep your own dinner \n3. Consider alternatives to our bubble-tea and St*rbucks ice coffee straws and plastic cups?\n4. Ask if a grocery receipt must be printed?\n5. ...\n\nWe could obviously compose a collective Kaggle listicle with this.\n \nThe important point is that these are small favours we *can* do right now for our Planet Earth. \n\nWhile they are baby-elephant steps, we must remember the collective power we bring.\n\nAnd if everyone adopts sustainable practices, we have greater hopes of bringing these practices to our workplaces (which, for instance, will facilitate more water-cooler conversations after biking to work) and spreading the message to all our social and family networks. We will collectively feel more motivated to push forward sustainable actions. \n\n\nAnd last week (July 15, 2023), [Markus Pukonen](https://beachmetro.com/2023/07/15/beacher-markus-pukonen-completes-motor-free-circumnavigation-of-the-globe-eight-years-after-paddling-out-into-lake-ontario/) finished his **\"non-motorized circumnavigation of the planet\", without putting a step on any airplane, jet, train, automobile... Not even a single elevator!** \n\n\n> ...shrinking my ecological footprint and learning more about social and environmental justice issues. I started to feel like that wasn’t enough and that I needed to do more to help.\n\nWithout the help of air-balloons, his travel could not be finished in 80 days, but it was still possible thanks to the countless bikes, canoes, paddleboards, ... and his **perseverance**.\n\n> What I am doing is inspiring the public and students at presentations and connecting with local non-profit organizations to share their stories and raise support for them.\n \n\nHope this true story will bring you inspirations today. Cheers!\n\n \n\n\n\nMarkus Pukonen on a stand-up paddleboard (Source: [BeachMetro](https://beachmetro.com/2023/07/15/beacher-markus-pukonen-completes-motor-free-circumnavigation-of-the-globe-eight-years-after-paddling-out-into-lake-ontario/))","metadata":{"_uuid":"658c821a-437e-4825-b18c-5b55a22af6b2","_cell_guid":"ff509b7b-8424-4f8d-b2ee-fbd48d756323","trusted":true}},{"cell_type":"markdown","source":"**References**\n\n1. The Associated Press (July 9, 2023) Backlash builds as Japan prepares to release wastewater from Fukushima nuclear plant https://www.npr.org/2023/07/09/1186677021/japan-fukushima-nuclear-plant-wastewater-release\n\n\n1. Ahmed, N., Slipski, M. M., Venzor-Cárdenas, I., Molina, M. J., Sénay, G., Cheung, M., ... & Renard, G. (2020). Leveraging Lightning with Convolutional Recurrent AutoEncoder and ROCKET for Severe Weather Detection. In AI for Earth Sciences Workshop at NeurIPS.\n\n\n1. Athanasopoulos, G., Hyndman, R. J., Song, H., & Wu, D. C. (2011). The tourism forecasting competition. International Journal of Forecasting, 27(3), 822-844.\n\n\n1. Barnes, E. A., Hurrell, J. W., Ebert‐Uphoff, I., Anderson, C., & Anderson, D. (2019). Viewing forced climate patterns through an AI lens. Geophysical Research Letters, 46(22), 13389-13398. \n\n\n1. BenDriss, H., Chakir, E. M., El Bakkali, J., & Doudouh, A. (2022). InterDosi Monte Carlo study of radiation exposure of a reference crab phantom due to radioactive wastewater deposited in marine environment following the Fukushima nuclear accident. Radiation and Environmental Biophysics, 61(4), 623-629.\n\n\n1. Bommer, P., Kretschmer, M., Hedström, A., Bareeva, D., & Höhne, M. M. C. (2023). Finding the right XAI method--A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science. arXiv preprint arXiv:2303.00652.\n\n\n1. Bui, D. T., Khosravi, K., Tiefenbacher, J., Nguyen, H., & Kazakis, N. (2020). Improving prediction of water quality indices using novel hybrid machine-learning algorithms. Science of the Total Environment, 721, 137612.\n\n\n1. Civitarese, D. S., Szwarcman, D., Zadrozny, B., & Watson, C. (2021). Extreme precipitation seasonal forecast using a transformer neural network. arXiv preprint arXiv:2107.06846.\n\n\n1. Dempster, A., Schmidt, D. F., & Webb, G. I. (2023). Hydra: Competing convolutional kernels for fast and accurate time series classification. Data Mining and Knowledge Discovery, 1-27.\n \n\n1. Diffenbaugh, Noah S., and Elizabeth A. Barnes. \"Data-driven predictions of the time remaining until critical global warming thresholds are reached.\" _Proceedings of the National Academy of Sciences_ 120, no. 6 (2023): e2207183120.\n\n\n1. Eisenmann M, Reinke A, Weru V, Tizabi MD, Isensee F, Adler TJ, Ali S, Andrearczyk V, Aubreville M, Baid U, Bakas S. Why is the winner the best?. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 19955-19966). \n\n \n1. Foumani, N. M., Miller, L., Tan, C. W., Webb, G. I., Forestier, G., & Salehi, M. (2023). Deep learning for time series classification and extrinsic regression: A current survey. arXiv preprint arXiv:2302.02515.\n\n\n1. Fuller, R., Landrigan, P. J., Balakrishnan, K., Bathan, G., Bose-O'Reilly, S., Brauer, M., ... & Yan, C. (2022). Pollution and health: a progress update. The Lancet Planetary Health, 6(6), e535-e547.\n\n\n1. Hayashi T, Shimizu T, Fukami Y. Collaborative Problem Solving on a Data Platform Kaggle. arXiv preprint arXiv:2107.11929. 2021 Jul 26. \n \n\n1. Hyndman, R. J., & Athanasopoulos, G. (2018). Forecasting: principles and practice. OTexts.\n\n\n1. Iglesias, G., Talavera, E., González-Prieto, Á., Mozo, A., & Gómez-Canaval, S. (2023). Data Augmentation techniques in time series domain: a survey and taxonomy. Neural Computing and Applications, 35(14), 10123-10145.\n\n\n4. Labe, Z. M., & Barnes, E. A. (2022). Comparison of climate model large ensembles with observations in the arctic using simple neural networks. Earth and Space Science, 9(7), e2022EA002348.\n\n\n3. Law, S. (2019). STUMPY: A Powerful and Scalable Python Library for Time Series Data Mining. Journal of Open Source Software, 4(39), 1504.\n\n\n1. Lenton, T. M. (2021). Tipping points in the climate system. *Weather*, 76 (10)\n\n\n1. Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1), 54-74.\n\n\n1. Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). The M5 competition: Background, organization, and implementation. International Journal of Forecasting,38(4), 1325-1336.\n\n\n2. Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4), 1346-1364.\n \n\n1. Manisalidis, I., Stavropoulou, E., Stavropoulos, A., & Bezirtzoglou, E. (2020). Environmental and health impacts of air pollution: a review. Frontiers in public health, 8, 14.\n\n\n1. Meisenbacher, S., Turowski, M., Phipps, K., Rätz, M., Müller, D., Hagenmeyer, V., & Mikut, R. (2022). Review of automated time series forecasting pipelines. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(6), e1475.\n\n\n1. Meta Kaggle. Accessed July 5, 2023 https://www.kaggle.com/datasets/kaggle/meta-kaggle \n\n\n1. Mirzavand Borujeni, S., Arras, L., Srinivasan, V., & Samek, W. (2023). Explainable sequence-to-sequence GRU neural network for pollution forecasting.*Scientific Reports*,13_(1), 9940. \n\n\n1. Nogrady, B. (2023). Is Fukushima wastewater release safe? What the science says. Nature. https://www.nature.com/articles/d41586-023-02057-y\n\n\n1. Oreshkin, B. N., Carpov, D., Chapados, N., & Bengio, Y. (2019). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437. \n\n\n1. Oreshkin, B. N., Dudek, G., Pełka, P., & Turkina, E. (2021). N-BEATS neural network for mid-term electricity load forecasting. Applied Energy, 293, 116918.\n\n\n1. Reed, E., Fellows, A., Lu, R., Rienks, M., Schmidt, L., Yin, X., & Mayr, M. (2022). Extracellular matrix profiling and disease modelling in engineered vascular smooth muscle cell tissues. *Matrix Biology Plus*, 16, 100122.\n\n\n1. Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., ... & Bengio, Y. (2022). Tackling climate change with machine learning. ACM Computing Surveys (CSUR), 55(2), 1-96\n\n\n1. Saha, P., Nath, P., Middya, A. I., & Roy, S. (2022). Improving temporal predictions through time-series labeling using matrix profile and motifs. Neural Computing and Applications, 1-17.\n\n\n1. Semenoglou, A. A., Spiliotis, E., & Assimakopoulos, V. (2023). Data augmentation for univariate time series forecasting with neural networks. Pattern Recognition, 134, 109132.\n\n\n1. Shrikumar, A., Greenside, P., & Kundaje, A. (2017, July). Learning important features through propagating activation differences. In International conference on machine learning (pp. 3145-3153). PMLR. \n\n\n1. Siebert, J., Seifert, D., Kelbert, P., Kläs, M., & Trendowicz, A. (2023). Badgers: generating data quality deficits with Python. arXiv preprint arXiv:2307.04468.\n\n\n1. Sivill, T., & Flach, P. (2022, May). Limesegment: Meaningful, realistic time series explanations. In International Conference on Artificial Intelligence and Statistics (pp. 3418-3433). PMLR \n\n\n1. Tauchert C, Buxmann P, Lambinus J. Crowdsourcing data science: A qualitative analysis of organizations’ usage of kaggle competitions. \n \n\n1. Xu, L., Zhao, X. Y., & Chen, J. H. (2022). Exploring the governance dilemma of nuclear wastewater in Fukushima: A tripartite evolutionary game model. Ocean & Coastal Management, 225, 106220.\n\n","metadata":{"_uuid":"2edce34f-f331-4d36-84bb-1cc64b924971","_cell_guid":"c674fc40-2412-40ab-b22f-466af6b5da06","trusted":true}},{"cell_type":"markdown","source":"# Appendix","metadata":{"_uuid":"ff574b2f-e657-4c2e-a341-4f7d12699c4f","_cell_guid":"e5901679-1f5d-48a6-9405-4bdbe637994b","trusted":true}},{"cell_type":"markdown","source":"# - Trendy themes that were not discussed in discussion posts in Meta Kaggle","metadata":{"_uuid":"24ccd2bd-52c0-4447-bb6c-d69c951fe3fa","_cell_guid":"80856146-ee96-47c0-9e45-affb26e361a9","trusted":true}},{"cell_type":"code","source":"'''\nArrow\nAutoTS\nDarts\nFlint\nGreyKite\nKats (Kits to Analyze TS)\nProphet\nPyflux\nTSFRESH\n\nR packages that you may consider:\n\n'forecast'\n'tsibble'\n'TimeTk'\n'Zoo'\n''';\n\nif 0:\n\n phrases = [ ['stumpy'], ['AutoTS'], ['Prophet'], ['TSFRESH'], ['GreyKite'], ['TimeTk'], ['tsibble'] ] \n tit = \"Python/ R packages\" \n summaries = [tit]*len(phrases) \n tally_writeup_multi_phrases( phrases, summaries, tit, 1 );\n\n tit='Statistical tests' \n phrases = [ ['confidence interval','parametric fit', 'non-parametric', 'Mann Whitney U-test', 'statistical ana']]\n summaries = [tit]*len(phrases) \n tally_writeup_multi_phrases( phrases, summaries, tit, 1 );\n\n tit='Survival analysis' \n phrases = [ ['survival times','survival analysis'], [' Kaplan(–| )?Meier','kaplan' ]]\n summaries = [tit]*len(phrases) \n tally_writeup_multi_phrases( phrases, summaries, tit, 1 );\n\n","metadata":{"execution":{"iopub.status.busy":"2023-07-19T05:04:55.391719Z","iopub.execute_input":"2023-07-19T05:04:55.392710Z","iopub.status.idle":"2023-07-19T05:04:55.400990Z","shell.execute_reply.started":"2023-07-19T05:04:55.392676Z","shell.execute_reply":"2023-07-19T05:04:55.400195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"046a3663-beb8-47eb-a779-f10ebf121cd3","_cell_guid":"f3affc04-2aac-4369-b0fc-228895f8a654","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# - Research Notes\n\n## HECAP & Sustainable Development Goals (SDG) \n\nHECAP+ stands for the **High Energy Physics**, **Cosmology**, **Astroparticle Physics**, and **Hadron and Nuclear Physics**. In their 2023 report, these communities of experts presented their views on bringing practical solutions through multipliciplinary research, teaching, and evidence-based advice on policy changes. They also outline ways to support the 16 Sustainable Development Goals mandated by the United Nations. Omitting goals that are outside the scope of environmental mandates, one would see these priority tasks:\n\n- Sustainalby use the oceans to conserve life below water\n- Responsible consumption and production\n- Clean water and sanitation\n- Affordable and clean energy \n- Climate action\n\n\n## Kaggle-specific\n\n\n### Unexpected finding from the Water Potability dataset [aside]\n\nOne controversial observation is the world-wide misuses of the Water Potability dataset that was shared to the public on Kaggle. Many Kaggle users have [questioned](https://www.kaggle.com/datasets/adityakadiwal/water-potability/discussion) the authenticity of this dataset. Eventually, in a response to a discussion thread, one [Kaggle user directed the community to the meta description](https://www.kaggle.com/datasets/adityakadiwal/water-potability/discussion/248220#1390648), quoting that this dataset was in fact \"generated synthetically'' (under Provenance of the Data Card).\n\nUnfortunately, at least five independent research groups from [Ireland/ USA](https://www.mdpi.com/2073-4441/14/21/3359), [the United Arab Emirates](https://www.mdpi.com/2071-1050/14/18/11478), [Turkey](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4423215), [Peru](https://ieeexplore.ieee.org/document/9574673) and [Algeria](https://www.researchgate.net/profile/Samy-Ali-Touati/publication/368302550_CLASSIFYING_WATER_INFESTATION_A_DEEP_CONVOLUTIONAL_NEURAL_NETWORK_APPROACH_FOR_BACTERIAL_IDENTIFICATION_AND_DENSITY_CALCULATION/links/63e149132f0d126cd18d531d/CLASSIFYING-WATER-INFESTATION-A-DEEP-CONVOLUTIONAL-NEURAL-NETWORK-APPROACH-FOR-BACTERIAL-IDENTIFICATION-AND-DENSITY-CALCULATION.pdf) have treated this dataset as real, archiving their \"results\" on pre-print servers, or, publishing them as conference proceedings [such as an IEEE conference in 2021](https://ieeexplore.ieee.org/document/9574673) and peer-reviewed journal articles under the MDPI publishing group (Multidisciplinary Digital Publishing Institute) in [2022](https://www.mdpi.com/2073-4441/14/17/2592) and [2023](https://www.mdpi.com/2071-1050/14/18/11478). \n\nOn the other hand, this dataset sparked lively interactions between Kagglers when it got used in demo [notebooks](https://www.kaggle.com/code/d4rklucif3r/water-quality-luciferml-76-deployment/notebook). \n \n\n### Sustainable Development Goals (SDG) driven by Kaggle \n\n#### Wildlife conservation\n\nMonitoring of animal populations is the first step to wildlife conservation. \n \nNOAA hosted a competition called [Fisheries Stella Sea lion Population Count](https://www.kaggle.com/competitions/noaa-fisheries-steller-sea-lion-population-count/overview) where submissions aimed to estimate counts of sea lions captured in aerial photographs.\n\nWhile the competition has ended six years ago, public interest continued in the form of a Master’s thesis study.\n\nNOAA also hosted a Kaggle challenge in 2015 to build a \"face recognition system\" for whales using a training set of over 500 images of right whale from aerial views, the downstream impact of that challenge led to [peer-reviewed journal articles by different research groups](https://doi.org/10.1111/cobi.13226). It also led to [worldwide collaborations](https://link.springer.com/article/10.1007/s42991-022-00253-3) and a web-based platform that allow users with no expertise in machine learning to upload photos for instant answers. The web system vastly expanded to include features for identification of both southern and North Atlantic right whales, using enlarged catalogue of photographs taken from Australia, South Africa, Argentina, Brazil, US, Canada, and New Zealand by uncrewed systems or researchers on vessels, shore, cliffs, aircrafts and helicopters. The technical advancements behind the platform is made possible via algorithm-specific for matching of different body parts of the whale: [Deepsense](https://doi.org/10.1111/cobi.13226), the original winning solution in 2015, and [pose invariant embedding (PIE)](http://arxiv.org/abs/1902.10847) are respectively used to operate on aerial and lateral views of the heads, while [CurvRank v2](https://openaccess.thecvf.com/content_WACV_2020/papers/Weideman_Extracting_identifying_contours_for_African_elephants_and_humpback_whales_using_WACV_2020_paper.pdf) and [HotSpotter](https://doi.org/10.1109/WACV.2013.6475023) algorithms are respectively used for the identification of flukes and peduncle scarring. \n\nWhen concluding on their lessons learned, the authors advocate on algorithms that enable reliable similarity score calculations between images that would not require the retraining of models on images capturing new species. They also attributed their successful worldwide collaborations to persistence, project management skills, active and ongoing multidisciplinary dialogs between researchers, biologists, data scientists, and computer engineers knowledgeable of automated workflows, which is something [Kaggle capable of attracting and facilitating](https://arxiv.org/ftp/arxiv/papers/2107/2107.11929.pdf).","metadata":{"_uuid":"51aad897-4f64-4506-86b5-340db1ccc3d0","_cell_guid":"c4bfccab-9c2a-459f-8ba6-e503ba709e90","trusted":true}},{"cell_type":"markdown","source":"## Technical notes\n\n### Time-series metrics\n\n1. Mean Absolute Percentage Error \n1. Correlation\n1. Euclidean distance\n1. ...\n \n\n","metadata":{}},{"cell_type":"markdown","source":"","metadata":{"_uuid":"5b921ab7-d483-4e05-a175-8a882115bb37","_cell_guid":"33830624-bd4c-4232-85ab-299a120e1d50","trusted":true}},{"cell_type":"markdown","source":"## Review of previous findings from recent data science competitions\n\nA [group of experts](#eisenmann) from the machine learning community administered a survey in 2021 to participants from 80 international benchmarking competitions held at MICCAI and ISBI, two top-tiers international conferences on computer vision and medical applications. To avoid biases, participants responded to the surveys before the competition results were announced. Participants from some challenges had 11% response rate (median and maximum rates were 72% and 100%, respectively); data from 249 surveys that met the inclusion criteria were analyzed. They found that among the survey responses, the most popular frameworks were PyTorch (76%) and NumPy (64%); this pattern is also observed in the [2022 Kaggle User Survey](#survey2022).\n\n\nFrom the hosts' perspectives, the following observations were made:\n
\n
\n
O1: Cheating behaviours such as using different user accounts to make multiple submissions were observed in 4% of the challenges.
\n
O2: One challenge mimicking natural settings for model deployment reported that submitted models failed to generalize in 3 out of 21 cases tested.
\n
O3: Sometimes, the metrics used to evaluate solutions did not reflect the domain of interest.
\n
O4: Despite the new trend where hosts provided additional data not traditionally available (e.g. hospital identifiers), most of the respondents could not leverage these additional data
\n
O5: The trend of complex algorithms outperformed by simpler continues
\n
O6: At times, hosts reported low diversity of methods employed; e.g. submissions differed only in the fourth decimal digit in terms of scores on a normalized scale).
\n
\n
\n\nAccording to the respondents, GPU memory constraints were their major obstacles, although many of them overcame this problem with these data augmentation strategies:\n- **Data augmentation**: \n\t- Use of downsampling of training data to a lower resolution\n\t- Solving an n-dimensional problem as a series of n-1 or lower dimensionality (e.g. 3D as 2D)\n - Patch-based strategies that use one data sample as multiple data samples\n - 37% and 6% of respondents did not use k-fold validation and internal evaluation, respectively.\n- **Post-processing**: 48% of respondents did not use post-processing.\n- **Data curation** : in the 80 challenges studied, 24% of them forbid the use of external data. Among the approaches based on deep learning, 9% of respondents actively sought additional data.\n- **Combining models**: Ensembles of model with identical (and different) architectures were used in 21% (and 9%) of the respondents. A median of 5 models were used in these settings.\n\n\n## My summary of data from past competitions\n\n\n**How might enrolments into competitions be affected by seasonal trends, prize values offered, and the administrative aspects of competitions?**\n - Duration\n - Seasonal trends\n - Reward values \n - We observed significant correlation between prize values and number of contestants only for the **research competitions** \n - Research: Spearman's correlation: 0.702 ($p$<1.57e-05)\n - Featured: Spearman's correlation: 0.155 ($p$=0.32570291447652666)\n**What are the characteristics of winning solutions?** \n - We list some of our observations in the context of the findnings from a 2023 study by [Eisenmann et al.](#eisenmann) examining competitions from the computer vision domain:\n - Winners spent twice amount of effort on failure analysis\n - Compared to non-winning submissions, ensemble approaches are at least 5.6 times more popular among winning solutions; in these approaches, ensembles determined by random seeds are most common, followed by those determined by data splits and the use of heterogeneous models.\n - The table below sets up the context of Kaggle competitions; it first lists the variables that were found by Eisenmann et al. to be statistically most significant in computer vision competitions.\n\n| Variables | Significant in 80 challenges from ISBI/MICCAI 2021 | Meta Kaggle |\n| :-- | :-- | :-- |\n| Number of team members | Y | Constrained to a maximum of 5 members |\n| Time invested on planning | Y | *Future work: Potentially available in Kaggle 2022* |\n| Time invested on data preprocessing/ augmentation | Y | *Future work: Potentially available in Kaggle 2022* |\n| Use of GPU cluster | Y | Constrained to a maximum of 30 hours|\n| Architecture type | Y | ? |\n| Account for the metrics used by the leaderboard when conducting hyper-parameter search | Y | ? |\n| Approach used for model development | Y | ? |\n| Approached used in augmentation | Y | ? |\n \n\n## References used in my research on Kaggle competitons \n\n1. Eisenmann M, Reinke A, Weru V, Tizabi MD, Isensee F, Adler TJ, Ali S, Andrearczyk V, Aubreville M, Baid U, Bakas S. Why is the winner the best?. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 19955-19966). \n\n1. Hayashi T, Shimizu T, Fukami Y. Collaborative Problem Solving on a Data Platform Kaggle. arXiv preprint arXiv:2107.11929. 2021 Jul 26. \n \n1. Tauchert C, Buxmann P, Lambinus J. Crowdsourcing data science: A qualitative analysis of organizations’ usage of kaggle competitions. \n \n1. Whitehill J. Climbing the kaggle leaderboard by exploiting the log-loss oracle. arXiv preprint arXiv:1707.01825. 2017 Jul 6. \n1. Whitehill J. Exploiting an oracle that reports AUC scores in machine learning contests. In AAAI, pages 1345–1351, 2016. \n1. Blum, Hardt M. The ladder: A reliable leaderboard for machine learning competitions. arXiv preprint arXiv:1502.04585, 2015. \n1. Zheng W. Toward a better understanding of leaderboard. arXiv preprint arXiv:1510.03349, 2015 \n1. Brackbill D, Centola D. Impact of network structure on collective learning: An experimental study in a data science competition. Plos one. 2020 Sep 4;15(9):e0237978. \n1. Can Competition Though Leaderboards Lead to Better Engagement and Learning of Data Science Concepts? An Experimental Study \n1. How to Use Kaggle [Internet]. Kaggle; 2023 [cited July 1 2023]. Available from https://www.kaggle.com/docs/competitions \n1. Why a Kaggle Competition? [Internet]. Kaggle; 2023 [cited June 23 2023]. Available from https://www.kaggle.com/c/about/host \n1. Internal discussion [Internet]. Kaggle; 2023 [cited June 23 2023]. Available from https://www.kaggle.com/discussions/general/35318 \n1. NFL Punt Analytics Competition [Internet]. Kaggle; 2023 [cited June 23 2023]. Available from https://www.kaggle.com/competitions/NFL-Punt-Analytics-Competition/discussion/73664 ","metadata":{"_uuid":"ccf05ba1-b85f-4f96-8860-460b97f3d000","_cell_guid":"309c2828-0425-44ce-92b2-cc0814f9b0e6","trusted":true}},{"cell_type":"markdown","source":"# Appendix\n\n## Resources\n\n\n- [CESM-LENS 40-member ensemble data, the Climate Data Gateway at NCAR](https://www.cesm.ucar.edu/projects/community-projects/LENS/data-sets.html)\n\n- [NMME data were provided by the IRI Data Library](http://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/ERA5) \n\n- [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/#!/search?text=ERA5&type=dataset) \n\n- [ERSSTv5 data were provided by NOAA/OAR/ESRL PSL](https://psl.noaa.gov/data/gridded/data.noaa.ersst.v5.html) \n\n- [CPC US Unified Precipitation data, NOAA/OAR/ESRL PSL](https://psl.noaa.gov/data/gridded/data.unified.daily.conus.html)\n","metadata":{"_uuid":"aa0033d6-6152-4fb6-bfa6-a959f585aac8","_cell_guid":"ae6ea7e4-ed8a-41b1-adb5-1178ac41fed6","trusted":true}},{"cell_type":"markdown","source":"## Summary of literature within Kaggle platform","metadata":{"_uuid":"43dfd5e9-bb81-4d57-91dc-b454c7641105","_cell_guid":"7cefe88e-7f87-49d1-ae6a-2c83cbafe1d0","trusted":true}},{"cell_type":"code","source":"import polars as pl \n\nif ( 'arxiv_ds' in globals() )==False: \n arxiv_ds = pl.read_ndjson(\"/kaggle/input/arxiv/arxiv-metadata-oai-snapshot.json\")\n \ndef check_arxiv( df, phr='climate action', required_phr='kaggle' ):\n if required_phr is not None:\n df = df.filter( (pl.col(\"title\").str.contains(\"(?i)%s\"% required_phr ) | pl.col(\"abstract\").str.contains(\"(?i)%s\" % required_phr ) ) ) \n df = df.filter( pl.col(\"abstract\").str.contains(\"(?i)%s\" %phr) ) \n for i,t in enumerate(df['title']):\n print(i,df['update_date'][i], t ) \n print(df.shape, '\\n')\n \nfor w in ['pollution' ,'climate', 'tipping point']:\n print(w)\n check_arxiv( arxiv_ds, w, 'time( |-)?series')","metadata":{"_uuid":"f59f64eb-9254-4250-8e45-c2a3c4c3a1ff","_cell_guid":"fa5c38d7-cdb3-45bd-9816-79634fe46aee","execution":{"iopub.status.busy":"2023-08-14T23:46:03.754837Z","iopub.execute_input":"2023-08-14T23:46:03.755176Z","iopub.status.idle":"2023-08-14T23:46:45.074976Z","shell.execute_reply.started":"2023-08-14T23:46:03.755151Z","shell.execute_reply":"2023-08-14T23:46:45.073912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Latest tips about expectations of the essay criteria\n\n> [1] This section (competitions) is meant to summarize the overall impact of Kaggle's competitions on the broader ML community. Did the ML community benefit from these competitions? Did these competitions drive forward any important advancements in the field of ML?\n\n> [2] This section (ethics) is meant to contain discussion of AI ethics as well as discussion of tools and approaches that can be used for monitoring, understanding, and mitigating risk in high-risk ML applications.\n\n> [3] This section (other) is meant to review anything that does not fit into any of the other categories e.g. simulations, robotics, multi-modal models, on-device ML, ML hardware, field-specific essays that discuss multiple categories, etc.\n","metadata":{"_uuid":"900c790a-b330-4823-bef9-f85baf574496","_cell_guid":"83c18a50-de89-4c14-83eb-3cf1b95cc7c9","trusted":true}},{"cell_type":"markdown","source":"## Final submission","metadata":{"_uuid":"a60f60cd-7b97-4440-9502-43f806fe39d4","_cell_guid":"23f1336b-b4bf-4bb9-8090-1d467b519c00","trusted":true}},{"cell_type":"code","source":"submission","metadata":{"execution":{"iopub.status.busy":"2023-07-19T05:10:00.159432Z","iopub.execute_input":"2023-07-19T05:10:00.160538Z","iopub.status.idle":"2023-07-19T05:10:00.185369Z","shell.execute_reply.started":"2023-07-19T05:10:00.160505Z","shell.execute_reply":"2023-07-19T05:10:00.184291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}
![\"Bird](\"https://img.freepik.com/premium-photo/bird-stands-pile-garbage-front-city-skyline_874904-4.jpg?w=1060\")
![](\"https://otexts.com/fpp3/fpp_files/figure-html/stationary-1.png\")
\n ![](\"https://github.com/TDAmeritrade/stumpy/blob/main/docs/images/stumpy_demo.gif?raw=true\")
\n ![](\"https://ars.els-cdn.com/content/image/1-s2.0-S0306261921003986-fx1_lrg.jpg\")
\n ![](\"https://ars.els-cdn.com/content/image/1-s2.0-S0306261921003986-gr7_lrg.jpg\")
\n