{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.display import HTML, Markdown\nimport time\n\nhandle = display(HTML(\"\"\"👌\"\"\"), display_id='html_marquee1')\ntime.sleep(2)\nhandle = display(HTML(\"\"\"~ \"No one will question your integrity if your integrity is not questionable.\" – Nathaniel Bronner Jr\n\"\"\"), display_id='html_marquee1', update=True)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:46:35.035247Z","iopub.execute_input":"2023-10-18T20:46:35.035592Z","iopub.status.idle":"2023-10-18T20:46:37.067195Z","shell.execute_reply.started":"2023-10-18T20:46:35.035568Z","shell.execute_reply":"2023-10-18T20:46:37.066494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Executive Summary
\n\n##
Preface
\n\n
📌 Summary: This notebook is intended to \n
\n\n##
Major Takeaways
\n\nDataset quality issues have been the unfortunate 'companion' to this competion from its inception. The criticality of the initial data quality problems justified the entire dataset re-generation and reloading to Kaggle by the contest host team (see the [respective discussion thread](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697) for more details about it).\n\nHowever, even after the dataset file overhaul, at least one of the glitches in the raw data persisted. \n\nThe DTS change time-related bug in the raw data manifests as the actual local timezone changes are happened a week after the real DTS change dates (as per the schedule in `2017-2019`).\n\nIt affects `43` series that overlap with DST change dates in 2017-2019.\n\nSuch a glitch may have impact on the data integrity of the training dataset under certain data loading strategies. However, the good news is the bug can be worked around if loading the training series timestamps in UTC (without converting to the local timezone time).\n\nAs displayed in the analysis flow down the road, there is also another problem with the train series step integrity that manifests in 13 series not overlapped with the DST change dates. It has a different root cause though.\n\nThe remaining sections of this notebook are structured as follows\n- **Prerequisites** set up the BigData-scale access method to the data (installing the lightweight in-memory SQL DB Engine - DuckDB; mounting the dataset files as tables in the in-memory DuckDB database)\n- **Dailight Saving Time Changes Check** contains the set of insights on how DST changes affect the training series as well as provides the comprehensive assessment/way to replicate the raw data glitch related to the DST changes listed in the timestamps one week below the actual change dates\n- **Root Cause Analysis** section discusses the reasons behind the data glitches observed\n- **Impact Analysis** section illustrate how such a data glitch in the raw data can affect the Data Science/ML applications being developed in the course of this competition; it also suggests a data loading strategy to work around this issue\n- **Check of the Series in 'Zzzs: Lightweight training dataset + target' Dataset** section provides the brief DTS change impact on the series preprocessed and packaged in a popular public lightweight dataset with preprocessed data used for model training in multiple ML contributions to this contest\n\nAt the end of the notebook, I posted \n- the references to my other EDA notebooks with the bunch of the insights mined from the contest data\n- the kudos to other Kaggle community members who pedalled the data quality issues in their notebooks and discussion threads","metadata":{}},{"cell_type":"markdown","source":"#
Prerequisites
\n\nBefore doing the BigData-scale analytics itself, we are going to complete the preparational steps below\n\n- import additional Python packages needed in our work (namely, duckdb)\n- start the instance of the in-memory database engine managed by DuckDB\n- mount the dataset files (train_series.parquet, test_series.parquet, and train_events.csv) as tables in the in-memory DuckDB database\n- create and publish the custom Python function to automate certain routines as for the data retrieval from DuckDB as well as building the visualizations","metadata":{}},{"cell_type":"code","source":"!pip install duckdb --quiet","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-18T20:46:37.068388Z","iopub.execute_input":"2023-10-18T20:46:37.069316Z","iopub.status.idle":"2023-10-18T20:46:50.329677Z","shell.execute_reply.started":"2023-10-18T20:46:37.069276Z","shell.execute_reply":"2023-10-18T20:46:50.328145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Importing gc module\nimport gc\n\nimport pandas as pd\nimport numpy as np\nimport datetime as dt\n\nimport networkx as nx\n\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nimport matplotlib.dates as mdates\nimport seaborn as sns\n\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nimport plotly.offline\n\nfrom colorama import Fore, Style, init\nfrom pprint import pprint\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:46:50.331624Z","iopub.execute_input":"2023-10-18T20:46:50.331979Z","iopub.status.idle":"2023-10-18T20:46:52.173892Z","shell.execute_reply.started":"2023-10-18T20:46:50.331951Z","shell.execute_reply":"2023-10-18T20:46:52.173084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Color printing\n# inspired by https://www.kaggle.com/code/ravi20076/sleepstate-eda-baseline\ndef PrintColor(text:str, color = Fore.BLUE, style = Style.BRIGHT):\n \"Prints color outputs using colorama using a text F-string\";\n print(style + color + text + Style.RESET_ALL);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:46:52.176840Z","iopub.execute_input":"2023-10-18T20:46:52.177493Z","iopub.status.idle":"2023-10-18T20:46:52.182463Z","shell.execute_reply.started":"2023-10-18T20:46:52.177462Z","shell.execute_reply":"2023-10-18T20:46:52.181304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import duckdb\n\nparquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet'\ntesting_parquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet'\ntraining_events_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv'\n\ncon = duckdb.connect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-18T20:46:52.183675Z","iopub.execute_input":"2023-10-18T20:46:52.184118Z","iopub.status.idle":"2023-10-18T20:46:52.237818Z","shell.execute_reply.started":"2023-10-18T20:46:52.184083Z","shell.execute_reply":"2023-10-18T20:46:52.237074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n CREATE TABLE train_series AS\n SELECT * \n FROM read_parquet('{parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:46:52.239327Z","iopub.execute_input":"2023-10-18T20:46:52.239718Z","iopub.status.idle":"2023-10-18T20:47:17.380104Z","shell.execute_reply.started":"2023-10-18T20:46:52.239681Z","shell.execute_reply":"2023-10-18T20:47:17.379040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n CREATE TABLE test_series AS\n SELECT * \n FROM read_parquet('{testing_parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:17.381508Z","iopub.execute_input":"2023-10-18T20:47:17.382153Z","iopub.status.idle":"2023-10-18T20:47:17.394637Z","shell.execute_reply.started":"2023-10-18T20:47:17.382126Z","shell.execute_reply":"2023-10-18T20:47:17.393611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# we intentionally load every col as VARCHAR; we need to load timestamp as a varchar, to prevent the UTC time casting at load\ncon.execute(f\"\"\"\n CREATE TABLE train_events AS\n SELECT * \n FROM read_csv('{training_events_table}', ALL_VARCHAR=1, AUTO_DETECT=TRUE);\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:17.396056Z","iopub.execute_input":"2023-10-18T20:47:17.396556Z","iopub.status.idle":"2023-10-18T20:47:17.479321Z","shell.execute_reply.started":"2023-10-18T20:47:17.396520Z","shell.execute_reply":"2023-10-18T20:47:17.478344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# inspired by \n# - https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\n# - https://www.kaggle.com/code/carlmcbrideellis/zzzs-eda-utc-or-not-utc\ndef get_series_data_with_events(series_id):\n \n train_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM train_series WHERE series_id = '{series_id}'\n \"\"\"\n df = con.execute(train_series_sql).fetchdf()\n \n #df['date_time'] = pd.to_datetime(df['timestamp'], utc=True)\n df['date_time'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n df = df.drop('timestamp', axis=1)\n \n train_events_sql = f\"\"\"\n SELECT \n series_id,\n CAST (night as INTEGER) as night,\n event,\n CAST (step as INTEGER) as step,\n timestamp\n FROM train_events \n WHERE series_id = '{series_id}'\n \"\"\"\n \n ev = con.execute(train_events_sql).fetchdf()\n #ev['date_time'] = pd.to_datetime(ev['timestamp'], utc=True)\n ev['date_time'] = pd.to_datetime(ev['timestamp']).apply(lambda t: t.tz_localize(None))\n ev = ev.drop('timestamp', axis=1)\n \n df2 = pd.merge(df, ev, on='date_time', how='outer')\n df2 = df2.drop(['series_id_x','series_id_y','step_x','step_y'], axis=1)\n df2['event'] = df2['event'].fillna(0)\n \n # fill values for night feature\n df2['night'][0] = 1\n df2['night'] = df2['night'].interpolate()\n df2['night'] = round(df2['night'])\n # map numeric values to event feature\n df2.loc[df2['event']=='onset','event'] = -1\n df2.loc[df2['event']=='wakeup','event'] = 1\n \n return df2\n\ndef get_train_series_data(series_id):\n train_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM train_series WHERE series_id = '{series_id}'\n \"\"\"\n train_df = con.execute(train_series_sql).fetchdf()\n return train_df\n\ndef get_test_series_data(series_id):\n test_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM test_series WHERE series_id = '{series_id}'\n \"\"\"\n df = con.execute(test_series_sql).fetchdf()\n \n #df['date_time'] = pd.to_datetime(df['timestamp'], utc=True)\n #df['date_time'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n #df = df.drop('timestamp', axis=1)\n \n return df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:17.482553Z","iopub.execute_input":"2023-10-18T20:47:17.482849Z","iopub.status.idle":"2023-10-18T20:47:17.495775Z","shell.execute_reply.started":"2023-10-18T20:47:17.482825Z","shell.execute_reply":"2023-10-18T20:47:17.494984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = \"SELECT DISTINCT series_id FROM test_series\"\ndf_test_series_id = con.execute(query).fetchdf()\n\nquery = \"SELECT DISTINCT series_id FROM train_series\"\ndf_train_series_id = con.execute(query).fetchdf()\n\ntest_series_id_list = df_test_series_id[\"series_id\"].tolist()\n\n# list of series_id for training series with the fully labelled events\n# see https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive/ for the data analysis flow resulted in locating these seria\nfully_labelled_training_series_id_list = ['08db4255286f', '0a96f4993bd7', '0cfc06c129cc', '1087d7b0ff2e', '10f8bc1f7b07', '18b61dd5aae8', \n '29c75c018220', '31011ade7c0a', '3452b878e596', '349c5562ee2c', '3664fe9233f9', '483d6545417f', \n '55a47ff9dc8a', '5acc9d63b5fd', '5f94bb3e1bed', '655f19eabf1e', '67f5fc60e494', '72bbd1ac3edf', \n '76237b9406d5', '7822ee8fe3ec', '89bd631d1769', '8e32047cbc1f', '939932f1822d', '9ee455e4770d', \n 'a596ad0b82aa', 'a9a2f7fac455', 'a9e5f5314bcb', 'af91d9a50547', 'b364205aba43', 'c535634d7dcd', \n 'c6788e579967', 'c68260cc9e8f', 'ca730dbf521d', 'd150801f3145', 'd25e479ecbb7', 'd515236bdeec', \n 'd5e47b94477e']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:17.498995Z","iopub.execute_input":"2023-10-18T20:47:17.499661Z","iopub.status.idle":"2023-10-18T20:47:19.674971Z","shell.execute_reply.started":"2023-10-18T20:47:17.499633Z","shell.execute_reply":"2023-10-18T20:47:19.674273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Dailight Saving Time Changes Check
\n\nThis section will draw intelligence on how the training series are affected by DST change dates. \n\nSince the competion host (Child Mind Insitute) is located in New York, USA, its patient sleep records are provided on East time (East Cost, USA). \n\nThis area is affected by Dailight Saving Time (DST) changes twice a year. Its timezone is either *-0400* (EST) or *-0500* (EDT), depending on daylight saving.","metadata":{}},{"cell_type":"markdown","source":"##
USA Daylight Change Dates
\n\nAccording to https://kb.lathem.com/hc/en-us/articles/115005029348-Daylight-Saving-Time-Starting-And-Ending-Dates , the schedule of daylight changes since 2014 is as follows\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
YearDST BeginsDST EndsYearDST BeginsDST Ends
2014March 9November 22024March 10November 3
2015March 8November 12025March 9November 2
2016March 13November 62026March 8November 1
2017March 12November 52027March 14November 7
2018March 11November 42028March 12November 5
2019March 10November 32029March 11November 4
2020March 8November 12030March 10November 3
2021March 14November 72031March 9November 2
2022March 13November 62032March 14November 7
2023March 12November 52033March 13November 6
\n\nSince the training series data spreads on 2017-2019 sparsly, we need to check which series overlap with the dailight saving events.","metadata":{}},{"cell_type":"markdown","source":"##
Checking Training Series Boundaries
","metadata":{}},{"cell_type":"code","source":"def get_train_series_date_boundaries():\n sql = f\"\"\"\n WITH start_end AS (\n SELECT\n series_id,\n MIN (timestamp) as series_start,\n MAX (timestamp) as series_end\n FROM train_series\n GROUP BY 1\n ),\n over_dst_start AS (\n SELECT DISTINCT\n series_id,\n '1' AS is_over_dst_start\n FROM train_series\n WHERE \n timestamp LIKE '2017-03-12%' OR\n timestamp LIKE '2018-03-11%' OR\n timestamp LIKE '2019-03-10%'\n ),\n over_dst_end AS (\n SELECT DISTINCT\n series_id,\n '1' AS is_over_dst_end\n FROM train_series\n WHERE \n timestamp LIKE '2017-11-05%' OR\n timestamp LIKE '2018-11-04%' OR\n timestamp LIKE '2019-11-03%'\n )\n SELECT\n start_end.series_id,\n start_end.series_start,\n start_end.series_end,\n over_dst_start.is_over_dst_start,\n over_dst_end.is_over_dst_end\n FROM start_end\n LEFT JOIN over_dst_start ON\n (over_dst_start.series_id = start_end.series_id)\n LEFT JOIN over_dst_end ON\n (over_dst_end.series_id = start_end.series_id);\n \n \"\"\"\n df = con.execute(sql).fetchdf()\n \n return df\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:19.676569Z","iopub.execute_input":"2023-10-18T20:47:19.676939Z","iopub.status.idle":"2023-10-18T20:47:19.683265Z","shell.execute_reply.started":"2023-10-18T20:47:19.676902Z","shell.execute_reply":"2023-10-18T20:47:19.682172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's review the start and end timestamps of each of the training series as well as check if any of DST start/end dates for 2017-2019 falls into them.\n\nFirst of all, we are going to load train series data (start of their timestamps interval, end of their timestamps interval, and a set of binary 0/1 variables to indicate how each series is affected by DTS).","metadata":{}},{"cell_type":"code","source":"train_series_data = get_train_series_date_boundaries()\ntrain_series_data = train_series_data.fillna('0')\ntrain_series_data['is_over_dst_start'] = train_series_data['is_over_dst_start'].astype(int)\ntrain_series_data['is_over_dst_end'] = train_series_data['is_over_dst_end'].astype(int)\n\ndef sum_row(x):\n return x['is_over_dst_start'] + x['is_over_dst_end']\n\ntrain_series_data['affected_by_dst'] = train_series_data.apply(sum_row, axis=1)\ntrain_series_data.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:19.684555Z","iopub.execute_input":"2023-10-18T20:47:19.684805Z","iopub.status.idle":"2023-10-18T20:47:31.597085Z","shell.execute_reply.started":"2023-10-18T20:47:19.684784Z","shell.execute_reply":"2023-10-18T20:47:31.596418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check if any series goes through either the beginning of DST and/or the end of DST period","metadata":{}},{"cell_type":"code","source":"dst_df = train_series_data[ train_series_data['affected_by_dst'] > 1]\n\nmsg = f\"Number of training series going through both the DST start and DST end dates: {dst_df.shape[0]}\"\nPrintColor(f\"\\n---> {msg}\")\nif dst_df.shape[0] == 0:\n PrintColor(f\"\\n---> As we can see, none of the training series is long enough to go through both thee DST start and DST end dates of the same year.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:31.598191Z","iopub.execute_input":"2023-10-18T20:47:31.598842Z","iopub.status.idle":"2023-10-18T20:47:31.608994Z","shell.execute_reply.started":"2023-10-18T20:47:31.598816Z","shell.execute_reply":"2023-10-18T20:47:31.608072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's visualize the distribution of series that are\n- affeced by DTS change start\n- affected by DTS change end\n- not affected by DTS change","metadata":{}},{"cell_type":"code","source":"summmary_stat = {\n 'series_with': [\"DST start date\", \"DST end date\", \"None of DST dates\"],\n 'series_count': [\n train_series_data['is_over_dst_start'].sum(),\n train_series_data['is_over_dst_end'].sum(),\n train_series_data.shape[0] - train_series_data['is_over_dst_start'].sum() - train_series_data['is_over_dst_end'].sum()\n ]\n}\nsummary_df = pd.DataFrame(summmary_stat)\nfig = px.bar(summary_df, x='series_with', y='series_count', \n title='Number of series overlapping with any of DST dates',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:31.610057Z","iopub.execute_input":"2023-10-18T20:47:31.610352Z","iopub.status.idle":"2023-10-18T20:47:33.337165Z","shell.execute_reply.started":"2023-10-18T20:47:31.610328Z","shell.execute_reply":"2023-10-18T20:47:33.336255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that\n- the majority of the series in the training dataset (`234`) are not affected by DST change\n- `22` series overlap with DST change start dates\n- `21` series overlap with DST change end dates\n- none of the training series lasts through both DST change start and end dates","metadata":{}},{"cell_type":"code","source":"# recover memory\ndel dst_df\n\n# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:33.338363Z","iopub.execute_input":"2023-10-18T20:47:33.338888Z","iopub.status.idle":"2023-10-18T20:47:33.452884Z","shell.execute_reply.started":"2023-10-18T20:47:33.338855Z","shell.execute_reply":"2023-10-18T20:47:33.449752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Checking TZ Stamps in Training Series
\n\nLet's check if timestamp TZ in the training series timestamps are correct in terms of matchning to **Pandas's** timezone calculations.\n\nInitially, let's do a smoke test with one of the series affected by DTS and one that is not.","metadata":{}},{"cell_type":"code","source":"# inspired by https://www.kaggle.com/code/junkoda/checking-daylight-saving-times\ndef check_timezone(series_id):\n df1 = get_train_series_data(series_id)\n # Original datetime in str\n timestamp_str = df1['timestamp'].values\n # e.g., 2018-08-14T15:30:00-0400\n # timezone is -0400 or -0500 depending on daylight saving\n\n # Convert to pd.Timestamp\n timestamp_utc = pd.to_datetime(df1['timestamp'], utc=True)\n timestamp_et = timestamp_utc.dt.tz_convert('US/Eastern')\n\n # str in ET\n x = timestamp_et.dt.strftime('%Y-%m-%dT%H:%M:%S%z')\n\n check = (timestamp_str == x).all()\n return check","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:33.453998Z","iopub.execute_input":"2023-10-18T20:47:33.454331Z","iopub.status.idle":"2023-10-18T20:47:33.478280Z","shell.execute_reply.started":"2023-10-18T20:47:33.454294Z","shell.execute_reply":"2023-10-18T20:47:33.477077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check `series_id` = *'599ca4ed791b'* , the one of the series affected by DST change.","metadata":{}},{"cell_type":"code","source":"%%time\nseries_id = '599ca4ed791b'\ncheck_result = check_timezone(series_id)\n\nmsg = f\"Is timezone suffix in timestamps of the series '{series_id}' correct?\"\nPrintColor(f\"\\n---> {msg}\")\nmsg = f\"Answer: {check_result}\"\nPrintColor(f\"---> {msg}\")\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:33.479403Z","iopub.execute_input":"2023-10-18T20:47:33.479773Z","iopub.status.idle":"2023-10-18T20:47:42.298014Z","shell.execute_reply.started":"2023-10-18T20:47:33.479739Z","shell.execute_reply":"2023-10-18T20:47:42.296816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's check `series_id` = *'72bbd1ac3edf'* , the one of the series not affected by DST change.","metadata":{}},{"cell_type":"code","source":"%%time\nseries_id = '72bbd1ac3edf'\ncheck_result = check_timezone(series_id)\n\nmsg = f\"Is timezone suffix in timestamps of the series '{series_id}' correct?\"\nPrintColor(f\"\\n---> {msg}\")\nmsg = f\"Answer: {check_result}\"\nPrintColor(f\"---> {msg}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:42.299357Z","iopub.execute_input":"2023-10-18T20:47:42.300324Z","iopub.status.idle":"2023-10-18T20:47:48.120883Z","shell.execute_reply.started":"2023-10-18T20:47:42.300286Z","shell.execute_reply":"2023-10-18T20:47:48.119890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we can see that the series affected by DST changes are likely to have TZ suffix issues (due to presumable deficiencies in the data preparation pipelines on the contest host side). In turn, the series not overlapping with DST change dates are correct in terms of their ","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_series_data['is_tz_correct'] = train_series_data['series_id'].apply(lambda x: check_timezone(x))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-18T20:47:48.121898Z","iopub.execute_input":"2023-10-18T20:47:48.122164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's review a few series with issues in the TZ suffix in the timestamp strings.","metadata":{}},{"cell_type":"code","source":"train_series_data.head(10)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In turn, there are equally quite a few series without issues in the TZ suffix in the timestamp strings. We can see some of them below.","metadata":{}},{"cell_type":"code","source":"train_series_data.tail(10)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's plot the total distribution of the train series by corrupted consistency of TZ suffixes in their timestamp strings.","metadata":{}},{"cell_type":"code","source":"dfg = train_series_data['is_tz_correct'].value_counts().reset_index()\ndfg.columns = ['is_tz_correct', 'series_count']\n\nvalue_map = {True: \"TZ stamps correct\", False: \"TZ stamps incorrect\"}\ndfg['is_tz_correct'] = dfg['is_tz_correct'].map(value_map)\n\nfig = px.bar(dfg, x='is_tz_correct', y='series_count', \n title='Number of series with/without issues in locat TZ timestamps',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that\n- The majority of the training series (`221`) has correct TZ suffixes across its *timestamp* values\n- `56` training series have issues with TZ suffixes across its *timestamp* values\n- the number of series with timestamp issues (`56`) exceeds the number of series affected by DST changes (`43`); that is, there are `13` series on top of it where the root cause of the issue is not related to the DST changes-driven TZ labelling bug\n\nLet's look closer on such `56` series with the TZ suffix issues.","metadata":{}},{"cell_type":"code","source":"dst_df = train_series_data[ train_series_data['is_tz_correct'] == False ]\nsummmary_stat = {\n 'series_with': [\"DST start date\", \"DST end date\", \"None of DST dates\"],\n 'series_count': [\n dst_df['is_over_dst_start'].sum(),\n dst_df['is_over_dst_end'].sum(),\n dst_df.shape[0] - dst_df['is_over_dst_start'].sum() - dst_df['is_over_dst_end'].sum()\n ]\n}\nsummary_df = pd.DataFrame(summmary_stat)\nfig = px.bar(summary_df, x='series_with', y='series_count', \n title='Number of series overlapping with any of DST dates (series with TZ issues)',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, among the series with the TZ consistency issues are\n- `21` series that overlap with the DST change start dates\n- `21` series that overlap with the DST change end dates\n- `14` series without any overlapping with DST change dates\n\nIn the section below, we are going to drill down into raw data for such series to identify more granular root cause of the issue.\n\n
📌 Note: From the findings above is inherent that, out of 22 series overlapping with the DST change start dates, one series has correct TZ labelling in the *timestamp* strings in the raw data.
","metadata":{}},{"cell_type":"markdown","source":"##
Timestamp TZ Issues: DST change start date-affected training series
\n\nLet's output the `series_id` of training series with timestamp TZ issues that overlap with DST change start dates.","metadata":{}},{"cell_type":"code","source":"filtered_df = dst_df[dst_df['is_over_dst_start'] == 1]\nseries_list = filtered_df['series_id'].values.tolist()\nseries_list","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Timestamp TZ Issues: DST change end date-affected training series
\n\nLet's output the `series_id` of training series with timestamp TZ issues that overlap with DST change end dates.","metadata":{}},{"cell_type":"code","source":"filtered_df = dst_df[dst_df['is_over_dst_end'] == 1]\nseries_list = filtered_df['series_id'].values.tolist()\nseries_list","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Timestamp TZ Issues: training series not affected by DST changes
\n\nLet's output the `series_id` of training series with timestamp TZ issues that do not overlap with any DST change dates.","metadata":{}},{"cell_type":"code","source":"filtered_df = dst_df[ dst_df['affected_by_dst'] == 0 ]\nseries_not_affected_by_dst_list = filtered_df['series_id'].values.tolist()\nseries_not_affected_by_dst_list","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Full Event-Labelled Series Check
\n\nBefore we jump to the root cause analysis, let's quickly check if any of the fully labelled training series is within the list of the series with the problematic timestamp TZ suffixes.","metadata":{}},{"cell_type":"code","source":"malformed_series_list = dst_df['series_id'].values.tolist()\nseries_count = 0\nfor series_id in malformed_series_list:\n if series_id in fully_labelled_training_series_id_list:\n msg = f\"Fully labelled series affected by TZ issue: '{series_id}'\"\n PrintColor(f\"---> {msg}\")\n series_count = series_count + 1\n\nif series_count == 0:\n \n print(\"No fully labelled series affected by TZ issue\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Root Cause Analysis of TZ issues
\n\nWe are going to drill down into raw data for such series to identify more granular root cause of the issue.\n\nThe root cause for the issue in the DST change-overlapping series is that the string representations of *timestamp* values have the zone suffix change (from -0500 to -0400 in spring, and from -0400 to -0500 in autumn) a week later then it is directively set (see directive dates listed in the section **USA Daylight Change Dates** above).\n\nAs for the series not affected by DST change, we can see that, although they do not overlap with the legally set DST change dates, they contain the data for the dates where DST change erroneously labelled in the data (see the case of the series *'d5e47b94477e'* reviewed below). Therefore, they are also affected by the same data glitch.","metadata":{}},{"cell_type":"markdown","source":"##
DST-affected Case #1: Timeline of train series '16fe2798ed0f'
\n\nLet's draw a directed graph chart to illustrate the timeline of the train series `'16fe2798ed0f'`","metadata":{}},{"cell_type":"code","source":"np.random_state=779\n\ndst_change_dates = [\n '2017-03-12', '2018-03-11', '2019-03-10',\n '2017-11-05', '2018-11-04', '2019-11-03'\n]\n\ndef get_series_graph_data(series_id):\n # 2018-11-22T09:44:55-0500\n \n sql = f\"\"\"\n WITH series_data AS (\n SELECT DISTINCT \n LEFT(timestamp,10) as step_date,\n RIGHT(timestamp, 4) as step_tz\n FROM train_series\n WHERE series_id = '{series_id}'\n )\n SELECT \n step_date,\n COUNT(*) as tz_change_flag\n FROM series_data\n GROUP BY step_date\n ORDER BY step_date;\n \"\"\"\n df = con.execute(sql).fetchdf()\n \n return df\n\ndef createDiGraph(nodes, edges):\n # Create a directed graph (digraph) object; i.e., a graph in which the edges\n # have a direction associated with them.\n G = nx.DiGraph()\n\n # Add nodes:\n G.add_nodes_from(nodes)\n # Add edges or links between the nodes:\n G.add_edges_from(edges)\n \n return G \n\ndef plot_time_series_graph(nodes, edges, node_colors, series_id):\n g = createDiGraph(nodes, edges)\n \n # Get a layout for the nodes according to some algorithm.\n # See https://networkx.github.io/documentation/stable/reference/drawing.html#layout\n # for alternative algorithms that are available.\n # Set random_state (default=None) if you want the layout to be deterministic\n # and repeatable.\n node_positions = nx.spring_layout(g)\n\n node_trace_x = []\n node_trace_y = []\n node_trace_text = []\n\n for node in node_positions:\n x, y = node_positions[node]\n node_trace_x.append(x)\n node_trace_y.append(y)\n node_trace_text.append(node)\n \n # The nodes will be plotted as a scatter plot of markers with their names\n # above each circle:\n node_trace = go.Scatter(\n x=node_trace_x,\n y=node_trace_y,\n text=node_trace_text,\n mode='markers+text',\n textposition='top center',\n textfont=dict(\n family='arial',\n size=7,\n color='rgb(0,0,0)'\n ),\n hoverinfo='none',\n marker_color = node_colors,\n marker=go.Marker(\n showscale=False,\n #color='rgb(200,0,0)',\n size=10,\n line=go.Line(width=1, color='rgb(0,0,0)'))\n )\n\n edge_trace_x = []\n edge_trace_y = []\n\n for edge in g.edges:\n x0, y0 = node_positions[edge[0]]\n x1, y1 = node_positions[edge[1]]\n edge_trace_x.extend([x0, x1, None])\n edge_trace_y.extend([y0, y1, None])\n\n # The edges will be drawn as lines:\n edge_trace = go.Scatter(\n x=edge_trace_x,\n y=edge_trace_y,\n line=go.Line(\n width=1, \n color='rgb(150,150,150)' \n #,marker= dict(size=8, symbol= \"arrow-bar-up\", angleref=\"previous\") \n ),\n hoverinfo='none',\n mode='lines'\n )\n\n # Create figure:\n fig = go.Figure(data = go.Data([edge_trace, node_trace]),\n layout = go.Layout(\n title = f\"Training Series Timeline: series '{series_id}'\",\n titlefont = dict(size=16),\n showlegend = False,\n hovermode = 'closest',\n margin = dict(b=20,l=5,r=5,t=40),\n xaxis = go.XAxis(showgrid=False, zeroline=False, showticklabels=False),\n yaxis = go.YAxis(showgrid=False, zeroline=False, showticklabels=False)\n )\n )\n \n return fig\n\n\ndef set_legal_change_date_flag(date_str):\n result = False\n \n if date_str in dst_change_dates:\n result = True\n \n return result\n\ndef update_dts_flag(is_legal_dst, tz_change_flag):\n result = tz_change_flag\n if is_legal_dst:\n result = 0\n \n return result\n \n \ndef set_color(int_flag):\n node_color = 'blue' # regular day node\n if int_flag == 0:\n node_color = 'green' # day node with the legitimate DST change scheduled\n elif int_flag == 2:\n node_color = 'red' # day node with the erroneous DST change as labelled in the raw data\n return node_color\n\ndef build_series_timeline_graph(series_id):\n df = get_series_graph_data(series_id)\n \n df['actual_DST'] = df['tz_change_flag'].apply(lambda x: True if x==2 else False)\n df['legal_DST'] = df['step_date'].apply(lambda x: set_legal_change_date_flag(x))\n \n df[\"tz_change_flag\"] = df.apply(lambda x: update_dts_flag(x[\"legal_DST\"], x[\"tz_change_flag\"]), axis=1)\n \n df['node_color'] = df['tz_change_flag'].apply(lambda x: set_color(x))\n \n nodes = [] \n edges = []\n node_colors = []\n \n nodes = df['step_date'].values.tolist()\n node_colors = df['node_color'].values.tolist()\n \n for index, row in df.iterrows():\n if index < df.shape[0]-1:\n node_start = df.iloc[index]['step_date']\n node_end = df.iloc[index + 1]['step_date']\n \n edge = (node_start, node_end)\n edges.append(edge)\n \n return nodes, edges, node_colors","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"warnings.filterwarnings('ignore')\nseries_id = '16fe2798ed0f'\nnodes, edges, node_colors = build_series_timeline_graph(series_id)\nfig = plot_time_series_graph(nodes, edges, node_colors, series_id)\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the actual DTS change date recorded in the data to be at `'2017-11-12'`, which is not the legitimate DTS change date (see the red point in the graph). The correct one should have been `'2017-11-05'` (indicated as a green point in the graph).","metadata":{}},{"cell_type":"markdown","source":"##
DST-affected Case #2: Timeline of train series '1d4569cbac0f'
\n\nLet's draw a directed graph chart to illustrate the timeline of the train series `'1d4569cbac0f'`.","metadata":{}},{"cell_type":"code","source":"warnings.filterwarnings('ignore')\nseries_id = '1d4569cbac0f'\nnodes, edges, node_colors = build_series_timeline_graph(series_id)\nfig = plot_time_series_graph(nodes, edges, node_colors, series_id)\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the actual DTS change date recorded in the data to be at `'2018-11-11'`, which is not the legitimate DTS change date (see the red point in the graph). The correct one should have been `'2018-11-04'` (indicated as a green point in the graph).","metadata":{}},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Case not affected by DTS: Timeline of train series 'd5e47b94477e'
\n\nLet's draw a directed graph chart to illustrate the timeline of the train series `'d5e47b94477e'`\n\nAs we can see, this series does not overlap with the legally set DST change date (for 2017, it is `'2017-11-04`). However, it does contain the date erroneously labelled as DST change date in the raw data (that is, `'2017-11-12'`). The latter fact leads to the issues with it, as shown in the diagram below.","metadata":{}},{"cell_type":"code","source":"warnings.filterwarnings('ignore')\nseries_id = 'd5e47b94477e'\n\nnodes, edges, node_colors = build_series_timeline_graph(series_id)\nfig = plot_time_series_graph(nodes, edges, node_colors, series_id)\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Impact Analysis
\n\nLet's illustrate how such a data glitch in the raw data provided by the contest host can affect the Data Science/ML applications being developed in the course of this competition.\n\nAs a case study, we will delve into the steps anatomy of series `'16fe2798ed0f'`, one of the series affected by the DST change.","metadata":{}},{"cell_type":"markdown","source":"##
Loading series '16fe2798ed0f' data in the local time
","metadata":{}},{"cell_type":"code","source":"%%time\ndef load_series_data(series_id):\n df1 = get_train_series_data(series_id)\n #df1['timestamp'] = pd.to_datetime(df1['timestamp'], utc=True)\n df1['timestamp'] = pd.to_datetime(df1['timestamp']).apply(lambda t: t.tz_localize(None))\n df1['step_date'] = df1['timestamp'].dt.date\n df1['step_time'] = df1['timestamp'].dt.time\n \n return df1\n\n\nseries_id = '16fe2798ed0f'\n\ndf_series = load_series_data(series_id)\ndf_series.head()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_series.groupby('step_date').step.count()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that there is an anomaly for `'2018-11-12'` as we see `18000` steps on the sample for this day. In a normal day, steps with 5 s increment (as stated for the accelerometer records provided by the contest host) will result in `17280` steps per day. \n\n18000 steps = 25 hours. It indicates the data loading resulted in duplicated records for `'2018-11-12'`, the day they erroneously recorded the DTS change date.","metadata":{}},{"cell_type":"markdown","source":"##
Loading series '16fe2798ed0f' data in UTC
","metadata":{}},{"cell_type":"code","source":"%%time\ndef load_series_data_utc(series_id):\n df1 = get_train_series_data(series_id)\n df1['timestamp'] = pd.to_datetime(df1['timestamp'], utc=True)\n # df1['timestamp'] = pd.to_datetime(df1['timestamp']).apply(lambda t: t.tz_localize(None))\n df1['step_date'] = df1['timestamp'].dt.date\n df1['step_time'] = df1['timestamp'].dt.time\n \n return df1\n\n\nseries_id = '16fe2798ed0f'\n\ndf_series = load_series_data_utc(series_id)\ndf_series.head()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_series.groupby('step_date').step.count()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At this data loading scenario, there are no any anomaly for `'2018-11-12'`. We see `17280` steps on the full days in the series. The number of steps in the first and last (incomplete) days of the series is different vs. the scenario of loading data in the local time.","metadata":{}},{"cell_type":"markdown","source":"##
Series data loading takeaways
\n\n
📌 Takeaways: The investigation above clearly indicates the time series data should be loaded with timestamps casted to UTC, in order to work around the raw data deficiency. For the purpose of ML modelling, having the time values in UTC (rather then in a local time zone's time) does not really matter.\n
\n\n
✅ Note: It was Veronika Sarikaya who paid attention to the issue with the duplicated steps records upon loading the series data (as per It's a wonderful, wonderful sleep (EDA) notebook)\n
","metadata":{}},{"cell_type":"markdown","source":"##
Notes to Non-Pandas-based Data Loading Strategies
\n\nThe above-mentioned issue with DST change dates labelling in the training series timestamps has been demonstrated with usage of Pandas as an instrument for data extraction/manipulation. However, as such, it will also impact other technology stacks to use for the data reading into memory.\n\nFor example, the following code using Polar will also generate the series step duplicates\n
\n rain_series = (pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\n .with_columns(\n (\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.year().alias(\"year\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.month().alias(\"month\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.day().alias(\"day\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.hour().alias(\"hour\")),\n )\n )\n .collect()\n .to_pandas()\n )\n
\n\nAt the same time, the Polar-based data loading strategy that takes the timestamps in UTC does not produce the duplicates:\n\n
\n dt_transforms = [\n pl.col('timestamp').str.to_datetime(), \n (pl.col('timestamp').str.to_datetime().dt.date()).alias('date'), \n pl.col('timestamp').str.to_datetime().dt.time().alias('time')\n ]\n \n train_series = (pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\n .with_columns(\n (dt_transforms)\n )\n .collect()\n .to_pandas()\n ) \n
","metadata":{}},{"cell_type":"markdown","source":"#
Check of the Series in 'Zzzs: Lightweight training dataset + target' Dataset
\n\nMany public competition notebooks rely on **'Zzzs: Lightweight training dataset + target'** dataset (https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target) prepared by [@Carl Ellis](https://www.kaggle.com/carlmcbrideellis). It consists of 35 (more or less formed well) series selected from the original 277 series; therefore, it addresses the challenge of training the ML models for the contest within the limited resources of Kaggle notebooks premise (as dictated by Code competition requirements).\n\nLet's do a quick sanity check of the integrity of the series data selected for this dataset.","metadata":{}},{"cell_type":"code","source":"file = '/kaggle/input/zzzs-lightweight-training-dataset-target/Zzzs_train.parquet'\nseries_id = pd.read_parquet(file, columns=['series_id'])\nseries_id_list = series_id.series_id.unique()\n\nmsg = f\"Number of series in the lightweight dataset: {len(series_id_list)}\"\nPrintColor(f\"---> {msg}\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for series_id in series_id_list:\n display(Markdown(\"##
Timeline: Series '\" + series_id + \"'
\" ))\n \n nodes, edges, node_colors = build_series_timeline_graph(series_id)\n fig = plot_time_series_graph(nodes, edges, node_colors, series_id)\n fig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Lightweight Series Boundaries / Impact by DST Change
","metadata":{}},{"cell_type":"code","source":"lightweight_series_df = train_series_data[train_series_data['series_id'].isin(series_id_list)]\nlightweight_series_df.style.set_caption(\"Series in the lightweight dataset\"). \\\nset_properties(**{'border': '1.3px solid violet',\n 'color': 'blue'})","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Lightweight Series Check: Major Findings
\n\nFrom reviewing the graphs and the table above, we can draw the insights below\n\n- there are series that overlap with the DST change dates, being affected by the erroneour DST change date labelling in the raw data (these are series with `id = '55a47ff9dc8a', 'a9a2f7fac455', 'c6788e579967'`) \n- there are series that do not overlap with DST change dates yet contain the date of the erroneous DST change, being affected by the erroneour DST change date labelling in the raw data (these are series with `id = '08db4255286f', '1087d7b0ff2e', '655f19eabf1e', '67f5fc60e494', 'd5e47b94477e'`)\n- finally, there is a set of the series with the well-formed timestamps data (the rest of the series not mentioned in the previous points of this list)","metadata":{}},{"cell_type":"markdown","source":"#
References
\n\nThis notebook extends the set of EDA insights/findings for **Child Mind Institute - Detect Sleep States** competition. You can find other notebooks of mine with interesting EDA insights as per the links below\n\n- [To Sleep or Not to Sleep? Deep EDA Dive](https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive)\n- [Plotting Big Data: Sleeping Series](https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series)\n- [TS Windows Around Events](https://www.kaggle.com/code/gvyshnya/ts-windows-around-events)","metadata":{}},{"cell_type":"markdown","source":"#
Kudos
\n\nThis notebook has been inspired by the contributors below\n- *Jun Koda* [@junkoda](https://www.kaggle.com/junkoda) who paid attention to local TZ inconsistency due to DTS change dates recorded in the data incorrectly, as per [Checking Daylight Saving Times](https://www.kaggle.com/code/junkoda/checking-daylight-saving-times)\n- *Veronika Sarikaya* [@veronikasarikaya](https://www.kaggle.com/code/veronikasarikaya/) who contributed to the body of knowledge for this contest with the bunch of nice insights (as per [It's a wonderful, wonderful sleep (EDA)](https://www.kaggle.com/code/veronikasarikaya/it-s-a-wonderful-wonderful-sleep-eda)); in particular, she paid attention to the duplicated *timestamp* values in data of certain series when such data is fetched with the option of converting the *timestamp* strings into *datetime*, using *local time zone (TZ)*\n\nIn his original note, *Jun Koda* mentioned\n
\n\"I think the switch between EST (-5) and EDT (-4) are done at wrong time (as of 2023-10-02), one week later than it should be. ...\n\nDaylight saving time bug discussion thread: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697\n\nThe bug is probably unimportant, only the string representation is wrong and physical time (e.g. in UTC) seems to be correct.\"\n
\n\nAs for the bug impact, however, it could be possible to partially disagree with his statement though. As displayed above, such a bug has an impact if someone reads the train series data from parquet with option of converting it to the datetime in the local time zone (rather then in UTC where there is no issues indeed).","metadata":{}}]}