{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.display import HTML, Markdown\nimport time\n\nhandle = display(HTML(\"\"\"👌\"\"\"), display_id='html_marquee1')\ntime.sleep(2)\nhandle = display(HTML(\"\"\"~ \"The world is one BigData problem.\" – Andrew McAfee\"\"\"), display_id='html_marquee1', update=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:53:48.462455Z","iopub.execute_input":"2023-09-29T07:53:48.462867Z","iopub.status.idle":"2023-09-29T07:53:50.482928Z","shell.execute_reply.started":"2023-09-29T07:53:48.462833Z","shell.execute_reply":"2023-09-29T07:53:50.481723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Executive Summary
\n\n
📌 Summary: This notebook is intended to \n
\n\nIn order to perform the analytics operations to get the insights on training and testing series data (BigData-scale data, indeed), we leveraged DuckDB, the fast, lightweight, and easy-to-use analytics database with a good API/wrappers for Python.\n\nThe key takeaways for the BigData-scale EDA drills on training and testing series are listed below\n\n- there is a lack of the systematic patterns in the population-level and series-level distribution and aggregated statistics for *enmo* and *angllez* in both training and test seria sets\n- distribution and summary statistics of *enmo* and *angllez* are significantly different in training seria vs. test seria\n- test series data contain the *enmo* and *angllez* feeds for just three seria, 1 h long each; such a structure of the test set may impose some complications/road blocks to the ML solutions attempting to tackle the contest problem in the regression setup\n- the average values of *enmo* are typically lower for the hours where *onset* events are more frequent (in the training set)\n- the average values of *enmo* are typically lower for the hours where *wakeup* events are more frequent (in the training set)\n- the average values of *anglez* are typically higher for the hours where *wakeup* events are more frequent (although we can see some hours without major number of *wakeup* events to have the similar average values of *anglez*)\n- the average values of *anglez* are typically lower for the hours where *onset* events are more frequent\n\nRegarding the intervals of sleeping and being awake, we also noticed the facts below\n- the average values of *anglez* are even more high in the time intervals where a person is awake (it can be even higher then the average values for the hours with the bigger number of *wakeup* events registered)\n- higher values of *enmo* are observed for the awake state of the observable humans\n\n
💥 Highlights: The above-mentioned finding about the hourly patterns in matching onset/wakeup events to the average hourly values of enmo and angllez, combined with the earlier detected findings on the lack of the systematic patterns in the population-level and series-level distribution and aggregated statistics for enmo and angllez, may highlight an interesting forecasting strategy.

\n \nThe ML solution can focus on scanning the time series breakpoints where simultaneoursly observed are\n\n
\n\n
✅ Note: The data insights about the training and testing series collected across this notebook extend the body of EDA findings per the master EDA notebook.
","metadata":{}},{"cell_type":"markdown","source":"#
Prerequisites
\n\nBefore doing the BigData-scale analytics itself, we are going to complete the preparational steps below\n- import additional Python packages needed in our work (namely, *duckdb*)\n- start the instance of the in-memory database engine managed by DuckDB\n- mount the dataset files (*train_series.parquet*, *test_series.parquet*, and *train_events.csv*) as tables in the in-memory DuckDB database\n- create and publish the custom Python function to automate certain routines as for the data retrieval from DuckDB as well as building the visualizations","metadata":{}},{"cell_type":"code","source":"# !pip install jupysql duckdb duckdb-engine pyarrow --quiet","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:53:50.484962Z","iopub.execute_input":"2023-09-29T07:53:50.485368Z","iopub.status.idle":"2023-09-29T07:53:50.490204Z","shell.execute_reply.started":"2023-09-29T07:53:50.485265Z","shell.execute_reply":"2023-09-29T07:53:50.488843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install duckdb --quiet","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:53:50.491734Z","iopub.execute_input":"2023-09-29T07:53:50.492207Z","iopub.status.idle":"2023-09-29T07:54:00.826134Z","shell.execute_reply.started":"2023-09-29T07:53:50.492162Z","shell.execute_reply":"2023-09-29T07:54:00.824333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Importing gc module\nimport gc\n\nimport pandas as pd\nimport numpy as np\nimport datetime as dt\n\n\nfrom scipy.signal import spectrogram\n\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nimport matplotlib.dates as mdates\nimport seaborn as sns\n\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nimport plotly.offline","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:00.829990Z","iopub.execute_input":"2023-09-29T07:54:00.830499Z","iopub.status.idle":"2023-09-29T07:54:02.172587Z","shell.execute_reply.started":"2023-09-29T07:54:00.830452Z","shell.execute_reply":"2023-09-29T07:54:02.171506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import duckdb\n\nparquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet'\ntesting_parquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet'\ntraining_events_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv'\n\ncon = duckdb.connect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:02.175038Z","iopub.execute_input":"2023-09-29T07:54:02.175820Z","iopub.status.idle":"2023-09-29T07:54:02.224699Z","shell.execute_reply.started":"2023-09-29T07:54:02.175773Z","shell.execute_reply":"2023-09-29T07:54:02.223820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n CREATE TABLE train_series AS\n SELECT * \n FROM read_parquet('{parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:02.226256Z","iopub.execute_input":"2023-09-29T07:54:02.226934Z","iopub.status.idle":"2023-09-29T07:54:26.514505Z","shell.execute_reply.started":"2023-09-29T07:54:02.226902Z","shell.execute_reply":"2023-09-29T07:54:26.513082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n CREATE TABLE test_series AS\n SELECT * \n FROM read_parquet('{testing_parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.516272Z","iopub.execute_input":"2023-09-29T07:54:26.516795Z","iopub.status.idle":"2023-09-29T07:54:26.530031Z","shell.execute_reply.started":"2023-09-29T07:54:26.516752Z","shell.execute_reply":"2023-09-29T07:54:26.528821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# we intentionally load every col as VARCHAR; we need to load timestamp as a varchar, to prevent the UTC time casting at load\ncon.execute(f\"\"\"\n CREATE TABLE train_events AS\n SELECT * \n FROM read_csv('{training_events_table}', ALL_VARCHAR=1, AUTO_DETECT=TRUE);\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.531573Z","iopub.execute_input":"2023-09-29T07:54:26.532319Z","iopub.status.idle":"2023-09-29T07:54:26.623103Z","shell.execute_reply.started":"2023-09-29T07:54:26.532280Z","shell.execute_reply":"2023-09-29T07:54:26.621976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_all_series_boxplot_data(table, col_name, where_clause = None):\n min_max_sql = \"\"\n quantile_sql = \"\"\n if where_clause:\n min_max_sql = f\"\"\"\n SELECT series_id, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n GROUP BY series_id\n WHERE {where_clause}\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n series_id,\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n GROUP BY series_id\n WHERE {where_clause}\n )\n SELECT\n series_id,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n else:\n min_max_sql = f\"\"\"\n SELECT series_id, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n GROUP BY series_id\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n series_id,\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n GROUP BY series_id\n )\n SELECT\n series_id,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n df_mean_max = con.execute(min_max_sql).fetchdf()\n df_quantile = con.execute(quantile_sql).fetchdf()\n \n df = pd.merge(\n df_mean_max,\n df_quantile,\n how=\"inner\",\n on='series_id')\n \n #rearrange the order of cols to make it a canonical boxplot style\n df = df[['series_id', 'min_value', 'q1', 'median', 'q3', 'max_value', 'mean_value', 'stddev']]\n \n return df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.625139Z","iopub.execute_input":"2023-09-29T07:54:26.625585Z","iopub.status.idle":"2023-09-29T07:54:26.634432Z","shell.execute_reply.started":"2023-09-29T07:54:26.625544Z","shell.execute_reply":"2023-09-29T07:54:26.633171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_boxplot_data(table, col_name, where_clause = None):\n min_max_sql = \"\"\n quantile_sql = \"\"\n if where_clause:\n min_max_sql = f\"\"\"\n SELECT '{col_name}' AS attribute, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n WHERE {where_clause}\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n WHERE {where_clause}\n )\n SELECT\n '{col_name}' AS attribute,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n else:\n min_max_sql = f\"\"\"\n SELECT '{col_name}' AS attribute, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n )\n SELECT\n '{col_name}' AS attribute,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n df_mean_max = con.execute(min_max_sql).fetchdf()\n df_quantile = con.execute(quantile_sql).fetchdf()\n \n df = pd.merge(\n df_mean_max,\n df_quantile,\n how=\"inner\",\n on='attribute')\n \n #rearrange the order of cols to make it a canonical boxplot style\n df = df[['attribute', 'min_value', 'q1', 'median', 'q3', 'max_value', 'mean_value', 'stddev']]\n \n return df\n\ndef build_boxplot(table, col_name, where_clause = None):\n df = get_boxplot_data(table, col_name, where_clause)\n fig_title = \"\"\n \n bar_name = \"\".join([table, \".\", col_name])\n \n if where_clause:\n fig_title = \"\".join([table, \".\", col_name, \": \", where_clause])\n else:\n fig_title = \"\".join([table, \".\", col_name, \": Entire table population\"])\n \n fig = go.Figure()\n\n fig.add_trace(go.Box(\n y= [ df['min_value'].iloc[0], df['max_value'].iloc[0] ],\n name=bar_name, boxpoints=False,)\n )\n\n fig.update_traces(q1=[ df['q1'].iloc[0] ], median=[ df['median'].iloc[0] ],\n q3=[ df['q3'].iloc[0] ], lowerfence=[df['min_value'].iloc[0]],\n upperfence=[df['max_value'].iloc[0]], mean=[ df['mean_value'].iloc[0] ],\n sd=[ df['stddev'].iloc[0]]\n )\n # Update visual layout\n fig.update_layout(\n showlegend=False,\n width=600,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n title=dict(text=fig_title, font=dict(size=16), automargin=True, yref='paper'),\n title_x=0.5, # center the title\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n )\n \n # update font size at the axes\n fig.update_coloraxes(colorbar_tickfont_size=10)\n # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n fig.update_annotations(font_size=12)\n # Reduce opacity\n fig.update_traces(opacity=0.75)\n \n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.638285Z","iopub.execute_input":"2023-09-29T07:54:26.639225Z","iopub.status.idle":"2023-09-29T07:54:26.658963Z","shell.execute_reply.started":"2023-09-29T07:54:26.639186Z","shell.execute_reply":"2023-09-29T07:54:26.657624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://duckdb.org/docs/sql/aggregates\ndef get_histogram_data(table, col_name, where_clause = None):\n query = \"\"\n if where_clause:\n query = f\"\"\"\n SELECT histogram({col_name}) FROM {table}\n WHERE {where_clause};\n \"\"\"\n else:\n query = f\"\"\"\n SELECT histogram({col_name}) FROM {table};\n \"\"\"\n df = con.execute(query).fetchdf()\n \n hist_col_name = f\"histogram({col_name})\"\n\n histogram_dict = df[hist_col_name].iloc[0] # dictionary: dict_keys(['key', 'value'])\n\n frame = {\n col_name: histogram_dict.get('key'),\n 'record_count': histogram_dict.get('value')}\n \n # Creating DataFrame by passing Dictionary\n agg_data = pd.DataFrame(frame)\n return agg_data\n \ndef build_histogram(table, col_name, where_clause = None):\n figure_title = \"\"\n if where_clause:\n figure_title = f\"Distribution of {col_name} in {table}; filter: {where_clause}\"\n else:\n figure_title = f\"Distribution of {col_name} in {table}\"\n \n agg_data = get_histogram_data(table, col_name, where_clause)\n \n fig = px.histogram(agg_data, x=col_name, marginal=\"rug\",\n title=figure_title,\n color_discrete_sequence=px.colors.qualitative.Prism)\n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.660685Z","iopub.execute_input":"2023-09-29T07:54:26.661174Z","iopub.status.idle":"2023-09-29T07:54:26.677586Z","shell.execute_reply.started":"2023-09-29T07:54:26.661090Z","shell.execute_reply":"2023-09-29T07:54:26.676026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_histogram_data_2(table, col_name, where_clause = None):\n query = \"\"\n if where_clause:\n query = f\"\"\"\n SELECT FLOOR({col_name}/1.00)*1 AS {col_name}, \n COUNT(*) AS record_count\n FROM {table}\n WHERE {where_clause}\n GROUP BY FLOOR({col_name}/1.00)*1\n ORDER BY 1\n \"\"\"\n else:\n query = f\"\"\"\n SELECT FLOOR({col_name}/1.00)*1 AS {col_name}, \n COUNT(*) AS record_count\n FROM {table}\n GROUP BY FLOOR({col_name}/1.00)*1\n ORDER BY 1\n \"\"\"\n df_agg = con.execute(query).fetchdf()\n return df_agg\n\ndef build_histogram_2(table, col_name, where_clause = None):\n fig_title = \"\"\n if where_clause:\n fig_title = f\"Distribution of {col_name}: {table}; filtered: {where_clause}\"\n else:\n fig_title = f\"Distribution of {col_name}: {table}\"\n df_agg = get_histogram_data_2(table, col_name, where_clause)\n fig = px.bar(df_agg, x=col_name, y=\"record_count\",\n title=\"Distribution of anglez: training series\",\n color_discrete_sequence=px.colors.qualitative.Prism)\n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.679596Z","iopub.execute_input":"2023-09-29T07:54:26.680064Z","iopub.status.idle":"2023-09-29T07:54:26.697441Z","shell.execute_reply.started":"2023-09-29T07:54:26.680022Z","shell.execute_reply":"2023-09-29T07:54:26.695626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_hourly_averages(table, col_name, where_clause = None):\n query = \"\"\n if where_clause:\n query = f\"\"\"\n WITH hours_of_day AS (\n SELECT extract('hour' FROM \n strptime(substring(replace(timestamp, 'T', ' '), 1, length(timestamp)-5), '%Y-%m-%d %H:%M:%S')) AS hour_of_day,\n {col_name}\n FROM {table}\n WHERE {where_clause}\n )\n SELECT\n hour_of_day, \n AVG({col_name}) AS avg_{col_name}\n FROM hours_of_day\n GROUP BY 1\n ORDER BY 1\n \"\"\"\n else:\n query = f\"\"\"\n WITH hours_of_day AS (\n SELECT extract('hour' FROM \n strptime(substring(replace(timestamp, 'T', ' '), 1, length(timestamp)-5), '%Y-%m-%d %H:%M:%S')) AS hour_of_day,\n {col_name}\n FROM {table}\n )\n SELECT\n hour_of_day, \n AVG({col_name}) AS avg_{col_name}\n FROM hours_of_day\n GROUP BY 1\n ORDER BY 1\n \"\"\"\n df_agg = con.execute(query).fetchdf()\n return df_agg\n\n\ndef build_hourly_averages_bar(table, col_name, where_clause = None):\n y_variable = f\"avg_{col_name}\"\n fig_title = \"\"\n if where_clause:\n fig_title = f\"Average {col_name} by hours in a day (local time); filtered: {where_clause}\"\n else:\n fig_title = f\"Average {col_name} by hours in a day (local time)\"\n df_agg = get_hourly_averages(table, col_name, where_clause)\n fig = px.bar(df_agg, x='hour_of_day', y=y_variable,\n title=fig_title,\n color_discrete_sequence=px.colors.qualitative.Prism)\n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.699146Z","iopub.execute_input":"2023-09-29T07:54:26.699556Z","iopub.status.idle":"2023-09-29T07:54:26.715309Z","shell.execute_reply.started":"2023-09-29T07:54:26.699517Z","shell.execute_reply":"2023-09-29T07:54:26.713938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_events_hourly_counts(where_clause = None):\n query = \"\"\n if where_clause:\n query = f\"\"\"\n WITH hours_of_day AS (\n SELECT extract('hour' FROM \n strptime(substring(replace(timestamp, 'T', ' '), 1, length(timestamp)-5), '%Y-%m-%d %H:%M:%S')) AS hour_of_day,\n event\n FROM train_events\n WHERE {where_clause}\n )\n SELECT\n event,\n hour_of_day, \n COUNT(*) AS event_count\n FROM hours_of_day\n GROUP BY 1,2\n ORDER BY 1,2\n \"\"\"\n else:\n query = f\"\"\"\n WITH hours_of_day AS (\n SELECT extract('hour' FROM \n strptime(substring(replace(timestamp, 'T', ' '), 1, length(timestamp)-5), '%Y-%m-%d %H:%M:%S')) AS hour_of_day,\n event\n FROM train_events\n )\n SELECT\n event,\n hour_of_day, \n COUNT(*) AS event_count\n FROM hours_of_day\n GROUP BY 1,2\n ORDER BY 1,2\n \"\"\"\n df_agg = con.execute(query).fetchdf()\n return df_agg","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.717453Z","iopub.execute_input":"2023-09-29T07:54:26.718251Z","iopub.status.idle":"2023-09-29T07:54:26.734525Z","shell.execute_reply.started":"2023-09-29T07:54:26.718205Z","shell.execute_reply":"2023-09-29T07:54:26.732965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Training vs. Testing Series in Comparison
\n\nLet's collect and compare the summmary statistics about the training and testing series in terms of\n- distribution of enmo and anglez\n- summary statistis on enmo and anglez usually visualized in box plots (min, max, median, mean, std, Q1, Q3)\n- average values of enmo by hours in a day\n- everage values of anglez by hours in a day","metadata":{}},{"cell_type":"markdown","source":"##
Distribution of enmo
\n\nLet's calculate and visualize the histograms of *enmo* for both the training and testing series.","metadata":{}},{"cell_type":"code","source":"fig = build_histogram(table='train_series', col_name='enmo')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:26.736709Z","iopub.execute_input":"2023-09-29T07:54:26.737185Z","iopub.status.idle":"2023-09-29T07:54:32.286837Z","shell.execute_reply.started":"2023-09-29T07:54:26.737139Z","shell.execute_reply":"2023-09-29T07:54:32.285922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_histogram(table='test_series', col_name='enmo')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:32.288090Z","iopub.execute_input":"2023-09-29T07:54:32.288944Z","iopub.status.idle":"2023-09-29T07:54:32.361260Z","shell.execute_reply.started":"2023-09-29T07:54:32.288908Z","shell.execute_reply":"2023-09-29T07:54:32.360061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that\n- the distribution of *enmo* on training and testing series is quite different (although both distributions are severely skewed to the left)\n- the most of the *enmo* values in the training series belong to the interval of *0.025..2.175* (within this interval, evey bucket with 0.05 incremental step contains either 500 or 501 observations)\n- the most of the *enmo* values in the testing series belong to the interval of *-0.025..0.0749*","metadata":{}},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:32.362717Z","iopub.execute_input":"2023-09-29T07:54:32.363609Z","iopub.status.idle":"2023-09-29T07:54:32.495106Z","shell.execute_reply.started":"2023-09-29T07:54:32.363574Z","shell.execute_reply":"2023-09-29T07:54:32.493558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Summary statistics on enmo
\n\nLet's extract the summary statistis on enmo and then visualize it for both the training and testing serias.\n\nLet's start with the training series.","metadata":{"execution":{"iopub.status.busy":"2023-09-15T08:33:08.549492Z","iopub.execute_input":"2023-09-15T08:33:08.550076Z","iopub.status.idle":"2023-09-15T08:33:08.556371Z","shell.execute_reply.started":"2023-09-15T08:33:08.550041Z","shell.execute_reply":"2023-09-15T08:33:08.554934Z"}}},{"cell_type":"code","source":"fig = build_boxplot('train_series', 'enmo', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:32.496621Z","iopub.execute_input":"2023-09-29T07:54:32.497226Z","iopub.status.idle":"2023-09-29T07:54:38.168093Z","shell.execute_reply.started":"2023-09-29T07:54:32.497170Z","shell.execute_reply":"2023-09-29T07:54:38.167117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's do the same for the testing series","metadata":{}},{"cell_type":"code","source":"fig = build_boxplot('test_series', 'enmo', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:38.169635Z","iopub.execute_input":"2023-09-29T07:54:38.169989Z","iopub.status.idle":"2023-09-29T07:54:38.218384Z","shell.execute_reply.started":"2023-09-29T07:54:38.169962Z","shell.execute_reply":"2023-09-29T07:54:38.217441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Hourly averages: enmo
","metadata":{}},{"cell_type":"markdown","source":"Let's visualize the hourly average values on *enmo* for the training series data.","metadata":{}},{"cell_type":"code","source":"fig = build_hourly_averages_bar('train_series', 'enmo')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:54:38.219450Z","iopub.execute_input":"2023-09-29T07:54:38.220365Z","iopub.status.idle":"2023-09-29T07:55:00.001043Z","shell.execute_reply.started":"2023-09-29T07:54:38.220333Z","shell.execute_reply":"2023-09-29T07:54:59.999769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's visualize the hourly average values on *enmo* for the test series data.","metadata":{}},{"cell_type":"code","source":"fig = build_hourly_averages_bar('test_series', 'enmo')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:00.002507Z","iopub.execute_input":"2023-09-29T07:55:00.003386Z","iopub.status.idle":"2023-09-29T07:55:00.057452Z","shell.execute_reply.started":"2023-09-29T07:55:00.003351Z","shell.execute_reply":"2023-09-29T07:55:00.056296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:00.059051Z","iopub.execute_input":"2023-09-29T07:55:00.059990Z","iopub.status.idle":"2023-09-29T07:55:00.192008Z","shell.execute_reply.started":"2023-09-29T07:55:00.059943Z","shell.execute_reply":"2023-09-29T07:55:00.190855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Distribution of anglez
\n\nLet's visualize distribution of *anglez* in both training and testing sets as histograms.","metadata":{}},{"cell_type":"code","source":"fig = build_histogram_2('train_series', 'anglez')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:00.193526Z","iopub.execute_input":"2023-09-29T07:55:00.193981Z","iopub.status.idle":"2023-09-29T07:55:02.069179Z","shell.execute_reply.started":"2023-09-29T07:55:00.193950Z","shell.execute_reply":"2023-09-29T07:55:02.067902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_histogram(table='test_series', col_name='anglez')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:02.071191Z","iopub.execute_input":"2023-09-29T07:55:02.072502Z","iopub.status.idle":"2023-09-29T07:55:02.142384Z","shell.execute_reply.started":"2023-09-29T07:55:02.072454Z","shell.execute_reply":"2023-09-29T07:55:02.141208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:02.144155Z","iopub.execute_input":"2023-09-29T07:55:02.144886Z","iopub.status.idle":"2023-09-29T07:55:02.273715Z","shell.execute_reply.started":"2023-09-29T07:55:02.144843Z","shell.execute_reply":"2023-09-29T07:55:02.272244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Summary statistics on anglez
","metadata":{}},{"cell_type":"code","source":"df = get_boxplot_data('train_series', 'anglez', where_clause = None)\ndf","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:02.275331Z","iopub.execute_input":"2023-09-29T07:55:02.275767Z","iopub.status.idle":"2023-09-29T07:55:07.244182Z","shell.execute_reply.started":"2023-09-29T07:55:02.275705Z","shell.execute_reply":"2023-09-29T07:55:07.242990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_boxplot('train_series', 'anglez', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:07.245423Z","iopub.execute_input":"2023-09-29T07:55:07.245734Z","iopub.status.idle":"2023-09-29T07:55:12.542611Z","shell.execute_reply.started":"2023-09-29T07:55:07.245707Z","shell.execute_reply":"2023-09-29T07:55:12.541730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = get_boxplot_data('test_series', 'anglez', where_clause = None)\ndf","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:12.549773Z","iopub.execute_input":"2023-09-29T07:55:12.550923Z","iopub.status.idle":"2023-09-29T07:55:12.573198Z","shell.execute_reply.started":"2023-09-29T07:55:12.550862Z","shell.execute_reply":"2023-09-29T07:55:12.571851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_boxplot('test_series', 'anglez', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:12.574455Z","iopub.execute_input":"2023-09-29T07:55:12.575149Z","iopub.status.idle":"2023-09-29T07:55:12.620089Z","shell.execute_reply.started":"2023-09-29T07:55:12.575114Z","shell.execute_reply":"2023-09-29T07:55:12.618768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:12.621458Z","iopub.execute_input":"2023-09-29T07:55:12.621853Z","iopub.status.idle":"2023-09-29T07:55:12.754836Z","shell.execute_reply.started":"2023-09-29T07:55:12.621819Z","shell.execute_reply":"2023-09-29T07:55:12.753497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Hourly averages: anglez
","metadata":{}},{"cell_type":"code","source":"fig = build_hourly_averages_bar('train_series', 'anglez')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:12.756616Z","iopub.execute_input":"2023-09-29T07:55:12.757138Z","iopub.status.idle":"2023-09-29T07:55:34.009232Z","shell.execute_reply.started":"2023-09-29T07:55:12.757092Z","shell.execute_reply":"2023-09-29T07:55:34.008047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_hourly_averages_bar('test_series', 'anglez')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:34.011576Z","iopub.execute_input":"2023-09-29T07:55:34.012105Z","iopub.status.idle":"2023-09-29T07:55:34.069239Z","shell.execute_reply.started":"2023-09-29T07:55:34.012060Z","shell.execute_reply":"2023-09-29T07:55:34.068453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:34.070460Z","iopub.execute_input":"2023-09-29T07:55:34.071396Z","iopub.status.idle":"2023-09-29T07:55:34.200317Z","shell.execute_reply.started":"2023-09-29T07:55:34.071362Z","shell.execute_reply":"2023-09-29T07:55:34.198968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Training Events
\n\nLet's visualize the frequency of the *onset* and *wakeup* events in the training set.","metadata":{}},{"cell_type":"code","source":"df_agg = get_events_hourly_counts().dropna()\n\ndf_onset = df_agg[df_agg['event'] == 'onset']\ndf_wakeup = df_agg[df_agg['event'] == 'wakeup']\n\nfig = make_subplots(rows=1, cols=2, subplot_titles=('Hourly Frequency of Onset Events',\n 'Hourly Frequency of Wakeup Events',\n ))\n\nfig.add_trace(go.Bar(x=df_onset['hour_of_day'], y=df_onset['event_count']), row=1, col=1)\nfig.add_trace(go.Bar(x=df_wakeup['hour_of_day'], y=df_wakeup['event_count']), row=1, col=2)\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=600,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show() ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:34.202035Z","iopub.execute_input":"2023-09-29T07:55:34.202492Z","iopub.status.idle":"2023-09-29T07:55:34.286163Z","shell.execute_reply.started":"2023-09-29T07:55:34.202458Z","shell.execute_reply":"2023-09-29T07:55:34.285173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the chart displayed, it is evident that the majority of individuals whose sleeping series are represented in the training dataset tend to\n\n- wake up between 6-8 am;\n- go to sleep around 8-11 pm.\n\nSuch a temporal pattern matches well with general intuition as well as provides a useful baseline for further analyses.","metadata":{}},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:34.287316Z","iopub.execute_input":"2023-09-29T07:55:34.287619Z","iopub.status.idle":"2023-09-29T07:55:34.418079Z","shell.execute_reply.started":"2023-09-29T07:55:34.287592Z","shell.execute_reply":"2023-09-29T07:55:34.416603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Training Series: Entire Population vs. Individual Series
\n\nLet's look at the selective individual training seria to appraise who similar or equally differnt they are (both from each other and from the entire population of the training seria) in terms of\n- distribution of *enmo* and *anglez*\n- summary statistics charts for *enmo* and *anglez*\n- hourly averages for *enmo*, *anglez*, and training events\n\nWe are going to specifically scrutinize the subset of *37* training series with fully labelled *onset* and *wakeup* events.\n\n
✅ Note: You can refer to the master EDA notebook to see more info on how such 37 training series have been located.
","metadata":{}},{"cell_type":"markdown","source":"##
Distribution of enmo: selective training series
\n\nLet's look at the *enmo* distribution in the selective training seria.","metadata":{}},{"cell_type":"code","source":"list_of_fully_labelled_seria = ['08db4255286f', '0a96f4993bd7', '0cfc06c129cc', '1087d7b0ff2e', '10f8bc1f7b07', '18b61dd5aae8', \n '29c75c018220', '31011ade7c0a', '3452b878e596', '349c5562ee2c', '3664fe9233f9', '483d6545417f', \n '55a47ff9dc8a', '5acc9d63b5fd', '5f94bb3e1bed', '655f19eabf1e', '67f5fc60e494', '72bbd1ac3edf', \n '76237b9406d5', '7822ee8fe3ec', '89bd631d1769', '8e32047cbc1f', '939932f1822d', '9ee455e4770d', \n 'a596ad0b82aa', 'a9a2f7fac455', 'a9e5f5314bcb', 'af91d9a50547', 'b364205aba43', 'c535634d7dcd', \n 'c6788e579967', 'c68260cc9e8f', 'ca730dbf521d', 'd150801f3145', 'd25e479ecbb7', 'd515236bdeec', \n 'd5e47b94477e']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:34.419532Z","iopub.execute_input":"2023-09-29T07:55:34.420577Z","iopub.status.idle":"2023-09-29T07:55:34.431305Z","shell.execute_reply.started":"2023-09-29T07:55:34.420542Z","shell.execute_reply":"2023-09-29T07:55:34.430282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"col_name_to_review = \"enmo\"\nsubplot_titles = [ f\"Distr. of {col_name_to_review}: series {series_id}\" for series_id in list_of_fully_labelled_seria ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=13, cols=3, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in list_of_fully_labelled_seria:\n df = get_histogram_data(table=\"train_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Histogram(x=df[col_name_to_review], y=df['record_count']), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=900,\n height=1000,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:34.433933Z","iopub.execute_input":"2023-09-29T07:55:34.434540Z","iopub.status.idle":"2023-09-29T07:55:36.771678Z","shell.execute_reply.started":"2023-09-29T07:55:34.434492Z","shell.execute_reply":"2023-09-29T07:55:36.770296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:36.773298Z","iopub.execute_input":"2023-09-29T07:55:36.773967Z","iopub.status.idle":"2023-09-29T07:55:36.902321Z","shell.execute_reply.started":"2023-09-29T07:55:36.773884Z","shell.execute_reply":"2023-09-29T07:55:36.901044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Summary statistics for enmo: selective training series
\nLet's look at the summary statistics for *enmo* in the selective training seria. We are going to visualize it in a set of boxplot diagrams.","metadata":{}},{"cell_type":"code","source":"n=37\ndf_series = get_all_series_boxplot_data('train_series', 'enmo')\ndf_series_slice = df_series[df_series['series_id'].isin(list_of_fully_labelled_seria[0:n])]\n\ndummy_values = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]\ndummmy_y = []\nfor i in range(n):\n dummmy_y.append(dummy_values)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:36.904352Z","iopub.execute_input":"2023-09-29T07:55:36.904690Z","iopub.status.idle":"2023-09-29T07:55:45.851577Z","shell.execute_reply.started":"2023-09-29T07:55:36.904663Z","shell.execute_reply":"2023-09-29T07:55:45.850480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\n\nfig.add_trace(go.Box(\n y=dummmy_y, \n x = df_series_slice['series_id'],\n name=\"Precompiled\", boxpoints=False,)\n)\n\nfig.update_traces(q1=df_series_slice['q1'], median=df_series_slice['median'],\n q3=df_series_slice['q3'], lowerfence=df_series_slice['min_value'],\n upperfence=df_series_slice['max_value'], mean=df_series_slice['mean_value'],\n sd=df_series_slice['stddev'] )\n\n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=400,\n title_text='Summary of enmo by selective training seria', \n title_x=0.5,\n margin=dict(t=30, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:45.853285Z","iopub.execute_input":"2023-09-29T07:55:45.853640Z","iopub.status.idle":"2023-09-29T07:55:45.896123Z","shell.execute_reply.started":"2023-09-29T07:55:45.853610Z","shell.execute_reply":"2023-09-29T07:55:45.894754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:45.897832Z","iopub.execute_input":"2023-09-29T07:55:45.898185Z","iopub.status.idle":"2023-09-29T07:55:46.033022Z","shell.execute_reply.started":"2023-09-29T07:55:45.898154Z","shell.execute_reply":"2023-09-29T07:55:46.031809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Average hourly enmo: selective training series
\nLet's look at the hourly averages for *enmo* distribution in the selective training seria.","metadata":{}},{"cell_type":"code","source":"col_name_to_review = \"enmo\"\nsubplot_titles = [ f\"Hourly averages of {col_name_to_review}: series {series_id}\" for series_id in list_of_fully_labelled_seria ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=13, cols=3, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in list_of_fully_labelled_seria:\n df = get_hourly_averages(table=\"train_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Bar(x=df['hour_of_day'], y=df[f\"avg_{col_name_to_review}\"]), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=1000,\n height=1000,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:46.034289Z","iopub.execute_input":"2023-09-29T07:55:46.034644Z","iopub.status.idle":"2023-09-29T07:55:49.466563Z","shell.execute_reply.started":"2023-09-29T07:55:46.034613Z","shell.execute_reply":"2023-09-29T07:55:49.465201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:49.468418Z","iopub.execute_input":"2023-09-29T07:55:49.468893Z","iopub.status.idle":"2023-09-29T07:55:49.603827Z","shell.execute_reply.started":"2023-09-29T07:55:49.468850Z","shell.execute_reply":"2023-09-29T07:55:49.602497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Distribution of anglez: selective training series
\nLet's look at the *anglez* distribution in the selective training seria.","metadata":{}},{"cell_type":"code","source":"col_name_to_review = \"anglez\"\nsubplot_titles = [ f\"Distr. of {col_name_to_review}: series {series_id}\" for series_id in list_of_fully_labelled_seria ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=13, cols=3, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in list_of_fully_labelled_seria:\n df = get_histogram_data_2(table=\"train_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Bar(x=df[col_name_to_review], y=df['record_count']), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=900,\n height=1000,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:49.605868Z","iopub.execute_input":"2023-09-29T07:55:49.606370Z","iopub.status.idle":"2023-09-29T07:55:50.461028Z","shell.execute_reply.started":"2023-09-29T07:55:49.606324Z","shell.execute_reply":"2023-09-29T07:55:50.459744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:50.462933Z","iopub.execute_input":"2023-09-29T07:55:50.463371Z","iopub.status.idle":"2023-09-29T07:55:50.597076Z","shell.execute_reply.started":"2023-09-29T07:55:50.463333Z","shell.execute_reply":"2023-09-29T07:55:50.595645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Summary statistics for anglez: selective training series
\nLet's look at the summary statistics for *anglez* in the selective training seria. We are going to visualize it in a set of boxplot diagrams.","metadata":{}},{"cell_type":"code","source":"n=37\ndf_series = get_all_series_boxplot_data('train_series', 'anglez')\ndf_series_slice = df_series[df_series['series_id'].isin(list_of_fully_labelled_seria[0:n])]\n\ndummy_values = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]\ndummmy_y = []\nfor i in range(n):\n dummmy_y.append(dummy_values)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:50.598379Z","iopub.execute_input":"2023-09-29T07:55:50.598717Z","iopub.status.idle":"2023-09-29T07:55:58.998793Z","shell.execute_reply.started":"2023-09-29T07:55:50.598678Z","shell.execute_reply":"2023-09-29T07:55:58.997504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\n\nfig.add_trace(go.Box(\n y=dummmy_y, \n x = df_series_slice['series_id'],\n name=\"Precompiled\", boxpoints=False,)\n)\n\nfig.update_traces(q1=df_series_slice['q1'], median=df_series_slice['median'],\n q3=df_series_slice['q3'], lowerfence=df_series_slice['min_value'],\n upperfence=df_series_slice['max_value'], mean=df_series_slice['mean_value'],\n sd=df_series_slice['stddev'] )\n\n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=400,\n title_text='Summary of anglez by selective training seria', \n title_x=0.5,\n margin=dict(t=30, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:59.000498Z","iopub.execute_input":"2023-09-29T07:55:59.000992Z","iopub.status.idle":"2023-09-29T07:55:59.046521Z","shell.execute_reply.started":"2023-09-29T07:55:59.000947Z","shell.execute_reply":"2023-09-29T07:55:59.045698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:59.047590Z","iopub.execute_input":"2023-09-29T07:55:59.048720Z","iopub.status.idle":"2023-09-29T07:55:59.180561Z","shell.execute_reply.started":"2023-09-29T07:55:59.048672Z","shell.execute_reply":"2023-09-29T07:55:59.179194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Average hourly anglez: selective training series
\nLet's look at the hourly averages for *anglez* in the selective training seria.","metadata":{}},{"cell_type":"code","source":"col_name_to_review = \"anglez\"\nsubplot_titles = [ f\"Hourly averages of {col_name_to_review}: series {series_id}\" for series_id in list_of_fully_labelled_seria ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=13, cols=3, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in list_of_fully_labelled_seria:\n df = get_hourly_averages(table=\"train_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Bar(x=df['hour_of_day'], y=df[f\"avg_{col_name_to_review}\"]), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=1000,\n height=1000,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:55:59.182145Z","iopub.execute_input":"2023-09-29T07:55:59.183029Z","iopub.status.idle":"2023-09-29T07:56:02.698978Z","shell.execute_reply.started":"2023-09-29T07:55:59.182998Z","shell.execute_reply":"2023-09-29T07:56:02.697872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:02.700484Z","iopub.execute_input":"2023-09-29T07:56:02.701214Z","iopub.status.idle":"2023-09-29T07:56:02.831030Z","shell.execute_reply.started":"2023-09-29T07:56:02.701177Z","shell.execute_reply":"2023-09-29T07:56:02.829629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Testing Series: Entire Population vs. Individual Series
\n\nLet's look at the selective individual test seria to appraise who similar or equally differnt they are (both from each other and from the entire population of the training seria) in terms of\n- distribution of *enmo* and *anglez*\n- summary statistics charts for *enmo* and *anglez*\n- hourly averages for *enmo* and *anglez*\n\nLet's get the list of series_id in the testing set.","metadata":{}},{"cell_type":"code","source":"query = \"SELECT DISTINCT series_id FROM test_series\"\ndf_testing_series_id = con.execute(query).fetchdf()\n\ntest_series_id_list = df_testing_series_id[\"series_id\"].tolist()\ntest_series_id_list","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:02.832380Z","iopub.execute_input":"2023-09-29T07:56:02.832747Z","iopub.status.idle":"2023-09-29T07:56:02.847883Z","shell.execute_reply.started":"2023-09-29T07:56:02.832715Z","shell.execute_reply":"2023-09-29T07:56:02.846710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Distribution of enmo: test series
\n\nLet's look at the *enmo* distribution in the test seria.","metadata":{}},{"cell_type":"code","source":"col_name_to_review = \"enmo\"\nsubplot_titles = [ f\"Distr. of {col_name_to_review}: series {series_id}\" for series_id in test_series_id_list ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=1, cols=number_of_cols_in_subplot, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in test_series_id_list:\n df = get_histogram_data(table=\"test_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Histogram(x=df[col_name_to_review], y=df['record_count']), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=300,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:02.849431Z","iopub.execute_input":"2023-09-29T07:56:02.849797Z","iopub.status.idle":"2023-09-29T07:56:02.923066Z","shell.execute_reply.started":"2023-09-29T07:56:02.849765Z","shell.execute_reply":"2023-09-29T07:56:02.921746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:02.924644Z","iopub.execute_input":"2023-09-29T07:56:02.925020Z","iopub.status.idle":"2023-09-29T07:56:03.055158Z","shell.execute_reply.started":"2023-09-29T07:56:02.924990Z","shell.execute_reply":"2023-09-29T07:56:03.053907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Summary statistics on enmo: test series
\n\nLet's visualize the summary statistics on the test seria as boxplots.","metadata":{}},{"cell_type":"code","source":"n=3\ndf_series_slice = get_all_series_boxplot_data('test_series', 'enmo')\n\ndummy_values = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]\ndummmy_y = []\nfor i in range(n):\n dummmy_y.append(dummy_values)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.057083Z","iopub.execute_input":"2023-09-29T07:56:03.058037Z","iopub.status.idle":"2023-09-29T07:56:03.077532Z","shell.execute_reply.started":"2023-09-29T07:56:03.057990Z","shell.execute_reply":"2023-09-29T07:56:03.076221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\n\nfig.add_trace(go.Box(\n y=dummmy_y, \n x = df_series_slice['series_id'],\n name=\"Precompiled\", boxpoints=False,)\n)\n\nfig.update_traces(q1=df_series_slice['q1'], median=df_series_slice['median'],\n q3=df_series_slice['q3'], lowerfence=df_series_slice['min_value'],\n upperfence=df_series_slice['max_value'], mean=df_series_slice['mean_value'],\n sd=df_series_slice['stddev'] )\n\n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=300,\n title_text='Summary of enmo by test seria', \n title_x=0.5,\n margin=dict(t=30, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.079243Z","iopub.execute_input":"2023-09-29T07:56:03.080007Z","iopub.status.idle":"2023-09-29T07:56:03.121069Z","shell.execute_reply.started":"2023-09-29T07:56:03.079962Z","shell.execute_reply":"2023-09-29T07:56:03.119891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.122641Z","iopub.execute_input":"2023-09-29T07:56:03.123409Z","iopub.status.idle":"2023-09-29T07:56:03.253825Z","shell.execute_reply.started":"2023-09-29T07:56:03.123371Z","shell.execute_reply":"2023-09-29T07:56:03.252179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Average hourly enmo: test series
\nLet's look at the hourly averages for *enmo* distribution in the test seria.","metadata":{}},{"cell_type":"code","source":"col_name_to_review = \"enmo\"\nsubplot_titles = [ f\"Hourly averages of {col_name_to_review}: series {series_id}\" for series_id in test_series_id_list ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=1, cols=number_of_cols_in_subplot, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in test_series_id_list:\n df = get_hourly_averages(table=\"test_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Bar(x=df['hour_of_day'], y=df[f\"avg_{col_name_to_review}\"]), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=1000,\n height=600,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.255457Z","iopub.execute_input":"2023-09-29T07:56:03.255955Z","iopub.status.idle":"2023-09-29T07:56:03.327992Z","shell.execute_reply.started":"2023-09-29T07:56:03.255912Z","shell.execute_reply":"2023-09-29T07:56:03.326528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.329587Z","iopub.execute_input":"2023-09-29T07:56:03.330227Z","iopub.status.idle":"2023-09-29T07:56:03.462446Z","shell.execute_reply.started":"2023-09-29T07:56:03.330189Z","shell.execute_reply":"2023-09-29T07:56:03.461286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Distribution of anglez: test series
\n\nLet's look at the *anglez* distribution in the test seria.","metadata":{}},{"cell_type":"code","source":"col_name_to_review = \"anglez\"\nsubplot_titles = [ f\"Distr. of {col_name_to_review}: series {series_id}\" for series_id in test_series_id_list ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=1, cols=number_of_cols_in_subplot, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in test_series_id_list:\n df = get_histogram_data(table=\"test_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Histogram(x=df[col_name_to_review], y=df['record_count']), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=300,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.464258Z","iopub.execute_input":"2023-09-29T07:56:03.464871Z","iopub.status.idle":"2023-09-29T07:56:03.548877Z","shell.execute_reply.started":"2023-09-29T07:56:03.464816Z","shell.execute_reply":"2023-09-29T07:56:03.548000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.550277Z","iopub.execute_input":"2023-09-29T07:56:03.550911Z","iopub.status.idle":"2023-09-29T07:56:03.686808Z","shell.execute_reply.started":"2023-09-29T07:56:03.550877Z","shell.execute_reply":"2023-09-29T07:56:03.685594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Summary statistics on anglez: test series
\n\nLet's visualize the summary statistics on *anglez* on the test seria as boxplots.","metadata":{}},{"cell_type":"code","source":"n=3\ndf_series_slice = get_all_series_boxplot_data('test_series', 'anglez')\n\ndummy_values = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]\ndummmy_y = []\nfor i in range(n):\n dummmy_y.append(dummy_values)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.688710Z","iopub.execute_input":"2023-09-29T07:56:03.689149Z","iopub.status.idle":"2023-09-29T07:56:03.709191Z","shell.execute_reply.started":"2023-09-29T07:56:03.689109Z","shell.execute_reply":"2023-09-29T07:56:03.707946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\n\nfig.add_trace(go.Box(\n y=dummmy_y, \n x = df_series_slice['series_id'],\n name=\"Precompiled\", boxpoints=False,)\n)\n\nfig.update_traces(q1=df_series_slice['q1'], median=df_series_slice['median'],\n q3=df_series_slice['q3'], lowerfence=df_series_slice['min_value'],\n upperfence=df_series_slice['max_value'], mean=df_series_slice['mean_value'],\n sd=df_series_slice['stddev'] )\n\n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=300,\n title_text='Summary of anglez by test seria', \n title_x=0.5,\n margin=dict(t=30, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.711340Z","iopub.execute_input":"2023-09-29T07:56:03.712253Z","iopub.status.idle":"2023-09-29T07:56:03.757075Z","shell.execute_reply.started":"2023-09-29T07:56:03.712193Z","shell.execute_reply":"2023-09-29T07:56:03.755871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.759231Z","iopub.execute_input":"2023-09-29T07:56:03.759702Z","iopub.status.idle":"2023-09-29T07:56:03.895027Z","shell.execute_reply.started":"2023-09-29T07:56:03.759636Z","shell.execute_reply":"2023-09-29T07:56:03.893727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Average hourly anglez: test series
\n\nLet's look at the hourly averages for *anglez* in the test seria.","metadata":{}},{"cell_type":"code","source":"col_name_to_review = \"anglez\"\nsubplot_titles = [ f\"Hourly averages of {col_name_to_review}: series {series_id}\" for series_id in test_series_id_list ]\nnumber_of_cols_in_subplot = 3\n# TODO: auto-calculate number of rows in the subplot\n\nfig = make_subplots(rows=1, cols=number_of_cols_in_subplot, subplot_titles=(subplot_titles))\n \nrow_id = 1\ncol_id = 1 \n \nfor series_id in test_series_id_list:\n df = get_hourly_averages(table=\"test_series\", col_name=col_name_to_review, where_clause = f\"series_id = '{series_id}'\")\n fig.add_trace(go.Bar(x=df['hour_of_day'], y=df[f\"avg_{col_name_to_review}\"]), row=row_id, col=col_id)\n col_id += 1\n if col_id == number_of_cols_in_subplot + 1:\n row_id += 1\n col_id = 1\n\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=1000,\n height=600,\n margin=dict(t=15, b=15, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.896451Z","iopub.execute_input":"2023-09-29T07:56:03.896903Z","iopub.status.idle":"2023-09-29T07:56:03.976093Z","shell.execute_reply.started":"2023-09-29T07:56:03.896862Z","shell.execute_reply":"2023-09-29T07:56:03.974991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:03.977820Z","iopub.execute_input":"2023-09-29T07:56:03.978290Z","iopub.status.idle":"2023-09-29T07:56:04.108152Z","shell.execute_reply.started":"2023-09-29T07:56:03.978236Z","shell.execute_reply":"2023-09-29T07:56:04.106874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#from sqlalchemy import create_engine\n#from sql import plot\n\n#conn = create_engine('duckdb:///')\n#p = plot.bar(parquet_table, 'anglez')\n#p","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:04.110016Z","iopub.execute_input":"2023-09-29T07:56:04.110467Z","iopub.status.idle":"2023-09-29T07:56:04.120260Z","shell.execute_reply.started":"2023-09-29T07:56:04.110426Z","shell.execute_reply":"2023-09-29T07:56:04.118997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Average hourly events: selective training series
\n\nLet's look at the hourly average counts for *onset* and *wakeup* events in the selective training seria, matching them with the average hourly values of *enmo* and *anglez*.","metadata":{}},{"cell_type":"code","source":"def build_hourly_charts(training_series_id):\n series_id = training_series_id\n where_clause = f\"series_id = '{series_id}'\"\n df_agg = get_events_hourly_counts(where_clause=where_clause).dropna()\n\n df_onset = df_agg[df_agg['event'] == 'onset']\n df_wakeup = df_agg[df_agg['event'] == 'wakeup']\n df_enmo = get_hourly_averages(table=\"train_series\", col_name='enmo', where_clause = where_clause)\n df_anglez = get_hourly_averages(table=\"train_series\", col_name='anglez', where_clause = where_clause)\n\n fig = make_subplots(rows=2, cols=2, subplot_titles=('Hourly Frequency of Onset Events',\n 'Hourly Frequency of Wakeup Events',\n 'Avg. Hourly Values of enmo',\n 'Avg. Hourly Values of anglez'\n ))\n\n fig.add_trace(go.Bar(x=df_onset['hour_of_day'], y=df_onset['event_count']), row=1, col=1)\n fig.add_trace(go.Bar(x=df_wakeup['hour_of_day'], y=df_wakeup['event_count']), row=1, col=2)\n fig.add_trace(go.Bar(x=df_enmo['hour_of_day'], y=df_enmo['avg_enmo']), row=2, col=1)\n fig.add_trace(go.Bar(x=df_anglez['hour_of_day'], y=df_anglez['avg_anglez']), row=2, col=2)\n \n # Update visual layout\n fig.update_layout(\n showlegend=False,\n width=600,\n height=400,\n autosize=False,\n title_text=f'Events and Avg. enmo/anglez by Hours: {where_clause}', \n title_x=0.5,\n margin=dict(t=50, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n )\n\n # update font size at the axes\n fig.update_coloraxes(colorbar_tickfont_size=10)\n # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n fig.update_annotations(font_size=12)\n # Reduce opacity\n fig.update_traces(opacity=0.75)\n \n return fig\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:04.122585Z","iopub.execute_input":"2023-09-29T07:56:04.123162Z","iopub.status.idle":"2023-09-29T07:56:04.137378Z","shell.execute_reply.started":"2023-09-29T07:56:04.123119Z","shell.execute_reply":"2023-09-29T07:56:04.136343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Summary of the Event Findings
\n\nWe find that \n- the average values of *enmo* are typically lower for the hours where *onset* events are more frequent\n- the average values of *enmo* are typically lower for the hours where *wakeup* events are more frequent\n- the average values of *anglez* are typically higher for the hours where *wakeup* events are more frequent (although we can see some hours without major number of *wakeup* events to have the similar average values of *anglez*)\n- the average values of *anglez* are typically lower for the hours where *onset* events are more frequent\n\n\nRegarding the intervals of sleeping and being awake, we can also notice the facts below\n- the average values of *anglez* are even more high in the time intervals where a person is awake (it can be even higher then the average values for the hours with the bigger number of *wakeup* events registered)\n- higher values of *enmo* are observed for the awake state of the observable humans\n\n\nWe are going to visualize the patterns described above by plotting the data ffor three selective training seria\n- Series *'08db4255286f'*\n- Series *'0a96f4993bd7'*\n- Series *'0cfc06c129cc'*\n\nOther training seria within the list of *37* training series within the fully labelled (annotated) *onset* and *wakeup* events demonstrate similar patterns. We skipped the visualizations of the rest of the seria from the list of the fully annotated seria to save some time of the readers.\n\n
✨💥 Note: The above-mentioned finding about the hourly patterns in matching onset/wakeup events to the average hourly values of enmo and angllez, combined with the earlier detected findings on the lack of the systematic patterns in the population-level and series-level distribution and aggregated statistics for enmo and angllez, may highlight an interesting forecasting strategy.

\n \nThe ML solution can focus on scanning the time series breakpoints where simultaneoursly observed are\n\n
","metadata":{}},{"cell_type":"markdown","source":"##
Series '08db4255286f'
","metadata":{}},{"cell_type":"code","source":"\"\"\"\nlist_of_fully_labelled_seria = ['08db4255286f', '0a96f4993bd7', '0cfc06c129cc', '1087d7b0ff2e', '10f8bc1f7b07', '18b61dd5aae8', \n '29c75c018220', '31011ade7c0a', '3452b878e596', '349c5562ee2c', '3664fe9233f9', '483d6545417f', \n '55a47ff9dc8a', '5acc9d63b5fd', '5f94bb3e1bed', '655f19eabf1e', '67f5fc60e494', '72bbd1ac3edf', \n '76237b9406d5', '7822ee8fe3ec', '89bd631d1769', '8e32047cbc1f', '939932f1822d', '9ee455e4770d', \n 'a596ad0b82aa', 'a9a2f7fac455', 'a9e5f5314bcb', 'af91d9a50547', 'b364205aba43', 'c535634d7dcd', \n 'c6788e579967', 'c68260cc9e8f', 'ca730dbf521d', 'd150801f3145', 'd25e479ecbb7', 'd515236bdeec', \n 'd5e47b94477e']\n\"\"\"\nfig = build_hourly_charts('08db4255286f')\nfig.show() ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:04.139601Z","iopub.execute_input":"2023-09-29T07:56:04.140225Z","iopub.status.idle":"2023-09-29T07:56:04.410915Z","shell.execute_reply.started":"2023-09-29T07:56:04.140184Z","shell.execute_reply":"2023-09-29T07:56:04.409684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:04.412411Z","iopub.execute_input":"2023-09-29T07:56:04.412793Z","iopub.status.idle":"2023-09-29T07:56:04.542416Z","shell.execute_reply.started":"2023-09-29T07:56:04.412759Z","shell.execute_reply":"2023-09-29T07:56:04.541144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that \n- the average values of *enmo* are typically lower for the hours where *onset* events are more frequent\n- the average values of *enmo* are typically lower for the hours where *wakeup* events are more frequent\n- the average values of *anglez* are typically higher for the hours where *wakeup* events are more frequent (although we can see some hours without major number of *wakeup* events to have the similar average values of *anglez*)\n- the average values of *anglez* are typically lower for the hours where *onset* events are more frequent\n\n\nRegarding the intervals of sleeping and being awake, we can also notice the facts below\n- the average values of *anglez* are even more high in the time intervals where a person is awake (it can be even higher then the average values for the hours with the bigger number of *wakeup* events registered)\n- higher values of *enmo* are observed for the awake state of the observable humans","metadata":{}},{"cell_type":"markdown","source":"##
Series '0a96f4993bd7'
","metadata":{}},{"cell_type":"code","source":"\"\"\"\nlist_of_fully_labelled_seria = ['08db4255286f', '0a96f4993bd7', '0cfc06c129cc', '1087d7b0ff2e', '10f8bc1f7b07', '18b61dd5aae8', \n '29c75c018220', '31011ade7c0a', '3452b878e596', '349c5562ee2c', '3664fe9233f9', '483d6545417f', \n '55a47ff9dc8a', '5acc9d63b5fd', '5f94bb3e1bed', '655f19eabf1e', '67f5fc60e494', '72bbd1ac3edf', \n '76237b9406d5', '7822ee8fe3ec', '89bd631d1769', '8e32047cbc1f', '939932f1822d', '9ee455e4770d', \n 'a596ad0b82aa', 'a9a2f7fac455', 'a9e5f5314bcb', 'af91d9a50547', 'b364205aba43', 'c535634d7dcd', \n 'c6788e579967', 'c68260cc9e8f', 'ca730dbf521d', 'd150801f3145', 'd25e479ecbb7', 'd515236bdeec', \n 'd5e47b94477e']\n\"\"\"\nfig = build_hourly_charts('0a96f4993bd7')\nfig.show() ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:04.543899Z","iopub.execute_input":"2023-09-29T07:56:04.544238Z","iopub.status.idle":"2023-09-29T07:56:04.749723Z","shell.execute_reply.started":"2023-09-29T07:56:04.544203Z","shell.execute_reply":"2023-09-29T07:56:04.748616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:04.750810Z","iopub.execute_input":"2023-09-29T07:56:04.751123Z","iopub.status.idle":"2023-09-29T07:56:04.877142Z","shell.execute_reply.started":"2023-09-29T07:56:04.751098Z","shell.execute_reply":"2023-09-29T07:56:04.875972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that \n- the average values of *enmo* are typically lower for the hours where *onset* events are more frequent\n- the average values of *enmo* are typically lower for the hours where *wakeup* events are more frequent\n- the average values of *anglez* are typically higher for the hours where *wakeup* events are more frequent \n- the average values of *anglez* are typically lower for the hours where *onset* events are more frequent\n\nRegarding the intervals of sleeping and being avake, we can also notice the facts below\n- the average values of *anglez* are even more high in the time intervals where a person is awake (it can be even higher then the average values for the hours with the bigger number of *wakeup* events registered)\n- high peaks of the average value of *anglez* are observed for 2-3 am and 3-4 am (local time) intervals where the majority of the observations fall into the sleeping state\n- higher values of *enmo* are observed for the awake state of the observable humans","metadata":{}},{"cell_type":"markdown","source":"##
Series '0cfc06c129cc'
","metadata":{}},{"cell_type":"code","source":"\"\"\"\nlist_of_fully_labelled_seria = ['08db4255286f', '0a96f4993bd7', '0cfc06c129cc', '1087d7b0ff2e', '10f8bc1f7b07', '18b61dd5aae8', \n '29c75c018220', '31011ade7c0a', '3452b878e596', '349c5562ee2c', '3664fe9233f9', '483d6545417f', \n '55a47ff9dc8a', '5acc9d63b5fd', '5f94bb3e1bed', '655f19eabf1e', '67f5fc60e494', '72bbd1ac3edf', \n '76237b9406d5', '7822ee8fe3ec', '89bd631d1769', '8e32047cbc1f', '939932f1822d', '9ee455e4770d', \n 'a596ad0b82aa', 'a9a2f7fac455', 'a9e5f5314bcb', 'af91d9a50547', 'b364205aba43', 'c535634d7dcd', \n 'c6788e579967', 'c68260cc9e8f', 'ca730dbf521d', 'd150801f3145', 'd25e479ecbb7', 'd515236bdeec', \n 'd5e47b94477e']\n\"\"\"\nfig = build_hourly_charts('0cfc06c129cc')\nfig.show() ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:04.878727Z","iopub.execute_input":"2023-09-29T07:56:04.879063Z","iopub.status.idle":"2023-09-29T07:56:05.134899Z","shell.execute_reply.started":"2023-09-29T07:56:04.879036Z","shell.execute_reply":"2023-09-29T07:56:05.133506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:05.136176Z","iopub.execute_input":"2023-09-29T07:56:05.136487Z","iopub.status.idle":"2023-09-29T07:56:05.274308Z","shell.execute_reply.started":"2023-09-29T07:56:05.136460Z","shell.execute_reply":"2023-09-29T07:56:05.272444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that \n- the average values of *enmo* are typically lower for the hours where *onset* events are more frequent\n- the average values of *enmo* are typically lower for the hours where *wakeup* events are more frequent\n- the average values of *anglez* are typically higher for the hours where *wakeup* events are more frequent \n- the average values of *anglez* are typically lower for the hours where *onset* events are more frequent\n\nRegarding the intervals of sleeping and being avake, we can also notice the facts below\n- the average values of *anglez* are even more high in the time intervals where a person is awake (it can be even higher then the average values for the hours with the bigger number of *wakeup* events registered)\n- higher values of *enmo* are observed for the awake state of the observable humans","metadata":{}},{"cell_type":"markdown","source":"#
Temporal View on Training Seria: Night Views
\n\nLet's visualize time series representations of certain selective training series.\n\nIn order to do it, we are going to preprocess the series data as follows\n- merge train series and events data for a selective series_id (outer join the enmo/anglez measurements on the events data)\n- normalize the values of *enmo* and *anglez*, to smoothen the TS line curves down the road as well as make the hourly trends more visible in the charts\n- parse timestamp values as datetimes, with casting to the local time zones\n- map event attribute as numeric variable (*-1* for *'onset'*, *1* for *wakeup*, and *0* for all other time slots without events recorded)\n- interpolate *night* values across the merged dataset to clearly separate merged records for each individual night\n\n\nAs proved by the individual series night charts in the subsections below,\n\n- the value of *enmo* typically go to the local miminum value around *onset* events\n- the value of *enmo* typically go to the local miminum value around *wakeup* events\n- the value of *anglez* typically go to the local maximum value around *wakeup* events\n- the value of *anglez* typically go to the local miminum value around *onset* events\n\nIn the sections below, nightly timeseries line charts for *enmo*, *anglez*, and *events* are visualized for the selective series_id below\n- '0a96f4993bd7' (13 nights)\n- '08db4255286f' (25 nights)","metadata":{}},{"cell_type":"code","source":"# inspired by \n# - https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\n# - https://www.kaggle.com/code/carlmcbrideellis/zzzs-eda-utc-or-not-utc\ndef get_series_data(series_id):\n \n train_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM train_series WHERE series_id = '{series_id}'\n \"\"\"\n df = con.execute(train_series_sql).fetchdf()\n \n #df['date_time'] = pd.to_datetime(df['timestamp'], utc=True)\n df['date_time'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n df = df.drop('timestamp', axis=1)\n \n train_events_sql = f\"\"\"\n SELECT \n series_id,\n CAST (night as INTEGER) as night,\n event,\n CAST (step as INTEGER) as step,\n timestamp\n FROM train_events \n WHERE series_id = '{series_id}'\n \"\"\"\n \n ev = con.execute(train_events_sql).fetchdf()\n #ev['date_time'] = pd.to_datetime(ev['timestamp'], utc=True)\n ev['date_time'] = pd.to_datetime(ev['timestamp']).apply(lambda t: t.tz_localize(None))\n ev = ev.drop('timestamp', axis=1)\n \n df2 = pd.merge(df, ev, on='date_time', how='outer')\n df2 = df2.drop(['series_id_x','series_id_y','step_x','step_y'], axis=1)\n df2['event'] = df2['event'].fillna(0)\n \n # fill values for night feature\n df2['night'][0] = 1\n df2['night'] = df2['night'].interpolate()\n df2['night'] = round(df2['night'])\n # map numeric values to event feature\n df2.loc[df2['event']=='onset','event'] = -1\n df2.loc[df2['event']=='wakeup','event'] = 1\n \n return df2\n\ndef get_test_series_data(series_id):\n test_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM test_series WHERE series_id = '{series_id}'\n \"\"\"\n df = con.execute(test_series_sql).fetchdf()\n \n #df['date_time'] = pd.to_datetime(df['timestamp'], utc=True)\n df['date_time'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n df = df.drop('timestamp', axis=1)\n \n return df\n\n# inspired by https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\ndef visualize_nights_ts(series_id):\n series_data_df = get_series_data(series_id)\n for i in set(series_data_df['night']):\n df22 = series_data_df[series_data_df['night']==i]\n df22['anglez'] = df22['anglez'] / max(abs(max(df22['anglez'])),abs(min(df22['anglez'])))\n df22['enmo'] = df22['enmo'] / max(abs(max(df22['enmo'])),abs(min(df22['enmo'])))\n df22['event'] = df22['event'] / max(abs(max(df22['event'])),abs(min(df22['event'])),1e-10)\n if df22['event'].max() == 1:\n display(Markdown('### Control chart: ' + series_id + ', night ' + str(int(i)) ))\n fig = plt.figure(figsize=(10,5))\n ax = fig.add_subplot()\n ax.plot(df22['date_time'], df22['anglez'], color='k', linewidth=0.5, label=\"anglez\")\n ax.plot(df22['date_time'], df22['enmo'], color='b', linewidth=0.5, label=\"enmo\")\n ax.plot(df22['date_time'], df22['event'], color='r', label=\"event\")\n ax.legend()\n ax.set_title(\"\".join(['Series Id : ', series_id, ' [Night# ', str(int(i)), ']']))\n ax.set_xlabel('DateTime')\n ax.set_ylabel('Normalized anglez, enmo, event [-]')\n ax.xaxis.set_major_locator(mdates.HourLocator(byhour=range(0, 24, 3)))\n ax.xaxis.set_minor_locator(mdates.HourLocator(byhour=range(0, 24, 1)))\n ax.xaxis.set_major_formatter(mdates.DateFormatter(\"%Y-%m-%d\\n%H:%M:%S\"))\n labels = ax.get_xticklabels()\n plt.setp(labels, rotation=40, fontsize=8)\n plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:05.276726Z","iopub.execute_input":"2023-09-29T07:56:05.277204Z","iopub.status.idle":"2023-09-29T07:56:05.295213Z","shell.execute_reply.started":"2023-09-29T07:56:05.277159Z","shell.execute_reply":"2023-09-29T07:56:05.293829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Nights View: Series '0a96f4993bd7'
","metadata":{}},{"cell_type":"code","source":"series_id = '0a96f4993bd7'\nvisualize_nights_ts(series_id)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:05.296835Z","iopub.execute_input":"2023-09-29T07:56:05.297299Z","iopub.status.idle":"2023-09-29T07:56:19.526151Z","shell.execute_reply.started":"2023-09-29T07:56:05.297256Z","shell.execute_reply":"2023-09-29T07:56:19.524271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:19.528806Z","iopub.execute_input":"2023-09-29T07:56:19.529770Z","iopub.status.idle":"2023-09-29T07:56:19.752155Z","shell.execute_reply.started":"2023-09-29T07:56:19.529722Z","shell.execute_reply":"2023-09-29T07:56:19.750713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Nights View: Series '08db4255286f'
","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nvisualize_nights_ts(series_id)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:19.754078Z","iopub.execute_input":"2023-09-29T07:56:19.754840Z","iopub.status.idle":"2023-09-29T07:56:49.823628Z","shell.execute_reply.started":"2023-09-29T07:56:19.754789Z","shell.execute_reply":"2023-09-29T07:56:49.822360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:49.825431Z","iopub.execute_input":"2023-09-29T07:56:49.826538Z","iopub.status.idle":"2023-09-29T07:56:50.119534Z","shell.execute_reply.started":"2023-09-29T07:56:49.826505Z","shell.execute_reply":"2023-09-29T07:56:50.118203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Temporal View: OHLC Resampling of Training Series TS
\n\n\nIn finance (and some other industries as well), a popular way to aggregate a time series is to implement **Open-high-low-close(OHLC) resampling**.\n\nThe essence of this technique is to to compute four values for each bucket (time interval) as follows\n- the *first(open)* value, \n- the *last(close)* value, \n- the *maximum(high)* value, and \n- the *minimal(low)* values \n\nLet's apply OHLC resampling to visualize the correlated *anglez, enmo, and event* trends on a single diagram, for selected training series.","metadata":{}},{"cell_type":"code","source":"# inspired by https://www.kaggle.com/code/fengvm/accelerate-time-series-data-processing\ndef plot_series_ohlc(series_id, freq='1min', nights=2):\n \"\"\"\n freq = '1min' : Sampling frequency, Optional 5s, 1min, 5min, 1H\n \"\"\"\n series_df = get_series_data(series_id)\n series = series_df[series_df['night'] <= nights] # Only display data for a specified number of nights (days)\n series = series.set_index('date_time')\n \n tss_anglez = series['anglez'].resample(freq).ohlc()\n tss_enmo = series['enmo'].resample(freq).ohlc()\n \n fig = make_subplots(rows=3, cols=1, subplot_titles=('OHLC: anglez',\n 'OHLC: enmo',\n 'Event timeline',\n ))\n\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['open'], name='anglez: opening'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['high'], name='anglez: high'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['low'], name='anglez: low'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['close'], name='anglez: close'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['open'], name='enmo: open'), row=2, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['high'], name='enmo: high'), row=2, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['low'], name='enmo: low'), row=2, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['close'], name='enmo: close'), row=2, col=1)\n fig.add_trace(go.Scatter(x=series.index, y=series['event'], name='events'), row=3, col=1)\n \n # Update visual layout\n fig.update_layout(\n showlegend=True,\n width=600,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n )\n\n # update font size at the axes\n fig.update_coloraxes(colorbar_tickfont_size=10)\n # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n fig.update_annotations(font_size=12)\n # Reduce opacity\n fig.update_traces(opacity=0.75)\n \n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:50.121490Z","iopub.execute_input":"2023-09-29T07:56:50.122166Z","iopub.status.idle":"2023-09-29T07:56:50.138319Z","shell.execute_reply.started":"2023-09-29T07:56:50.122130Z","shell.execute_reply":"2023-09-29T07:56:50.137084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
OHLC View: Series '08db4255286f'
\n\nLet's build OHLC charts for initial *5* nights in series *'08db4255286f'*, resampling by the *15-min* intervals.","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nfig = plot_series_ohlc(series_id, freq='15min', nights=5)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:50.140019Z","iopub.execute_input":"2023-09-29T07:56:50.141101Z","iopub.status.idle":"2023-09-29T07:56:58.649125Z","shell.execute_reply.started":"2023-09-29T07:56:50.141051Z","shell.execute_reply":"2023-09-29T07:56:58.648182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the charts above, \n- curves of high and low *anglez* display drastic going up around *wakeup* events\n- curves of high and low *anglez* display going down around *onset* events\n- curves of high and low *enmo* display moderate going up around *wakeup* events\n- curves of high and low *enmo* display going down around *onset* events","metadata":{}},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:58.650680Z","iopub.execute_input":"2023-09-29T07:56:58.651746Z","iopub.status.idle":"2023-09-29T07:56:58.786894Z","shell.execute_reply.started":"2023-09-29T07:56:58.651708Z","shell.execute_reply":"2023-09-29T07:56:58.786054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
OHLC View: Series '0a96f4993bd7'
\n\nLet's build OHLC charts for initial *5* nights in series *'0a96f4993bd7'*, resampling by the *15-min* intervals.","metadata":{}},{"cell_type":"code","source":"series_id = '0a96f4993bd7'\nfig = plot_series_ohlc(series_id, freq='15min', nights=5)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:56:58.796284Z","iopub.execute_input":"2023-09-29T07:56:58.796760Z","iopub.status.idle":"2023-09-29T07:57:03.287213Z","shell.execute_reply.started":"2023-09-29T07:56:58.796724Z","shell.execute_reply":"2023-09-29T07:57:03.285825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:03.288617Z","iopub.execute_input":"2023-09-29T07:57:03.289000Z","iopub.status.idle":"2023-09-29T07:57:03.431396Z","shell.execute_reply.started":"2023-09-29T07:57:03.288968Z","shell.execute_reply":"2023-09-29T07:57:03.430509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe the same trends as above.","metadata":{}},{"cell_type":"markdown","source":"#
OHLC Resampling of Test Series Data
","metadata":{}},{"cell_type":"code","source":"def plot_test_series_ohlc(series_id, freq='1min'):\n \"\"\"\n freq = '1min' : Sampling frequency, Optional 5s, 1min, 5min, 1H\n \"\"\"\n series = get_test_series_data(series_id)\n\n series = series.set_index('date_time')\n \n tss_anglez = series['anglez'].resample(freq).ohlc()\n tss_enmo = series['enmo'].resample(freq).ohlc()\n \n fig = make_subplots(rows=2, cols=1, subplot_titles=(\n 'OHLC: anglez',\n 'OHLC: enmo',\n ))\n\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['open'], name='anglez: opening'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['high'], name='anglez: high'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['low'], name='anglez: low'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['close'], name='anglez: close'), row=1, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['open'], name='enmo: open'), row=2, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['high'], name='enmo: high'), row=2, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['low'], name='enmo: low'), row=2, col=1)\n fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['close'], name='enmo: close'), row=2, col=1)\n \n # Update visual layout\n fig.update_layout(\n showlegend=True,\n width=600,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n )\n\n # update font size at the axes\n fig.update_coloraxes(colorbar_tickfont_size=10)\n # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n fig.update_annotations(font_size=12)\n # Reduce opacity\n fig.update_traces(opacity=0.75)\n \n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:03.433299Z","iopub.execute_input":"2023-09-29T07:57:03.433923Z","iopub.status.idle":"2023-09-29T07:57:03.448394Z","shell.execute_reply.started":"2023-09-29T07:57:03.433886Z","shell.execute_reply":"2023-09-29T07:57:03.446563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for series_id in test_series_id_list:\n display(Markdown(\"##
OHLC Resampling: Series \" + series_id + \"
\" ))\n fig = plot_test_series_ohlc(series_id, freq='1min')\n fig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:58:48.157187Z","iopub.execute_input":"2023-09-29T07:58:48.157566Z","iopub.status.idle":"2023-09-29T07:58:48.371210Z","shell.execute_reply.started":"2023-09-29T07:58:48.157536Z","shell.execute_reply":"2023-09-29T07:58:48.370171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:03.664092Z","iopub.execute_input":"2023-09-29T07:57:03.664438Z","iopub.status.idle":"2023-09-29T07:57:03.797544Z","shell.execute_reply.started":"2023-09-29T07:57:03.664410Z","shell.execute_reply":"2023-09-29T07:57:03.796253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Polar View on Training Series Data
","metadata":{}},{"cell_type":"code","source":"def plot_polar_views(series_id):\n\n series_df = get_series_data(series_id)\n fig1 = px.scatter_polar(series_df, r=\"enmo\", theta=\"anglez\",\n color=\"event\", symbol=\"event\", size=\"enmo\",\n color_discrete_sequence=px.colors.sequential.Turbo,\n title=f\"Polar view on enmo/anglez: series {series_id}, all observations\")\n \n \n events_df = series_df[series_df['event'] != 0]\n fig2 = px.scatter_polar(events_df, r=\"enmo\", theta=\"anglez\",\n color=\"event\", symbol=\"event\", size=\"enmo\",\n color_discrete_sequence=px.colors.sequential.Turbo,\n title=f\"Polar view on enmo/anglez at event spots: series {series_id}, (onset=-1, awake=1)\")\n \n return fig1, fig2","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:03.798983Z","iopub.execute_input":"2023-09-29T07:57:03.799317Z","iopub.status.idle":"2023-09-29T07:57:03.813411Z","shell.execute_reply.started":"2023-09-29T07:57:03.799288Z","shell.execute_reply":"2023-09-29T07:57:03.812185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Polar View: Series '0a96f4993bd7'
","metadata":{}},{"cell_type":"code","source":"series_id = '0a96f4993bd7'\nfig1, fig2 = plot_polar_views(series_id)\ndisplay(Markdown('### Entire records for the series'))\nfig1.show()\ndisplay(Markdown('### Wakeup/Onset spots for the series'))\nfig2.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:03.815318Z","iopub.execute_input":"2023-09-29T07:57:03.816387Z","iopub.status.idle":"2023-09-29T07:57:07.433831Z","shell.execute_reply.started":"2023-09-29T07:57:03.816329Z","shell.execute_reply":"2023-09-29T07:57:07.432918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:07.435294Z","iopub.execute_input":"2023-09-29T07:57:07.435618Z","iopub.status.idle":"2023-09-29T07:57:07.573196Z","shell.execute_reply.started":"2023-09-29T07:57:07.435591Z","shell.execute_reply":"2023-09-29T07:57:07.572011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Polar View: Series '08db4255286f'
","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nfig1, fig2 = plot_polar_views(series_id)\ndisplay(Markdown('### Entire records for the series'))\nfig1.show()\ndisplay(Markdown('### Wakeup/Onset spots for the series'))\nfig2.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:07.575070Z","iopub.execute_input":"2023-09-29T07:57:07.575692Z","iopub.status.idle":"2023-09-29T07:57:15.347078Z","shell.execute_reply.started":"2023-09-29T07:57:07.575621Z","shell.execute_reply":"2023-09-29T07:57:15.345247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:15.349070Z","iopub.execute_input":"2023-09-29T07:57:15.349780Z","iopub.status.idle":"2023-09-29T07:57:15.496314Z","shell.execute_reply.started":"2023-09-29T07:57:15.349692Z","shell.execute_reply":"2023-09-29T07:57:15.495004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Polar View: Series '1087d7b0ff2e'
","metadata":{}},{"cell_type":"code","source":"series_id = '1087d7b0ff2e'\nfig1, fig2 = plot_polar_views(series_id)\ndisplay(Markdown('### Entire records for the series'))\nfig1.show()\ndisplay(Markdown('### Wakeup/Onset spots for the series'))\nfig2.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:15.498994Z","iopub.execute_input":"2023-09-29T07:57:15.500522Z","iopub.status.idle":"2023-09-29T07:57:22.588836Z","shell.execute_reply.started":"2023-09-29T07:57:15.500468Z","shell.execute_reply":"2023-09-29T07:57:22.587597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:22.590798Z","iopub.execute_input":"2023-09-29T07:57:22.592100Z","iopub.status.idle":"2023-09-29T07:57:22.734819Z","shell.execute_reply.started":"2023-09-29T07:57:22.592039Z","shell.execute_reply":"2023-09-29T07:57:22.733543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Spectral Analysis of Training Series Data
\n\nAs mentioned by [@andradaolteanu]() in her [notebook](https://www.kaggle.com/code/andradaolteanu/zzz-in-depth-explore-with-spectrograms-lines), this competition can be potentially tackled from multiple angles:\n\n- it can be set up as a tabular classification problem (the most popular approach in the public notebooks so far)\n- it can be an LSTM-RNN classifier using the sequential (time) data\n- it can be even a computer vision problem (*if we convert the time data to spectrograms*)\n\nThis chapter is dedicated to building and visualizing the spectrograms for the *anglez/enmo* data in the training seria.\n\n##
Understanding spectrograms
\n\nA spectrogram is a visual representation of the Short-Time Fourier Transform (see [this thread](https://stackoverflow.com/questions/29321696/what-is-a-spectrogram-and-how-do-i-set-its-parameters) for moree info). You can think of it as taking chunks of an input signal and applying a local Fourier Transform on each chunk. Each chunk has a specified width and you apply a Fourier Transform to this chunk. You should take note that each chunk has an associated frequency distribution. For each chunk that is centred at a specific time point in your time signal, you get a bunch of frequency components. The collection of all of these frequency components at each chunk and plotted all together is what is essentially a spectrogram.\n\nThe spectrogram is a 2D visual heat map where the horizontal axis represents the time of the signal and the vertical axis represents the frequency axis. What is visualized is an image where darker colours means that for a particular time point and a particular frequency, the lower in magnitude the frequency component is, the darker the colour. Similarly, the higher in magnitude the frequency component is, the lighter the colour.","metadata":{}},{"cell_type":"code","source":"# inspired by https://www.kaggle.com/code/andradaolteanu/zzz-in-depth-explore-with-spectrograms-lines#3.-Spectrogram-analysis\ndef create_spectrogram(value):\n \n # defines the width of each chunk in terms of samples\n # large window size -> better frequency resolution -> poor time localization\n window_size = 256 \n # ensures that each chunk has a certain number of samples \n # that are overlapping for each chunk\n # default is 0.5*window\n overlap = 128\n # how many FFT points are desired to be computed per chunk\n # how fine-grained the frequency resolution will be\n nfft = 256 \n # sampling frequency of your signal\n # default is 1\n fs = 1.0 \n\n frequencies, times, Sxx = spectrogram(value, fs=fs, nperseg=window_size, \n noverlap=overlap, nfft=nfft)\n return frequencies, times, Sxx\n\n\ndef build_spectrogram(series_id):\n \n series_data_df = get_series_data(series_id)\n for i in set(series_data_df['night']):\n spec_data = series_data_df[series_data_df['night']==i]\n display(Markdown('### Spectrogram: ' + series_id + ', night ' + str(int(i)) ))\n \n fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 10))\n axs = [ax1, ax2]\n fig.suptitle(f'A night for id: {series_id}')\n \n k=0\n\n feat = 'anglez'\n f, t, s = create_spectrogram(spec_data[feat].values)\n axs[k].pcolormesh(t, f, 10 * np.log10(s), cmap=\"RdYlBu\")\n axs[k].set_xlabel('Time (s)')\n axs[k].set_ylabel('Frequency (Hz)')\n axs[k].set_title(f'Spectrogram of {feat}')\n k = k+1\n \n feat = 'enmo'\n f, t, s = create_spectrogram(spec_data[feat].values)\n axs[k].pcolormesh(t, f, 10 * np.log10(s), cmap=\"RdYlBu\")\n axs[k].set_xlabel('Time (s)')\n axs[k].set_ylabel('Frequency (Hz)')\n axs[k].set_title(f'Spectrogram of {feat}')\n \n plt.show()\n ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:22.736838Z","iopub.execute_input":"2023-09-29T07:57:22.737220Z","iopub.status.idle":"2023-09-29T07:57:22.749442Z","shell.execute_reply.started":"2023-09-29T07:57:22.737189Z","shell.execute_reply":"2023-09-29T07:57:22.748472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Spectrogram: Series '08db4255286f'
","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nbuild_spectrogram(series_id)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:22.750629Z","iopub.execute_input":"2023-09-29T07:57:22.751769Z","iopub.status.idle":"2023-09-29T07:57:43.894177Z","shell.execute_reply.started":"2023-09-29T07:57:22.751733Z","shell.execute_reply":"2023-09-29T07:57:43.893100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:43.896237Z","iopub.execute_input":"2023-09-29T07:57:43.897246Z","iopub.status.idle":"2023-09-29T07:57:44.229745Z","shell.execute_reply.started":"2023-09-29T07:57:43.897199Z","shell.execute_reply":"2023-09-29T07:57:44.228297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Spectrogram: Series '0a96f4993bd7'
","metadata":{}},{"cell_type":"code","source":"series_id = '0a96f4993bd7'\nbuild_spectrogram(series_id)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:44.231391Z","iopub.execute_input":"2023-09-29T07:57:44.231771Z","iopub.status.idle":"2023-09-29T07:57:56.049609Z","shell.execute_reply.started":"2023-09-29T07:57:44.231741Z","shell.execute_reply":"2023-09-29T07:57:56.048369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:56.051092Z","iopub.execute_input":"2023-09-29T07:57:56.051727Z","iopub.status.idle":"2023-09-29T07:57:56.267103Z","shell.execute_reply.started":"2023-09-29T07:57:56.051689Z","shell.execute_reply":"2023-09-29T07:57:56.265542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Clustering Training Seria
\n\nLet's look at the training series through the prism of the summary statistics for *anglez* and *enmo* across the individual series.\n\nFirst of all, let's extract and review the summary statistics on *enmo* for every training series","metadata":{}},{"cell_type":"code","source":"df_series_enmo = get_all_series_boxplot_data('train_series', 'enmo')\ndf_series_enmo.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:57:56.269117Z","iopub.execute_input":"2023-09-29T07:57:56.269625Z","iopub.status.idle":"2023-09-29T07:58:04.470567Z","shell.execute_reply.started":"2023-09-29T07:57:56.269581Z","shell.execute_reply":"2023-09-29T07:58:04.469334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's extract and review the summary statistics on *anglez* for every training series","metadata":{}},{"cell_type":"code","source":"df_series_anglez = get_all_series_boxplot_data('train_series', 'anglez')\ndf_series_anglez.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-29T07:58:04.472630Z","iopub.execute_input":"2023-09-29T07:58:04.473543Z","iopub.status.idle":"2023-09-29T07:58:12.588197Z","shell.execute_reply.started":"2023-09-29T07:58:04.473496Z","shell.execute_reply":"2023-09-29T07:58:12.586967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"TBD","metadata":{}},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T07:58:12.589985Z","iopub.execute_input":"2023-09-29T07:58:12.590578Z","iopub.status.idle":"2023-09-29T07:58:12.735542Z","shell.execute_reply.started":"2023-09-29T07:58:12.590543Z","shell.execute_reply":"2023-09-29T07:58:12.734149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
References
\n\n- https://www.datacamp.com/blog/an-introduction-to-duckdb-what-is-it-and-why-should-you-use-it\n- https://duckdb.org/docs/sql/functions/timestamp\n- https://duckdb.org/docs/sql/functions/dateformat\n- https://duckdb.org/docs/sql/aggregates\n- https://pub.towardsai.net/3-best-often-better-alternatives-to-histograms-b588f3850ce8\n- https://towardsdatascience.com/probability-distribution-functions-demystified-a6e882759af3\n- https://cs.stackexchange.com/questions/118065/build-a-histogram-from-a-very-large-data-sample\n- https://chart-studio.plotly.com/~chris/18051/#code\n- https://community.plotly.com/t/boxplot-with-precomputed-quartiles-without-displaying-data/71031","metadata":{}},{"cell_type":"markdown","source":"#
Appendix: Methodology of EDA for large datasets
\n\n##
Concepts
\n\nTBD\n\n##
Implementation Notes
\n\nTBD","metadata":{}}]}