{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Guidebook for the Child Mind Institute - Detect Sleep States Competition \nmiltiades-the-general @2023\n\n### Competition Overview\nMerger Deadline: November 28, 2023  \nGoal: Detect sleep/wake states from wrist-worn accelerometer data.  \nEvaluation: Evaluated on **Average Precision** of predicted events based on timestamp error tolerance thresholds and event classes.  \n\n### Data\n**train_events.csv**, **train_series.parquet**, **test_series.parquet**  \n\n<u>train_events</u>: \nProvides the target of the training data in the form of event, given a step and timestamp. \nseries_id:str | night:int | event:str | step:int | timestamp:datetime\n\ncolumns:  \nseries_id: Unique identifier for each accelerometor series. Series are defined as continuous recordings for a test subject.  \nnight: an enumeration of potential onset/wake event pairs. At most, one event pair can occur per night.  \nevent: The type of event, either onset or wakeup.  \nstep: Integer timestep for each observation in the events. Corresponds to the step for train_series.   \ntimestamp: time when each event occurred.  \n\nnull values: series_id: 0 | night: 0 | event: 0 | step: 4923 | timestamp: 4923\n\npercent null columns: 33.9%\n\nvalue counts:  event- onset, wakeup (7254, 7254)\n\n<u>train_series</u>  \nProvides the main features of the training data.  \nseries_id:str | step:int | timestamp:datetime | anglez:float | enmo:float\n\ncolumns:  \nseries_id: Unique identifier. To be matched on train_events dataframe.  \nstep: Integer timestep for each observation in the series.  \ntimestamp: A corresponding datetime with ISO 8601 format %Y-%m-%dT%H:%M:%S%z.  \nanglez: z-angle is a metric derived from individual accelerometer components that is commonly used in sleep detection, and refers to the angle of the arm relative to the vertical axis of the body.  \nenmo: Euclidean Norm Minus One of all accelerometer signals, with negative values rounded to zero. Measures acceleration. \n\n<u>test_series</u>  \nContains the same columns as train_series. ","metadata":{}},{"cell_type":"markdown","source":"### Import Libraries\n\nImport standard libraries, etc. for data preprocessing and features engineering.\n\nUse polars for large parquet format data frames. Parquet uses lazy processes to make the data preprocessing much faster than pandas for large dataframes. ","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport random\nimport gc\nfrom tqdm import tqdm\nimport datetime\n# Uses lazy execution which is more efficient on large data\nimport polars as pl","metadata":{"execution":{"iopub.status.busy":"2023-09-26T18:31:04.974727Z","iopub.execute_input":"2023-09-26T18:31:04.975209Z","iopub.status.idle":"2023-09-26T18:31:07.789084Z","shell.execute_reply.started":"2023-09-26T18:31:04.975172Z","shell.execute_reply":"2023-09-26T18:31:07.787751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define a class called Config\n\nConfig will hold constants and variables for the data preparation, and model.  \nIf hyperparameters are needed, add them to the Config class and change them in Config when necessary. If evaluating hyperparameters iteratively, do not add them to config. Iterate them in the function. ","metadata":{}},{"cell_type":"code","source":"class Config:\n    # Define data paths here\n    base_path = \"/kaggle/input/child-mind-institute-detect-sleep-states/\"\n    train_events_csv = str(base_path + \"train_events.csv\")\n    train_series_parquet = str(base_path + \"train_series.parquet\")\n    test_series_parquet = str(base_path + \"test_series.parquet\")\n    sample_submission_csv = str(base_path + \"sample_submission.csv\")\n    \n    # Data \n    sample_size = 1e6","metadata":{"execution":{"iopub.status.busy":"2023-09-26T18:31:07.791450Z","iopub.execute_input":"2023-09-26T18:31:07.792371Z","iopub.status.idle":"2023-09-26T18:31:07.800673Z","shell.execute_reply.started":"2023-09-26T18:31:07.792330Z","shell.execute_reply":"2023-09-26T18:31:07.798671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Loading and Preprocessing\n\nData can be loaded into a polars dataframe object. Cast preprocessing functions onto the dataframes. These may be in the form of type casting, augmentation, or aggregation. For the train_series data the z-angle and enmo, the data may be aggregated by series_id and relative timestamp. \n\nThe timestamp column for the train_events, and train_series data is split up into a year, month, day, hour column to make time comparisons more straightforward.","metadata":{}},{"cell_type":"code","source":"# ---Data Transforms---\n\n# Transform the datetime column to a year, month, day, hour column \ndatetime_transforms = [\n    pl.col('timestamp').str.to_datetime(),\n    (pl.col('timestamp').str.to_datetime().dt.year()-2000).cast(pl.UInt8).alias('year'),\n    pl.col('timestamp').str.to_datetime().dt.month().cast(pl.UInt8).alias('month'),\n    pl.col('timestamp').str.to_datetime().dt.day().cast(pl.UInt8).alias('day'),\n    pl.col('timestamp').str.to_datetime().dt.hour().cast(pl.UInt8).alias('hour'),\n]\n\n# Transform the enmo, and z-angle data in the train_series dataframe \nseries_data_transforms = [\n    pl.col('anglez').cast(pl.Int16),\n    (pl.col('enmo')*1000).cast(pl.UInt16)\n]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ---Load Series and Event Data---\n\n# Load series data for train and test, cast the appropriate transforms onto the dataframes\ntrain_series = pl.scan_parquet(Config.train_series_parquet).with_columns(datetime_transforms + series_data_transforms)\ntest_series = pl.scan_parquet(Config.test_series_parquet)\n\n# Load event data, cast appropriate transforms\ntrain_events = pl.read_csv(Config.train_events_csv).with_columns(datetime_transforms)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the series_ids from the events dataframe, droppping null values\nseries_ids = train_events.drop_nulls()['series_id'].unique(maintain_order=True).to_list()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ---Feature Engineering---\n\n# Feature data will be aggregated by the hour \nfeatures, feature_cols = [pl.col('hour')], ['hour']\n\n# Aggregate the feature data by 30 mins and 120 mins\n# Use mean and max as aggregate functions\n# Explore different aggregation methods in the future\nfor mins in [30, 120]:\n    features += [\n        pl.col('enmo').rolling_mean(12 * mins, center=True, min_periods=1).abs().cast(pl.UInt16).alias(f'enmo_{mins}m_mean'),\n        pl.col('enmo').rolling_max(12 * mins, center=True, min_periods=1).abs().cast(pl.UInt16).alias(f'enmo_{mins}m_max'),\n    ]\n    \n    # Add new cols to feature columns\n    feature_cols += [\n        f'enmo_{mins}m_mean', f'enmo_{mins}m_max'\n    ]\n    \n    for var in ['anglez', 'enmo']:\n        # Aggregate the difference by rolling mean and max\n        features += [\n            (pl.col(var).diff().abs().rolling_mean(12 * mins, center=True, min_periods=1) * 10).abs().cast(pl.UInt32).alias(f'{var}_1v_{mins}m_mean'),\n            (pl.col(var).diff().abs().rolling_max(12 * mins, center=True, min_periods=1) * 10).abs().cast(pl.UInt32).alias(f'{var}_1v_{mins}m_max'),\n        ]\n        \n        feature_cols += [\n            f'{var}_1v_{mins}m_mean', f'{var}_1v_{mins}m_max'\n        ]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the columns used to identify the data to do matching on, included\n# in the series data\nid_cols = ['series_id', 'step', 'timestamp']\n\n# Cast the feature engineering steps onto the train_series and test_series, \n# select all of the columns needed\ntrain_series = train_series.with_columns(\n    features\n).select(id_cols + feature_cols)\n\ntest_series = test_series.with_columns(\n    features\n).select(id_cols + feature_cols)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ---Define a function to make a train dataset--\ndef make_train_dataset(train_series, train_events, drop_nulls=False):\n    \"\"\"\n    Takes data from the train_series, and train_events dataframes and returns the X, y values.\n    X values are the feature data from the series, y values are the target data from the events. \n    y data corresponds to sleep phase in between onset, and wakeup in the event columns. \n    \"\"\"\n    series_ids = train_series['series_id'].unique(maintain_order=True).to_list()\n    \n    X, y = pl.DataFrame(), pl.DataFrame()\n    \n    for ids in tqdm(series_ids):\n        # Take the appropriate data based on current series id\n        # Normalize the feature data\n        sample = train_series.filter(pl.col('series_id') == ids).with_columns(\n            [(pl.col(col) / pl.col(col).std()).cast(pl.Float32) for col in feature_cols if col != 'hour']\n        )\n        \n        # Get the applicable events from the train_events dataframe\n        events = train_events.filter(pl.col('series_id') == ids)\n        \n        if drop_nulls:\n            # Remove datapoints on times where no data was recorded\n            sample = sample.filter(\n                pl.col('timestamp').dt.date().is_in(events['timestamp'].dt.date())\n            )\n        \n        X = X.vstack(sample[id_cols + feature_cols])\n        \n        onsets = events.filter((pl.col('event') == 'onset') & (pl.col('step') != None))['step'].to_list()\n        wakeups = events.filter((pl.col('event') == 'wakeup') & (pl.col('step') != None))['step'].to_list()\n        \n        # NOTE: This will break if there are event series without any recorded onsets or wakeups\n        y = y.vstack(sample.with_columns(\n                sum([(onset <= pl.col('step')) & (pl.col('step') <= wakeup) for onset, wakeup in zip(onsets, wakeups)]).cast(pl.Boolean).alias('asleep')\n            ).select('asleep')\n        )\n    y = y.to_numpy().ravel()\n    \n    return X, y","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ---Takes a sample of the train_series, define x, y values for train set---\n# Filter the valid series_ids and take a sample of size \"sample_size\"\ntrain_series = train_series.filter(pl.col('series_id').is_in(series_ids)).collect().sample(Config.sample_size)\n\n# Create the x and y values for the training set\ntrain_X, train_y = make_train_dataset(train_series, train_events)\n\ndel train_data\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Build Classifier Model and apply it to the data","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Use the Classifier Model to Predict the test data and produce a submission_df","metadata":{}},{"cell_type":"code","source":"def get_events(series, classifier) :\n    '''\n    Takes a time series and a classifier and returns a formatted submission dataframe.\n    '''\n    series_ids = series['series_id'].unique(maintain_order=True).to_list()\n    events = pl.DataFrame(schema={'series_id':str, 'step':int, 'event':str, 'score':float})\n\n    for idx in tqdm(series_ids) : \n        scale_cols = [col for col in feature_cols if (col != 'hour') & (series[col].std() !=0)]\n        # Collecting sample and normalizing features\n        X = series.filter(pl.col('series_id') == idx).select(id_cols + feature_cols).with_columns(\n            [(pl.col(col) / series[col].std()).cast(pl.Float32) for col in scale_cols]\n        )\n\n        # Applying classifier to get predictions and scores\n        preds =  classifier.predict(X[feature_cols])\n        probs = classifier.predict_proba(X[feature_cols])[:, 1]\n\n        #NOTE: Considered using rolling max to get sleep periods excluding <30 min interruptions, but ended up decreasing performance\n        X = X.with_columns(\n            pl.lit(preds).cast(pl.Int8).alias('prediction'), \n            pl.lit(probs).alias('probability')\n                        )\n        \n        # Getting predicted onset and wakeup time steps\n        pred_onsets = X.filter(X['prediction'].diff() > 0)['step'].to_list()\n        pred_wakeups = X.filter(X['prediction'].diff() < 0)['step'].to_list()\n        \n        if len(pred_onsets) > 0 : \n            \n            # Ensuring all predicted sleep periods begin and end\n            if min(pred_wakeups) < min(pred_onsets) : \n                pred_wakeups = pred_wakeups[1:]\n\n            if max(pred_onsets) > max(pred_wakeups) :\n                pred_onsets = pred_onsets[:-1]\n\n            # Keeping sleep periods longer than 30 minutes\n            sleep_periods = [(onset, wakeup) for onset, wakeup in zip(pred_onsets, pred_wakeups) if wakeup - onset >= 12 * 30]\n\n            for onset, wakeup in sleep_periods :\n                # Scoring using mean probability over period\n                score = X.filter((pl.col('step') >= onset) & (pl.col('step') <= wakeup))['probability'].mean()\n\n                # Adding sleep event to dataframe\n                events = events.vstack(pl.DataFrame().with_columns(\n                    pl.Series([idx, idx]).alias('series_id'), \n                    pl.Series([onset, wakeup]).alias('step'),\n                    pl.Series(['onset', 'wakeup']).alias('event'),\n                    pl.Series([score, score]).alias('score')\n                ))\n\n    # Adding row id column\n    events = events.to_pandas().reset_index().rename(columns={'index':'row_id'})\n\n    return events","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a submission_df and export it to a .csv file for submission\nsubmission_df = get_events(test_series, my_classifier)","metadata":{},"execution_count":null,"outputs":[]}]}