{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport plotly.graph_objects as go\nimport gc\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nfrom warnings import simplefilter\nsimplefilter(action=\"ignore\", category=pd.errors.PerformanceWarning)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-28T15:01:24.118384Z","iopub.execute_input":"2023-10-28T15:01:24.118979Z","iopub.status.idle":"2023-10-28T15:01:24.549190Z","shell.execute_reply.started":"2023-10-28T15:01:24.118934Z","shell.execute_reply":"2023-10-28T15:01:24.547571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load and transform functions","metadata":{}},{"cell_type":"markdown","source":"The function \"feat_eng\" is designed for feature engineering on a DataFrame. It takes a DataFrame \"df\" as input and performs several operations to transform and create new features. Here is a description of what this function does:\n\n**Data Type Conversion**:\n    Converts the 'series_id' column to a categorical data type.\n    Converts the 'timestamp' column to a datetime data type, and removes the time zone information.\n\n**Time-Related Feature Creation**:\n    Creates a new 'hour' column by extracting the hour from the 'timestamp' column.\n    Sorts the DataFrame based on the 'timestamp' and sets 'timestamp' as the index.\n\n**Outlier Limiting**:\n    Limits outliers in the 'enmo' column by clipping values at an upper limit of 4.\n\n**lids**:\n    Calculates a modified 'lids' feature by taking the maximum of 0 and the difference between 'enmo' and 0.02.\n    Performs a rolling sum of 'lids' over a 5-minute window and scales the result to a percentage.\n    Computes the mean of the rolling 'lids' over a 15-minute window. This Feature I got from the great work on [LightGBM + Feature Implementation from Paper](https://www.kaggle.com/code/rimbax/lightgbm-feature-implementation-from-paper).\n    \n**anglezdiffabs**:\nIs a term formed from \"anglez\", that it represents the absolute differences between consecutive values of an angle, likely in degrees or radians. I got the anglez diff abs from the insightul analysis on [Feature Engineering and Random Forest Prediction](https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction)\n\n**Data Type Conversion**:\n    Converts 'enmo' and 'anglez' columns to integer types.\n    Calculates the absolute differences between consecutive values in the 'anglez' column and converts them to a float data type.\n\n**Feature Engineering with Rolling Statistics**:\n    For columns 'enmo', 'anglez', and 'anglezdiffabs', the function calculates various rolling statistics (median, mean, max, min, and variance) over different time periods (60, 360, and 720 seconds).\n    Additionally, it calculates the Median Absolute Deviation (MAD) for the last time period (720 seconds) for 'enmo'.\n    Computes the amplitude (difference between max and min values) and the minimum amplitude over rolling windows.\n    Calculates the maximum absolute differences between consecutive values over rolling windows for both 'anglez' and 'anglezdiffabs'.\n\n**Data Cleaning**:\n    Resets the index of the DataFrame and removes rows with missing values.\n\n**Output**:\n    Returns the modified DataFrame with the newly engineered features.","metadata":{}},{"cell_type":"code","source":"def load_serie_by_id(idx):\n    \n    df = pd.read_parquet(path+'train_series.parquet', filters=[('series_id','=',idx)])\n    df['series_id'] = df['series_id'].astype('category')\n    df['timestamp'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n    \n    return df\n\n\ndef get_df_by_freq(df, freq, col):\n    \n    df = df.copy()\n    df.set_index('timestamp', inplace=True)\n    df_ohlc = df[col].resample(freq).ohlc()\n    df_ohlc.columns = [f'{c}_{col}' for c in df_ohlc.columns]\n    \n    df_ohlc = pd.merge(df_ohlc, df[['event', 'series_id']], left_index=True, right_index=True)\n    \n    return df_ohlc","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:01:24.551436Z","iopub.execute_input":"2023-10-28T15:01:24.551979Z","iopub.status.idle":"2023-10-28T15:01:24.561511Z","shell.execute_reply.started":"2023-10-28T15:01:24.551941Z","shell.execute_reply":"2023-10-28T15:01:24.560251Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot functions","metadata":{}},{"cell_type":"code","source":"def plot_by_freq(df, freq, col):\n    \n    serie_id = df['series_id'].unique()[0]\n    \n    fig = go.Figure()\n\n    # set up trace for wakeup\n    df_wakeup = df[df['event']=='wakeup']\n    fig.add_traces(go.Ohlc(x=df_wakeup.index,\n                    open=df_wakeup[f'open_{col}'], high=df_wakeup[f'high_{col}'],\n                    low=df_wakeup[f'low_{col}'], close=df_wakeup[f'close_{col}'],\n                   increasing_line_color= 'orange', decreasing_line_color= 'orange', name='awake'\n                ))\n\n    # set up traces for onset\n    df_onset = df[df['event']=='onset']\n    fig.add_traces(go.Ohlc(x=df_onset.index,\n                    open=df_onset[f'open_{col}'], high=df_onset[f'high_{col}'],\n                    low=df_onset[f'low_{col}'], close=df_onset[f'close_{col}'],\n                   increasing_line_color= 'blue', decreasing_line_color= 'blue', name='sleeping'\n                ))\n\n    fig.update_layout(title=f'Serie_id: {serie_id} - {freq} OLHC {col}', xaxis_rangeslider_visible=True)\n    fig.show()\n    \n\ndef plot_columns_by_freq(df, freq, columns):\n    \n    for col in columns:\n        df_ohlc = get_df_by_freq(df, freq, col)\n        plot_by_freq(df_ohlc, freq, col)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:01:24.563189Z","iopub.execute_input":"2023-10-28T15:01:24.563671Z","iopub.status.idle":"2023-10-28T15:01:24.579007Z","shell.execute_reply.started":"2023-10-28T15:01:24.563626Z","shell.execute_reply":"2023-10-28T15:01:24.577413Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SIGMA = 720 # 12 * 60\ndef gauss(n=SIGMA,sigma=SIGMA*0.15):\n    # guassian distribution function\n    r = range(-int(n/2),int(n/2)+1)\n    return [1 / (sigma * sqrt(2*pi)) * exp(-float(x)**2/(2*sigma**2)) for x in r]\n\n\n\ndef get_macd(s, hfast=4, hslow=48, macd=15, center=False):\n    \n    sma_fast = s.rolling(hfast*12*60, min_periods=1, center=center).agg('mean')\n    sma_slow = s.rolling(hslow*12*60, min_periods=1, center=center).agg('mean')\n    macd = (sma_fast - sma_slow).rolling(12*macd, min_periods=1, center=center).mean().astype(np.float32)\n    \n    return macd\n\n\ndef macd_feat(df, col='anglezdiffabs', dif=120):\n    \n    s = df[f'{col}']\n    df[f'{col}_macd'] = get_macd(s)\n\n    s = df[f'{col}'].sort_index(ascending=False)\n    df[f'{col}_macd_rev'] = get_macd(s).sort_index()\n    \n    df[f'{col}_spred'] = df[f'{col}_macd']-df[f'{col}_macd_rev']\n    df[f'{col}_spread_diff']=df[f'{col}_spred'].diff(dif)\n    \n#     df.loc[df[f'{col}_spred']<3, 'wakeup_zone'] = 1\n#     df['wakeup_zone'].fillna(0, inplace=True)\n#     df.loc[df[f'{col}_spred']>-3, 'onset_zone'] = 1\n#     df['onset_zone'].fillna(0, inplace=True)\n    \n    return df\n\n\ndef feat_eng(df):\n    \n    df['series_id'] = df['series_id'].astype('category')\n    df['timestamp'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n    df['hour'] = df[\"timestamp\"].dt.hour\n    df['month'] = df[\"timestamp\"].dt.month\n    \n    df.sort_values(['timestamp'], inplace=True)\n    df.set_index('timestamp', inplace=True)\n    \n    # limit outiliers in enmo\n    df['enmo'] = df['enmo'].clip(upper=4)\n    \n    df['lids'] = np.maximum(0., df['enmo'] - 0.02)\n    df['lids'] = df['lids'].rolling(f'{120*5}s', center=True, min_periods=1).agg('sum')\n    df['lids'] = 100 / (df['lids'] + 1)\n    df['lids'] = df['lids'].rolling(f'{360*5}s', center=True, min_periods=1).agg('mean').astype(np.float32)\n    \n    df['lids_sma_fast'] = df['lids'].rolling('2h', center=True, min_periods=1).agg('mean').astype(np.float32)\n    df['lids_sma_macd'] = (df['lids'].rolling('18h', center=True, min_periods=1).agg('mean') - df['lids_sma_fast']\n                                 ).rolling(f'1h', center=True, min_periods=1).agg('mean').astype(np.float32)\n    \n    df[\"enmo\"] = (df[\"enmo\"]*1000).astype(np.int16)\n    df[\"anglez\"] = df[\"anglez\"].astype(np.int16)\n    df[\"anglezdiffabs\"] = df[\"anglez\"].diff().abs().astype(np.float32)\n    \n#     df = macd_feat(df, col='anglezdiffabs')\n#     df = macd_feat(df, col='lids')\n\n#     for col in ['enmo', 'anglez', 'anglezdiffabs']:\n    for col in ['enmo', 'anglez']:\n#         df[f'{col}_sma_fast'] = df[f'{col}'].rolling(f'4h', center=True, min_periods=1).agg('mean').astype(np.float32)\n#         df[f'{col}_sf_macd'] = (df[f'{col}_sma_fast'] - df[f'{col}'].rolling(f'48h', center=True, min_periods=1).agg('mean')\n#                                    ).rolling(f'1h', center=True, min_periods=1).agg('mean').astype(np.float32)\n        \n        # periods in seconds        \n        periods = [720] \n        \n        for n in periods:\n            \n            rol_args = {'window':f'{n+5}s', 'min_periods':10, 'center':True}\n            \n#             for agg in ['median', 'mean', 'max', 'min', 'std']:\n            for agg in ['std']:\n                df[f'{col}_{agg}_{n}'] = df[col].rolling(**rol_args).agg(agg).astype(np.float32).values\n                gc.collect()\n                \n            rol_args = {'window':f'8h', 'min_periods':15, 'center':True}\n            df[f'{col}_std_max_8h'] = df[f'{col}_std_{n}'].rolling(**rol_args).max().astype(np.float32).values\n            \n#             if n == max(periods):\n#                 df[f'{col}_mad_{n}'] = (df[col] - df[f'{col}_median_{n}']).abs().rolling(**rol_args).median().astype(np.float32)\n            \n#             df[f'{col}_amplit_{n}'] = df[f'{col}_max_{n}']-df[f'{col}_min_{n}']\n#             df[f'{col}_amplit_{n}_min'] = df[f'{col}_amplit_{n}'].rolling(**rol_args).min().astype(np.float32).values\n            \n#             df[f'{col}_diff_{n}_max'] = df[f'{col}_max_{n}'].diff().abs().rolling(**rol_args).max().astype(np.float32)\n#             df[f'{col}_diff_{n}_mean'] = df[f'{col}_max_{n}'].diff().abs().rolling(**rol_args).mean().astype(np.float32)\n\n    \n    df.reset_index(inplace=True)\n#     df.dropna(inplace=True)\n\n    return df\n\n\n\n\n\n\n# df = feat_eng_by_id(series_id[0])","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:02:41.811397Z","iopub.execute_input":"2023-10-28T15:02:41.811886Z","iopub.status.idle":"2023-10-28T15:02:41.839498Z","shell.execute_reply.started":"2023-10-28T15:02:41.811849Z","shell.execute_reply":"2023-10-28T15:02:41.837979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading data","metadata":{}},{"cell_type":"code","source":"path = '/kaggle/input/child-mind-institute-detect-sleep-states/'\n\ntrain_events = pd.read_csv(path + 'train_events.csv')\ntrain_events['timestamp'] = pd.to_datetime(train_events['timestamp']).apply(lambda t: t.tz_localize(None))\ncount_na = train_events.groupby('series_id')['timestamp'].apply(lambda x: x.isna().sum())","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:02:43.550807Z","iopub.execute_input":"2023-10-28T15:02:43.551194Z","iopub.status.idle":"2023-10-28T15:02:43.861247Z","shell.execute_reply.started":"2023-10-28T15:02:43.551162Z","shell.execute_reply":"2023-10-28T15:02:43.860287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Anglez standard deviation in 8h","metadata":{}},{"cell_type":"markdown","source":"## Plot series with null events","metadata":{}},{"cell_type":"code","source":"for idx in count_na[count_na>=0].index[:3]:\n# for idx in ['6ca4f4fca6a2']:\n    print(f'ID: {idx}', \"=\"*120, sep='\\n')\n    train_serie = load_serie_by_id(idx)\n    df = train_serie.merge(train_events[train_events['series_id']==idx], on=['series_id', 'timestamp'], how='left').ffill()\n    df = feat_eng(df)\n    plot_columns_by_freq(df, '5min', ['anglez', 'anglez_std_720', 'anglez_std_max_8h'])","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:24:47.838078Z","iopub.execute_input":"2023-10-28T15:24:47.838552Z","iopub.status.idle":"2023-10-28T15:25:57.066313Z","shell.execute_reply.started":"2023-10-28T15:24:47.838513Z","shell.execute_reply":"2023-10-28T15:25:57.064883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot Series without null events","metadata":{}},{"cell_type":"code","source":"for idx in count_na[count_na==0].index[:2]:\n# for idx in ['6ca4f4fca6a2']:\n    print(f'ID: {idx}', \"=\"*120, sep='\\n')\n    train_serie = load_serie_by_id(idx)\n    df = train_serie.merge(train_events[train_events['series_id']==idx], on=['series_id', 'timestamp'], how='left').ffill()\n    df = feat_eng(df)\n    plot_columns_by_freq(df, '5min', ['anglez', 'anglez_std_max_8h'])","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:07:35.529368Z","iopub.execute_input":"2023-10-28T15:07:35.529826Z","iopub.status.idle":"2023-10-28T15:08:14.925115Z","shell.execute_reply.started":"2023-10-28T15:07:35.529762Z","shell.execute_reply":"2023-10-28T15:08:14.923761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot serie with weird patern","metadata":{}},{"cell_type":"code","source":"for idx in ['6ca4f4fca6a2']:\n    print(f'ID: {idx}', \"=\"*120, sep='\\n')\n    train_serie = load_serie_by_id(idx)\n    df = train_serie.merge(train_events[train_events['series_id']==idx], on=['series_id', 'timestamp'], how='left').ffill()\n    df = feat_eng(df)\n    plot_columns_by_freq(df, '5min', ['anglez', 'anglez_std_max_8h'])","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:08:24.533917Z","iopub.execute_input":"2023-10-28T15:08:24.534371Z","iopub.status.idle":"2023-10-28T15:08:55.641288Z","shell.execute_reply.started":"2023-10-28T15:08:24.534331Z","shell.execute_reply":"2023-10-28T15:08:55.640006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot all features grouping by time","metadata":{}},{"cell_type":"code","source":"for idx in count_na[count_na==0].index[:1]:\n    print(f'ID: {idx}', \"=\"*120, sep='\\n')\n    train_serie = load_serie_by_id(idx)\n    df = train_serie.merge(train_events[train_events['series_id']==idx], on=['series_id', 'timestamp'], how='left').ffill()\n    df = feat_eng(df)\n    plot_columns_by_freq(df, '30min', sorted(df.select_dtypes(include=['float32']).columns[:5]))","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:01:58.102626Z","iopub.status.idle":"2023-10-28T15:01:58.103505Z","shell.execute_reply.started":"2023-10-28T15:01:58.103187Z","shell.execute_reply":"2023-10-28T15:01:58.103216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}