{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Feature Engineering and Random Forest Prediction to Detect Sleep States\n\nI took Lucas's public notebook and add Optuna to find better hyperparameters. \n\nThanks to Lucas Burke!\nPlease upvote his great notebook: https://www.kaggle.com/code/lccburk/feature-engineering-and-random-forest-prediction\n","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport polars as pl\nimport datetime \nfrom tqdm import tqdm\n\nimport optuna\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import f1_score\n\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\nfrom metric import score # Import event detection ap score function\n\n# These are variables to be used by the score function\ncolumn_names = {\n    'series_id_column_name': 'series_id',\n    'time_column_name': 'step',\n    'event_column_name': 'event',\n    'score_column_name': 'score',\n}\n\ntolerances = {\n    'onset': [12, 36, 60, 90, 120, 150, 180, 240, 300, 360], \n    'wakeup': [12, 36, 60, 90, 120, 150, 180, 240, 300, 360]\n}\n\n#\"Dla obu klas używamy progów tolerancji błędu wynoszących 1, 3, 5, 7,5, 10, 12,5, 15, 20, 25, 30 minut \n# lub 12, 36, 60, 90, 120, 150, 180, 240, 300, 360 step\"","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-15T15:00:40.705672Z","iopub.execute_input":"2023-10-15T15:00:40.706499Z","iopub.status.idle":"2023-10-15T15:00:40.717270Z","shell.execute_reply.started":"2023-10-15T15:00:40.706454Z","shell.execute_reply":"2023-10-15T15:00:40.716039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -q /kaggle/input/xgboost-2-0-0-whl/xgboost-2.0.0-py3-none-manylinux2014_x86_64.whl\n\nimport xgboost as xgb\n\nxgb.__version__","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:02:41.675414Z","iopub.execute_input":"2023-10-15T15:02:41.675896Z","iopub.status.idle":"2023-10-15T15:03:25.876053Z","shell.execute_reply.started":"2023-10-15T15:02:41.675858Z","shell.execute_reply":"2023-10-15T15:03:25.874830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Importing data","metadata":{}},{"cell_type":"code","source":"# Importing data \n\n# Column transformations\n\n# przekształcenia daty:\ndt_transforms = [\n    pl.col('timestamp').str.to_datetime(), \n    (pl.col('timestamp').str.to_datetime().dt.year()-2000).cast(pl.UInt8).alias('year'), \n    pl.col('timestamp').str.to_datetime().dt.month().cast(pl.UInt8).alias('month'),\n    pl.col('timestamp').str.to_datetime().dt.day().cast(pl.UInt8).alias('day'), \n    pl.col('timestamp').str.to_datetime().dt.hour().cast(pl.UInt8).alias('hour')\n]\n\n# przekształcenia anglez i enmo\ndata_transforms = [\n    pl.col('anglez').cast(pl.Int16), # Casting anglez to 16 bit integer\n    (pl.col('enmo')*1000).cast(pl.UInt16), # Convert enmo to 16 bit uint\n]\n\n\n#wczytanie train i test\ntrain_series = pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet').with_columns(\n    dt_transforms + data_transforms\n    )\n\ntrain_events = pl.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv').with_columns(\n    dt_transforms\n    )\n\ntest_series = pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet').with_columns(\n    dt_transforms + data_transforms\n    )","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:31.034480Z","iopub.execute_input":"2023-10-15T15:03:31.035275Z","iopub.status.idle":"2023-10-15T15:03:31.387032Z","shell.execute_reply.started":"2023-10-15T15:03:31.035235Z","shell.execute_reply":"2023-10-15T15:03:31.385736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:34.860183Z","iopub.execute_input":"2023-10-15T15:03:34.860580Z","iopub.status.idle":"2023-10-15T15:03:34.882120Z","shell.execute_reply.started":"2023-10-15T15:03:34.860549Z","shell.execute_reply":"2023-10-15T15:03:34.880913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Tworzy listę unikalnych identyfikatorów serii (series_id) z danych zawartych w kolumnie 'series_id' w train_events. Te identyfikatory serii są przechowywane w zmiennej series_ids.\n\n* Usuwa serie, które mają niezgodne liczby wystąpień zdarzeń 'onset' i 'wakeup'. Aby to zrobić, najpierw grupuje dane w train_events według 'series_id' i zlicza wystąpienia 'onset' i 'wakeup' w każdej serii. Następnie tworzy ramkę danych counts, która zawiera informacje o identyfikatorze serii oraz liczbach wystąpień 'onset' i 'wakeup' w każdej serii. count_mismatches to ramka danych zawierająca serie, dla których liczby wystąpień 'onset' i 'wakeup' są różne.\n\n* Filtruje train_series i train_events, usuwając serie z identyfikatorami, które występują w count_mismatches. W ten sposób usuwa serie, które mają niezgodne liczby wystąpień 'onset' i 'wakeup'.\n\n* Aktualizuje listę identyfikatorów serii series_ids, nie zawierając serii bez wartości nie-null (usuwa serie, które nie mają wartości 'series_id' w train_events)\n\nPo co? w danych train_events są przypadki, że człowiek poszedł spać a się nie obudził lub na odwrót ;)","metadata":{}},{"cell_type":"code","source":"# Getting series ids as a list for convenience\nseries_ids = train_events['series_id'].unique(maintain_order=True).to_list()\n\n# Removing series with mismatched counts: \nonset_counts = train_events.filter(pl.col('event')=='onset').group_by('series_id').count().sort('series_id')['count']\nwakeup_counts = train_events.filter(pl.col('event')=='wakeup').group_by('series_id').count().sort('series_id')['count']\n\ncounts = pl.DataFrame({'series_id':sorted(series_ids), 'onset_counts':onset_counts, 'wakeup_counts':wakeup_counts})\ncount_mismatches = counts.filter(counts['onset_counts'] != counts['wakeup_counts'])\n\ntrain_series = train_series.filter(~pl.col('series_id').is_in(count_mismatches['series_id']))\ntrain_events = train_events.filter(~pl.col('series_id').is_in(count_mismatches['series_id']))\n\n# Updating list of series ids, not including series with no non-null values.\nseries_ids = train_events.drop_nulls()['series_id'].unique(maintain_order=True).to_list()","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:37.697822Z","iopub.execute_input":"2023-10-15T15:03:37.698181Z","iopub.status.idle":"2023-10-15T15:03:37.764618Z","shell.execute_reply.started":"2023-10-15T15:03:37.698155Z","shell.execute_reply":"2023-10-15T15:03:37.763600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_mismatches","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:39.502644Z","iopub.execute_input":"2023-10-15T15:03:39.503063Z","iopub.status.idle":"2023-10-15T15:03:39.511689Z","shell.execute_reply.started":"2023-10-15T15:03:39.503036Z","shell.execute_reply":"2023-10-15T15:03:39.510561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Trzeba sprawdzić, czy ta funkcja jest potrzebna i/lub ją zmodyfikować. \nLink do dyskusji o aktualizacji danych: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697","metadata":{}},{"cell_type":"markdown","source":"## Feature Engineering","metadata":{}},{"cell_type":"code","source":"features, feature_cols = [pl.col('hour')], ['hour']\n\nfor mins in [5, 30, 60*2, 60*8] :\n    \n    for var in ['enmo', 'anglez'] :\n        \n        features += [\n            pl.col(var).rolling_mean(12 * mins, center=True, min_periods=1).abs().cast(pl.UInt16).alias(f'{var}_{mins}m_mean'),\n            pl.col(var).rolling_max(12 * mins, center=True, min_periods=1).abs().cast(pl.UInt16).alias(f'{var}_{mins}m_max'),\n            pl.col(var).rolling_std(12 * mins, center=True, min_periods=1).abs().cast(pl.UInt16).alias(f'{var}_{mins}m_std')\n        ]\n\n        feature_cols += [ \n            f'{var}_{mins}m_mean', f'{var}_{mins}m_max', f'{var}_{mins}m_std'\n        ]\n\n        # Getting first variations\n        features += [\n            (pl.col(var).diff().abs().rolling_mean(12 * mins, center=True, min_periods=1)*10).abs().cast(pl.UInt32).alias(f'{var}_1v_{mins}m_mean'),\n            (pl.col(var).diff().abs().rolling_max(12 * mins, center=True, min_periods=1)*10).abs().cast(pl.UInt32).alias(f'{var}_1v_{mins}m_max'),\n            (pl.col(var).diff().abs().rolling_std(12 * mins, center=True, min_periods=1)*10).abs().cast(pl.UInt32).alias(f'{var}_1v_{mins}m_std')\n        ]\n\n        feature_cols += [ \n            f'{var}_1v_{mins}m_mean', f'{var}_1v_{mins}m_max', f'{var}_1v_{mins}m_std'\n        ]\n\nid_cols = ['series_id', 'step', 'timestamp']\n\ntrain_series = train_series.with_columns(\n    features\n).select(id_cols + feature_cols)\n\ntest_series = test_series.with_columns(\n    features\n).select(id_cols + feature_cols)","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:42.518773Z","iopub.execute_input":"2023-10-15T15:03:42.520407Z","iopub.status.idle":"2023-10-15T15:03:42.542264Z","shell.execute_reply.started":"2023-10-15T15:03:42.520347Z","shell.execute_reply":"2023-10-15T15:03:42.540949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_train_dataset(train_data, train_events, drop_nulls=False) :\n    \n    series_ids = train_data['series_id'].unique(maintain_order=True).to_list()\n    X, y = pl.DataFrame(), pl.DataFrame()\n    for idx in tqdm(series_ids) : \n        \n        # Normalizing sample features\n        sample = train_data.filter(pl.col('series_id')==idx).with_columns(\n            [(pl.col(col) / pl.col(col).std()).cast(pl.Float32) for col in feature_cols if col != 'hour']\n        )\n        \n        events = train_events.filter(pl.col('series_id')==idx)\n        \n        if drop_nulls : \n            # Removing datapoints on dates where no data was recorded\n            sample = sample.filter(\n                pl.col('timestamp').dt.date().is_in(events['timestamp'].dt.date())\n            )\n        \n        X = X.vstack(sample[id_cols + feature_cols])\n\n        onsets = events.filter((pl.col('event') == 'onset') & (pl.col('step') != None))['step'].to_list()\n        wakeups = events.filter((pl.col('event') == 'wakeup') & (pl.col('step') != None))['step'].to_list()\n\n        # NOTE: This will break if there are event series without any recorded onsets or wakeups\n        y = y.vstack(sample.with_columns(\n            sum([(onset <= pl.col('step')) & (pl.col('step') <= wakeup) for onset, wakeup in zip(onsets, wakeups)]).cast(pl.Boolean).alias('asleep')\n            ).select('asleep')\n            )\n    \n    y = y.to_numpy().ravel()\n    \n    return X, y","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:44.130012Z","iopub.execute_input":"2023-10-15T15:03:44.131095Z","iopub.status.idle":"2023-10-15T15:03:44.140518Z","shell.execute_reply.started":"2023-10-15T15:03:44.131054Z","shell.execute_reply":"2023-10-15T15:03:44.139347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_events(series, classifier) :\n    '''\n    Takes a time series and a classifier and returns a formatted submission dataframe.\n    '''\n    \n    series_ids = series['series_id'].unique(maintain_order=True).to_list()\n    events = pl.DataFrame(schema={'series_id':str, 'step':int, 'event':str, 'score':float})\n\n    for idx in tqdm(series_ids) : \n\n        # Collecting sample and normalizing features\n        scale_cols = [col for col in feature_cols if (col != 'hour') & (series[col].std() !=0)]\n        X = series.filter(pl.col('series_id') == idx).select(id_cols + feature_cols).with_columns(\n            [(pl.col(col) / series[col].std()).cast(pl.Float32) for col in scale_cols]\n        )\n\n        # Applying classifier to get predictions and scores\n        preds, probs = classifier.predict(X[feature_cols]), classifier.predict_proba(X[feature_cols])[:, 1]\n\n        #NOTE: Considered using rolling max to get sleep periods excluding <30 min interruptions, but ended up decreasing performance\n        X = X.with_columns(\n            pl.lit(preds).cast(pl.Int8).alias('prediction'), \n            pl.lit(probs).alias('probability')\n                        )\n        \n        # Getting predicted onset and wakeup time steps\n        pred_onsets = X.filter(X['prediction'].diff() > 0)['step'].to_list()\n        pred_wakeups = X.filter(X['prediction'].diff() < 0)['step'].to_list()\n        \n        if len(pred_onsets) > 0 : \n            \n            # Ensuring all predicted sleep periods begin and end\n            if min(pred_wakeups) < min(pred_onsets) : \n                pred_wakeups = pred_wakeups[1:]\n\n            if max(pred_onsets) > max(pred_wakeups) :\n                pred_onsets = pred_onsets[:-1]\n\n            # Keeping sleep periods longer than 30 minutes\n            sleep_periods = [(onset, wakeup) for onset, wakeup in zip(pred_onsets, pred_wakeups) if wakeup - onset >= 12 * 30]\n\n            for onset, wakeup in sleep_periods :\n                # Scoring using mean probability over period\n                score = X.filter((pl.col('step') >= onset) & (pl.col('step') <= wakeup))['probability'].mean()\n\n                # Adding sleep event to dataframe\n                events = events.vstack(pl.DataFrame().with_columns(\n                    pl.Series([idx, idx]).alias('series_id'), \n                    pl.Series([onset, wakeup]).alias('step'),\n                    pl.Series(['onset', 'wakeup']).alias('event'),\n                    pl.Series([score, score]).alias('score')\n                ))\n\n    # Adding row id column\n    events = events.to_pandas().reset_index().rename(columns={'index':'row_id'})\n\n    return events","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:46.356614Z","iopub.execute_input":"2023-10-15T15:03:46.357023Z","iopub.status.idle":"2023-10-15T15:03:46.371839Z","shell.execute_reply.started":"2023-10-15T15:03:46.356992Z","shell.execute_reply":"2023-10-15T15:03:46.369975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# def reduce_mem_usage(df):\n#     \"\"\"\n#     Iterate through all numeric columns of a Polars DataFrame and modify the data type\n#     to reduce memory usage.\n#     \"\"\"\n\n#     col_names = df.schema.get_field_names()\n#     df_dict = {}\n\n#     for col_name in col_names:\n#         col = df[col_name]\n#         col_type = col.data_type()\n\n#         if isinstance(col_type, (Int8, Int16, Int32, Int64)):\n#             c_min = col.min().unwrap()\n#             c_max = col.max().unwrap()\n\n#             if isinstance(col_type, Int8):\n#                 if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n#                     col = col.cast(pl.Int8)\n#                 elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n#                     col = col.cast(pl.Int16)\n#                 elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n#                     col = col.cast(pl.Int32)\n#                 elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n#                     col = col.cast(pl.Int64)\n#             elif isinstance(col_type, Float32):\n#                 if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n#                     col = col.cast(pl.Float32)\n#                 else:\n#                     col = col.cast(pl.Float64)\n\n#         df_dict[col_name] = col\n\n#     df_optimized = pl.DataFrame(df_dict)\n#     df_optimized = df_optimized.with_column('series_id', df['series_id'].cast(Categorical))\n\n#     return df_optimized","metadata":{"execution":{"iopub.status.busy":"2023-10-13T11:13:18.973177Z","iopub.execute_input":"2023-10-13T11:13:18.973530Z","iopub.status.idle":"2023-10-13T11:13:18.990242Z","shell.execute_reply.started":"2023-10-13T11:13:18.973502Z","shell.execute_reply":"2023-10-13T11:13:18.988950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Models","metadata":{}},{"cell_type":"code","source":"# reduce_mem_usage(train_series)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T11:13:18.995019Z","iopub.execute_input":"2023-10-13T11:13:18.995539Z","iopub.status.idle":"2023-10-13T11:13:19.008201Z","shell.execute_reply.started":"2023-10-13T11:13:18.995508Z","shell.execute_reply":"2023-10-13T11:13:19.006784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ids, val_ids = train_test_split(series_ids, train_size=0.7, random_state=42, shuffle = False)\n\n# We will collect datapoints at 5 minute intervals for training for validating\n# train_data = train_series.filter(pl.col('series_id').is_in(train_ids)).take_every(12 * 5).collect()\n\n# train_data = train_series.filter(pl.col('series_id').is_in(train_ids)).collect()\n\ntrain_data = train_series.filter(pl.col('series_id').is_in(series_ids)).collect().sample(int(1e6))\n\n\nval_data = train_series.filter(pl.col('series_id').is_in(val_ids)).collect()\nval_solution = train_events.filter(pl.col('series_id').is_in(val_ids)).select(['series_id', 'event', 'step']).to_pandas()","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:03:52.020669Z","iopub.execute_input":"2023-10-15T15:03:52.021000Z","iopub.status.idle":"2023-10-15T15:11:56.740004Z","shell.execute_reply.started":"2023-10-15T15:03:52.020974Z","shell.execute_reply":"2023-10-15T15:11:56.738854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating train dataset\nX_train, y_train = make_train_dataset(train_data, train_events)","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:11:56.746181Z","iopub.execute_input":"2023-10-15T15:11:56.747381Z","iopub.status.idle":"2023-10-15T15:12:03.232952Z","shell.execute_reply.started":"2023-10-15T15:11:56.747333Z","shell.execute_reply":"2023-10-15T15:12:03.231990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training and validating random forest","metadata":{}},{"cell_type":"code","source":"X_train.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:12:03.233965Z","iopub.execute_input":"2023-10-15T15:12:03.235024Z","iopub.status.idle":"2023-10-15T15:12:03.254159Z","shell.execute_reply.started":"2023-10-15T15:12:03.234991Z","shell.execute_reply":"2023-10-15T15:12:03.253046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Optuna","metadata":{}},{"cell_type":"code","source":"# def objective(trial):\n#     # Define the hyperparameters to search over\n#     n_estimators = trial.suggest_int('n_estimators', 100, 1000, step=100)\n#     min_samples_leaf = trial.suggest_int('min_samples_leaf', 100, 500)\n#     max_depth = trial.suggest_int('max_depth', 5, 20)  # Add max_depth as a hyperparameter\n\n#     # Create and train the random forest classifier\n#     rf_classifier = RandomForestClassifier(\n#         n_estimators=n_estimators,\n#         min_samples_leaf=min_samples_leaf,\n#         max_depth=max_depth,  # Use the suggested max_depth value\n#         random_state=42,\n#         n_jobs=-1\n#     )\n\n#     rf_classifier.fit(X_train[feature_cols], y_train)\n\n#     # Make predictions on the validation set\n#     y_pred = rf_classifier.predict(X_val[feature_cols])\n\n#     # Calculate the F1 score as the objective to minimize\n#     f1 = f1_score(y_val, y_pred)\n\n#     return 1 - f1  # Optuna minimizes, so we return 1 - f1 to minimize F1 score\n\n# X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.3, random_state=42)\n\n# study = optuna.create_study(direction='minimize')  # We want to minimize the F1 score\n# study.optimize(objective, n_trials=100)  # You can adjust the number of trials as needed\n\n# best_params = study.best_params\n# best_score = 1 - study.best_value  # Convert back to F1 score\n\n# print(\"Best Hyperparameters:\", best_params)\n# print(\"Best F1 Score:\", best_score)\n\n# [I 2023-09-25 11:12:55,468] A new study created in memory with name: no-name-9526f79b-21e9-4e9d-ad84-c314ece354ab\n# [I 2023-09-25 11:52:54,599] Trial 0 finished with value: 0.06946539490413939 and parameters: {'n_estimators': 700, 'min_samples_leaf': 146, 'max_depth': 13}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 13:03:29,482] Trial 1 finished with value: 0.07323151639082381 and parameters: {'n_estimators': 1000, 'min_samples_leaf': 377, 'max_depth': 19}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 13:37:44,074] Trial 2 finished with value: 0.07325825955419085 and parameters: {'n_estimators': 700, 'min_samples_leaf': 187, 'max_depth': 11}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 13:51:40,476] Trial 3 finished with value: 0.07031010973254115 and parameters: {'n_estimators': 200, 'min_samples_leaf': 243, 'max_depth': 18}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 14:16:29,696] Trial 4 finished with value: 0.07151886891478065 and parameters: {'n_estimators': 400, 'min_samples_leaf': 275, 'max_depth': 15}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 14:38:45,216] Trial 5 finished with value: 0.07515495515949588 and parameters: {'n_estimators': 500, 'min_samples_leaf': 269, 'max_depth': 10}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 14:47:46,083] Trial 6 finished with value: 0.07710140184366987 and parameters: {'n_estimators': 200, 'min_samples_leaf': 467, 'max_depth': 10}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 15:14:30,617] Trial 7 finished with value: 0.07310587260317014 and parameters: {'n_estimators': 500, 'min_samples_leaf': 288, 'max_depth': 12}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 15:23:29,831] Trial 8 finished with value: 0.07534936936446102 and parameters: {'n_estimators': 200, 'min_samples_leaf': 283, 'max_depth': 10}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 16:02:09,101] Trial 9 finished with value: 0.07403436401137597 and parameters: {'n_estimators': 800, 'min_samples_leaf': 222, 'max_depth': 11}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 16:26:45,116] Trial 10 finished with value: 0.08593876559209446 and parameters: {'n_estimators': 1000, 'min_samples_leaf': 103, 'max_depth': 5}. Best is trial 0 with value: 0.06946539490413939.\n# [I 2023-09-25 16:34:26,003] Trial 11 finished with value: 0.06653632867100934 and parameters: {'n_estimators': 100, 'min_samples_leaf': 122, 'max_depth': 19}. Best is trial 11 with value: 0.06653632867100934.\n# [I 2023-09-25 17:26:56,238] Trial 12 finished with value: 0.06678471353857662 and parameters: {'n_estimators': 800, 'min_samples_leaf': 112, 'max_depth': 16}. Best is trial 11 with value: 0.06653632867100934.","metadata":{"execution":{"iopub.status.busy":"2023-10-13T11:21:59.306822Z","iopub.execute_input":"2023-10-13T11:21:59.307729Z","iopub.status.idle":"2023-10-13T11:21:59.322248Z","shell.execute_reply.started":"2023-10-13T11:21:59.307697Z","shell.execute_reply":"2023-10-13T11:21:59.321149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Plotting feature importances\n# px.bar(x=feature_cols, \n#        y=rf_classifier.feature_importances_,\n#        title='Random forest feature importances'\n#       )","metadata":{"execution":{"iopub.status.busy":"2023-10-13T11:21:59.323784Z","iopub.execute_input":"2023-10-13T11:21:59.324240Z","iopub.status.idle":"2023-10-13T11:21:59.343512Z","shell.execute_reply.started":"2023-10-13T11:21:59.324198Z","shell.execute_reply":"2023-10-13T11:21:59.342231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#for 1mln and 10 minutes:\n# rf_classifier = RandomForestClassifier(n_estimators=700,\n#                                     min_samples_leaf=101,\n#                                     random_state=42,\n#                                     n_jobs=-1)\n\n#RF with hyperparameters from Optuna\n\n# Random Forest with hyperparameters from Optuna\n\n# rf_classifier = RandomForestClassifier(n_estimators=100,\n#                                     min_samples_leaf=122,\n#                                     max_depth = 19,\n#                                     random_state=42,\n#                                     n_jobs=-1)\n\n# rf_classifier.fit(X_train[feature_cols], y_train)\n","metadata":{"execution":{"iopub.status.busy":"2023-10-13T11:21:59.344836Z","iopub.execute_input":"2023-10-13T11:21:59.345267Z","iopub.status.idle":"2023-10-13T11:21:59.355295Z","shell.execute_reply.started":"2023-10-13T11:21:59.345240Z","shell.execute_reply":"2023-10-13T11:21:59.354304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Checking performance on validation set\n# rf_submission = get_events(val_data, rf_classifier)\n\n# print(f\"Random forest score: {score(val_solution, rf_submission, tolerances, **column_names)}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-13T11:21:59.356677Z","iopub.execute_input":"2023-10-13T11:21:59.357106Z","iopub.status.idle":"2023-10-13T11:21:59.366686Z","shell.execute_reply.started":"2023-10-13T11:21:59.357064Z","shell.execute_reply":"2023-10-13T11:21:59.365664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving classifier \n# import pickle\n# with open('rf_classifier_5m_8h.pkl', 'wb') as f:\n#     pickle.dump(rf_classifier, f)\n\n#with open('rf_classifier.pkl', 'rb') as f:\n#    rf_classifier = pickle.load(f)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T11:21:59.368094Z","iopub.execute_input":"2023-10-13T11:21:59.368401Z","iopub.status.idle":"2023-10-13T11:21:59.383408Z","shell.execute_reply.started":"2023-10-13T11:21:59.368375Z","shell.execute_reply":"2023-10-13T11:21:59.382063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# XGBoost 2.0","metadata":{}},{"cell_type":"code","source":"# import gc\nfrom xgboost import XGBClassifier\n\nmodel = XGBClassifier(n_estimators=520, objective=\"binary:logistic\", learning_rate=0.02, max_depth=7, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:12:08.983784Z","iopub.execute_input":"2023-10-15T15:12:08.984232Z","iopub.status.idle":"2023-10-15T15:12:08.990358Z","shell.execute_reply.started":"2023-10-15T15:12:08.984203Z","shell.execute_reply":"2023-10-15T15:12:08.988734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(X_train[feature_cols], y_train)\n\n# Checking performance on validation set\nval_submission = get_events(val_data, model)\n\nprint(f\"Model score: {score(val_solution, val_submission, tolerances, **column_names)}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-15T15:12:11.467686Z","iopub.execute_input":"2023-10-15T15:12:11.468066Z","iopub.status.idle":"2023-10-15T16:10:13.379949Z","shell.execute_reply.started":"2023-10-15T15:12:11.468038Z","shell.execute_reply":"2023-10-15T16:10:13.378909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Applying to test data","metadata":{}},{"cell_type":"code","source":"# Recovering memory\ndel train_data ","metadata":{"execution":{"iopub.status.busy":"2023-10-13T12:49:44.579320Z","iopub.execute_input":"2023-10-13T12:49:44.579672Z","iopub.status.idle":"2023-10-13T12:49:44.603929Z","shell.execute_reply.started":"2023-10-13T12:49:44.579641Z","shell.execute_reply":"2023-10-13T12:49:44.602534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Getting event predictions for test set and saving submission\nsubmission = get_events(test_series.collect(), model)\nsubmission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T12:49:44.605514Z","iopub.execute_input":"2023-10-13T12:49:44.605830Z","iopub.status.idle":"2023-10-13T12:49:44.735347Z","shell.execute_reply.started":"2023-10-13T12:49:44.605805Z","shell.execute_reply":"2023-10-13T12:49:44.734451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission","metadata":{"execution":{"iopub.status.busy":"2023-10-13T12:49:44.736556Z","iopub.execute_input":"2023-10-13T12:49:44.737072Z","iopub.status.idle":"2023-10-13T12:49:44.754482Z","shell.execute_reply.started":"2023-10-13T12:49:44.737041Z","shell.execute_reply":"2023-10-13T12:49:44.753209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Why submission is empty? https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/444673\n\n* Playing with the competition metric: https://www.kaggle.com/code/carlmcbrideellis/zzzs-playing-with-the-competition-metric","metadata":{}}]}