{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"}],"dockerImageVersionId":30558,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# データの読み込みと確認","metadata":{}},{"cell_type":"markdown","source":"## train_series\n\n* series_id\n    * 各加速度センサシリーズの一意な識別子．\n* step \n    * シリーズ内の各観測のタイムステップ（整数）．\n* timestamp\n    * ISO 8601 フォーマット %Y-%m-%dT%H:%M:%S%z に対応する日時．\n* anglez\n    * GGIR パッケージで計算・記述されているように、z-angle は睡眠検出で一般的に使用される個々の加速度計コンポーネントから導出されるメトリックで、身体の垂直軸に対する腕の角度を指す。\n* ENMO\n    * GGIRパッケージによって計算され、説明されているように、ENMOはすべての加速度計信号のユークリッド正規マイナス1であり、負の値はゼロに丸められます。\n    * この空間に加速度の標準的な尺度は存在しないが、これは一般的に計算されるいくつかの特徴の1つである。","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ntrain_series = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\")\n\ntrain_series","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-11-10T00:05:36.198470Z","iopub.execute_input":"2023-11-10T00:05:36.199002Z","iopub.status.idle":"2023-11-10T00:06:44.105609Z","shell.execute_reply.started":"2023-11-10T00:05:36.198960Z","shell.execute_reply":"2023-11-10T00:06:44.104188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series[train_series['series_id']=='038441c925bb']","metadata":{"execution":{"iopub.status.busy":"2023-11-10T00:06:44.108010Z","iopub.execute_input":"2023-11-10T00:06:44.108487Z","iopub.status.idle":"2023-11-10T00:07:08.337339Z","shell.execute_reply.started":"2023-11-10T00:06:44.108448Z","shell.execute_reply":"2023-11-10T00:07:08.336197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## train_events\n\n* train_events.csv\n    * トレーニングセット内の各シリーズの睡眠ログ．\n* series_id\n    * train_series.parquet内の加速度センサデータの各シリーズの一意な識別子。\n* night\n    * オンセットイベントとウェイクアップイベントのペアの列挙。各夜に発生するイベントのペアは最大1つ。\n* event\n    * オンセットまたはウェイクアップのイベントのタイプ．\n* step and timestamp\n    * 加速度センサシリーズにおけるイベントの発生時刻。","metadata":{}},{"cell_type":"code","source":"train_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\n\ntrain_events","metadata":{"execution":{"iopub.status.busy":"2023-11-10T00:07:08.339484Z","iopub.execute_input":"2023-11-10T00:07:08.339859Z","iopub.status.idle":"2023-11-10T00:07:08.402631Z","shell.execute_reply.started":"2023-11-10T00:07:08.339829Z","shell.execute_reply":"2023-11-10T00:07:08.401157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet\")\n\ntest_series","metadata":{"execution":{"iopub.status.busy":"2023-11-10T00:07:08.405704Z","iopub.execute_input":"2023-11-10T00:07:08.406135Z","iopub.status.idle":"2023-11-10T00:07:08.436922Z","shell.execute_reply.started":"2023-11-10T00:07:08.406087Z","shell.execute_reply":"2023-11-10T00:07:08.435535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series[test_series['series_id']=='038441c925bb']","metadata":{"execution":{"iopub.status.busy":"2023-11-10T00:07:08.438547Z","iopub.execute_input":"2023-11-10T00:07:08.438974Z","iopub.status.idle":"2023-11-10T00:07:08.460849Z","shell.execute_reply.started":"2023-11-10T00:07:08.438938Z","shell.execute_reply":"2023-11-10T00:07:08.459215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## sample_submission\n\n* \"row_id\": 予測のID\n* \"series_id\": 各シリーズのID\n* \"event\": イベントのタイプ\n* \"step\": イベントが発生したステップ\n* \"score\": イベントの信頼度スコア","metadata":{}},{"cell_type":"code","source":"sample_submission = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/sample_submission.csv\")\n\nsample_submission","metadata":{"execution":{"iopub.status.busy":"2023-11-07T23:04:45.079659Z","iopub.execute_input":"2023-11-07T23:04:45.080238Z","iopub.status.idle":"2023-11-07T23:04:45.097514Z","shell.execute_reply.started":"2023-11-07T23:04:45.080210Z","shell.execute_reply":"2023-11-07T23:04:45.096538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# データの加工\n\nデータを加工して `train_events`のイベントを、前後1時間のセンサ情報から回帰により予測する  \n\n## 学習用データの加工\n\n* 目的変数： \n    * `train_events`のイベント\n* 説明変数\n    1. イベント発生の6分前から5分前のセンサデータの平均\n    1. イベント発生の5分前から4分前のセンサデータの平均\n    1. イベント発生の4分前から3分前のセンサデータの平均\n    1. イベント発生の3分前から2分前のセンサデータの平均\n    1. イベント発生の2分前から1分前のセンサデータの平均\n    1. イベント発生の1分前から0分前のセンサデータの平均\n    1. イベント発生の0分後から1分後のセンサデータの平均\n    1. イベント発生の1分後から2分後のセンサデータの平均\n    1. イベント発生の2分後から3分後のセンサデータの平均\n    1. イベント発生の3分後から4分後のセンサデータの平均\n    1. イベント発生の4分後から5分後のセンサデータの平均\n    1. イベント発生の5分後から6分後のセンサデータの平均\n","metadata":{}},{"cell_type":"code","source":"train_series_ids = train_series['series_id'].unique()","metadata":{"execution":{"iopub.status.busy":"2023-11-07T23:04:45.098685Z","iopub.execute_input":"2023-11-07T23:04:45.099027Z","iopub.status.idle":"2023-11-07T23:04:53.534640Z","shell.execute_reply.started":"2023-11-07T23:04:45.099000Z","shell.execute_reply":"2023-11-07T23:04:53.533859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import datetime\n\nexp_period = '1MIN'\nexp_range_min = 5\n\n_xs = {}\n_ys = {}\nfor series_id in train_series_ids:\n    print(series_id)\n    \n    _train_series = train_series[train_series['series_id']==series_id]\n    _train_df = _train_series[['anglez', 'enmo']]\n    _train_df.index = pd.to_datetime(_train_series['timestamp'], utc=True)\n    _train_df = _train_df.resample(exp_period).mean()\n    \n    _train_event = train_events[train_events['series_id']==series_id].dropna(how='any')\n    _train_event.index = pd.to_datetime(_train_event['timestamp'], utc=True)\n\n    for _event_time, _step in zip(_train_event.index, _train_event['step']):\n        _time_list = pd.date_range(\n            start=_event_time.round(exp_period) + datetime.timedelta(minutes=-exp_range_min), \n            end=_event_time.round(exp_period) + datetime.timedelta(minutes=exp_range_min), \n            freq=exp_period, tz='UTC')\n        \n        _x = {}\n        for _i, _time in enumerate(_time_list):\n            _d = _train_df.loc[_time].to_dict() if _time in _train_df.index else {'anglez': float('nan'), 'enmo': float('nan')}\n            _x.update({_k+str(_i): _v for _k, _v in _d.items()})\n        \n        _xs[(series_id, _step)] = _x\n        _ys[(series_id, _step)] = {'event': 0 if _train_event.loc[_event_time]['event']=='onset' else 1}","metadata":{"execution":{"iopub.status.busy":"2023-11-07T23:04:53.535849Z","iopub.execute_input":"2023-11-07T23:04:53.536163Z","iopub.status.idle":"2023-11-08T00:03:47.222094Z","shell.execute_reply.started":"2023-11-07T23:04:53.536135Z","shell.execute_reply":"2023-11-08T00:03:47.221163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series_ids = test_series['series_id'].unique()\n\ntest_period = '1MIN' if len(test_series) < 500 else '30MIN'\n\n_test_xs = {}\nfor series_id in test_series_ids:\n    print(series_id)\n    \n    _test_series = test_series[test_series['series_id']==series_id]\n    _test_df = _test_series[['step', 'anglez', 'enmo']]\n    _test_df.index = pd.to_datetime(_test_series['timestamp'], utc=True)\n    _test_df = _test_df.resample(exp_period).mean()\n    _test_df.loc[:, 'step'] = _test_df['step'].astype('int')\n    \n    _event_time_list = pd.date_range(\n            start=_test_df.index.min(), \n            end=_test_df.index.max(), \n            freq=test_period, tz='UTC')\n    \n    for _event_time in _event_time_list:\n        _step = _test_df.loc[_event_time, 'step']\n        _time_list = pd.date_range(\n            start=_event_time.round(exp_period) + datetime.timedelta(minutes=-exp_range_min), \n            end=_event_time.round(exp_period) + datetime.timedelta(minutes=exp_range_min), \n            freq=exp_period, tz='UTC')\n        \n        _test_x = {}\n        for _i, _time in enumerate(_time_list):\n            _d = _test_df.loc[_time, ['anglez', 'enmo']].to_dict() if _time in _test_df.index else {'anglez': float('nan'), 'enmo': float('nan')}\n            _test_x.update({_k+str(_i): _v for _k, _v in _d.items()})\n        \n        _test_xs[(series_id, _step)] = _test_x\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:03:47.223351Z","iopub.execute_input":"2023-11-08T00:03:47.224049Z","iopub.status.idle":"2023-11-08T00:03:47.503573Z","shell.execute_reply.started":"2023-11-08T00:03:47.224016Z","shell.execute_reply":"2023-11-08T00:03:47.502417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_df = pd.DataFrame(_ys).T\ny_df","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:03:47.507017Z","iopub.execute_input":"2023-11-08T00:03:47.507402Z","iopub.status.idle":"2023-11-08T00:03:47.741851Z","shell.execute_reply.started":"2023-11-08T00:03:47.507362Z","shell.execute_reply":"2023-11-08T00:03:47.740612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_df = pd.DataFrame(_xs).T\nprint(x_df.columns)\nx_df","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:03:47.743122Z","iopub.execute_input":"2023-11-08T00:03:47.743431Z","iopub.status.idle":"2023-11-08T00:03:48.032929Z","shell.execute_reply.started":"2023-11-08T00:03:47.743403Z","shell.execute_reply":"2023-11-08T00:03:48.031806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_x_df = pd.DataFrame(_test_xs).T\nprint(test_x_df.columns)\ntest_x_df","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:03:48.034255Z","iopub.execute_input":"2023-11-08T00:03:48.034546Z","iopub.status.idle":"2023-11-08T00:03:48.089964Z","shell.execute_reply.started":"2023-11-08T00:03:48.034521Z","shell.execute_reply":"2023-11-08T00:03:48.089048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# モデルの作成とテストデータへの推論\n\n## 学習\n\nNANのある行を削除する","metadata":{}},{"cell_type":"code","source":"nan_index = x_df.isnull().any(axis=1)\n\n_x_df = x_df[~nan_index]\n_y_df = y_df[~nan_index]\n\nfrom sklearn.ensemble import RandomForestRegressor\n\nclf = RandomForestRegressor(random_state=123)\nclf.fit(_x_df, _y_df)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:03:48.091861Z","iopub.execute_input":"2023-11-08T00:03:48.092313Z","iopub.status.idle":"2023-11-08T00:04:05.472331Z","shell.execute_reply.started":"2023-11-08T00:03:48.092274Z","shell.execute_reply":"2023-11-08T00:04:05.471205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## テストデータへ推論","metadata":{}},{"cell_type":"code","source":"nan_index = test_x_df.isnull().any(axis=1)\n_test_x_df = test_x_df[~nan_index]\n_test_predict = clf.predict(_test_x_df)\n_test_predict","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:04:05.473642Z","iopub.execute_input":"2023-11-08T00:04:05.473979Z","iopub.status.idle":"2023-11-08T00:04:05.490552Z","shell.execute_reply.started":"2023-11-08T00:04:05.473952Z","shell.execute_reply":"2023-11-08T00:04:05.489441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_submission = []\nfor _i, (_p, (series_id, _step)) in enumerate(zip(_test_predict, _test_x_df.index)):\n    if _p < 0.5:\n        _event = 'onset'\n        _score = (0.5-_p)*2 if (0.5-_p)*2>0 else 0.\n    else:\n        _event = 'wakeup'\n        _score = (_p-0.5)*2 if (_p-0.5)*2<1 else 1.\n        \n    _submission.append(\n        {\n            'row_id': _i,\n            'series_id': series_id,\n            'step': _step,\n            'event': _event,\n            'score': _score\n        }\n    ) ","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:24:37.021270Z","iopub.execute_input":"2023-11-08T00:24:37.021746Z","iopub.status.idle":"2023-11-08T00:24:37.028444Z","shell.execute_reply.started":"2023-11-08T00:24:37.021709Z","shell.execute_reply":"2023-11-08T00:24:37.027339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame(_submission)\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:24:46.954416Z","iopub.execute_input":"2023-11-08T00:24:46.954837Z","iopub.status.idle":"2023-11-08T00:24:46.969324Z","shell.execute_reply.started":"2023-11-08T00:24:46.954803Z","shell.execute_reply":"2023-11-08T00:24:46.968262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = submission.loc[submission.groupby([\"series_id\", \"event\"])[\"score\"].idxmax(), :]\nsubmission = submission.sort_values('row_id')\nsubmission['row_id'] = list(range(len(submission)))\n\nsubmission","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:25:21.097101Z","iopub.execute_input":"2023-11-08T00:25:21.097905Z","iopub.status.idle":"2023-11-08T00:25:21.103614Z","shell.execute_reply.started":"2023-11-08T00:25:21.097866Z","shell.execute_reply":"2023-11-08T00:25:21.102465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_df.to_csv('x_df.csv', index=False)\ny_df.to_csv('y_df.csv', index=False)\ntest_x_df.to_csv('test_x_df.csv', index=False)\n","metadata":{"execution":{"iopub.status.busy":"2023-11-08T00:25:19.103938Z","iopub.execute_input":"2023-11-08T00:25:19.104968Z","iopub.status.idle":"2023-11-08T00:25:19.509247Z","shell.execute_reply.started":"2023-11-08T00:25:19.104929Z","shell.execute_reply":"2023-11-08T00:25:19.508266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}