{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np \nimport pandas as pd \nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport datetime\nimport warnings\n#warnings.filterwarnings('ignore')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\n#df = pd.read_csv('data/train_events.csv')\ndf_series = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet', engine='pyarrow')\n#df_series = pd.read_parquet('data/train_series.parquet', engine='pyarrow')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_series.query(\"series_id == '038441c925bb' & step >= 4990\").head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explanation\n\nThe series_id is the device (i.e. the child). \n\nThe parquet files contains all the data (anglez and enmo) of the sensor, for all the night. \n\nThe .csv file contains labeled event occuring at some steps of the night. We can link both of them using the step and the series_id.","metadata":{}},{"cell_type":"markdown","source":"first serie has 389880 step\n\nwe have 277 unique series","metadata":{}},{"cell_type":"markdown","source":"## Data visualization","metadata":{}},{"cell_type":"code","source":"sample = df_series.query(\"series_id == '038441c925bb' & 4800 <= step <= 5200\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(10, 5))\naxes = axes.flatten()\nfor i, col in enumerate(['anglez', 'enmo']):\n    sns.lineplot(data=sample, x='step', y=col, ax=axes[i])\n    axes[i].set_title(f\"Bar Plot for {col}\")\nplt.tight_layout()\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data engineering\n\n### Adjust Date format \n\n- split the current timestamp into year/month/day/hour/minute/seconds fields","metadata":{}},{"cell_type":"markdown","source":"### Data cleaning\n\nWe start by removing rows without timestamp, as they leads to error when converting the datetime. We will see later if it's possible \nto infer the timestamp for these values, but as the step is also missing, it might be really difficult.\n\nThis concerns 51% of the data which is quite huge, but I will not be able to train a model without the timestamp at the moment so we will need\nto deal with it.","metadata":{}},{"cell_type":"code","source":"print(f\"Percentage data without timestamp: {round(df.timestamp.isna().sum() / df.timestamp.count() * 100,0)}%\")\nprint(f\"Percentage data without step: {round(df.step.isna().sum() / df.step.count() * 100,0)}%\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# remove the rows with no timestamp\ndf_cleaned = df[df.timestamp.notna()].copy()\nprint(f\"Percentage data without timestamp: {round(df_cleaned.timestamp.isna().sum() / df_cleaned.timestamp.count() * 100,0)}%\")\nprint(f\"Percentage data without step: {round(df_cleaned.step.isna().sum() / df_cleaned.step.count() * 100,0)}%\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we attemp to convert the datetime field using pd.to_datetime, we might have errors due to the following rows for example:","metadata":{}},{"cell_type":"code","source":"df_cleaned['timestamp'] = pd.to_datetime(df_cleaned['timestamp'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"problematic_row = df_cleaned.loc[54]\nprint(problematic_row)\nproblematic_row['timestamp'] = problematic_row['timestamp'].astimezone(datetime.timezone.utc)\nprint(problematic_row)\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This timestamp is in the format YYYY-MM-DD HH:MM:SS±HH:MM, which includes the time zone offset of \"-04:00\" (meaning it's 4 hours behind UTC).\n\nThe issue arises because pandas is trying to convert this timestamp to a datetime64 object. However, datetime64 objects in pandas are not designed to handle time zone offsets. To convert a timezone-aware datetime to datetime64, we will convert it to UTC first:","metadata":{}},{"cell_type":"code","source":"def convert_df_timestamp_to_utc(to_convert_df):\n    to_convert_df['timestamp'] = to_convert_df['timestamp'].apply(lambda x: x.astimezone(datetime.timezone.utc) if x.tzinfo else x)\n    return to_convert_df\n\ndf_cleaned = convert_df_timestamp_to_utc(df_cleaned)\ndf_cleaned.dtypes","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def split_timestamp(dataframe):\n    dataframe['year'] = dataframe.timestamp.dt.year\n    dataframe['month'] = dataframe.timestamp.dt.month\n    dataframe['day'] = dataframe.timestamp.dt.day\n    dataframe['hour'] = dataframe.timestamp.dt.hour\n    dataframe['minute'] = dataframe.timestamp.dt.minute\n    dataframe['second'] = dataframe.timestamp.dt.second\n    return dataframe","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_cleaned = split_timestamp(df_cleaned)\ndf_cleaned.drop(columns=['timestamp'], inplace=True)\ndf_cleaned.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# we can combine everything into one function\ndef rm_timestamp_nan_and_split_date(to_convert):\n    n_null = to_convert.timestamp.isna().sum() \n    if n_null > 0:\n        # print(f\"Percentage data without timestamp before cleaning: {round(to_convert.timestamp.isna().sum() / to_convert.timestamp.count() * 100,0)}%\")\n        to_convert = to_convert[to_convert.timestamp.notna()]\n        # print(f\"Percentage data without timestamp after cleaning: {round(to_convert.timestamp.isna().sum() / to_convert.timestamp.count() * 100,0)}%\"))\n    print(\"1\")\n    to_convert['timestamp'] = pd.to_datetime(to_convert['timestamp'])\n    print(\"2\")\n    to_convert = convert_df_timestamp_to_utc(to_convert)\n    print(\"3\")\n    to_convert = split_timestamp(to_convert)\n    print(\"4\")\n    to_convert.drop(columns=['timestamp'], inplace=True)\n    print(\"5\")\n    return to_convert","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# only on a subset of the data atm as takes ~ 700 seconds on full dataset on my machine so should be even slower on Kaggle\ndf_series_cleaned = rm_timestamp_nan_and_split_date(df_series.head(1_000_000))\n#df_series_cleaned = rm_timestamp_nan_and_split_date(df_series.head(127_946_340))\ndf_series_cleaned.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature ideas:\n\n- this is time series problem, look at kaggle tutorial\n- create some indicator, e.g moving average, or # spike in last hour,...\n- other ","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}}]}