---
title: 'Prepare ZZZ in R'
author: 'Peter Hahn'
date: '`r Sys.Date()`'
output:
  html_document:
    number_sections: true
    toc: true
---

# Prepare data for ML
I read the notebook by https://www.kaggle.com/carlmcbrideellis   

https://www.kaggle.com/code/carlmcbrideellis/zzzs-make-clean-starter-dataset-target/notebook and tried to transfer it to R.
Maybe helpul for R-users who wants to dive deeper into the competition using R

## Load libraries
The library arrow is required for reading .parquet files. 
```{r message = FALSE}
library(tidyverse) # metapackage of all tidyverse packages
library(arrow)
library(skimr)
```

### Read train_events and modify timestamp with lubridate
Be careful with timestamps: I substracted 4 hours to get the right time and days.
See: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440868
Timestamps are removed because loading them from csv and paquet caused incompabilities ;-)

## Read events
```{r message = FALSE}
events <- read_csv("../input/child-mind-institute-detect-sleep-states/train_events.csv") %>%
        mutate(dt = as_datetime(timestamp)) %>% 
        mutate(dt = dt - hours(4))  %>% mutate(hr = hour(dt)) %>% 
        select(-timestamp)
```

```{r}
head(events)
```
### Detect NA in events
```{r}
events_na <- events %>% group_by(series_id,step) %>% filter(is.na(step))
num_na <- length(unique(events_na$series_id))
na_id <- unique(events_na$series_id)
```
`r num_na`series_id have NA events

### Series_id without any NA events
There are 37 series
```{r}
all_id <- unique(events$series_id)
nna_id <- setdiff(all_id,na_id)
nna_id
```
### Remove two truncated event series
See the above mentioned script 
```{r}
trunc <- c("31011ade7c0a","a596ad0b82aa")
nna_id <- setdiff(nna_id,trunc)
df_nna <- tibble(nna_id) %>% rename(series_id = nna_id)
```
# Cleaning 
Try to do all without a for loop
## Read training data
This will take some time
```{r}
train <- read_parquet("../input/child-mind-institute-detect-sleep-states/train_series.parquet") %>%
        mutate(dt = as_datetime(timestamp)) %>% 
        mutate(dt = dt - hours(4))  %>% mutate(hr = hour(dt)) %>% 
        select(-timestamp)

nna_train <- right_join(train,df_nna)
nna_event <- right_join(events,df_nna)
## prepare awake
nna_event <- nna_event %>% mutate(awake = ifelse(event == "onset",1,0))
by <- join_by(series_id,step == step)
train_new <- left_join(nna_train,nna_event)
### Use fill from tidyr
train_new <- train_new %>% fill(awake, .direction= c("up"))
```

## Finally
I hope this is helpul to the R-people