{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"}],"dockerImageVersionId":30553,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

EDA Simplified: Child Mind Institute Detect Sleep States\n

","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"

Introduction

\n\nSince July, we took a break on writing on all of the data analysis based on various datasets in most of the competitions, from education-related ones to the medical-related ones. But now, we awoken from our hiatus of our EDA Simplified series because we are going to step onto the data analysis of detecting sleep onsets and wake, because the \"Zzzs\" can lead us to our overall health, both physically and mentally. However, we noted that there's a lot of challenges researchers need to jump through because there are lacking naturalistic data capture as well as accuracy on annotation. On the positive side, data science can help researchers analyze wrist-worn accelometer data for sleep monitoring, resulting in how they understand the sleep importance and function. In summary about this problem, our goal is to detect sleep onsets and wakes from accelometer data, so that we will make the researchers understand how important sleep can benefit us, followed by making them conducting large-scale studies of sleep and monitor sleep. Without further talking, let's jump from fence-to-fence towards exploring data analysis in this competition like a flock of sheep!","metadata":{}},{"cell_type":"markdown","source":"

Importing the Modules / DataFrame Setup

\n\nFirst things first, let's import the modules needed for our data analysis! This time from our previous data analysis notebooks, we use two modules: pandas as pd for data analysis and dataframe creation, and seaborn as sns for data plotting and visualization.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:29:31.687866Z","iopub.execute_input":"2023-11-20T20:29:31.688556Z","iopub.status.idle":"2023-11-20T20:29:33.954552Z","shell.execute_reply.started":"2023-11-20T20:29:31.688521Z","shell.execute_reply":"2023-11-20T20:29:33.953340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Following from importing the two modules, let's generate two dataframes that were from the csv file and the parquet file! We simply characterize the train_df dataframe to read the csv file, the train_events one, with the pd module's read_csv function, and the train_par_df dataframe to read the parquet file of the train_series, with the read_parquet function. Thenceforth, we use the head function to the newly-created two dataframes for displaying out the first five rows of them.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:29:33.956492Z","iopub.execute_input":"2023-11-20T20:29:33.956987Z","iopub.status.idle":"2023-11-20T20:29:34.020584Z","shell.execute_reply.started":"2023-11-20T20:29:33.956956Z","shell.execute_reply":"2023-11-20T20:29:34.019428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_par_df = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\")\ntrain_par_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:29:34.021929Z","iopub.execute_input":"2023-11-20T20:29:34.022313Z","iopub.status.idle":"2023-11-20T20:30:25.312374Z","shell.execute_reply.started":"2023-11-20T20:29:34.022280Z","shell.execute_reply":"2023-11-20T20:30:25.311532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we created two dataframes, let's begin our short peek on the train_df and train_par_df dataframes! First, let's look at the overall number of entities in both dataframes with the len function.","metadata":{}},{"cell_type":"code","source":"len(train_df)","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:25.314800Z","iopub.execute_input":"2023-11-20T20:30:25.315718Z","iopub.status.idle":"2023-11-20T20:30:25.322335Z","shell.execute_reply.started":"2023-11-20T20:30:25.315680Z","shell.execute_reply":"2023-11-20T20:30:25.321007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_par_df)","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:25.323830Z","iopub.execute_input":"2023-11-20T20:30:25.324268Z","iopub.status.idle":"2023-11-20T20:30:25.338375Z","shell.execute_reply.started":"2023-11-20T20:30:25.324227Z","shell.execute_reply":"2023-11-20T20:30:25.336430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the train_df dataframe, we counted 14510 data entities, while we counted 127945340 entities in the train_par_df dataframe. To be specific, the huge number of data entities in the train_par_df dataframe hinted us that there's more data to gather on sleep onsets and wakes in this parquet file based on each patient.","metadata":{}},{"cell_type":"markdown","source":"Now let's find how many missing values are there in the train_df and train_par_df dataframes! We apply the isna function to both dataframes to determine how many missing values were there and then plug in the sum function to add up the missing values that were present in both dataframes.","metadata":{}},{"cell_type":"code","source":"train_df.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:25.340185Z","iopub.execute_input":"2023-11-20T20:30:25.341059Z","iopub.status.idle":"2023-11-20T20:30:25.356939Z","shell.execute_reply.started":"2023-11-20T20:30:25.341013Z","shell.execute_reply":"2023-11-20T20:30:25.355884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_par_df.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:25.358682Z","iopub.execute_input":"2023-11-20T20:30:25.359091Z","iopub.status.idle":"2023-11-20T20:30:52.388081Z","shell.execute_reply.started":"2023-11-20T20:30:25.359053Z","shell.execute_reply":"2023-11-20T20:30:52.386820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While the train_par_df dataframe contains no missing values, we spotted 4923 missing values in the step and timestamp columns in the train_df dataframe, counting an overall 9846 missing values. Specifically, the 9846 missing values we glimpsed from the train_df dataframe gave us clues that there are some data entities have no timestamps specified as well as having unclear values from the step column.","metadata":{}},{"cell_type":"markdown","source":"Lastly for our short look on the two dataframes, let's sum up the number of columns in the two dataframes! We use the shape attribute to the train_df and train_par_df dataframes for getting the shape of them, then extract the last value of the dataframes' shape.","metadata":{}},{"cell_type":"code","source":"train_df.shape[1]","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:52.389461Z","iopub.execute_input":"2023-11-20T20:30:52.389868Z","iopub.status.idle":"2023-11-20T20:30:52.396149Z","shell.execute_reply.started":"2023-11-20T20:30:52.389839Z","shell.execute_reply":"2023-11-20T20:30:52.395389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_par_df.shape[1]","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:52.397878Z","iopub.execute_input":"2023-11-20T20:30:52.398193Z","iopub.status.idle":"2023-11-20T20:30:52.409995Z","shell.execute_reply.started":"2023-11-20T20:30:52.398167Z","shell.execute_reply":"2023-11-20T20:30:52.408759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, there are five columns in both train_df and train_par_df dataframes. In other words, the five columns seen from both dataframes implied to us that it shows the timestamp, event, and other important data of each recording.","metadata":{}},{"cell_type":"markdown","source":"As we finished our short peek on our two dataframes, let's move towards on visualizing the train_df dataframe by plotting the data with seaborn!","metadata":{}},{"cell_type":"markdown","source":"

Chapter 1: The train_df DataFrame Visualization

\n\nUpon entering the first section of our EDA analysis in detecting sleep states, we have gotten some context that this dataframe was used for gathering sleep log data in the training set, recording onset, and wake events. Without further ado, let's go over the columns seen in the train_df dataframe!\n* **series_id**: The identifier specified for each series of accelerometer data from the train_series.parquet file.\n* **step** & **timestamp**: The recorded time of occurence specified in the event from the accelerometer. \n* **night**: An enumeration of possible onset/wakeup event pairs. One pair of events can occur each night at most.\n* **event**: Specifies the type of event, whether it's onset or wakeup.","metadata":{}},{"cell_type":"markdown","source":"Now that we summarized all five columns in the train_df dataframe with some brief explanation, let's plot the data from the night column to our histplot! To get started, we use the sns module's displot function to distribute the specified data to the histogram chart, setting the train_df dataframe as the data specified for creating the displot followed by configuring the x parameter to night for setting up the x-axis.","metadata":{}},{"cell_type":"code","source":"sns.displot(train_df, x=\"night\")","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:52.415835Z","iopub.execute_input":"2023-11-20T20:30:52.416223Z","iopub.status.idle":"2023-11-20T20:30:52.952337Z","shell.execute_reply.started":"2023-11-20T20:30:52.416186Z","shell.execute_reply":"2023-11-20T20:30:52.951205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the graph containing the data distribution of the night column in our histogram (or histplot), we caught sight of the bars skewing to the right, as there's more data plotted in the left side of the diagram. Additionally, the highest data recorded is in the data range between 0 and 1, with roughly 1200 entities, while the least data counted is in the data range around 62 to 68, 69 to 76, and 78 to 83, with approximately one or two data entities. To be specific, the right skew we witnessed from the left of our graph hinted us that there are less numbers of onset / onwake event pairs.","metadata":{}},{"cell_type":"markdown","source":"Now that we plotted the sleep data to our histplot, let's plot the event data columns to the bar chart! For plotting the event data column to our bar chart with seaborn, we use the sns module's barplot to configure our bar chart, setting the x parameter to the train_df dataframe's event column for setting up the x-axis of our bar chart.","metadata":{}},{"cell_type":"code","source":"sns.barplot(x=train_df[\"event\"].value_counts().index, y=train_df[\"event\"].value_counts().values)","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:52.954673Z","iopub.execute_input":"2023-11-20T20:30:52.955126Z","iopub.status.idle":"2023-11-20T20:30:53.206919Z","shell.execute_reply.started":"2023-11-20T20:30:52.955086Z","shell.execute_reply":"2023-11-20T20:30:53.205117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From what we saw in the bar chart distribution of the event column, we noticed that the bar heights of the onset and wakeup data values are the same, as both of them were counted up to approxminately 7300 entities. Henceforth, the same number of data values distributed from the event column implied to us that each night there is each pair of onset and wakeup event recorded in the train_df dataframe.","metadata":{}},{"cell_type":"markdown","source":"Last but not least, let's visualize the train_df dataframe's step column by distributing them to a displot graph (histogram)! Once again, we use the sns module's displot function, setting the train_df dataframe as the data for the displot followed by the x parameter to the step column for configuring the x-axis.","metadata":{}},{"cell_type":"code","source":"sns.displot(train_df, x=\"step\")","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:53.209473Z","iopub.execute_input":"2023-11-20T20:30:53.209959Z","iopub.status.idle":"2023-11-20T20:30:53.726223Z","shell.execute_reply.started":"2023-11-20T20:30:53.209917Z","shell.execute_reply":"2023-11-20T20:30:53.725135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Despite seeing the numbers in the x-axis clumped closely to each other, we glimpsed that the bars' height were tall at the left of the graph, implying that the bars were skewed to the right of the diagram. Additionally, the highest number of data recorded in the step column is around 2500 to 3000, with approximately 580 entities, while the lowest number of data recorded is in another data range roughly between 700000 to 740000, with around 1 or 2 entities. Overall, the right-skewed data distribution seen in the step column suggested to us that there are a few occurences of some events recorded by the accelerometer series.","metadata":{}},{"cell_type":"markdown","source":"As we finished visualizing three columns in the train_df dataframe, we completed visualizing most parts of the train_df dataframe! What's next we're going to visualize is the immense number of data recorded in the train_par_df dataframe (which it was created from the parquet file).","metadata":{}},{"cell_type":"markdown","source":"

Chapter 2: The train_par_df DataFrame Visualization

\n\nAfter we finished visualizing the data inside the train_df dataframe, we proceed to visualize another dataframe we created from the parquet file, known as the train_par_df, as it contained most of the accelerometer data that is used for training. Aside from the overview of the train_par_df dataframe, let's take a look at this dataframe we're about to visualize! \n* **series_id**: Specifies the identifier for each data series.\n* **step**: Specifies the integer timestep for each observation in each series.\n* **timestamp**: Specifies the datetime of when the data series was recorded.\n* **anglez**: Refers to the angle of the arm relative to the vertical axis of the body according to the GGIR package.\n* **enmo**: Specifies the Euclidean Norm Minus One of all accelerometer signals, with negative values rounded to zero, according to the GGIR package.","metadata":{}},{"cell_type":"markdown","source":"With all of the five data columns specified from the train_par_df dataframe, let's get started to visualize the data in this dataframe by visualizing the step data column in the histogram! We use the sns module's displot to plot a histogram diagram with Seaborn, setting the train_par_df as the data specified for the graph, and the x parameter set to the step column for configuring the x-axes for the graph.","metadata":{}},{"cell_type":"code","source":"sns.displot(train_par_df, x=\"step\")","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:30:53.727717Z","iopub.execute_input":"2023-11-20T20:30:53.728038Z","iopub.status.idle":"2023-11-20T20:32:11.163939Z","shell.execute_reply.started":"2023-11-20T20:30:53.728010Z","shell.execute_reply":"2023-11-20T20:32:11.162425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After waiting for a long time for the histogram graph to compile, we noticed that there is a big peak on the left side of the histogram chart, as the left peak of the diagram made a right-skewed distribution. In other words, the highest data recorded in the diagram is approximately 0.0 to 0.30 in 1e-6 scale, with around 280000 entities, while we can't determine the lowest data in the diagram because of the step data being scaled to 1e-6. Overall, the right-skew distribution we saw from the graph gave us some hints that most of the data series has less timesteps in the train_par_df dataframe.","metadata":{}},{"cell_type":"markdown","source":"Now let's distribute the relationship between the anglez and enmo data columns to the relplot diagram! But first, we redefine the train_par_df dataframe to limit the data to 1000 entities with the head function so that we will not allocate more memory in this notebook, then we use the sns module's relplot function to create a relationship graph, setting the data parameter to the train_par_df dataframe, the x parameter to the anglez column for configuring the x-axis, and the y parameter to the enmo column for configuring the y-axis.","metadata":{}},{"cell_type":"code","source":"train_par_df = train_par_df.head(1000)\nsns.relplot(data=train_par_df, x=\"anglez\", y=\"enmo\")","metadata":{"execution":{"iopub.status.busy":"2023-11-20T20:32:11.166008Z","iopub.execute_input":"2023-11-20T20:32:11.166321Z","iopub.status.idle":"2023-11-20T20:32:11.647733Z","shell.execute_reply.started":"2023-11-20T20:32:11.166294Z","shell.execute_reply":"2023-11-20T20:32:11.646340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After we created our relplot diagram, we noted that there's a bunch of plots plotted in the bottom of the graph, while there are a few plots being plotted on the top of the graph. Specifically, the number of data points at the bottom of the graph indicated that the less data in the enmo column means more common data for the anglez column from the train_par_df dataframe.","metadata":{}},{"cell_type":"markdown","source":"Now that we plotted two graphs in Seaborn with just three data columns from the train_par_df dataframe, we finished our entire data analysis of detecting sleep states!","metadata":{}},{"cell_type":"markdown","source":"

Conclusion

\n\nSo what are the takeaways from visualizing the data in the competition based on detecting the sleep states? Well, from the train_df dataframe, we highlighted that there are less numbers of onset and onwake events from the night data column, as well as seeing similar numbers of onset and wakeup events from the event column, since they both form into a pair. Meanwhile from the train_par_df dataframe, we reflected on how the step column as less timesteps for each data series of sleep events and observing how there's big clusters of data forming between the enmo and anglez data columns when the enmo column's data values is less. With our wrap up to our exploratory data analysis in detecting sleep states, always remember to get enough sleep because it will improve your memory and your health.","metadata":{}}]}