I read both the parquet file and the csv file, but the step data in both are very different. In the CVS file steps are vey large float type data, while the parquet file, has \"step\" in integers starting from 0.
\nAre the step columns in both files supposed to be the same data?
", "messages": [ { "id": 2429990, "postDate": "2023-09-09T01:18:48.333Z", "content": "I read both the parquet file and the csv file, but the step data in both are very different. In the CVS file steps are vey large float type data, while the parquet file, has \"step\" in integers starting from 0.
\nAre the step columns in both files supposed to be the same data?
", "rawMarkdown": "I read both the parquet file and the csv file, but the step data in both are very different. In the CVS file steps are vey large float type data, while the parquet file, has \"step\" in integers starting from 0. \n\nAre the step columns in both files supposed to be the same data?", "votes": 2 }, { "id": 2435425, "postDate": "2023-09-13T01:08:38.490Z", "content": "@renatoreggiani has done a great job of developing features from the data. You can find his work here
\nHis work makes you think about what could potentially be inferred from the train data and how you may go about the prediction.
", "rawMarkdown": "@renatoreggiani has done a great job of developing features from the data. You can find his work [here](https://www.kaggle.com/code/renatoreggiani/feat-eng-ideas-60-memory-reduction-cmi?scriptVersionId=142568316)\n\nHis work makes you think about what could potentially be inferred from the train data and how you may go about the prediction.", "votes": 1, "replies": [ { "id": 2435448, "postDate": "2023-09-13T01:58:46.430Z", "content": "Tks @nadereafshar , I did this dataset from this feature engineering ideas, fell free to use it.
", "rawMarkdown": "Tks @nadereafshar , I did this [dataset](https://www.kaggle.com/datasets/renatoreggiani/zzzs-training-dataset-target-feat-eng) from this feature engineering ideas, fell free to use it.", "replies": [ { "id": 2438732, "postDate": "2023-09-14T13:17:17.480Z", "content": "Thank you @renatoreggiani - Beautiful work !
", "rawMarkdown": "Thank you @renatoreggiani - Beautiful work !" } ] } ] }, { "id": 2435430, "postDate": "2023-09-13T01:22:35.947Z", "content": "There is apparently a \"clean dataset\" for this competition since most of the 277 series are missing important data. The specific sets with complete data have been compiled here, along with a \"Target\" column:
", "rawMarkdown": "There is apparently a \"clean dataset\" for this competition since most of the 277 series are missing important data. The specific sets with complete data have been compiled [here](https://www.kaggle.com/code/carlmcbrideellis/zzzs-make-clean-starter-dataset-target), along with a \"Target\" column:\n\n\n" }, { "id": 2431504, "postDate": "2023-09-10T06:53:41.870Z", "content": "The step property in the series dataset can be thought of as an index, it starts from 0 and increments by one for every row. There are a subset of rows for every series which have been annotated withonset and wakeup events. One can combine the two datasets by joining it on [series_id, step] like we see below to get the merged dataset.
merged_df = pd.merge(\n train_series_df,\n train_events_df[[, , ]],\n on=[, ],\n how=\n)\nThank you kindly for the clarification. Can you also expand on the \"prediction\" requirements of the ultimate model? I am not exactly sure what is being asked to predict. Is the model supposed to predict the state of \"onset\" & \"awake\" from:
\na) given time of day?
\nb) combination of time and z-angle, and enmo?
\nc) some other combination of input?
Thanks agian
", "rawMarkdown": "Thank you kindly for the clarification. Can you also expand on the \"prediction\" requirements of the ultimate model? I am not exactly sure what is being asked to predict. Is the model supposed to predict the state of \"onset\" & \"awake\" from:\n\na) given time of day?\nb) combination of time and z-angle, and enmo?\nc) some other combination of input?\n\nThanks agian ", "replies": [ { "id": 2433805, "postDate": "2023-09-11T21:04:20.080Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2434383, "postDate": "2023-09-12T09:11:06.860Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2434169, "postDate": "2023-09-12T06:23:00.897Z", "content": "Do you mean that a \"step\" in parquet can be defined as a counter of every onset / wakeup event for a unique series ID? If yes than that part is clear. However, what does a step signify in CSV file? Doesn't look like the step in both the files are related.
", "rawMarkdown": "Do you mean that a \"step\" in parquet can be defined as a counter of every onset / wakeup event for a unique series ID? If yes than that part is clear. However, what does a step signify in CSV file? Doesn't look like the step in both the files are related.", "replies": [ { "id": 2436903, "postDate": "2023-09-13T20:45:33.023Z", "content": "The parquet time series, for a given series_id, represents the data from a single person's wrist accelerometer: its anglez and enmo values are sampled (and stored) every 5 seconds.
\nThe step=0 corresponds to when the accelerometer was reset and/or started on the person (one would expect this to be while they are awake 😃 )
\nEvery 5 seconds is another step, 12 steps (samples) per minute or 17280 steps per day; a single series covers many days, about 27 on average.
The csv file lists two step values for every 24 hours in each series_id. One of these is \"onset\" indicating when the person began their main sleep period that day and \"wakeup\" when they next woke up. These step values are the appropriate ones from the parquet file.
\nSo for every 17280 steps in the parquet series file, we expect (only) 2 rows in the csv events file.
I hope this helps!
", "rawMarkdown": "The parquet time series, for a given series_id, represents the data from a single person's wrist accelerometer: its anglez and enmo values are sampled (and stored) every 5 seconds.\nThe step=0 corresponds to when the accelerometer was reset and/or started on the person (one would expect this to be while they are awake 😃 )\nEvery 5 seconds is another step, 12 steps (samples) per minute or 17280 steps per day; a single series covers many days, about 27 on average.\n\nThe csv file lists two step values for every 24 hours in each series_id. One of these is \"onset\" indicating when the person began their main sleep period that day and \"wakeup\" when they next woke up. These step values are the appropriate ones from the parquet file.\nSo for every 17280 steps in the parquet series file, we expect (only) 2 rows in the csv events file.\n\nI hope this helps!", "votes": 1 } ] } ] }, { "id": 2431204, "postDate": "2023-09-09T22:14:43.633Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 2433433, "postDate": "2023-09-11T15:31:05.180Z", "content": "Thank you Simon
", "rawMarkdown": "Thank you Simon" } ] } ], "comments": [ { "id": 2435425, "author_name": "Nader E Afshar", "author_url": "", "post_date": "2023-09-13T01:08:38.490000", "content": "@renatoreggiani has done a great job of developing features from the data. You can find his work here
\nHis work makes you think about what could potentially be inferred from the train data and how you may go about the prediction.
", "votes": 1, "replies": [ { "id": 2435448, "author_name": "Renato Reggiani", "author_url": "", "post_date": "2023-09-13T01:58:46.430000", "content": "Tks @nadereafshar , I did this dataset from this feature engineering ideas, fell free to use it.
", "votes": 0, "replies": [ { "id": 2438732, "author_name": "Nader E Afshar", "author_url": "", "post_date": "2023-09-14T13:17:17.480000", "content": "Thank you @renatoreggiani - Beautiful work !
", "votes": 0, "replies": [] } ] } ] }, { "id": 2435430, "author_name": "Nader E Afshar", "author_url": "", "post_date": "2023-09-13T01:22:35.947000", "content": "There is apparently a \"clean dataset\" for this competition since most of the 277 series are missing important data. The specific sets with complete data have been compiled here, along with a \"Target\" column:
", "votes": 0, "replies": [] }, { "id": 2431504, "author_name": "Bhargav Kowshik", "author_url": "", "post_date": "2023-09-10T06:53:41.870000", "content": "The step property in the series dataset can be thought of as an index, it starts from 0 and increments by one for every row. There are a subset of rows for every series which have been annotated withonset and wakeup events. One can combine the two datasets by joining it on [series_id, step] like we see below to get the merged dataset.
merged_df = pd.merge(\n train_series_df,\n train_events_df[[, , ]],\n on=[, ],\n how=\n)\nThank you kindly for the clarification. Can you also expand on the \"prediction\" requirements of the ultimate model? I am not exactly sure what is being asked to predict. Is the model supposed to predict the state of \"onset\" & \"awake\" from:
\na) given time of day?
\nb) combination of time and z-angle, and enmo?
\nc) some other combination of input?
Thanks agian
", "votes": 0, "replies": [ { "id": 2433805, "author_name": "", "author_url": "", "post_date": "2023-09-11T21:04:20.080000", "content": "", "votes": 1, "replies": [] }, { "id": 2434383, "author_name": "", "author_url": "", "post_date": "2023-09-12T09:11:06.860000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2434169, "author_name": "TheChessEngineer", "author_url": "", "post_date": "2023-09-12T06:23:00.897000", "content": "Do you mean that a \"step\" in parquet can be defined as a counter of every onset / wakeup event for a unique series ID? If yes than that part is clear. However, what does a step signify in CSV file? Doesn't look like the step in both the files are related.
", "votes": 0, "replies": [ { "id": 2436903, "author_name": "Daniel Dewey", "author_url": "", "post_date": "2023-09-13T20:45:33.023000", "content": "The parquet time series, for a given series_id, represents the data from a single person's wrist accelerometer: its anglez and enmo values are sampled (and stored) every 5 seconds.
\nThe step=0 corresponds to when the accelerometer was reset and/or started on the person (one would expect this to be while they are awake 😃 )
\nEvery 5 seconds is another step, 12 steps (samples) per minute or 17280 steps per day; a single series covers many days, about 27 on average.
The csv file lists two step values for every 24 hours in each series_id. One of these is \"onset\" indicating when the person began their main sleep period that day and \"wakeup\" when they next woke up. These step values are the appropriate ones from the parquet file.
\nSo for every 17280 steps in the parquet series file, we expect (only) 2 rows in the csv events file.
I hope this helps!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2431204, "author_name": "", "author_url": "", "post_date": "2023-09-09T22:14:43.633000", "content": "", "votes": 2, "replies": [ { "id": 2433433, "author_name": "Nader E Afshar", "author_url": "", "post_date": "2023-09-11T15:31:05.180000", "content": "Thank you Simon
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2429990": "I read both the parquet file and the csv file, but the step data in both are very different. In the CVS file steps are vey large float type data, while the parquet file, has \"step\" in integers starting from 0. \n\nAre the step columns in both files supposed to be the same data?", "2435425": "@renatoreggiani has done a great job of developing features from the data. You can find his work [here](https://www.kaggle.com/code/renatoreggiani/feat-eng-ideas-60-memory-reduction-cmi?scriptVersionId=142568316)\n\nHis work makes you think about what could potentially be inferred from the train data and how you may go about the prediction.", "2435430": "There is apparently a \"clean dataset\" for this competition since most of the 277 series are missing important data. The specific sets with complete data have been compiled [here](https://www.kaggle.com/code/carlmcbrideellis/zzzs-make-clean-starter-dataset-target), along with a \"Target\" column:\n\n\n", "2431504": "The `step` property in the series dataset can be thought of as an `index`, it starts from `0` and increments by one for every row. There are a subset of rows for every series which have been annotated with`onset` and `wakeup` events. One can combine the two datasets by joining it on `[series_id, step]` like we see below to get the merged dataset.\n\n```python\nmerged_df = pd.merge(\n train_series_df,\n train_events_df[['series_id', 'step', 'event']],\n on=['series_id', 'step'],\n how='left'\n)\n```", "2431204": "" } }