Stuck on a problem or have a question about the competition? Want to ask fellow Kagglers what infrastructure they're using? Want to share what you've learned so far?
\nAsk and share with other competition participants on this discussion board!
", "messages": [ { "id": "2524764", "postDate": "11/14/2023 14:15:27", "content": "Stuck on a problem or have a question about the competition? Want to ask fellow Kagglers what infrastructure they're using? Want to share what you've learned so far?
\nAsk and share with other competition participants on this discussion board!
", "rawMarkdown": "Stuck on a problem or have a question about the competition? Want to ask fellow Kagglers what infrastructure they're using? Want to share what you've learned so far?\n\nAsk and share with other competition participants on this discussion board!", "votes": null }, { "id": "2547376", "postDate": "12/03/2023 12:59:11", "content": "Seeking Clarifications on Timestamp Mapping and Data Consistency in Cool Quiet City Competition
\nDear Organizers,
\nI hope this message finds you well. My name is LAU, and I am currently participating in the Cool Quiet City competition on Kaggle. I am writing to seek clarification on some aspects of the provided data to ensure a better understanding and accurate analysis. Below are the identified issues along with references to the relevant statements and corresponding questions:
\nIssue 1: Timestamp Mapping in Survey Data
\n**Reference: **
\nhttps://www.kaggle.com/competitions/cool-quiet-city-competition/data
\n[Micro-survey data] module states that 'ws_timestamp_start' is the 'Timestamp of when the micro-survey was started'.
\n[Physiological / Environmental / Activity data] module states that ‘index’ is the ‘Timestamp (UTC) of when micro-survey was submitted, also serves as index’.
**Question: **
\nIn 'cozie_responses_and_physiological_data_test_public.csv' and 'cozie_responses_and_physiological_data_survey_only.csv,' both with 'time' as the index, it seems that 'time' is different from 'ws_timestamp_start' in the same row. Could you provide insights into the mapping relationship between 'time' and 'ws_timestamp_start'?
Issue 2: Timestamps for Physiological Data (e.g., ts_heart_rate)
\n**Reference: **
\nhttps://www.kaggle.com/code/mariofrei/02-example-minimal-viable-ml-example/notebook,
\n[Interpolate health data] module mentions that 'Health data, e.g., ts_heart_rate, and the watch survey responses, e.g., q_thermal_preference, do not have timestamps.'
\nhttps://www.kaggle.com/code/mariofrei/01-example-basic-visualizations#Plot-Cozie-data,
\n[Plot heart rate of all participants] module utilizes 'time' for plotting heart rate changes over time.
\nhttps://developer.apple.com/documentation/healthkit/hkquantitytypeidentifier/3174858-applestandtime
\n[restingHeartRate] states that ‘These samples use count/time units (described in HKUnit) and measure discrete values (described in HKQuantityAggregationStyle)’.
\n[cozie_responses_and_physiological_data]
\nId_unique time ts_step_count ts_walking_distance
\n23607 2022/10/18 22:46 13 10.4
\n23608 2022/10/18 22:46 527 359.1
\n23609 2022/10/18 22:48 112 69.9
**Question: **
\nFor columns like 'ts_heart_rate' and other 'ts'-prefixed indicators in csv files, do these indicators have associated timestamps? If so, could you elaborate on the mapping relationship between the timestamps for physiological data and the general 'time' column? Because it seems impossible that the participant walk hundreds of steps in one minute. Additionally, if there is any sampling involved in collecting these physiological data points, could you provide information on the time intervals used for sampling in the original data?
I greatly appreciate your time and assistance in clarifying these points. Understanding these aspects is crucial for ensuring the accuracy and reliability of the analyses conducted during the competition. Your guidance will be immensely valuable in enhancing the quality of our submissions.
\nThank you for organizing this competition and supporting participants. Looking forward to your response.
\nBest regards,
\nLAU
Hi Lau,
\nThank you for your interest in the data. Please find my responses to your questions below.
\nIssue 1: Could you provide insights into the mapping relationship between 'time' and 'ws_timestamp_start'?
\ntime is the timestamp of the end of the survey, i.e., when the submit button is pressed by the participant at the end of the watch survey. It would make sense to also name it ws_timestamp_end for these cases.
\nws_timestamp_start is the time when the participant starts the watch survey by opening the Cozie app on the Apple Watch, i.e., it marks the beginning of the watch survey. The timestamp in ws_timestamp_start is expected to be older then the timestamp in time. Usually, the two timestamps are about 10 seconds apart. The difference time - ws_timestamp_start is the time it took the participant to complete the watch survey, i.e., the duration.
Below is short snippet of code to illustrate the response.
\n\ndf_train = pd.read_csv(, index_col=[], keep_default_na=, na_values=[], parse_dates=[, , ], date_format=)\ndf = df_train.copy()\n\n\ndf = df[df.ws_survey_count.notna()]\n\n\ndf = df.reset_index()\ndf[] = df[] - df[]\ndf[] = df[].dt.total_seconds()\n\n\n(, df[].())\n(, df[].mean())\n(, df[].median())\n(, df[].())\n\n\n_=df[].hist(bins=((,,)))\nIssue 2: For columns like 'ts_heart_rate' and other 'ts'-prefixed indicators in csv files, do these indicators have associated timestamps? If so, could you elaborate on the mapping relationship between the timestamps for physiological data and the general 'time' column?
\nThe text is incorrect. I meant to write that these two categories usually do not have coinciding timestamps, i.e., when there is a value for q_thermal_preference, there is usually no value with the same timestamp forts_heart_rate.
\nThe timestamp in the time column is the timestamp of the sample time for 'ts'-prefixed columns.
\nI updated the notebook accordingly. Thank you for pointing this out.
Original text:
\n\n\nHealth data, e.g.,
\nts_heart_reate, and the watch survey responses, e.g.,q_thermal_preference, do not have the timestamps. Hence, the health data needs to be interpolated at the timestamps of the watch survey responses.
Updated text:
\n\n\nHealth data, e.g.,
\nts_heart_reate, and the watch survey responses, e.g.,q_thermal_preference, do not have coinciding timestamps. Hence, the health data needs to be interpolated at the timestamps of the watch survey responses.
Issue 2: […] Because it seems impossible that the participant walk hundreds of steps in one minute. Additionally, if there is any sampling involved in collecting these physiological data points, could you provide information on the time intervals used for sampling in the original data?
\nThe Cozie app retrieves raw data from the HealthKit API on iOS. There is intentional resampling or aggregation happening in the app or in the data pre-processing of the data for this competition.
\nThe timestamp for ts_step_count is forward-looking, i.e., it marks the first step in the sample.
\nIn the example below, the participant took 13 steps between 22:46:40.883 and 22:46:43.935 and 527 steps between 22:46:43.935 and 22:48:36.234, which results in a cadence of 255 and 281 steps per minute respectively and a running speed of 12.3 and 11.5 km/h which should be within the limits of physics and human biology.
| Id_unique | \ntime | \nts_step_count | \nts_walking_distance | \n
|---|---|---|---|
| 23607 | \n2022/10/18 22:46 | \n13 | \n10.4 | \n
| 23608 | \n2022/10/18 22:46 | \n527 | \n359.1 | \n
| 23609 | \n2022/10/18 22:48 | \n112 | \n69.9 | \n
Below is a short snippet of code to illustrate the example
\n\ndf_train = pd.read_csv(, index_col=[], keep_default_na=, na_values=[], parse_dates=[, , ], date_format=)\ndf = df_train.copy()\ndf = df.reset_index()\n\n\nid_participant_focus = \ndf = df[df.id_participant == id_participant_focus]\n\n\ndf = df[df.ts_step_count.notna()]\n\n\ndf[] = df.time.shift(-)\ndf[] = (df.timestamp_next - df.time).dt.total_seconds()/ \ndf[] = df.ts_step_count / df.timestamp_delta \ndf[] = (df.ts_walking_distance/) / (df.timestamp_delta/) \n\n\ndf = df[(df.id_unique>=)&(df.id_unique<=)]\ndf[[, , , , , , , , ]]\nAnyhow, this is a valid question and led me to make a short amendment to the table on the data tab of this competition. Thank you for the input.
\nI hope the responses above bring clarity to your questions. If you need more clarification on these questions or have any more questions, feel free to reach out again. I appreciate your thorough analysis of the data and the questions.
\nKind regards,
\nMario
Hi Mario,
\nThank you for your quick response and for shedding light on certain aspects of the dataset. I now have a better understanding of the details you provided.
\nI appreciate the clarification regarding the \"data_training\" and \"data_test_public\" dataset:
\nFor questionnaire data, the 'time' field indicates submission time, while for wristwatch monitoring data, it denotes the sampling time. The consolidation of these two datasets demonstrates thoughtful preprocessing.
It's clear that fields like 'ts_step_count' include activities like walking, running and other states, and represent incremental changes in specific intervals relative to the row's time(excluding questionnaire instances).
Thank you for your efforts in making this information accessible.
\nWarm regards,
\nLAU
Hello, is there any reason on why we need to predict the result of only 4 survey questions instead of all survey questions? I feel like because the way the data is generated, if you already know a participant's answer to one survey question, you should already know their answer to the entire survey. However, that doesn't seem to be the case in the test dataset.
\nI wonder if the model we are building can actually be deployed in real life situation, because right now, there is bound to be leakage due to the fact we can use features that we shouldn't have known when predicting the result of those 4 survey questions in my understanding.
", "rawMarkdown": "Hello, is there any reason on why we need to predict the result of only 4 survey questions instead of all survey questions? I feel like because the way the data is generated, if you already know a participant's answer to one survey question, you should already know their answer to the entire survey. However, that doesn't seem to be the case in the test dataset.\n\nI wonder if the model we are building can actually be deployed in real life situation, because right now, there is bound to be leakage due to the fact we can use features that we shouldn't have known when predicting the result of those 4 survey questions in my understanding.", "votes": null }, { "id": "2558561", "postDate": "12/12/2023 08:07:24", "content": "Dear Iqbal,
\nThank you for your question and your interest in the competition.
\nWe generally agree with your assessment that some of the survey responses could be used to predict the responses required by the competition.
\nHowever, to us, it would be interesting to see a demonstration that it is, in fact, possible and how good that works.
\nIf it turns out to be doable in a reliable way, that would help us to shorten the watch survey in future deployments and make future research experiments more efficient.
\nBut you are, of course, right. Ideally, this whole endeavor would eventually lead to reliable comfort predictions that don't require any user inputs.
\nOne could even take that thought further and try to predict the comfort using as little data collected with the Apple Watch. So, the predictive strength of each feature is interesting to us.
I hope the response above answers your question.
\nPlease feel free to reach out again if you have any more questions.
Best,
\nMario
Thank you @mariofrei! May I ask another question? Could I know how you split the dataset into train and test? The way I see it now, it seems to be completely random. I have also done adversarial validation and it seems that both train and test datasets have almost completely distinct distribution. This is understandable if you split them in a way that observations in the test dataset always happen after observations in train dataset. However, that doesn't seem to be the case here.
", "rawMarkdown": "Thank you @mariofrei! May I ask another question? Could I know how you split the dataset into train and test? The way I see it now, it seems to be completely random. I have also done adversarial validation and it seems that both train and test datasets have almost completely distinct distribution. This is understandable if you split them in a way that observations in the test dataset always happen after observations in train dataset. However, that doesn't seem to be the case here.", "votes": null }, { "id": "2667260", "postDate": "02/25/2024 02:51:40", "content": "Hello, since the competition is over, is there a date for when the winner announcement will be made?
", "rawMarkdown": "Hello, since the competition is over, is there a date for when the winner announcement will be made?", "votes": null }, { "id": "2669125", "postDate": "02/26/2024 05:43:43", "content": "Hey Iqbal -- we took the weekend to verify the results and have just released the Private Leaderboard! Congrats on your 2nd placing!!!
", "rawMarkdown": "Hey Iqbal -- we took the weekend to verify the results and have just released the Private Leaderboard! Congrats on your 2nd placing!!!", "votes": null } ], "comments": [ { "id": 2547376, "author_name": "liuxinwhu", "author_url": "", "post_date": "12/03/2023 12:59:11", "content": "Seeking Clarifications on Timestamp Mapping and Data Consistency in Cool Quiet City Competition
\nDear Organizers,
\nI hope this message finds you well. My name is LAU, and I am currently participating in the Cool Quiet City competition on Kaggle. I am writing to seek clarification on some aspects of the provided data to ensure a better understanding and accurate analysis. Below are the identified issues along with references to the relevant statements and corresponding questions:
\nIssue 1: Timestamp Mapping in Survey Data
\n**Reference: **
\nhttps://www.kaggle.com/competitions/cool-quiet-city-competition/data
\n[Micro-survey data] module states that 'ws_timestamp_start' is the 'Timestamp of when the micro-survey was started'.
\n[Physiological / Environmental / Activity data] module states that ‘index’ is the ‘Timestamp (UTC) of when micro-survey was submitted, also serves as index’.
**Question: **
\nIn 'cozie_responses_and_physiological_data_test_public.csv' and 'cozie_responses_and_physiological_data_survey_only.csv,' both with 'time' as the index, it seems that 'time' is different from 'ws_timestamp_start' in the same row. Could you provide insights into the mapping relationship between 'time' and 'ws_timestamp_start'?
Issue 2: Timestamps for Physiological Data (e.g., ts_heart_rate)
\n**Reference: **
\nhttps://www.kaggle.com/code/mariofrei/02-example-minimal-viable-ml-example/notebook,
\n[Interpolate health data] module mentions that 'Health data, e.g., ts_heart_rate, and the watch survey responses, e.g., q_thermal_preference, do not have timestamps.'
\nhttps://www.kaggle.com/code/mariofrei/01-example-basic-visualizations#Plot-Cozie-data,
\n[Plot heart rate of all participants] module utilizes 'time' for plotting heart rate changes over time.
\nhttps://developer.apple.com/documentation/healthkit/hkquantitytypeidentifier/3174858-applestandtime
\n[restingHeartRate] states that ‘These samples use count/time units (described in HKUnit) and measure discrete values (described in HKQuantityAggregationStyle)’.
\n[cozie_responses_and_physiological_data]
\nId_unique time ts_step_count ts_walking_distance
\n23607 2022/10/18 22:46 13 10.4
\n23608 2022/10/18 22:46 527 359.1
\n23609 2022/10/18 22:48 112 69.9
**Question: **
\nFor columns like 'ts_heart_rate' and other 'ts'-prefixed indicators in csv files, do these indicators have associated timestamps? If so, could you elaborate on the mapping relationship between the timestamps for physiological data and the general 'time' column? Because it seems impossible that the participant walk hundreds of steps in one minute. Additionally, if there is any sampling involved in collecting these physiological data points, could you provide information on the time intervals used for sampling in the original data?
I greatly appreciate your time and assistance in clarifying these points. Understanding these aspects is crucial for ensuring the accuracy and reliability of the analyses conducted during the competition. Your guidance will be immensely valuable in enhancing the quality of our submissions.
\nThank you for organizing this competition and supporting participants. Looking forward to your response.
\nBest regards,
\nLAU
Hi Lau,
\nThank you for your interest in the data. Please find my responses to your questions below.
\nIssue 1: Could you provide insights into the mapping relationship between 'time' and 'ws_timestamp_start'?
\ntime is the timestamp of the end of the survey, i.e., when the submit button is pressed by the participant at the end of the watch survey. It would make sense to also name it ws_timestamp_end for these cases.
\nws_timestamp_start is the time when the participant starts the watch survey by opening the Cozie app on the Apple Watch, i.e., it marks the beginning of the watch survey. The timestamp in ws_timestamp_start is expected to be older then the timestamp in time. Usually, the two timestamps are about 10 seconds apart. The difference time - ws_timestamp_start is the time it took the participant to complete the watch survey, i.e., the duration.
Below is short snippet of code to illustrate the response.
\n\ndf_train = pd.read_csv(, index_col=[], keep_default_na=, na_values=[], parse_dates=[, , ], date_format=)\ndf = df_train.copy()\n\n\ndf = df[df.ws_survey_count.notna()]\n\n\ndf = df.reset_index()\ndf[] = df[] - df[]\ndf[] = df[].dt.total_seconds()\n\n\n(, df[].())\n(, df[].mean())\n(, df[].median())\n(, df[].())\n\n\n_=df[].hist(bins=((,,)))\nIssue 2: For columns like 'ts_heart_rate' and other 'ts'-prefixed indicators in csv files, do these indicators have associated timestamps? If so, could you elaborate on the mapping relationship between the timestamps for physiological data and the general 'time' column?
\nThe text is incorrect. I meant to write that these two categories usually do not have coinciding timestamps, i.e., when there is a value for q_thermal_preference, there is usually no value with the same timestamp forts_heart_rate.
\nThe timestamp in the time column is the timestamp of the sample time for 'ts'-prefixed columns.
\nI updated the notebook accordingly. Thank you for pointing this out.
Original text:
\n\n\nHealth data, e.g.,
\nts_heart_reate, and the watch survey responses, e.g.,q_thermal_preference, do not have the timestamps. Hence, the health data needs to be interpolated at the timestamps of the watch survey responses.
Updated text:
\n\n\nHealth data, e.g.,
\nts_heart_reate, and the watch survey responses, e.g.,q_thermal_preference, do not have coinciding timestamps. Hence, the health data needs to be interpolated at the timestamps of the watch survey responses.
Issue 2: […] Because it seems impossible that the participant walk hundreds of steps in one minute. Additionally, if there is any sampling involved in collecting these physiological data points, could you provide information on the time intervals used for sampling in the original data?
\nThe Cozie app retrieves raw data from the HealthKit API on iOS. There is intentional resampling or aggregation happening in the app or in the data pre-processing of the data for this competition.
\nThe timestamp for ts_step_count is forward-looking, i.e., it marks the first step in the sample.
\nIn the example below, the participant took 13 steps between 22:46:40.883 and 22:46:43.935 and 527 steps between 22:46:43.935 and 22:48:36.234, which results in a cadence of 255 and 281 steps per minute respectively and a running speed of 12.3 and 11.5 km/h which should be within the limits of physics and human biology.
| Id_unique | \ntime | \nts_step_count | \nts_walking_distance | \n
|---|---|---|---|
| 23607 | \n2022/10/18 22:46 | \n13 | \n10.4 | \n
| 23608 | \n2022/10/18 22:46 | \n527 | \n359.1 | \n
| 23609 | \n2022/10/18 22:48 | \n112 | \n69.9 | \n
Below is a short snippet of code to illustrate the example
\n\ndf_train = pd.read_csv(, index_col=[], keep_default_na=, na_values=[], parse_dates=[, , ], date_format=)\ndf = df_train.copy()\ndf = df.reset_index()\n\n\nid_participant_focus = \ndf = df[df.id_participant == id_participant_focus]\n\n\ndf = df[df.ts_step_count.notna()]\n\n\ndf[] = df.time.shift(-)\ndf[] = (df.timestamp_next - df.time).dt.total_seconds()/ \ndf[] = df.ts_step_count / df.timestamp_delta \ndf[] = (df.ts_walking_distance/) / (df.timestamp_delta/) \n\n\ndf = df[(df.id_unique>=)&(df.id_unique<=)]\ndf[[, , , , , , , , ]]\nAnyhow, this is a valid question and led me to make a short amendment to the table on the data tab of this competition. Thank you for the input.
\nI hope the responses above bring clarity to your questions. If you need more clarification on these questions or have any more questions, feel free to reach out again. I appreciate your thorough analysis of the data and the questions.
\nKind regards,
\nMario
Hi Mario,
\nThank you for your quick response and for shedding light on certain aspects of the dataset. I now have a better understanding of the details you provided.
\nI appreciate the clarification regarding the \"data_training\" and \"data_test_public\" dataset:
\nFor questionnaire data, the 'time' field indicates submission time, while for wristwatch monitoring data, it denotes the sampling time. The consolidation of these two datasets demonstrates thoughtful preprocessing.
It's clear that fields like 'ts_step_count' include activities like walking, running and other states, and represent incremental changes in specific intervals relative to the row's time(excluding questionnaire instances).
Thank you for your efforts in making this information accessible.
\nWarm regards,
\nLAU
Hello, is there any reason on why we need to predict the result of only 4 survey questions instead of all survey questions? I feel like because the way the data is generated, if you already know a participant's answer to one survey question, you should already know their answer to the entire survey. However, that doesn't seem to be the case in the test dataset.
\nI wonder if the model we are building can actually be deployed in real life situation, because right now, there is bound to be leakage due to the fact we can use features that we shouldn't have known when predicting the result of those 4 survey questions in my understanding.
", "votes": null, "replies": [ { "id": 2558561, "author_name": "mariofrei", "author_url": "", "post_date": "12/12/2023 08:07:24", "content": "Dear Iqbal,
\nThank you for your question and your interest in the competition.
\nWe generally agree with your assessment that some of the survey responses could be used to predict the responses required by the competition.
\nHowever, to us, it would be interesting to see a demonstration that it is, in fact, possible and how good that works.
\nIf it turns out to be doable in a reliable way, that would help us to shorten the watch survey in future deployments and make future research experiments more efficient.
\nBut you are, of course, right. Ideally, this whole endeavor would eventually lead to reliable comfort predictions that don't require any user inputs.
\nOne could even take that thought further and try to predict the comfort using as little data collected with the Apple Watch. So, the predictive strength of each feature is interesting to us.
I hope the response above answers your question.
\nPlease feel free to reach out again if you have any more questions.
Best,
\nMario
Thank you @mariofrei! May I ask another question? Could I know how you split the dataset into train and test? The way I see it now, it seems to be completely random. I have also done adversarial validation and it seems that both train and test datasets have almost completely distinct distribution. This is understandable if you split them in a way that observations in the test dataset always happen after observations in train dataset. However, that doesn't seem to be the case here.
", "votes": null, "replies": [] } ] } ] }, { "id": 2667260, "author_name": "iqbalsyahakbar", "author_url": "", "post_date": "02/25/2024 02:51:40", "content": "Hello, since the competition is over, is there a date for when the winner announcement will be made?
", "votes": null, "replies": [ { "id": 2669125, "author_name": "claytonmiller", "author_url": "", "post_date": "02/26/2024 05:43:43", "content": "Hey Iqbal -- we took the weekend to verify the results and have just released the Private Leaderboard! Congrats on your 2nd placing!!!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2524764": "Stuck on a problem or have a question about the competition? Want to ask fellow Kagglers what infrastructure they're using? Want to share what you've learned so far?\n\nAsk and share with other competition participants on this discussion board!", "2547376": "Seeking Clarifications on Timestamp Mapping and Data Consistency in Cool Quiet City Competition\n\nDear Organizers,\n\nI hope this message finds you well. My name is LAU, and I am currently participating in the Cool Quiet City competition on Kaggle. I am writing to seek clarification on some aspects of the provided data to ensure a better understanding and accurate analysis. Below are the identified issues along with references to the relevant statements and corresponding questions:\n\n****Issue 1: Timestamp Mapping in Survey Data****\n\n**Reference: **\n[https://www.kaggle.com/competitions/cool-quiet-city-competition/data](url)\n[Micro-survey data] module states that 'ws_timestamp_start' is the 'Timestamp of when the micro-survey was started'.\n[Physiological / Environmental / Activity data] module states that ‘index’ is the ‘Timestamp (UTC) of when micro-survey was submitted, also serves as index’.\n\n**Question: **\nIn 'cozie_responses_and_physiological_data_test_public.csv' and 'cozie_responses_and_physiological_data_survey_only.csv,' both with 'time' as the index, it seems that 'time' is different from 'ws_timestamp_start' in the same row. Could you provide insights into the mapping relationship between 'time' and 'ws_timestamp_start'?\n\n****Issue 2: Timestamps for Physiological Data (e.g., ts_heart_rate)****\n\n**Reference: **\n\n[https://www.kaggle.com/code/mariofrei/02-example-minimal-viable-ml-example/notebook](url), \n[Interpolate health data] module mentions that 'Health data, e.g., ts_heart_rate, and the watch survey responses, e.g., q_thermal_preference, do not have timestamps.'\n[https://www.kaggle.com/code/mariofrei/01-example-basic-visualizations#Plot-Cozie-data](url), \n[Plot heart rate of all participants] module utilizes 'time' for plotting heart rate changes over time.\n[https://developer.apple.com/documentation/healthkit/hkquantitytypeidentifier/3174858-applestandtime](url)\n[restingHeartRate] states that ‘These samples use count/time units (described in HKUnit) and measure discrete values (described in HKQuantityAggregationStyle)’.\n[cozie_responses_and_physiological_data]\nId_unique\ttime\t\tts_step_count\t\tts_walking_distance\n23607\t2022/10/18 22:46\t\t13\t\t10.4\n23608\t2022/10/18 22:46\t\t527\t\t359.1\n23609\t2022/10/18 22:48\t\t112\t\t69.9\n\n**Question: **\nFor columns like 'ts_heart_rate' and other 'ts'-prefixed indicators in csv files, do these indicators have associated timestamps? If so, could you elaborate on the mapping relationship between the timestamps for physiological data and the general 'time' column? Because it seems impossible that the participant walk hundreds of steps in one minute. Additionally, if there is any sampling involved in collecting these physiological data points, could you provide information on the time intervals used for sampling in the original data?\n\nI greatly appreciate your time and assistance in clarifying these points. Understanding these aspects is crucial for ensuring the accuracy and reliability of the analyses conducted during the competition. Your guidance will be immensely valuable in enhancing the quality of our submissions.\n\nThank you for organizing this competition and supporting participants. Looking forward to your response.\n\nBest regards,\nLAU", "2547597": "Hi Lau,\n\nThank you for your interest in the data. Please find my responses to your questions below.\n\n**Issue 1: Could you provide insights into the mapping relationship between 'time' and 'ws_timestamp_start'?**\n`time` is the timestamp of the end of the survey, i.e., when the submit button is pressed by the participant at the end of the watch survey. It would make sense to also name it `ws_timestamp_end` for these cases. \n`ws_timestamp_start` is the time when the participant starts the watch survey by opening the Cozie app on the Apple Watch, i.e., it marks the beginning of the watch survey. The timestamp in `ws_timestamp_start` is expected to be older then the timestamp in `time`. Usually, the two timestamps are about 10 seconds apart. The difference `time` - `ws_timestamp_start` is the time it took the participant to complete the watch survey, i.e., the duration.\n\nBelow is short snippet of code to illustrate the response.\n```python\n# Read data\ndf_train = pd.read_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_training.csv', index_col=[0], keep_default_na=False, na_values=[''], parse_dates=['time', 'ws_timestamp_location', 'ws_timestamp_start'], date_format='%Y-%m-%d %H:%M:%S.%f%z')\ndf = df_train.copy()\n\n# Filter for watch surveys\ndf = df[df.ws_survey_count.notna()]\n\n# Compute watch survey duration\ndf = df.reset_index()\ndf['ws_duration'] = df['time'] - df['ws_timestamp_start']\ndf['ws_duration'] = df['ws_duration'].dt.total_seconds()\n\n# Stats\nprint('Min', df['ws_duration'].min())\nprint('Mean', df['ws_duration'].mean())\nprint('Median', df['ws_duration'].median())\nprint('Max', df['ws_duration'].max())\n\n# Plot histogram\n_=df['ws_duration'].hist(bins=list(range(1,60,1)))\n```\n\n**Issue 2: For columns like 'ts_heart_rate' and other 'ts'-prefixed indicators in csv files, do these indicators have associated timestamps? If so, could you elaborate on the mapping relationship between the timestamps for physiological data and the general 'time' column?**\nThe text is incorrect. I meant to write that these two categories usually do not have coinciding timestamps, i.e., when there is a value for `q_thermal_preference`, there is usually no value with the same timestamp for`ts_heart_rate`. \nThe timestamp in the `time` column is the timestamp of the sample time for 'ts'-prefixed columns.\nI [updated the notebook ](https://www.kaggle.com/code/mariofrei/02-example-minimal-viable-ml-example)accordingly. Thank you for pointing this out.\n\nOriginal text:\n>Health data, e.g., `ts_heart_reate`, and the watch survey responses, e.g., `q_thermal_preference`, do not have the timestamps. Hence, the health data needs to be interpolated at the timestamps of the watch survey responses.\n\nUpdated text:\n>Health data, e.g., `ts_heart_reate`, and the watch survey responses, e.g., `q_thermal_preference`, do not have coinciding timestamps. Hence, the health data needs to be interpolated at the timestamps of the watch survey responses.\n\n**Issue 2: [...] Because it seems impossible that the participant walk hundreds of steps in one minute. Additionally, if there is any sampling involved in collecting these physiological data points, could you provide information on the time intervals used for sampling in the original data?**\nThe Cozie app retrieves raw data from the HealthKit API on iOS. There is intentional resampling or aggregation happening in the app or in the data pre-processing of the data for this competition. \nThe timestamp for `ts_step_count` is forward-looking, i.e., it marks the first step in the sample.\nIn the example below, the participant took 13 steps between 22:46:40.883 and 22:46:43.935 and 527 steps between 22:46:43.935 and 22:48:36.234, which results in a cadence of 255 and 281 steps per minute respectively and a running speed of 12.3 and 11.5 km/h which should be within the limits of physics and human biology.\n\n| Id_unique | time | ts_step_count | ts_walking_distance |\n| --- | --- | --- | ---|\n| 23607 | 2022/10/18 22:46 | 13 | 10.4 |\n| 23608 | 2022/10/18 22:46 | 527 | 359.1 |\n| 23609 | 2022/10/18 22:48 | 112 | 69.9 |\n\nBelow is a short snippet of code to illustrate the example\n```python\n# Read data\ndf_train = pd.read_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_training.csv', index_col=[0], keep_default_na=False, na_values=[''], parse_dates=['time', 'ws_timestamp_location', 'ws_timestamp_start'], date_format='%Y-%m-%d %H:%M:%S.%f%z')\ndf = df_train.copy()\ndf = df.reset_index()\n\n# Filter for single participant\nid_participant_focus = 'xesh001'\ndf = df[df.id_participant == id_participant_focus]\n\n# Filter for step count\ndf = df[df.ts_step_count.notna()]\n\n# Compute cadance and running/walking speed\ndf['timestamp_next'] = df.time.shift(-1)\ndf['timestamp_delta'] = (df.timestamp_next - df.time).dt.total_seconds()/60 # [Minutes]\ndf['cadence'] = df.ts_step_count / df.timestamp_delta # [SPM: Steps per Minute]\ndf['speed_running'] = (df.ts_walking_distance/1000) / (df.timestamp_delta/60) # [km/h]\n\n# Show data\ndf = df[(df.id_unique>=23607)&(df.id_unique<=23609)]\ndf[['time', 'timestamp_next', 'timestamp_delta', 'id_participant', 'id_unique', 'ts_step_count', 'ts_walking_distance', 'cadence', 'speed_running']]\n```\n\nAnyhow, this is a valid question and led me to make a short amendment to the table on the [data tab of this competition](https://www.kaggle.com/competitions/cool-quiet-city-competition/data). Thank you for the input.\n\nI hope the responses above bring clarity to your questions. If you need more clarification on these questions or have any more questions, feel free to reach out again. I appreciate your thorough analysis of the data and the questions.\n\nKind regards,\nMario", "2548021": "Hi Mario,\n\nThank you for your quick response and for shedding light on certain aspects of the dataset. I now have a better understanding of the details you provided.\n\nI appreciate the clarification regarding the \"data_training\" and \"data_test_public\" dataset:\n\n1. For questionnaire data, the 'time' field indicates submission time, while for wristwatch monitoring data, it denotes the sampling time. The consolidation of these two datasets demonstrates thoughtful preprocessing.\n\n2. It's clear that fields like 'ts_step_count' include activities like walking, running and other states, and represent incremental changes in specific intervals relative to the row's time(excluding questionnaire instances).\n\nThank you for your efforts in making this information accessible.\n\nWarm regards,\nLAU", "2557444": "Hello, is there any reason on why we need to predict the result of only 4 survey questions instead of all survey questions? I feel like because the way the data is generated, if you already know a participant's answer to one survey question, you should already know their answer to the entire survey. However, that doesn't seem to be the case in the test dataset.\n\nI wonder if the model we are building can actually be deployed in real life situation, because right now, there is bound to be leakage due to the fact we can use features that we shouldn't have known when predicting the result of those 4 survey questions in my understanding.", "2558561": "Dear Iqbal,\n\nThank you for your question and your interest in the competition.\nWe generally agree with your assessment that some of the survey responses could be used to predict the responses required by the competition.\nHowever, to us, it would be interesting to see a demonstration that it is, in fact, possible and how good that works.\nIf it turns out to be doable in a reliable way, that would help us to shorten the watch survey in future deployments and make future research experiments more efficient.\nBut you are, of course, right. Ideally, this whole endeavor would eventually lead to reliable comfort predictions that don't require any user inputs.\nOne could even take that thought further and try to predict the comfort using as little data collected with the Apple Watch. So, the predictive strength of each feature is interesting to us.\n\nI hope the response above answers your question.\nPlease feel free to reach out again if you have any more questions.\n\nBest,\nMario", "2563515": "Thank you @mariofrei! May I ask another question? Could I know how you split the dataset into train and test? The way I see it now, it seems to be completely random. I have also done adversarial validation and it seems that both train and test datasets have almost completely distinct distribution. This is understandable if you split them in a way that observations in the test dataset always happen after observations in train dataset. However, that doesn't seem to be the case here.", "2667260": "Hello, since the competition is over, is there a date for when the winner announcement will be made?", "2669125": "Hey Iqbal -- we took the weekend to verify the results and have just released the Private Leaderboard! Congrats on your 2nd placing!!!" }, "source": "meta" }