Hi,
\nIs there a hidden test set used during the score evaluation? There are 277 unique series id in the train dataset, so does the hidden test set have the exact same 277 unique series id? And what about the number of nights? Do we only need to predict one night event of each series id or actually there is more?
\nThanks!
", "messages": [ { "id": "2427519", "postDate": "09/07/2023 09:38:58", "content": "Hi,
\nIs there a hidden test set used during the score evaluation? There are 277 unique series id in the train dataset, so does the hidden test set have the exact same 277 unique series id? And what about the number of nights? Do we only need to predict one night event of each series id or actually there is more?
\nThanks!
", "rawMarkdown": "Hi,\n\nIs there a hidden test set used during the score evaluation? There are 277 unique series id in the train dataset, so does the hidden test set have the exact same 277 unique series id? And what about the number of nights? Do we only need to predict one night event of each series id or actually there is more? \n\nThanks!", "votes": null }, { "id": "2427749", "postDate": "09/07/2023 13:06:31", "content": "Hi @yus002,
\nThere is a hidden dataset for score evaluation -- actually two hidden datasets: one for the Public leaderboard (what you see now) and one for the Private leaderboard (which you will see once the competition ends). You can see from the Leaderboard page that the Public LB has about 25% of the total test data. The series used in train / public lb / private lb are distinct.
", "rawMarkdown": "Hi @yus002,\n\nThere is a hidden dataset for score evaluation -- actually two hidden datasets: one for the Public leaderboard (what you see now) and one for the Private leaderboard (which you will see once the competition ends). You can see from the Leaderboard page that the Public LB has about 25% of the total test data. The series used in train / public lb / private lb are distinct.", "votes": null }, { "id": "2427870", "postDate": "09/07/2023 14:15:50", "content": "Dear @ryanholbrook
\n\n\nThe series used in train / public lb / private lb are distinct.
\n
Are series_id of train and test data the same? I think that of test data is wrapped around that of train data.
Thanks for the information! I am still confused though. I read the \"Data\" section again and it says
\n\n\nNote that this is a Code Competition, in which the actual test set is hidden. In this public version, we give some sample data in the correct format to help you author your solutions. The full test set contains about 200 series.
\n
It only mentions the number of series id in the test set, but what about the number of nights? Should we predict the onset and wakeup time points for each of these 200 series for just one night, or should we say that each of these 200 series has many nights to predict?
\nThanks!
", "rawMarkdown": "Thanks for the information! I am still confused though. I read the \"Data\" section again and it says \n\n>Note that this is a Code Competition, in which the actual test set is hidden. In this public version, we give some sample data in the correct format to help you author your solutions. The full test set contains about 200 series.\n\nIt only mentions the number of series id in the test set, but what about the number of nights? Should we predict the onset and wakeup time points for each of these 200 series for just one night, or should we say that each of these 200 series has many nights to predict?\n\nThanks!", "votes": null }, { "id": "2429299", "postDate": "09/08/2023 14:18:02", "content": "In general, the series in the test set will have events for more than one night, just like those in the training set.
", "rawMarkdown": "In general, the series in the test set will have events for more than one night, just like those in the training set.", "votes": null }, { "id": "2430723", "postDate": "09/09/2023 14:29:33", "content": "I see. Thanks for the clarification!
", "rawMarkdown": "I see. Thanks for the clarification!", "votes": null }, { "id": "2430737", "postDate": "09/09/2023 14:42:53", "content": "Understood. Perhaps silence is the answer.
\nAlthough it is possible to create an individual predictive model for the 277 IDs that appear in the training data, I will design the model with versatility in mind. Thank you.
I'm quite confused about the test_series:
\nIt seems the test_series are the first 150 steps of the train_series for each id?
But then events in the test_series should be in the train_events?
Are we supposed to assume that the final evaluation is performed on such short periods (i.e. 150 steps)?
Could you provide an actual full test_series to understand better?
The answer from @ryanholbrook is below.
\n\n\nIn general, the series in the test set will have events for more than one night, just like those in the training set.
\n
In other words, there are IDs where \"step\" is continuous and lasts for more than one night, you may need to define \"night\" (onset/wakeup set) for yourself.
", "rawMarkdown": "The answer from @ryanholbrook is below.\n> In general, the series in the test set will have events for more than one night, just like those in the training set.\n\nIn other words, there are IDs where \"step\" is continuous and lasts for more than one night, you may need to define \"night\" (onset/wakeup set) for yourself.", "votes": null }, { "id": "2431541", "postDate": "09/10/2023 07:28:36", "content": "OK, but since the test_series are in the train_series for which we know the events, what's the point of submitting computed events for the test_series? Shouldn't those results already be in the train_events??
", "rawMarkdown": "OK, but since the test_series are in the train_series for which we know the events, what's the point of submitting computed events for the test_series? Shouldn't those results already be in the train_events??", "votes": null }, { "id": "2431553", "postDate": "09/10/2023 07:41:38", "content": "@dariobiasini You should probably check the test data \"test_series.parquet\" (what you're seeing now is a dummy sample) and the format of the submission file. We need to infer the \"step\", \"event\", and \"score\" corresponding to each \"series_id\" using only the features present in the test data.
", "rawMarkdown": "dariobiasini You should probably check the test data \"test_series.parquet\" (what you're seeing now is a dummy sample) and the format of the submission file. We need to infer the \"step\", \"event\", and \"score\" corresponding to each \"series_id\" using only the features present in the test data.", "votes": null }, { "id": "2431576", "postDate": "09/10/2023 08:07:00", "content": "@mitsuyasuhoshino ah ok i think i got it, so we only submit notebook and that's just the format of the output, thanks
", "rawMarkdown": "mitsuyasuhoshino ah ok i think i got it, so we only submit notebook and that's just the format of the output, thanks", "votes": null }, { "id": "2431829", "postDate": "09/10/2023 12:45:18", "content": "In such cases, it is understandable that Kaggle staff choose not to answer. And I personally will assume that the series in train and test set may fully intersect or not intersecting at all.
", "rawMarkdown": "In such cases, it is understandable that Kaggle staff choose not to answer. And I personally will assume that the series in train and test set may fully intersect or not intersecting at all.", "votes": null }, { "id": "2431834", "postDate": "09/10/2023 12:52:21", "content": "Thanks @rimbax. It is natural for test data to have unknown IDs, so I create a model with that assumption in mind.
", "rawMarkdown": "Thanks @rimbax. It is natural for test data to have unknown IDs, so I create a model with that assumption in mind.", "votes": null }, { "id": "2432728", "postDate": "09/11/2023 05:12:44", "content": "Hi @ryanholbrook what would (roughly) be the overall test size in term of rows / initial memory size ?
", "rawMarkdown": "Hi @ryanholbrook what would (roughly) be the overall test size in term of rows / initial memory size ?", "votes": null }, { "id": "2433356", "postDate": "09/11/2023 14:14:44", "content": "As mentioned in the data documentation, there are roughly 200 series in the test set. The test set series are similar to those in the training set.
", "rawMarkdown": "As mentioned in the data documentation, there are roughly 200 series in the test set. The test set series are similar to those in the training set.", "votes": null }, { "id": "2440155", "postDate": "09/15/2023 10:04:54", "content": "Hi @ryanholbrook.
\nIs there any overlap between series_id in train_series.parquet and series_id in test_series.parquet?
Hi @yudaiueno. I also asked the same question 8 days ago, but no answer. Judging from that, I think that completely different IDs are used for train and test.
", "rawMarkdown": "Hi @yudaiueno. I also asked the same question 8 days ago, but no answer. Judging from that, I think that completely different IDs are used for train and test.", "votes": null }, { "id": "2440396", "postDate": "09/15/2023 13:24:41", "content": "The series used in train / public lb / private lb are distinct. Since each series has its own unique id, there will also be no overlap of ids between splits.
Hi @ryanholbrook thank you very much for the information esp the dataset series ID separation
", "rawMarkdown": "Hi @ryanholbrook thank you very much for the information esp the dataset series ID separation", "votes": null }, { "id": "2447889", "postDate": "09/20/2023 09:52:15", "content": "Hi @ryanholbrook , during submission, is our notebook scored on both 25% and 75% set immediately? or do the 75% test set will only be tested after the competition ends by rerunning participants' notebooks?
", "rawMarkdown": "Hi @ryanholbrook , during submission, is our notebook scored on both 25% and 75% set immediately? or do the 75% test set will only be tested after the competition ends by rerunning participants' notebooks?", "votes": null }, { "id": "2448784", "postDate": "09/20/2023 20:02:58", "content": "@rimbax Your notebook is run against the entire test set (both the 25% and the 75%) at once for every submission.
", "rawMarkdown": "rimbax Your notebook is run against the entire test set (both the 25% and the 75%) at once for every submission.", "votes": null }, { "id": "2449002", "postDate": "09/20/2023 23:39:16", "content": "Great, thank you Ryan for your answer
", "rawMarkdown": "Great, thank you Ryan for your answer", "votes": null }, { "id": "2450042", "postDate": "09/21/2023 15:22:56", "content": "Dear @ryanholbrook
\nis it possible to get some information regarding code submissions - I'm getting failures due to (most likely) incorrect number of rows in the submission.csv - does the submission evaluation code expect at least one row per series or what is the restriction there? How could the csv not have the correct nr of rows? Couldn't find any info on this in the descriptions.
\nthanks
", "rawMarkdown": "Dear @ryanholbrook\n\nis it possible to get some information regarding code submissions - I'm getting failures due to (most likely) incorrect number of rows in the submission.csv - does the submission evaluation code expect at least one row per series or what is the restriction there? How could the csv not have the correct nr of rows? Couldn't find any info on this in the descriptions.\n\nthanks", "votes": null }, { "id": "2483848", "postDate": "10/16/2023 03:41:20", "content": "@ryanholbrook thank you for giving detail. It appears there might be some discrepancy or ambiguity in the responses regarding the test set. While YUE SUN's original question response you initially mentioned that the test set series are distinct from the training set, later Lucas Morin's response you mentioned that the test set series are similar to those in the training set.
\nPlease clarify!
", "rawMarkdown": "ryanholbrook thank you for giving detail. It appears there might be some discrepancy or ambiguity in the responses regarding the test set. While YUE SUN's original question response you initially mentioned that the test set series are distinct from the training set, later Lucas Morin's response you mentioned that the test set series are similar to those in the training set.\n\nPlease clarify!", "votes": null }, { "id": "2522824", "postDate": "11/13/2023 05:16:48", "content": "@ryanholbrook
\nHi, the length of the series for each of the series_ids is multiple of 180 steps. (Of course, in training set)
\nCan I assume it is also true for test set?
Hi @yus002,
\nThere is a hidden dataset for score evaluation -- actually two hidden datasets: one for the Public leaderboard (what you see now) and one for the Private leaderboard (which you will see once the competition ends). You can see from the Leaderboard page that the Public LB has about 25% of the total test data. The series used in train / public lb / private lb are distinct.
", "votes": null, "replies": [ { "id": 2428808, "author_name": "yus002", "author_url": "", "post_date": "09/08/2023 07:12:00", "content": "Thanks for the information! I am still confused though. I read the \"Data\" section again and it says
\n\n\nNote that this is a Code Competition, in which the actual test set is hidden. In this public version, we give some sample data in the correct format to help you author your solutions. The full test set contains about 200 series.
\n
It only mentions the number of series id in the test set, but what about the number of nights? Should we predict the onset and wakeup time points for each of these 200 series for just one night, or should we say that each of these 200 series has many nights to predict?
\nThanks!
", "votes": null, "replies": [ { "id": 2429299, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/08/2023 14:18:02", "content": "In general, the series in the test set will have events for more than one night, just like those in the training set.
", "votes": null, "replies": [ { "id": 2430723, "author_name": "yus002", "author_url": "", "post_date": "09/09/2023 14:29:33", "content": "I see. Thanks for the clarification!
", "votes": null, "replies": [ { "id": 2432728, "author_name": "lucasmorin", "author_url": "", "post_date": "09/11/2023 05:12:44", "content": "Hi @ryanholbrook what would (roughly) be the overall test size in term of rows / initial memory size ?
", "votes": null, "replies": [ { "id": 2433356, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/11/2023 14:14:44", "content": "As mentioned in the data documentation, there are roughly 200 series in the test set. The test set series are similar to those in the training set.
", "votes": null, "replies": [ { "id": 2440155, "author_name": "yudaiueno", "author_url": "", "post_date": "09/15/2023 10:04:54", "content": "Hi @ryanholbrook.
\nIs there any overlap between series_id in train_series.parquet and series_id in test_series.parquet?
Hi @yudaiueno. I also asked the same question 8 days ago, but no answer. Judging from that, I think that completely different IDs are used for train and test.
", "votes": null, "replies": [] }, { "id": 2440396, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/15/2023 13:24:41", "content": "The series used in train / public lb / private lb are distinct. Since each series has its own unique id, there will also be no overlap of ids between splits.
Hi @ryanholbrook thank you very much for the information esp the dataset series ID separation
", "votes": null, "replies": [] }, { "id": 2447889, "author_name": "rimbax", "author_url": "", "post_date": "09/20/2023 09:52:15", "content": "Hi @ryanholbrook , during submission, is our notebook scored on both 25% and 75% set immediately? or do the 75% test set will only be tested after the competition ends by rerunning participants' notebooks?
", "votes": null, "replies": [ { "id": 2448784, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/20/2023 20:02:58", "content": "@rimbax Your notebook is run against the entire test set (both the 25% and the 75%) at once for every submission.
", "votes": null, "replies": [ { "id": 2449002, "author_name": "rimbax", "author_url": "", "post_date": "09/20/2023 23:39:16", "content": "Great, thank you Ryan for your answer
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2427870, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/07/2023 14:15:50", "content": "Dear @ryanholbrook
\n\n\nThe series used in train / public lb / private lb are distinct.
\n
Are series_id of train and test data the same? I think that of test data is wrapped around that of train data.
Understood. Perhaps silence is the answer.
\nAlthough it is possible to create an individual predictive model for the 277 IDs that appear in the training data, I will design the model with versatility in mind. Thank you.
In such cases, it is understandable that Kaggle staff choose not to answer. And I personally will assume that the series in train and test set may fully intersect or not intersecting at all.
", "votes": null, "replies": [ { "id": 2431834, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/10/2023 12:52:21", "content": "Thanks @rimbax. It is natural for test data to have unknown IDs, so I create a model with that assumption in mind.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2431440, "author_name": "dariobiasini", "author_url": "", "post_date": "09/10/2023 06:03:05", "content": "\nI'm quite confused about the test_series:
\nIt seems the test_series are the first 150 steps of the train_series for each id?
But then events in the test_series should be in the train_events?
Are we supposed to assume that the final evaluation is performed on such short periods (i.e. 150 steps)?
Could you provide an actual full test_series to understand better?
The answer from @ryanholbrook is below.
\n\n\nIn general, the series in the test set will have events for more than one night, just like those in the training set.
\n
In other words, there are IDs where \"step\" is continuous and lasts for more than one night, you may need to define \"night\" (onset/wakeup set) for yourself.
", "votes": null, "replies": [ { "id": 2431541, "author_name": "dariobiasini", "author_url": "", "post_date": "09/10/2023 07:28:36", "content": "OK, but since the test_series are in the train_series for which we know the events, what's the point of submitting computed events for the test_series? Shouldn't those results already be in the train_events??
", "votes": null, "replies": [ { "id": 2431553, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/10/2023 07:41:38", "content": "@dariobiasini You should probably check the test data \"test_series.parquet\" (what you're seeing now is a dummy sample) and the format of the submission file. We need to infer the \"step\", \"event\", and \"score\" corresponding to each \"series_id\" using only the features present in the test data.
", "votes": null, "replies": [ { "id": 2431576, "author_name": "dariobiasini", "author_url": "", "post_date": "09/10/2023 08:07:00", "content": "@mitsuyasuhoshino ah ok i think i got it, so we only submit notebook and that's just the format of the output, thanks
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2450042, "author_name": "dariobiasini", "author_url": "", "post_date": "09/21/2023 15:22:56", "content": "Dear @ryanholbrook
\nis it possible to get some information regarding code submissions - I'm getting failures due to (most likely) incorrect number of rows in the submission.csv - does the submission evaluation code expect at least one row per series or what is the restriction there? How could the csv not have the correct nr of rows? Couldn't find any info on this in the descriptions.
\nthanks
", "votes": null, "replies": [] }, { "id": 2483848, "author_name": "oraware", "author_url": "", "post_date": "10/16/2023 03:41:20", "content": "@ryanholbrook thank you for giving detail. It appears there might be some discrepancy or ambiguity in the responses regarding the test set. While YUE SUN's original question response you initially mentioned that the test set series are distinct from the training set, later Lucas Morin's response you mentioned that the test set series are similar to those in the training set.
\nPlease clarify!
", "votes": null, "replies": [] }, { "id": 2522824, "author_name": "dongyk", "author_url": "", "post_date": "11/13/2023 05:16:48", "content": "@ryanholbrook
\nHi, the length of the series for each of the series_ids is multiple of 180 steps. (Of course, in training set)
\nCan I assume it is also true for test set?