I was reading the data paper and it indicates that there were 6 weight series, 2 friction series, and 2 mixed series.
\n\nSince we're predicting for series 9 and 10, I was wondering if these series numbers meant anything (whether we are given information on whether each series corresponds to a particular type of series)
\n\nI know this is off-topic but I am also wondering if there is a run-time constraint on this competition.
\n\nThanks,
", "messages": [ { "id": "86764", "postDate": "07/24/2015 06:56:11", "content": "I was reading the data paper and it indicates that there were 6 weight series, 2 friction series, and 2 mixed series.
\n\nSince we're predicting for series 9 and 10, I was wondering if these series numbers meant anything (whether we are given information on whether each series corresponds to a particular type of series)
\n\nI know this is off-topic but I am also wondering if there is a run-time constraint on this competition.
\n\nThanks,
", "rawMarkdown": "I was reading the data paper and it indicates that there were 6 weight series, 2 friction series, and 2 mixed series.\r\n\r\nSince we're predicting for series 9 and 10, I was wondering if these series numbers meant anything (whether we are given information on whether each series corresponds to a particular type of series)\r\n\r\nI know this is off-topic but I am also wondering if there is a run-time constraint on this competition.\r\n\r\nThanks,", "votes": null }, { "id": "86782", "postDate": "07/24/2015 10:23:16", "content": "For the given task the type of series shouldn't matter that much and both series 9 and 10 are always weight-series. So don't worry too much about that.
\n\n@off-topic: We decided not to set any runtime constraints for this competition.
", "rawMarkdown": "For the given task the type of series shouldn't matter that much and both series 9 and 10 are always weight-series. So don't worry too much about that.\r\n\r\n@off-topic: We decided not to set any runtime constraints for this competition.", "votes": null }, { "id": "89192", "postDate": "08/12/2015 15:52:46", "content": "I have some additional question on the topic of series numbering:
\n\n1) Does the series number represent the chronological order of experiments? If not, and the series number is some permutation of the chronological order, is it the same permutation for all subjects?
\n\n2) Is the test set split to private and public subsets according to the series number (e.g. series 9 is public and series 10 is private), or does the public test set contain parts of both series?
\n\nThanks,
", "rawMarkdown": "I have some additional question on the topic of series numbering:\r\n\r\n1) Does the series number represent the chronological order of experiments? If not, and the series number is some permutation of the chronological order, is it the same permutation for all subjects?\r\n\r\n2) Is the test set split to private and public subsets according to the series number (e.g. series 9 is public and series 10 is private), or does the public test set contain parts of both series?\r\n\r\nThanks,", "votes": null }, { "id": "89204", "postDate": "08/12/2015 18:10:51", "content": "@Yair, I second your question #2. I was wondering about the public/private split myself. Setting up local validation is tricker depending on which series you choose to validate with and I was wondering if the public is series 9 only and the private is 10, then, there is a good chance to overfit to 9 only.
", "rawMarkdown": "Yair, I second your question #2. I was wondering about the public/private split myself. Setting up local validation is tricker depending on which series you choose to validate with and I was wondering if the public is series 9 only and the private is 10, then, there is a good chance to overfit to 9 only.", "votes": null }, { "id": "89207", "postDate": "08/12/2015 18:22:23", "content": "Hi Yair, Hi Deep,
\n\nFor the #2, the admin stated in another thread that the public/private split is indeed corresponding to series 9/10. This is also because the labels of series9 are openly available as part of the dataset they released on nature.com
\n\nAbout #1, according to the supplementary information given with the paper, series for 1 to 7 are in chronological order. then series 8,9 and 10 are randomized. So S10 is not always the last series, but s1 is always the first.
\n\nNote that explicitly using these information is a grey area, since it can be seen as a use of external data.
", "rawMarkdown": "Hi Yair, Hi Deep,\r\n\r\nFor the #2, the admin stated in another thread that the public/private split is indeed corresponding to series 9/10. This is also because the labels of series9 are openly available as part of the dataset they released on nature.com\r\n\r\nAbout #1, according to the supplementary information given with the paper, series for 1 to 7 are in chronological order. then series 8,9 and 10 are randomized. So S10 is not always the last series, but s1 is always the first.\r\n\r\nNote that explicitly using these information is a grey area, since it can be seen as a use of external data.", "votes": null }, { "id": "89223", "postDate": "08/12/2015 21:26:06", "content": "Thanks for the answers Alexandre - I should have given the paper a more thorough look...
\n\nAdmins - can you comment about fair use of this information? To me using labels from series 9 seems unfair, but making cross-validation splits between series 1-8 based on chronological order seems reasonable.
", "rawMarkdown": "Thanks for the answers Alexandre - I should have given the paper a more thorough look...\r\n\r\nAdmins - can you comment about fair use of this information? To me using labels from series 9 seems unfair, but making cross-validation splits between series 1-8 based on chronological order seems reasonable.", "votes": null }, { "id": "89228", "postDate": "08/12/2015 22:04:40", "content": "Thanks, @Alexandre!
\n\nBut, I am still concerned about overfitting to 9. If you use one full series as a validation set and train on others, depending on the choice of the validation series and how similar that choice is to 9 vs. 10, there's a very good chance for overfitting to 9. It would have been nice if the public/private split was random samples from both 9 and 10. So, it begs to be careful in your local validation sample choice. Am I the only one concered about this?
\n\nThe fact that we know which one is public and which one is private and other factors including that nature paper makes this competition different from others and there seems to be a lot is offered to exploit. But, at the same time, one has to be careful not to fall for overfitting for 9.
", "rawMarkdown": "Thanks, @Alexandre! \r\n\r\nBut, I am still concerned about overfitting to 9. If you use one full series as a validation set and train on others, depending on the choice of the validation series and how similar that choice is to 9 vs. 10, there's a very good chance for overfitting to 9. It would have been nice if the public/private split was random samples from both 9 and 10. So, it begs to be careful in your local validation sample choice. Am I the only one concered about this? \r\n\r\nThe fact that we know which one is public and which one is private and other factors including that nature paper makes this competition different from others and there seems to be a lot is offered to exploit. But, at the same time, one has to be careful not to fall for overfitting for 9.", "votes": null }, { "id": "89232", "postDate": "08/12/2015 22:40:57", "content": "@Yair, i believe that using information provided by the paper describing the dataset is fair. After-all, these informations are publicly available and could be considered as knowledge and not data. However, using additional data from the dataset is forbiden. This include labels from the serie9 as well as other data from other events (there is tons of events provided). In both case, a word from the admin will be welcome.
\n\n@Deep, you are right. Using leaked labels from serie9 to do a local validation only help to artificially increase your number of submission per day, and therefore will increase your chance to overfit serie9. \nThen, for this challenge, EEG data have been recorded in a single session of ~3h. This clearly reduce the variability of EEG, and judging by our validation procedure, there is only a little difference between series.\nAs compare to past M/EEG challenge (DecMeg and BCIChallenge), i'm pretty relax about overfitting. Saying that, having a good Validation procedure is a key in any challenge, but for this one i will be very surprised to see a shakeout of 0.1 AUC like in the other M/EEG challenges.
\n\nThis competition is a bit different from the others, because of all the leaks and rules that are impossible to enforce. If your are not aiming for the top3, then you could completely ignore all the rules (and more specifically the rule of future data) that will give you a definite advantage over the other sub Top3 contestant. More than ever, the ranking is only based of the belief that contestant are playing fair.
", "rawMarkdown": "Yair, i believe that using information provided by the paper describing the dataset is fair. After-all, these informations are publicly available and could be considered as knowledge and not data. However, using additional data from the dataset is forbiden. This include labels from the serie9 as well as other data from other events (there is tons of events provided). In both case, a word from the admin will be welcome.\r\n\r\n@Deep, you are right. Using leaked labels from serie9 to do a local validation only help to artificially increase your number of submission per day, and therefore will increase your chance to overfit serie9. \r\nThen, for this challenge, EEG data have been recorded in a single session of ~3h. This clearly reduce the variability of EEG, and judging by our validation procedure, there is only a little difference between series.\r\nAs compare to past M/EEG challenge (DecMeg and BCIChallenge), i'm pretty relax about overfitting. Saying that, having a good Validation procedure is a key in any challenge, but for this one i will be very surprised to see a shakeout of 0.1 AUC like in the other M/EEG challenges.\r\n\r\nThis competition is a bit different from the others, because of all the leaks and rules that are impossible to enforce. If your are not aiming for the top3, then you could completely ignore all the rules (and more specifically the rule of future data) that will give you a definite advantage over the other sub Top3 contestant. More than ever, the ranking is only based of the belief that contestant are playing fair.", "votes": null }, { "id": "89290", "postDate": "08/13/2015 12:06:19", "content": "Thank you @Alexandre for the good summary.
\n\nYou are right that using the information from the nature publication to guide your choices is allowed. But using additional data for training is not. So if you add the labels for series 9, further events, or other sidechannel information to your training data we'd consider that (unfair) use of external data.
\n\n@Deep: In our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.
", "rawMarkdown": "Thank you @Alexandre for the good summary. \r\n\r\nYou are right that using the information from the nature publication to guide your choices is allowed. But using additional data for training is not. So if you add the labels for series 9, further events, or other sidechannel information to your training data we'd consider that (unfair) use of external data.\r\n\r\n@Deep: In our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.", "votes": null }, { "id": "89293", "postDate": "08/13/2015 12:29:29", "content": "@Klaus, Thank you for your answer.
\n\n\n\n\nIn our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.
\n
I'm curious, What is the score you achieved in your own experiments ?
", "rawMarkdown": "Klaus, Thank you for your answer.\r\n\r\n> In our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.\r\n\r\nI'm curious, What is the score you achieved in your own experiments ?", "votes": null }, { "id": "89311", "postDate": "08/13/2015 17:57:50", "content": "@Klaus, thanks for your response. That clears some air. By being so flexible in describing the private dataset to this extent and even allowing to use the information from the nature paper, it seems that your goal is to get the best possible model without use of external data. So, letting us know your inhouse AUC could be instructive.
", "rawMarkdown": "Klaus, thanks for your response. That clears some air. By being so flexible in describing the private dataset to this extent and even allowing to use the information from the nature paper, it seems that your goal is to get the best possible model without use of external data. So, letting us know your inhouse AUC could be instructive.", "votes": null }, { "id": "89347", "postDate": "08/14/2015 09:55:47", "content": "Unfortunately our best results are not directly comparable, since we've changed the evaluation for the Kaggle competition. I am in the process of rerunning them within the setup of the competition, but I can't tell you the exact numbers just yet. It does look like they'll be somewhere between 0.9 and 0.95 AUC.
", "rawMarkdown": "Unfortunately our best results are not directly comparable, since we've changed the evaluation for the Kaggle competition. I am in the process of rerunning them within the setup of the competition, but I can't tell you the exact numbers just yet. It does look like they'll be somewhere between 0.9 and 0.95 AUC.", "votes": null } ], "comments": [ { "id": 86782, "author_name": "qwlouse", "author_url": "", "post_date": "07/24/2015 10:23:16", "content": "For the given task the type of series shouldn't matter that much and both series 9 and 10 are always weight-series. So don't worry too much about that.
\n\n@off-topic: We decided not to set any runtime constraints for this competition.
", "votes": null, "replies": [] }, { "id": 89192, "author_name": "yaircarmon", "author_url": "", "post_date": "08/12/2015 15:52:46", "content": "I have some additional question on the topic of series numbering:
\n\n1) Does the series number represent the chronological order of experiments? If not, and the series number is some permutation of the chronological order, is it the same permutation for all subjects?
\n\n2) Is the test set split to private and public subsets according to the series number (e.g. series 9 is public and series 10 is private), or does the public test set contain parts of both series?
\n\nThanks,
", "votes": null, "replies": [] }, { "id": 89204, "author_name": "deepcnn", "author_url": "", "post_date": "08/12/2015 18:10:51", "content": "@Yair, I second your question #2. I was wondering about the public/private split myself. Setting up local validation is tricker depending on which series you choose to validate with and I was wondering if the public is series 9 only and the private is 10, then, there is a good chance to overfit to 9 only.
", "votes": null, "replies": [] }, { "id": 89207, "author_name": "alexandrebarachant", "author_url": "", "post_date": "08/12/2015 18:22:23", "content": "Hi Yair, Hi Deep,
\n\nFor the #2, the admin stated in another thread that the public/private split is indeed corresponding to series 9/10. This is also because the labels of series9 are openly available as part of the dataset they released on nature.com
\n\nAbout #1, according to the supplementary information given with the paper, series for 1 to 7 are in chronological order. then series 8,9 and 10 are randomized. So S10 is not always the last series, but s1 is always the first.
\n\nNote that explicitly using these information is a grey area, since it can be seen as a use of external data.
", "votes": null, "replies": [] }, { "id": 89223, "author_name": "yaircarmon", "author_url": "", "post_date": "08/12/2015 21:26:06", "content": "Thanks for the answers Alexandre - I should have given the paper a more thorough look...
\n\nAdmins - can you comment about fair use of this information? To me using labels from series 9 seems unfair, but making cross-validation splits between series 1-8 based on chronological order seems reasonable.
", "votes": null, "replies": [] }, { "id": 89228, "author_name": "deepcnn", "author_url": "", "post_date": "08/12/2015 22:04:40", "content": "Thanks, @Alexandre!
\n\nBut, I am still concerned about overfitting to 9. If you use one full series as a validation set and train on others, depending on the choice of the validation series and how similar that choice is to 9 vs. 10, there's a very good chance for overfitting to 9. It would have been nice if the public/private split was random samples from both 9 and 10. So, it begs to be careful in your local validation sample choice. Am I the only one concered about this?
\n\nThe fact that we know which one is public and which one is private and other factors including that nature paper makes this competition different from others and there seems to be a lot is offered to exploit. But, at the same time, one has to be careful not to fall for overfitting for 9.
", "votes": null, "replies": [] }, { "id": 89232, "author_name": "alexandrebarachant", "author_url": "", "post_date": "08/12/2015 22:40:57", "content": "@Yair, i believe that using information provided by the paper describing the dataset is fair. After-all, these informations are publicly available and could be considered as knowledge and not data. However, using additional data from the dataset is forbiden. This include labels from the serie9 as well as other data from other events (there is tons of events provided). In both case, a word from the admin will be welcome.
\n\n@Deep, you are right. Using leaked labels from serie9 to do a local validation only help to artificially increase your number of submission per day, and therefore will increase your chance to overfit serie9. \nThen, for this challenge, EEG data have been recorded in a single session of ~3h. This clearly reduce the variability of EEG, and judging by our validation procedure, there is only a little difference between series.\nAs compare to past M/EEG challenge (DecMeg and BCIChallenge), i'm pretty relax about overfitting. Saying that, having a good Validation procedure is a key in any challenge, but for this one i will be very surprised to see a shakeout of 0.1 AUC like in the other M/EEG challenges.
\n\nThis competition is a bit different from the others, because of all the leaks and rules that are impossible to enforce. If your are not aiming for the top3, then you could completely ignore all the rules (and more specifically the rule of future data) that will give you a definite advantage over the other sub Top3 contestant. More than ever, the ranking is only based of the belief that contestant are playing fair.
", "votes": null, "replies": [] }, { "id": 89290, "author_name": "qwlouse", "author_url": "", "post_date": "08/13/2015 12:06:19", "content": "Thank you @Alexandre for the good summary.
\n\nYou are right that using the information from the nature publication to guide your choices is allowed. But using additional data for training is not. So if you add the labels for series 9, further events, or other sidechannel information to your training data we'd consider that (unfair) use of external data.
\n\n@Deep: In our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.
", "votes": null, "replies": [] }, { "id": 89293, "author_name": "alexandrebarachant", "author_url": "", "post_date": "08/13/2015 12:29:29", "content": "@Klaus, Thank you for your answer.
\n\n\n\n\nIn our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.
\n
I'm curious, What is the score you achieved in your own experiments ?
", "votes": null, "replies": [] }, { "id": 89311, "author_name": "deepcnn", "author_url": "", "post_date": "08/13/2015 17:57:50", "content": "@Klaus, thanks for your response. That clears some air. By being so flexible in describing the private dataset to this extent and even allowing to use the information from the nature paper, it seems that your goal is to get the best possible model without use of external data. So, letting us know your inhouse AUC could be instructive.
", "votes": null, "replies": [] }, { "id": 89347, "author_name": "qwlouse", "author_url": "", "post_date": "08/14/2015 09:55:47", "content": "Unfortunately our best results are not directly comparable, since we've changed the evaluation for the Kaggle competition. I am in the process of rerunning them within the setup of the competition, but I can't tell you the exact numbers just yet. It does look like they'll be somewhere between 0.9 and 0.95 AUC.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86764": "I was reading the data paper and it indicates that there were 6 weight series, 2 friction series, and 2 mixed series.\r\n\r\nSince we're predicting for series 9 and 10, I was wondering if these series numbers meant anything (whether we are given information on whether each series corresponds to a particular type of series)\r\n\r\nI know this is off-topic but I am also wondering if there is a run-time constraint on this competition.\r\n\r\nThanks,", "86782": "For the given task the type of series shouldn't matter that much and both series 9 and 10 are always weight-series. So don't worry too much about that.\r\n\r\n@off-topic: We decided not to set any runtime constraints for this competition.", "89192": "I have some additional question on the topic of series numbering:\r\n\r\n1) Does the series number represent the chronological order of experiments? If not, and the series number is some permutation of the chronological order, is it the same permutation for all subjects?\r\n\r\n2) Is the test set split to private and public subsets according to the series number (e.g. series 9 is public and series 10 is private), or does the public test set contain parts of both series?\r\n\r\nThanks,", "89204": "Yair, I second your question #2. I was wondering about the public/private split myself. Setting up local validation is tricker depending on which series you choose to validate with and I was wondering if the public is series 9 only and the private is 10, then, there is a good chance to overfit to 9 only.", "89207": "Hi Yair, Hi Deep,\r\n\r\nFor the #2, the admin stated in another thread that the public/private split is indeed corresponding to series 9/10. This is also because the labels of series9 are openly available as part of the dataset they released on nature.com\r\n\r\nAbout #1, according to the supplementary information given with the paper, series for 1 to 7 are in chronological order. then series 8,9 and 10 are randomized. So S10 is not always the last series, but s1 is always the first.\r\n\r\nNote that explicitly using these information is a grey area, since it can be seen as a use of external data.", "89223": "Thanks for the answers Alexandre - I should have given the paper a more thorough look...\r\n\r\nAdmins - can you comment about fair use of this information? To me using labels from series 9 seems unfair, but making cross-validation splits between series 1-8 based on chronological order seems reasonable.", "89228": "Thanks, @Alexandre! \r\n\r\nBut, I am still concerned about overfitting to 9. If you use one full series as a validation set and train on others, depending on the choice of the validation series and how similar that choice is to 9 vs. 10, there's a very good chance for overfitting to 9. It would have been nice if the public/private split was random samples from both 9 and 10. So, it begs to be careful in your local validation sample choice. Am I the only one concered about this? \r\n\r\nThe fact that we know which one is public and which one is private and other factors including that nature paper makes this competition different from others and there seems to be a lot is offered to exploit. But, at the same time, one has to be careful not to fall for overfitting for 9.", "89232": "Yair, i believe that using information provided by the paper describing the dataset is fair. After-all, these informations are publicly available and could be considered as knowledge and not data. However, using additional data from the dataset is forbiden. This include labels from the serie9 as well as other data from other events (there is tons of events provided). In both case, a word from the admin will be welcome.\r\n\r\n@Deep, you are right. Using leaked labels from serie9 to do a local validation only help to artificially increase your number of submission per day, and therefore will increase your chance to overfit serie9. \r\nThen, for this challenge, EEG data have been recorded in a single session of ~3h. This clearly reduce the variability of EEG, and judging by our validation procedure, there is only a little difference between series.\r\nAs compare to past M/EEG challenge (DecMeg and BCIChallenge), i'm pretty relax about overfitting. Saying that, having a good Validation procedure is a key in any challenge, but for this one i will be very surprised to see a shakeout of 0.1 AUC like in the other M/EEG challenges.\r\n\r\nThis competition is a bit different from the others, because of all the leaks and rules that are impossible to enforce. If your are not aiming for the top3, then you could completely ignore all the rules (and more specifically the rule of future data) that will give you a definite advantage over the other sub Top3 contestant. More than ever, the ranking is only based of the belief that contestant are playing fair.", "89290": "Thank you @Alexandre for the good summary. \r\n\r\nYou are right that using the information from the nature publication to guide your choices is allowed. But using additional data for training is not. So if you add the labels for series 9, further events, or other sidechannel information to your training data we'd consider that (unfair) use of external data.\r\n\r\n@Deep: In our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.", "89293": "Klaus, Thank you for your answer.\r\n\r\n> In our own experiments we didn't have any problems with overfitting. Series 9 and 10 seem to be very similar in all relevant ways.\r\n\r\nI'm curious, What is the score you achieved in your own experiments ?", "89311": "Klaus, thanks for your response. That clears some air. By being so flexible in describing the private dataset to this extent and even allowing to use the information from the nature paper, it seems that your goal is to get the best possible model without use of external data. So, letting us know your inhouse AUC could be instructive.", "89347": "Unfortunately our best results are not directly comparable, since we've changed the evaluation for the Kaggle competition. I am in the process of rerunning them within the setup of the competition, but I can't tell you the exact numbers just yet. It does look like they'll be somewhere between 0.9 and 0.95 AUC." }, "source": "meta" }