On page linked in Get the Data section (http://www.nature.com/articles/sdata201447) we can find a lot of useful information about experiment. Specifically there is detailed information about (almost) all series for each subject. Can we use this information in our solutions? For example can we use different preprocessing methods for each file, based only on information found on this site? It's even possible to find some information the use of which may be treated as cheating, but would be undetectable to organizators.
", "messages": [ { "id": "86475", "postDate": "07/22/2015 14:11:47", "content": "On page linked in Get the Data section (http://www.nature.com/articles/sdata201447) we can find a lot of useful information about experiment. Specifically there is detailed information about (almost) all series for each subject. Can we use this information in our solutions? For example can we use different preprocessing methods for each file, based only on information found on this site? It's even possible to find some information the use of which may be treated as cheating, but would be undetectable to organizators.
", "rawMarkdown": "On page linked in Get the Data section (http://www.nature.com/articles/sdata201447) we can find a lot of useful information about experiment. Specifically there is detailed information about (almost) all series for each subject. Can we use this information in our solutions? For example can we use different preprocessing methods for each file, based only on information found on this site? It's even possible to find some information the use of which may be treated as cheating, but would be undetectable to organizators.", "votes": null }, { "id": "86513", "postDate": "07/22/2015 19:12:15", "content": "The rules imply that you cannot:
\n\n"EXTERNAL DATA
\n\nUnless otherwise expressly stated on the Competition Website, Participants must not use data other than the Data to develop and test their models and Submissions. Competition Sponsor reserves the right in its sole discretion to disqualify any Participant who Competition Sponsor discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Competition Website and in these Competition Rules, in the course of the Competition."
\n\nhttps://www.kaggle.com/c/grasp-and-lift-eeg-detection/rules
", "rawMarkdown": "The rules imply that you cannot: \r\n\r\n\"EXTERNAL DATA\r\n\r\nUnless otherwise expressly stated on the Competition Website, Participants must not use data other than the Data to develop and test their models and Submissions. Competition Sponsor reserves the right in its sole discretion to disqualify any Participant who Competition Sponsor discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Competition Website and in these Competition Rules, in the course of the Competition.\"\r\n\r\nhttps://www.kaggle.com/c/grasp-and-lift-eeg-detection/rules", "votes": null }, { "id": "86514", "postDate": "07/22/2015 19:14:08", "content": "Also: "Use of external data is not permitted."
", "rawMarkdown": "Also: \"Use of external data is not permitted.\"", "votes": null }, { "id": "86519", "postDate": "07/22/2015 19:45:55", "content": "Hi,\nThis clearly call for an answer from the organizers.\nOf course, use of external data are not permitted.
\n\nThis case is a bit more tricky, because the dataset we are working on is a subset of the original dataset, which is a public dataset. The public dataset incorporates EMG as well as extra information such as a lot of different events and information about the sequences.\nThe most disturbing thing is that this dataset includes label for serie9, which is part of the test set in this challenge. I sincerely hope that the public/private split is done according to that, but i have no idea.
\n\nSo yeah, we are not allowed to use these data. But as for every rule, the problem is the enforcement. The only thing that can be enforced is to check that winning solution are not making obvious use of these data (such as submitting the events for serie9 extracted from the public dataset.) For all the other "use" of these data, no one will never know.
\n\nDeveloping a classification pipeline require tons of parameters tuning. A step that is usually done by hand. You can use extra information from this dataset to ensure maximal performance on the test set (by overfitting) while making no apparent use of the data.
", "rawMarkdown": "Hi,\r\nThis clearly call for an answer from the organizers.\r\nOf course, use of external data are not permitted.\r\n\r\nThis case is a bit more tricky, because the dataset we are working on is a subset of the original dataset, which is a public dataset. The public dataset incorporates EMG as well as extra information such as a lot of different events and information about the sequences.\r\nThe most disturbing thing is that **this dataset includes label for serie9**, which is part of the test set in this challenge. I sincerely hope that the public/private split is done according to that, but i have no idea.\r\n\r\nSo yeah, we are not allowed to use these data. But as for every rule, the problem is the enforcement. The only thing that can be enforced is to check that winning solution are not making obvious use of these data (such as submitting the events for serie9 extracted from the public dataset.) For all the other \"use\" of these data, no one will never know.\r\n\r\nDeveloping a classification pipeline require tons of parameters tuning. A step that is usually done by hand. You can use extra information from this dataset to ensure maximal performance on the test set (by overfitting) while making no apparent use of the data.", "votes": null }, { "id": "86522", "postDate": "07/22/2015 19:57:18", "content": "Well, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might yield worse performance in 10 without alerting the cheater.
\n\nStill, this is very concerning and I see no way they can enforce the rules in this instance other than punishing suspicious choices of hyperparameters that don't seem to come from cross-validation.
", "rawMarkdown": "Well, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might yield worse performance in 10 without alerting the cheater.\r\n\r\nStill, this is very concerning and I see no way they can enforce the rules in this instance other than punishing suspicious choices of hyperparameters that don't seem to come from cross-validation.", "votes": null }, { "id": "86524", "postDate": "07/22/2015 20:05:53", "content": "\n\n\nWell, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might give worse performance in 10.
\n
I sincerely hope this is the case. Can we have a word from the admin about that ?
", "rawMarkdown": "> Well, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might give worse performance in 10.\r\n\r\nI sincerely hope this is the case. Can we have a word from the admin about that ?", "votes": null }, { "id": "86576", "postDate": "07/23/2015 06:30:12", "content": "There's a possibility for Kaggle to reduce the most obvious use of this data for all participants: if performance for series 9 is much better than for series 10 -> commence leaderboard removal.
\n\nDifferences between scores for different held-out series are rather small (~0.05), as according to my local validations. If this holds not only for my models then a big difference in test set performance of series 9 and 10 should raise a red flag.
\n\nBut the subtle ways this data can be used are disturbing.
", "rawMarkdown": "There's a possibility for Kaggle to reduce the most obvious use of this data for all participants: if performance for series 9 is much better than for series 10 -> commence leaderboard removal. \r\n\r\nDifferences between scores for different held-out series are rather small (~0.05), as according to my local validations. If this holds not only for my models then a big difference in test set performance of series 9 and 10 should raise a red flag.\r\n\r\nBut the subtle ways this data can be used are disturbing.", "votes": null }, { "id": "86589", "postDate": "07/23/2015 09:03:57", "content": "Indeed. Scoring considerably better on 9 across the board would indeed be suspicious.
\n\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.
\n\nBack to the point however, I wonder if it's just better to score exclusively on 10 and just release the series 9 labels for training... The variance of the Leaderboard would increase ofc which is not ideal but at least we can be sure everyone is competing fairly.
", "rawMarkdown": "Indeed. Scoring considerably better on 9 across the board would indeed be suspicious. \r\n\r\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.\r\n\r\nBack to the point however, I wonder if it's just better to score exclusively on 10 and just release the series 9 labels for training... The variance of the Leaderboard would increase ofc which is not ideal but at least we can be sure everyone is competing fairly.", "votes": null }, { "id": "86594", "postDate": "07/23/2015 09:34:13", "content": "You are completely right: There is additional information available about the dataset including the labels for series 9. This series is indeed used for the public leaderboard, and the private leaderboard data comes from an unpublished 10th series for each participant.
\n\nSo while you could potentially cheat on the public leaderboard, the final scoring will be fair. And as you've already guessed we'll monitor the leaderboard for suspicious submissions. You guys already pointed out there are of course other undetectable ways to use it. Note though, that we'll check the winning solutions to make sure they only use the EEG data and that they do not use future data.
\n\nWe discourage the use of this external data, but in the end what matters most to us is that you come up with good algorithms to map the EEG to the given events, because that is what will drive the development of BCI controlled prosthetics.
", "rawMarkdown": "You are completely right: There is additional information available about the dataset **including the labels for series 9**. This series is indeed used for the public leaderboard, and the private leaderboard data comes from an *unpublished 10th series for each participant*. \r\n\r\nSo while you could potentially cheat on the public leaderboard, the final scoring will be fair. And as you've already guessed we'll monitor the leaderboard for suspicious submissions. You guys already pointed out there are of course other undetectable ways to use it. Note though, that we'll check the winning solutions to make sure they only use the EEG data and that they do not use future data. \r\n\r\nWe discourage the use of this external data, but in the end what matters most to us is that you come up with good algorithms to map the EEG to the given events, because that is what will drive the development of BCI controlled prosthetics.", "votes": null }, { "id": "86614", "postDate": "07/23/2015 12:59:02", "content": "@Klaus Greff, that's really good news!
\n\n[quote=ajoo;86589]
\n\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.
\n\n[/quote]
\n\nReason for lower differences between series in my case is probably that I'm not doing cross-validation, but rather using a hold-out set consisting of series 7&8 of each subject to enforce the "no future data" rule in local tests. Also, I'm using mostly logistic regression, which is a rather simple and usually consistent method.
", "rawMarkdown": "Klaus Greff, that's really good news!\r\n\r\n[quote=ajoo;86589]\r\n\r\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.\r\n\r\n[/quote]\r\n\r\nReason for lower differences between series in my case is probably that I'm not doing cross-validation, but rather using a hold-out set consisting of series 7&8 of each subject to enforce the \"no future data\" rule in local tests. Also, I'm using mostly logistic regression, which is a rather simple and usually consistent method.", "votes": null }, { "id": "86617", "postDate": "07/23/2015 13:26:49", "content": "@ajoo, @blain, i also think all series where recorded on one single day, which dramatically decrease the variability of EEG. Except the first serie where there is more variability on the timing, the subject is really consistant in the way he do the sequence. If you remove the first one, you will see a decrease in variance of your CV
", "rawMarkdown": "ajoo, @blain, i also think all series where recorded on one single day, which dramatically decrease the variability of EEG. Except the first serie where there is more variability on the timing, the subject is really consistant in the way he do the sequence. If you remove the first one, you will see a decrease in variance of your CV", "votes": null }, { "id": "86620", "postDate": "07/23/2015 13:41:03", "content": "I simply confirm that all recordings in single subjects were recorded sequentially in ~2 hours with short breaks, i.e., once the electrode cap was in place it remained there for the whole experiments. This does reduce EEG variability. The behavior (the task) as such is performed consistently across day in healthy subjects.
\n\nThe hidden datasets (for final testing) is not particular in any fashion but were picked at random.
", "rawMarkdown": "I simply confirm that all recordings in single subjects were recorded sequentially in ~2 hours with short breaks, i.e., once the electrode cap was in place it remained there for the whole experiments. This does reduce EEG variability. The behavior (the task) as such is performed consistently across day in healthy subjects. \r\n\r\nThe hidden datasets (for final testing) is not particular in any fashion but were picked at random.", "votes": null }, { "id": "86623", "postDate": "07/23/2015 13:52:55", "content": "\n\n\nThe hidden datasets (for final testing) is not particular in any fashion but were picked at random.
\n
By that you mean the 9 and 10 series are randomly pick, i.e. can be a serie that take place at the begining of the experiement ?
", "rawMarkdown": "> The hidden datasets (for final testing) is not particular in any fashion but were picked at random.\r\n\r\nBy that you mean the 9 and 10 series are randomly pick, i.e. can be a serie that take place at the begining of the experiement ?", "votes": null }, { "id": "86651", "postDate": "07/23/2015 15:22:03", "content": "Well, they were neither picked from the beginning of the experiment, nor from the end. The point is that the 9th and 10th series are not particular in any sense.
", "rawMarkdown": "Well, they were neither picked from the beginning of the experiment, nor from the end. The point is that the 9th and 10th series are not particular in any sense.", "votes": null }, { "id": "86656", "postDate": "07/23/2015 15:38:40", "content": "Well, i don't want to play with word, but accordingly to the Series schedule in the paper, S9 and S10 are picks among the last 3 series the subject did. And more specifically, S10 are all weight#6 series. That make them pretty specific to me. \nSaying that, i don't make any use of this information. But i think that to this point we don't really need to be elusive on what are the test data.
", "rawMarkdown": "Well, i don't want to play with word, but accordingly to the Series schedule in the paper, S9 and S10 are picks among the last 3 series the subject did. And more specifically, S10 are all weight#6 series. That make them pretty specific to me. \r\nSaying that, i don't make any use of this information. But i think that to this point we don't really need to be elusive on what are the test data.", "votes": null } ], "comments": [ { "id": 86513, "author_name": "jlachance", "author_url": "", "post_date": "07/22/2015 19:12:15", "content": "The rules imply that you cannot:
\n\n"EXTERNAL DATA
\n\nUnless otherwise expressly stated on the Competition Website, Participants must not use data other than the Data to develop and test their models and Submissions. Competition Sponsor reserves the right in its sole discretion to disqualify any Participant who Competition Sponsor discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Competition Website and in these Competition Rules, in the course of the Competition."
\n\nhttps://www.kaggle.com/c/grasp-and-lift-eeg-detection/rules
", "votes": null, "replies": [] }, { "id": 86514, "author_name": "jlachance", "author_url": "", "post_date": "07/22/2015 19:14:08", "content": "Also: "Use of external data is not permitted."
", "votes": null, "replies": [] }, { "id": 86519, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/22/2015 19:45:55", "content": "Hi,\nThis clearly call for an answer from the organizers.\nOf course, use of external data are not permitted.
\n\nThis case is a bit more tricky, because the dataset we are working on is a subset of the original dataset, which is a public dataset. The public dataset incorporates EMG as well as extra information such as a lot of different events and information about the sequences.\nThe most disturbing thing is that this dataset includes label for serie9, which is part of the test set in this challenge. I sincerely hope that the public/private split is done according to that, but i have no idea.
\n\nSo yeah, we are not allowed to use these data. But as for every rule, the problem is the enforcement. The only thing that can be enforced is to check that winning solution are not making obvious use of these data (such as submitting the events for serie9 extracted from the public dataset.) For all the other "use" of these data, no one will never know.
\n\nDeveloping a classification pipeline require tons of parameters tuning. A step that is usually done by hand. You can use extra information from this dataset to ensure maximal performance on the test set (by overfitting) while making no apparent use of the data.
", "votes": null, "replies": [] }, { "id": 86522, "author_name": "ajoo88", "author_url": "", "post_date": "07/22/2015 19:57:18", "content": "Well, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might yield worse performance in 10 without alerting the cheater.
\n\nStill, this is very concerning and I see no way they can enforce the rules in this instance other than punishing suspicious choices of hyperparameters that don't seem to come from cross-validation.
", "votes": null, "replies": [] }, { "id": 86524, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/22/2015 20:05:53", "content": "\n\n\nWell, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might give worse performance in 10.
\n
I sincerely hope this is the case. Can we have a word from the admin about that ?
", "votes": null, "replies": [] }, { "id": 86576, "author_name": "rafalcycon", "author_url": "", "post_date": "07/23/2015 06:30:12", "content": "There's a possibility for Kaggle to reduce the most obvious use of this data for all participants: if performance for series 9 is much better than for series 10 -> commence leaderboard removal.
\n\nDifferences between scores for different held-out series are rather small (~0.05), as according to my local validations. If this holds not only for my models then a big difference in test set performance of series 9 and 10 should raise a red flag.
\n\nBut the subtle ways this data can be used are disturbing.
", "votes": null, "replies": [] }, { "id": 86589, "author_name": "ajoo88", "author_url": "", "post_date": "07/23/2015 09:03:57", "content": "Indeed. Scoring considerably better on 9 across the board would indeed be suspicious.
\n\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.
\n\nBack to the point however, I wonder if it's just better to score exclusively on 10 and just release the series 9 labels for training... The variance of the Leaderboard would increase ofc which is not ideal but at least we can be sure everyone is competing fairly.
", "votes": null, "replies": [] }, { "id": 86594, "author_name": "qwlouse", "author_url": "", "post_date": "07/23/2015 09:34:13", "content": "You are completely right: There is additional information available about the dataset including the labels for series 9. This series is indeed used for the public leaderboard, and the private leaderboard data comes from an unpublished 10th series for each participant.
\n\nSo while you could potentially cheat on the public leaderboard, the final scoring will be fair. And as you've already guessed we'll monitor the leaderboard for suspicious submissions. You guys already pointed out there are of course other undetectable ways to use it. Note though, that we'll check the winning solutions to make sure they only use the EEG data and that they do not use future data.
\n\nWe discourage the use of this external data, but in the end what matters most to us is that you come up with good algorithms to map the EEG to the given events, because that is what will drive the development of BCI controlled prosthetics.
", "votes": null, "replies": [] }, { "id": 86614, "author_name": "rafalcycon", "author_url": "", "post_date": "07/23/2015 12:59:02", "content": "@Klaus Greff, that's really good news!
\n\n[quote=ajoo;86589]
\n\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.
\n\n[/quote]
\n\nReason for lower differences between series in my case is probably that I'm not doing cross-validation, but rather using a hold-out set consisting of series 7&8 of each subject to enforce the "no future data" rule in local tests. Also, I'm using mostly logistic regression, which is a rather simple and usually consistent method.
", "votes": null, "replies": [] }, { "id": 86617, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/23/2015 13:26:49", "content": "@ajoo, @blain, i also think all series where recorded on one single day, which dramatically decrease the variability of EEG. Except the first serie where there is more variability on the timing, the subject is really consistant in the way he do the sequence. If you remove the first one, you will see a decrease in variance of your CV
", "votes": null, "replies": [] }, { "id": 86620, "author_name": "", "author_url": "", "post_date": "07/23/2015 13:41:03", "content": "I simply confirm that all recordings in single subjects were recorded sequentially in ~2 hours with short breaks, i.e., once the electrode cap was in place it remained there for the whole experiments. This does reduce EEG variability. The behavior (the task) as such is performed consistently across day in healthy subjects.
\n\nThe hidden datasets (for final testing) is not particular in any fashion but were picked at random.
", "votes": null, "replies": [] }, { "id": 86623, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/23/2015 13:52:55", "content": "\n\n\nThe hidden datasets (for final testing) is not particular in any fashion but were picked at random.
\n
By that you mean the 9 and 10 series are randomly pick, i.e. can be a serie that take place at the begining of the experiement ?
", "votes": null, "replies": [] }, { "id": 86651, "author_name": "", "author_url": "", "post_date": "07/23/2015 15:22:03", "content": "Well, they were neither picked from the beginning of the experiment, nor from the end. The point is that the 9th and 10th series are not particular in any sense.
", "votes": null, "replies": [] }, { "id": 86656, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/23/2015 15:38:40", "content": "Well, i don't want to play with word, but accordingly to the Series schedule in the paper, S9 and S10 are picks among the last 3 series the subject did. And more specifically, S10 are all weight#6 series. That make them pretty specific to me. \nSaying that, i don't make any use of this information. But i think that to this point we don't really need to be elusive on what are the test data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86475": "On page linked in Get the Data section (http://www.nature.com/articles/sdata201447) we can find a lot of useful information about experiment. Specifically there is detailed information about (almost) all series for each subject. Can we use this information in our solutions? For example can we use different preprocessing methods for each file, based only on information found on this site? It's even possible to find some information the use of which may be treated as cheating, but would be undetectable to organizators.", "86513": "The rules imply that you cannot: \r\n\r\n\"EXTERNAL DATA\r\n\r\nUnless otherwise expressly stated on the Competition Website, Participants must not use data other than the Data to develop and test their models and Submissions. Competition Sponsor reserves the right in its sole discretion to disqualify any Participant who Competition Sponsor discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Competition Website and in these Competition Rules, in the course of the Competition.\"\r\n\r\nhttps://www.kaggle.com/c/grasp-and-lift-eeg-detection/rules", "86514": "Also: \"Use of external data is not permitted.\"", "86519": "Hi,\r\nThis clearly call for an answer from the organizers.\r\nOf course, use of external data are not permitted.\r\n\r\nThis case is a bit more tricky, because the dataset we are working on is a subset of the original dataset, which is a public dataset. The public dataset incorporates EMG as well as extra information such as a lot of different events and information about the sequences.\r\nThe most disturbing thing is that **this dataset includes label for serie9**, which is part of the test set in this challenge. I sincerely hope that the public/private split is done according to that, but i have no idea.\r\n\r\nSo yeah, we are not allowed to use these data. But as for every rule, the problem is the enforcement. The only thing that can be enforced is to check that winning solution are not making obvious use of these data (such as submitting the events for serie9 extracted from the public dataset.) For all the other \"use\" of these data, no one will never know.\r\n\r\nDeveloping a classification pipeline require tons of parameters tuning. A step that is usually done by hand. You can use extra information from this dataset to ensure maximal performance on the test set (by overfitting) while making no apparent use of the data.", "86522": "Well, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might yield worse performance in 10 without alerting the cheater.\r\n\r\nStill, this is very concerning and I see no way they can enforce the rules in this instance other than punishing suspicious choices of hyperparameters that don't seem to come from cross-validation.", "86524": "> Well, we can at least hope that it's series 9 that's used for the public LB so that overfitting this series might give worse performance in 10.\r\n\r\nI sincerely hope this is the case. Can we have a word from the admin about that ?", "86576": "There's a possibility for Kaggle to reduce the most obvious use of this data for all participants: if performance for series 9 is much better than for series 10 -> commence leaderboard removal. \r\n\r\nDifferences between scores for different held-out series are rather small (~0.05), as according to my local validations. If this holds not only for my models then a big difference in test set performance of series 9 and 10 should raise a red flag.\r\n\r\nBut the subtle ways this data can be used are disturbing.", "86589": "Indeed. Scoring considerably better on 9 across the board would indeed be suspicious. \r\n\r\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.\r\n\r\nBack to the point however, I wonder if it's just better to score exclusively on 10 and just release the series 9 labels for training... The variance of the Leaderboard would increase ofc which is not ideal but at least we can be sure everyone is competing fairly.", "86594": "You are completely right: There is additional information available about the dataset **including the labels for series 9**. This series is indeed used for the public leaderboard, and the private leaderboard data comes from an *unpublished 10th series for each participant*. \r\n\r\nSo while you could potentially cheat on the public leaderboard, the final scoring will be fair. And as you've already guessed we'll monitor the leaderboard for suspicious submissions. You guys already pointed out there are of course other undetectable ways to use it. Note though, that we'll check the winning solutions to make sure they only use the EEG data and that they do not use future data. \r\n\r\nWe discourage the use of this external data, but in the end what matters most to us is that you come up with good algorithms to map the EEG to the given events, because that is what will drive the development of BCI controlled prosthetics.", "86614": "Klaus Greff, that's really good news!\r\n\r\n[quote=ajoo;86589]\r\n\r\nI'm surprised however that you get such consistent results. In my cross-validation I sometimes get series that are classified much worse (~0.2) than others but I'm not doing any artifact removal or anything so maybe that's it.\r\n\r\n[/quote]\r\n\r\nReason for lower differences between series in my case is probably that I'm not doing cross-validation, but rather using a hold-out set consisting of series 7&8 of each subject to enforce the \"no future data\" rule in local tests. Also, I'm using mostly logistic regression, which is a rather simple and usually consistent method.", "86617": "ajoo, @blain, i also think all series where recorded on one single day, which dramatically decrease the variability of EEG. Except the first serie where there is more variability on the timing, the subject is really consistant in the way he do the sequence. If you remove the first one, you will see a decrease in variance of your CV", "86620": "I simply confirm that all recordings in single subjects were recorded sequentially in ~2 hours with short breaks, i.e., once the electrode cap was in place it remained there for the whole experiments. This does reduce EEG variability. The behavior (the task) as such is performed consistently across day in healthy subjects. \r\n\r\nThe hidden datasets (for final testing) is not particular in any fashion but were picked at random.", "86623": "> The hidden datasets (for final testing) is not particular in any fashion but were picked at random.\r\n\r\nBy that you mean the 9 and 10 series are randomly pick, i.e. can be a serie that take place at the begining of the experiement ?", "86651": "Well, they were neither picked from the beginning of the experiment, nor from the end. The point is that the 9th and 10th series are not particular in any sense.", "86656": "Well, i don't want to play with word, but accordingly to the Series schedule in the paper, S9 and S10 are picks among the last 3 series the subject did. And more specifically, S10 are all weight#6 series. That make them pretty specific to me. \r\nSaying that, i don't make any use of this information. But i think that to this point we don't really need to be elusive on what are the test data." }, "source": "meta" }