Just to make sure this is absolutely clear. Are we allowed to use the data from subj1_series6 in training for subj1_series5?
\nIt seems that this is allowed from reading the data page, but I would like to check because it is using future data.
", "messages": [ { "id": 82996, "postDate": "2015-06-30T14:48:56.467Z", "content": "[quote=William Cukierski;82992]
\nThe host will do their best to enforce the rule. This might entail removing any teams with performance greater than the "highest known legitimate score."
\n[/quote]
\nThis is what i was thinking, and i believe it is a good way to make this dissuasive enough so most of the people will play by the rules.
\n[quote]
\nAlso, as a neurobiologist, there are factors that are important in neural signals "chains". Removing the possibility to use it is bizarre, from a biological point of view.
\n[/quote]
\nFrom a applicative point of view, this rule make sense. From the challenge point of view, it also force participant to do a bit of EEG classification, and not just reverse-engineering of the paradigm.
", "votes": 3 }, { "id": 82993, "postDate": "2015-06-30T14:35:12.033Z", "content": "The goal of the competition is to do better than what's already known. So such kind of rule is puzzling. Also, If you don't want contestant to use data in an inappropriate fashion, why not beginning by deliver them properly? I am not trying to minimize kaggle efforts or trolling, but isn't there a good way to do it there?
\nAlso, as a neurobiologist, there are factors that are important in neural signals "chains". Removing the possibility to use it is bizarre, from a biological point of view.
", "votes": 3 }, { "id": 82937, "postDate": "2015-06-29T21:53:20.577Z", "content": "The original thing I posted was just a corner case. The main application is to data within the same series.
\nThe most important point is that we can't use information from the near future. As there is a sequence of 6 different events we are trying to predict it would have been helpful to use the information that the final event occurred to predict the first event but we can't do this because we can't use future data. The rule adds a decent amount of extra difficulty to this competition.
", "votes": 3 }, { "id": 83003, "postDate": "2015-06-30T16:09:55.607Z", "content": "Your are just introducing a lag in your system, so you can use the whole sequence to trig the movement. It wont change the problem, because you can not rollback time to execute the movement at the beginning of the sequence, i.e you can not change the probabilities on the detection you made in the past.
\nWhat you describe here can be done for the last element of the sequence, since you have all the past signal available.
", "votes": 4 }, { "id": 82997, "postDate": "2015-06-30T14:55:20.687Z", "content": "[quote=WhizWilde;82993]
\nThe goal of the competition is to do better than what's already known. So such kind of rule is puzzling. Also, If you don't want contestant to use data in an inappropriate fashion, why not beginning by deliver them properly? I am not trying to minimize kaggle efforts or trolling, but isn't there a good way to do it there?
\nAlso, as a neurobiologist, there are factors that are important in neural signals "chains". Removing the possibility to use it is bizarre, from a biological point of view.
\n[/quote]
\nUnless neurobiologists have invented a time machine without telling the rest of us, there shouldn't be anything puzzling about this rule :) An algorithm controlling a prosthetic device does not have access to future EEG signals, so neither should your algorithm. As I mentioned, the only way to not give out this data is to use code submission, but this is a complex bridge we have not yet crossed.
", "votes": 3 }, { "id": 83446, "postDate": "2015-07-04T22:10:14.497Z", "content": "When you filter a signal, you introduce a delay. So, I think you would need to see the future data to predict the label of the current sample the system is processing.
\nIs it valid a lagged response? (let's say 0.5 seconds)
", "votes": 1 }, { "id": 83017, "postDate": "2015-06-30T20:34:29.707Z", "content": "[quote=I can do the splits NO PROB;83016]
\nWhen predicting, you may not use data from the future! In other words, if you are predicting labels for id subj1_series9_11, you may not incorporate any frame after 11 of series 9 from subject 1 for any reason.
\n[/quote]
\nDone!
\n[quote=I can do the splits NO PROB;83016]
\nAnd btw.: Is there any possibility to extract the trial number in each series (for example like a fixed observation window given in ms)? Usually, one would average over all trials of one subject in order to increase the signal-to-noise ratio in the EEG-signal. Or is this part of the challenge?
\n[/quote]
\nPart of the challenge.
", "votes": 1 }, { "id": 82939, "postDate": "2015-06-29T21:57:46.833Z", "content": "[quote=J Kolb;82936]
\nSo we can ignore the entire "Important Note" section on the Data page since it is not possible to do, right?
\n[/quote]
\nIt still applies to future data that is in the same series.
", "votes": 1 }, { "id": 82936, "postDate": "2015-06-29T21:47:09.177Z", "content": "[quote=William Cukierski;82923]
\nYes. However, it's a moot point since you are predicting series 9 and 10 and don't have any "latter" series labels to train on.
\n[/quote]
\nSo we can ignore the entire "Important Note" section on the Data page since it is not possible to do, right?
", "votes": 1 }, { "id": 82914, "postDate": "2015-06-29T17:23:04.703Z", "content": "Just to make sure this is absolutely clear. Are we allowed to use the data from subj1_series6 in training for subj1_series5?
\nIt seems that this is allowed from reading the data page, but I would like to check because it is using future data.
", "votes": 1 }, { "id": 82990, "postDate": "2015-06-30T13:41:10.037Z", "content": "Hi William,
\nHow do you plan to enforce this rule ? I don't want to be the devil's advocate but people who aims at the kaggle rank only may ignore this rule and decline the prizes.
", "votes": 2 }, { "id": 82992, "postDate": "2015-06-30T14:16:06.127Z", "content": "Hi Alexandre,
\nThe host will do their best to enforce the rule. This might entail removing any teams with performance greater than the "highest known legitimate score."
\nCompetitions are too popular to check everybody's approach, but we do strive to make sure the winners are clean. We believe that most people play by the rules, understand their intent, and place the greater research goals above their own self interests. The alternative is that we never run time-series competitions like this, or have to move to a setup where participants submit code that runs on a holdout set.
", "votes": 1 }, { "id": 82923, "postDate": "2015-06-29T19:06:09.523Z", "content": "Yes. However, it's a moot point since you are predicting series 9 and 10 and don't have any "latter" series labels to train on.
", "votes": 1 }, { "id": 83000, "postDate": "2015-06-30T15:54:21.750Z", "content": "Well, I can give one example for which this is not relevant:
\nlet's say you must have a series of 4 specific waveforms to trigger the signal and prosthetic movement
\nYou also know that you have noise.
\nFor triggering the signal, you really have to be sure so there are no trembling and so on.
\nIf you have the 3rd waveform lacking, you could rule out that movement must never be started. But then you will have a clumsy prothetic. Cause noise will often drown this signal out.
\nBut you could model the thing to "foresee" that signal 3 exists with probability p when signal 4 too.
\nReturn to real life application. Your processor detects first and second waveform, not the third.
\nWith first solution, it will discard the series and wait again.
\nWith second, it will wait until detection of signal 4 to decide that movement can be triggered with a p confidence. It is not that it uses future signal as it is not possible.
\nit is that you modelized that some data could exist with some confidence given that an event has happened too, even if it is subsequent.
\nIs it clearer for you, or do I still look funny? ;)
", "votes": -1 }, { "id": 86318, "postDate": "2015-07-21T15:35:45.973Z", "content": "a ok, indeed I did not realize...now it's clear \nthanks for the quick response! ;)
", "rawMarkdown": "a ok, indeed I did not realize...now it's clear \r\nthanks for the quick response! ;)" }, { "id": 86316, "postDate": "2015-07-21T15:27:47.310Z", "content": "The idea of this restriction is that the resulting method should be applicable for controlling prosthetics using EEG readings. More concretely, if you want to send a control signal to a robotic arm at some point in time t, you can of course only use EEG data recorded up to time t.
\n\nIn the same way while predicting labels for t=11 for subject 1 and series 9, you are only allowed to use EEG data from that series up to t=11. You are of course free to train your algorithm on all the data from all other series of that subject, so this restriction does not affect training at all.
", "rawMarkdown": "The idea of this restriction is that the resulting method should be applicable for controlling prosthetics using EEG readings. More concretely, if you want to send a control signal to a robotic arm at some point in time t, you can of course only use EEG data recorded up to time t. \r\n\r\nIn the same way while predicting labels for t=11 for subject 1 and series 9, you are only allowed to use EEG data from that series up to t=11. You are of course free to train your algorithm on all the data from all other series of that subject, so this restriction does not affect training at all.\r\n" }, { "id": 86292, "postDate": "2015-07-21T13:20:53.573Z", "content": "I'll probably be deficient.. but I do not understand the meaning of this:\nWhen predicting, you may not use data from the future! In other words, if you are predicting labels for id subj1_series9_11, you may not incorporate any frame after 11 of series 9 from subject 1 for any reason.\nBut not incorporate in what? In training dataset? Then I can’t train on all subj1_series9’ frames to predict all ** subj1_series1_events labels???... I hope I have misunderstood…otherwise we would need a model for each frame...mha...\nplease inform me and sorry for my bad English...
", "rawMarkdown": "I'll probably be deficient.. but I do not understand the meaning of this:\r\n***When predicting, you may not use data from the future! In other words, if you are predicting labels for id subj1_series9_11, you may not incorporate any frame after 11 of series 9 from subject 1 for any reason.***\r\nBut not incorporate in what? In training dataset? Then I can’t train on all **subj1_series9’ frames to predict all ** subj1_series1_events** labels???... I hope I have misunderstood…otherwise we would need a model for each frame...mha...\r\nplease inform me and sorry for my bad English...\r\n" }, { "id": 83068, "postDate": "2015-07-01T10:04:49.623Z", "content": "[quote=Alexandre Barachant;83003]
\nYour are just introducing a lag in your system, so you can use the whole sequence to trig the movement. It wont change the problem, because you can not rollback time to execute the movement at the beginning of the sequence, i.e you can not change the probabilities on the detection you made in the past.
\nWhat you describe here can be done for the last element of the sequence, since you have all the past signal available.
\n[/quote]
\nNot to insist too much on that since rules are what they are, but this is not the problem. Actually, in my example you used data from the future of signal 3 to define more accurately the possibility of its presence (but drowning in noise).
\nThe fact that applicatively you can only use data that you have seen for such purpose is logical, but it doesn't prevent us to use signal 4 to define that signals1+2+4 is actually the full series and not only 1+2+4. If you consider that the signal 1-4 is usually just a bloc in a larger signal, maybe it would be 4 that could be predicted using 1 of the second repeat, as they are usually very close in time. For setting the system up, you would need to use "future data", but in application you won't.
\nIn this problem, you could not given the rules. That was my point, but anyway, you will find a lot of ways to do well without that.
" }, { "id": 82940, "postDate": "2015-06-29T22:00:09.480Z", "content": "Ok, thanks. Makes sense now.
" }, { "id": 83018, "postDate": "2015-06-30T20:37:07.770Z", "rawMarkdown": "", "isDeleted": true }, { "id": 83016, "postDate": "2015-06-30T20:09:08.927Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 82996, "author_name": "Alexandre Barachant", "author_url": "", "post_date": "2015-06-30T14:48:56.467000", "content": "[quote=William Cukierski;82992]
\nThe host will do their best to enforce the rule. This might entail removing any teams with performance greater than the "highest known legitimate score."
\n[/quote]
\nThis is what i was thinking, and i believe it is a good way to make this dissuasive enough so most of the people will play by the rules.
\n[quote]
\nAlso, as a neurobiologist, there are factors that are important in neural signals "chains". Removing the possibility to use it is bizarre, from a biological point of view.
\n[/quote]
\nFrom a applicative point of view, this rule make sense. From the challenge point of view, it also force participant to do a bit of EEG classification, and not just reverse-engineering of the paradigm.
", "votes": 3, "replies": [] }, { "id": 82993, "author_name": "WhizWilde", "author_url": "", "post_date": "2015-06-30T14:35:12.033000", "content": "The goal of the competition is to do better than what's already known. So such kind of rule is puzzling. Also, If you don't want contestant to use data in an inappropriate fashion, why not beginning by deliver them properly? I am not trying to minimize kaggle efforts or trolling, but isn't there a good way to do it there?
\nAlso, as a neurobiologist, there are factors that are important in neural signals "chains". Removing the possibility to use it is bizarre, from a biological point of view.
", "votes": 3, "replies": [] }, { "id": 82937, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2015-06-29T21:53:20.577000", "content": "The original thing I posted was just a corner case. The main application is to data within the same series.
\nThe most important point is that we can't use information from the near future. As there is a sequence of 6 different events we are trying to predict it would have been helpful to use the information that the final event occurred to predict the first event but we can't do this because we can't use future data. The rule adds a decent amount of extra difficulty to this competition.
", "votes": 3, "replies": [] }, { "id": 83003, "author_name": "Alexandre Barachant", "author_url": "", "post_date": "2015-06-30T16:09:55.607000", "content": "Your are just introducing a lag in your system, so you can use the whole sequence to trig the movement. It wont change the problem, because you can not rollback time to execute the movement at the beginning of the sequence, i.e you can not change the probabilities on the detection you made in the past.
\nWhat you describe here can be done for the last element of the sequence, since you have all the past signal available.
", "votes": 4, "replies": [] }, { "id": 82997, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-06-30T14:55:20.687000", "content": "[quote=WhizWilde;82993]
\nThe goal of the competition is to do better than what's already known. So such kind of rule is puzzling. Also, If you don't want contestant to use data in an inappropriate fashion, why not beginning by deliver them properly? I am not trying to minimize kaggle efforts or trolling, but isn't there a good way to do it there?
\nAlso, as a neurobiologist, there are factors that are important in neural signals "chains". Removing the possibility to use it is bizarre, from a biological point of view.
\n[/quote]
\nUnless neurobiologists have invented a time machine without telling the rest of us, there shouldn't be anything puzzling about this rule :) An algorithm controlling a prosthetic device does not have access to future EEG signals, so neither should your algorithm. As I mentioned, the only way to not give out this data is to use code submission, but this is a complex bridge we have not yet crossed.
", "votes": 3, "replies": [] }, { "id": 83446, "author_name": "lnicalo", "author_url": "", "post_date": "2015-07-04T22:10:14.497000", "content": "When you filter a signal, you introduce a delay. So, I think you would need to see the future data to predict the label of the current sample the system is processing.
\nIs it valid a lagged response? (let's say 0.5 seconds)
", "votes": 1, "replies": [] }, { "id": 83017, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-06-30T20:34:29.707000", "content": "[quote=I can do the splits NO PROB;83016]
\nWhen predicting, you may not use data from the future! In other words, if you are predicting labels for id subj1_series9_11, you may not incorporate any frame after 11 of series 9 from subject 1 for any reason.
\n[/quote]
\nDone!
\n[quote=I can do the splits NO PROB;83016]
\nAnd btw.: Is there any possibility to extract the trial number in each series (for example like a fixed observation window given in ms)? Usually, one would average over all trials of one subject in order to increase the signal-to-noise ratio in the EEG-signal. Or is this part of the challenge?
\n[/quote]
\nPart of the challenge.
", "votes": 1, "replies": [] }, { "id": 82939, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-06-29T21:57:46.833000", "content": "[quote=J Kolb;82936]
\nSo we can ignore the entire "Important Note" section on the Data page since it is not possible to do, right?
\n[/quote]
\nIt still applies to future data that is in the same series.
", "votes": 1, "replies": [] }, { "id": 82936, "author_name": "J Kolb", "author_url": "", "post_date": "2015-06-29T21:47:09.177000", "content": "[quote=William Cukierski;82923]
\nYes. However, it's a moot point since you are predicting series 9 and 10 and don't have any "latter" series labels to train on.
\n[/quote]
\nSo we can ignore the entire "Important Note" section on the Data page since it is not possible to do, right?
", "votes": 1, "replies": [] }, { "id": 82990, "author_name": "Alexandre Barachant", "author_url": "", "post_date": "2015-06-30T13:41:10.037000", "content": "Hi William,
\nHow do you plan to enforce this rule ? I don't want to be the devil's advocate but people who aims at the kaggle rank only may ignore this rule and decline the prizes.
", "votes": 2, "replies": [] }, { "id": 82992, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-06-30T14:16:06.127000", "content": "Hi Alexandre,
\nThe host will do their best to enforce the rule. This might entail removing any teams with performance greater than the "highest known legitimate score."
\nCompetitions are too popular to check everybody's approach, but we do strive to make sure the winners are clean. We believe that most people play by the rules, understand their intent, and place the greater research goals above their own self interests. The alternative is that we never run time-series competitions like this, or have to move to a setup where participants submit code that runs on a holdout set.
", "votes": 1, "replies": [] }, { "id": 82923, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-06-29T19:06:09.523000", "content": "Yes. However, it's a moot point since you are predicting series 9 and 10 and don't have any "latter" series labels to train on.
", "votes": 1, "replies": [] }, { "id": 83000, "author_name": "WhizWilde", "author_url": "", "post_date": "2015-06-30T15:54:21.750000", "content": "Well, I can give one example for which this is not relevant:
\nlet's say you must have a series of 4 specific waveforms to trigger the signal and prosthetic movement
\nYou also know that you have noise.
\nFor triggering the signal, you really have to be sure so there are no trembling and so on.
\nIf you have the 3rd waveform lacking, you could rule out that movement must never be started. But then you will have a clumsy prothetic. Cause noise will often drown this signal out.
\nBut you could model the thing to "foresee" that signal 3 exists with probability p when signal 4 too.
\nReturn to real life application. Your processor detects first and second waveform, not the third.
\nWith first solution, it will discard the series and wait again.
\nWith second, it will wait until detection of signal 4 to decide that movement can be triggered with a p confidence. It is not that it uses future signal as it is not possible.
\nit is that you modelized that some data could exist with some confidence given that an event has happened too, even if it is subsequent.
\nIs it clearer for you, or do I still look funny? ;)
", "votes": -1, "replies": [] }, { "id": 86318, "author_name": "madara", "author_url": "", "post_date": "2015-07-21T15:35:45.973000", "content": "a ok, indeed I did not realize...now it's clear \nthanks for the quick response! ;)
", "votes": 0, "replies": [] }, { "id": 86316, "author_name": "Klaus Greff", "author_url": "", "post_date": "2015-07-21T15:27:47.310000", "content": "The idea of this restriction is that the resulting method should be applicable for controlling prosthetics using EEG readings. More concretely, if you want to send a control signal to a robotic arm at some point in time t, you can of course only use EEG data recorded up to time t.
\n\nIn the same way while predicting labels for t=11 for subject 1 and series 9, you are only allowed to use EEG data from that series up to t=11. You are of course free to train your algorithm on all the data from all other series of that subject, so this restriction does not affect training at all.
", "votes": 0, "replies": [] }, { "id": 86292, "author_name": "madara", "author_url": "", "post_date": "2015-07-21T13:20:53.573000", "content": "I'll probably be deficient.. but I do not understand the meaning of this:\nWhen predicting, you may not use data from the future! In other words, if you are predicting labels for id subj1_series9_11, you may not incorporate any frame after 11 of series 9 from subject 1 for any reason.\nBut not incorporate in what? In training dataset? Then I can’t train on all subj1_series9’ frames to predict all ** subj1_series1_events labels???... I hope I have misunderstood…otherwise we would need a model for each frame...mha...\nplease inform me and sorry for my bad English...
", "votes": 0, "replies": [] }, { "id": 83068, "author_name": "WhizWilde", "author_url": "", "post_date": "2015-07-01T10:04:49.623000", "content": "[quote=Alexandre Barachant;83003]
\nYour are just introducing a lag in your system, so you can use the whole sequence to trig the movement. It wont change the problem, because you can not rollback time to execute the movement at the beginning of the sequence, i.e you can not change the probabilities on the detection you made in the past.
\nWhat you describe here can be done for the last element of the sequence, since you have all the past signal available.
\n[/quote]
\nNot to insist too much on that since rules are what they are, but this is not the problem. Actually, in my example you used data from the future of signal 3 to define more accurately the possibility of its presence (but drowning in noise).
\nThe fact that applicatively you can only use data that you have seen for such purpose is logical, but it doesn't prevent us to use signal 4 to define that signals1+2+4 is actually the full series and not only 1+2+4. If you consider that the signal 1-4 is usually just a bloc in a larger signal, maybe it would be 4 that could be predicted using 1 of the second repeat, as they are usually very close in time. For setting the system up, you would need to use "future data", but in application you won't.
\nIn this problem, you could not given the rules. That was my point, but anyway, you will find a lot of ways to do well without that.
", "votes": 0, "replies": [] }, { "id": 82940, "author_name": "J Kolb", "author_url": "", "post_date": "2015-06-29T22:00:09.480000", "content": "Ok, thanks. Makes sense now.
", "votes": 0, "replies": [] }, { "id": 83018, "author_name": "", "author_url": "", "post_date": "2015-06-30T20:37:07.770000", "content": "", "votes": 0, "replies": [] }, { "id": 83016, "author_name": "", "author_url": "", "post_date": "2015-06-30T20:09:08.927000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "82996": "", "82993": "", "82937": "", "83003": "", "82997": "", "83446": "", "83017": "", "82939": "", "82936": "", "82914": "", "82990": "", "82992": "", "82923": "", "83000": "", "86318": "a ok, indeed I did not realize...now it's clear \r\nthanks for the quick response! ;)", "86316": "The idea of this restriction is that the resulting method should be applicable for controlling prosthetics using EEG readings. More concretely, if you want to send a control signal to a robotic arm at some point in time t, you can of course only use EEG data recorded up to time t. \r\n\r\nIn the same way while predicting labels for t=11 for subject 1 and series 9, you are only allowed to use EEG data from that series up to t=11. You are of course free to train your algorithm on all the data from all other series of that subject, so this restriction does not affect training at all.\r\n", "86292": "I'll probably be deficient.. but I do not understand the meaning of this:\r\n***When predicting, you may not use data from the future! In other words, if you are predicting labels for id subj1_series9_11, you may not incorporate any frame after 11 of series 9 from subject 1 for any reason.***\r\nBut not incorporate in what? In training dataset? Then I can’t train on all **subj1_series9’ frames to predict all ** subj1_series1_events** labels???... I hope I have misunderstood…otherwise we would need a model for each frame...mha...\r\nplease inform me and sorry for my bad English...\r\n", "83068": "", "82940": "", "83018": "", "83016": "" } }