Hi all,
\n\nIn reviewing the competition dataset, we have discovered a preparation bug that unfortunately affects all event labels (both in training and testing). We therefore must swap the data with a corrected version and invalidate all previous submissions on the leaderboard. Once the new data is live, please download the new files and resubmit your submissions.
\n\nWe are very sorry for this inconvenience are both grateful and impressed with your work so far. As an attempt to make up for the lost time, we will be increasing the daily submission limit to 4/day.
\n\nKeep up the good work!
\n\nBest Regards,\nThe WAY Project Team
", "messages": [ { "id": "86282", "postDate": "07/21/2015 12:11:31", "content": "Hi all,
\n\nIn reviewing the competition dataset, we have discovered a preparation bug that unfortunately affects all event labels (both in training and testing). We therefore must swap the data with a corrected version and invalidate all previous submissions on the leaderboard. Once the new data is live, please download the new files and resubmit your submissions.
\n\nWe are very sorry for this inconvenience are both grateful and impressed with your work so far. As an attempt to make up for the lost time, we will be increasing the daily submission limit to 4/day.
\n\nKeep up the good work!
\n\nBest Regards,\nThe WAY Project Team
", "rawMarkdown": "Hi all,\r\n\r\nIn reviewing the competition dataset, we have discovered a preparation bug that unfortunately affects all event labels (both in training and testing). We therefore must swap the data with a corrected version and invalidate all previous submissions on the leaderboard. Once the new data is live, please download the new files and resubmit your submissions.\r\n\r\nWe are very sorry for this inconvenience are both grateful and impressed with your work so far. As an attempt to make up for the lost time, we will be increasing the daily submission limit to 4/day.\r\n\r\nKeep up the good work!\r\n\r\nBest Regards,\r\nThe WAY Project Team", "votes": null }, { "id": "86297", "postDate": "07/21/2015 13:38:06", "content": "Hi Klaus,
\n\nCan you give us more information about the nature of the bug? It might help us to understand how close our current solutions might be. Thanks in advance.
\n\nAll the best,\nJulie
", "rawMarkdown": "Hi Klaus,\r\n\r\nCan you give us more information about the nature of the bug? It might help us to understand how close our current solutions might be. Thanks in advance.\r\n\r\nAll the best,\r\nJulie", "votes": null }, { "id": "86302", "postDate": "07/21/2015 14:11:42", "content": "The new data is published and scoring has been reset. The data will shortly be updated in Scripts as well.
\n\n@Julie Your current methodology will likely still work (the bug was a time offset error, small but it affected the whole data set).
", "rawMarkdown": "The new data is published and scoring has been reset. The data will shortly be updated in Scripts as well.\r\n\r\n@Julie Your current methodology will likely still work (the bug was a time offset error, small but it affected the whole data set).", "votes": null }, { "id": "86306", "postDate": "07/21/2015 14:44:57", "content": "So only the *_events.csv files has been changed, right?
", "rawMarkdown": "So only the *_events.csv files has been changed, right?", "votes": null }, { "id": "86310", "postDate": "07/21/2015 14:59:50", "content": "[quote=Jasiek;86306]
\n\nSo only the *_events.csv files has been changed, right?
\n\n[/quote]
\n\nCorrect
", "rawMarkdown": "[quote=Jasiek;86306]\r\n\r\nSo only the *_events.csv files has been changed, right?\r\n\r\n[/quote]\r\n\r\nCorrect", "votes": null }, { "id": "86311", "postDate": "07/21/2015 15:04:15", "content": "I've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.
\n\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :)
", "rawMarkdown": "I've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.\r\n\r\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :)", "votes": null }, { "id": "86381", "postDate": "07/21/2015 20:40:23", "content": "@William One small problem still exists in the data:
\n\nThe same thing happens for subj7_series8_63084 - subj7_series8_63074.
\n\nAs far as I see there are only few instances of such problem, so its influence is probably negligible.
\n\n@Everyone It seems that all events were moved with almost constant shift (-1002, -1001 or -1000 ticks).
", "rawMarkdown": "William One small problem still exists in the data:\r\n\r\n - subj7_series7_40520 - end of FirstDigitTouch;\r\n - subj7_series7_40596 - start of LiftOff;\r\n - and there is no BothStartLoadPhase in the middle.\r\n\r\nThe same thing happens for subj7_series8_63084 - subj7_series8_63074.\r\n\r\nAs far as I see there are only few instances of such problem, so its influence is probably negligible.\r\n\r\n@Everyone It seems that all events were moved with almost constant shift (-1002, -1001 or -1000 ticks).", "votes": null }, { "id": "86382", "postDate": "07/21/2015 20:56:34", "content": "@mibel : Thanks for reporting that. 1000 ticks is two second ! \nThis simple fact make me doubt even more that our solutions are actually decoding brain signals.\nAn AUC ~0.95 is already too good to be true (for EEG classification)
", "rawMarkdown": "mibel : Thanks for reporting that. 1000 ticks is two second ! \r\nThis simple fact make me doubt even more that our solutions are actually decoding brain signals.\r\nAn AUC ~0.95 is already too good to be true (for EEG classification)", "votes": null }, { "id": "86385", "postDate": "07/21/2015 21:59:16", "content": "I was very surprised in the previous dataset when people get 98% and when simple rescaled time-domain, unfiltered raw signal gave 70%+. I still doubt this dataset. It is still too early and yet single submission trial by #1 is 92%. It is still too early to get such a hugh AUC.
\n\nI reran Elena's code (raw rescaled data) and it gave me 71%. Previous one was 70%. It actually improved. 2 seconds is so huge for so many other events to affect the 'elicited' response. I work with physiological signals (EKG, PPG, EDA, EMG, ...) and my colleague works with EEG and I know that these signals are very noisy. They can be corrupted with so many sources of artifacts: movement, EOG, EMG, EKG, powerline interference, ... I still don't believe this very high AUC.
", "rawMarkdown": "I was very surprised in the previous dataset when people get 98% and when simple rescaled time-domain, unfiltered raw signal gave 70%+. I still doubt this dataset. It is still too early and yet single submission trial by #1 is 92%. It is still too early to get such a hugh AUC.\r\n\r\nI reran Elena's code (raw rescaled data) and it gave me 71%. Previous one was 70%. It actually improved. 2 seconds is so huge for so many other events to affect the 'elicited' response. I work with physiological signals (EKG, PPG, EDA, EMG, ...) and my colleague works with EEG and I know that these signals are very noisy. They can be corrupted with so many sources of artifacts: movement, EOG, EMG, EKG, powerline interference, ... I still don't believe this very high AUC.", "votes": null }, { "id": "86387", "postDate": "07/21/2015 22:14:31", "content": "Yep,\nwe are probably triggering on some residual EMG activity + learning the experimental constraints.
\n\nAsynchronous BCI are considered as the hardest challenge to solve. This one is even more difficult since the event duration are really short (150ms).
\n\nI'm sure the organizer already tried to run a classifier on this data. i will be really interested to know the value of their in-house AUC. This is probably the crazy AUC we got that make them look for possible Leakage.
", "rawMarkdown": "Yep,\r\nwe are probably triggering on some residual EMG activity + learning the experimental constraints.\r\n\r\nAsynchronous BCI are considered as the hardest challenge to solve. This one is even more difficult since the event duration are really short (150ms). \r\n\r\nI'm sure the organizer already tried to run a classifier on this data. i will be really interested to know the value of their in-house AUC. This is probably the crazy AUC we got that make them look for possible Leakage.", "votes": null }, { "id": "86388", "postDate": "07/21/2015 22:24:48", "content": "@Deep @Alex: I have a hypothesis for why Elena's code (raw rescaled data) has better AUC than Alex's CSP code. This may be because the finger and arm sensors interfered the EEG sensors. The finger and arm sensors generated a lot high-frequency (HF) signals, which easily modulated with EEG sensors (it could be regarded as leakage). Alex's CSP code firstly removes the HF signals, which removes the leakage equivalently .
\n\nDo you think the EEG here having too much HF signals compared to other EEG signals?
\n\nBTW, my education background was wireless and signal processing.
", "rawMarkdown": "Deep @Alex: I have a hypothesis for why Elena's code (raw rescaled data) has better AUC than Alex's CSP code. This may be because the finger and arm sensors interfered the EEG sensors. The finger and arm sensors generated a lot high-frequency (HF) signals, which easily modulated with EEG sensors (it could be regarded as leakage). Alex's CSP code firstly removes the HF signals, which removes the leakage equivalently . \r\n\r\nDo you think the EEG here having too much HF signals compared to other EEG signals?\r\n\r\nBTW, my education background was wireless and signal processing.", "votes": null }, { "id": "86390", "postDate": "07/21/2015 22:36:36", "content": "There seems to be a lot of useful predictive information contained in very low frequencies (like < 3Hz). Elena's code does not filter low frequencies too. My wild guess is that head/body movements (i.e. not the related brain activity, but movements themselves) got somehow leaked into EEG signals. But I'm not an expert though.
", "rawMarkdown": "There seems to be a lot of useful predictive information contained in very low frequencies (like < 3Hz). Elena's code does not filter low frequencies too. My wild guess is that head/body movements (i.e. not the related brain activity, but movements themselves) got somehow leaked into EEG signals. But I'm not an expert though.", "votes": null }, { "id": "86391", "postDate": "07/21/2015 22:39:28", "content": "@ blaine, Indeed, this is also my main hypothesis.
\n\nIf you plot the coefs of the LR on a topomap, you will see it picks the electrodes on the frontal or on the occipital part, known to be corrupted by many source of artefact (movement, muscle, EOG)
", "rawMarkdown": "blaine, Indeed, this is also my main hypothesis.\r\n\r\nIf you plot the coefs of the LR on a topomap, you will see it picks the electrodes on the frontal or on the occipital part, known to be corrupted by many source of artefact (movement, muscle, EOG)", "votes": null }, { "id": "86416", "postDate": "07/22/2015 03:54:21", "content": "Also, how is that possible that each target action has exactly 150 points ? Fo each series, you have either 28 or 34 sub series for each target of exactly 150 value 1. \nHow was the data labelled ? Actions cant be always exactly the same length ...
", "rawMarkdown": "Also, how is that possible that each target action has exactly 150 points ? Fo each series, you have either 28 or 34 sub series for each target of exactly 150 value 1. \r\nHow was the data labelled ? Actions cant be always exactly the same length ...", "votes": null }, { "id": "86425", "postDate": "07/22/2015 06:42:16", "content": "Hi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?
", "rawMarkdown": "Hi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?", "votes": null }, { "id": "86427", "postDate": "07/22/2015 06:56:38", "content": "@Guocong, you might be right that the absense of the bandpass filter could be the reason the raw rescaled signal is performing well. However, I doubt if it is just EMG (muscle) interference alone. EMG signal is ussually very high frequency (~50-500Hz). Obviously, it would be one contributing factor. But, from the time-domain plot of the EEG signal, it doesn't look to contain very high frequency noise. Rather I could see low frequency baseline wander and motion artifact. So, @blaine could be right. We could indeed be fitting low frequency noise.
\n\nBut, I second Alex's call for the organizers to give us benchmark in-house AUC so we could have some clue as to what we are really doing here. Organizers? Admins?
", "rawMarkdown": "Guocong, you might be right that the absense of the bandpass filter could be the reason the raw rescaled signal is performing well. However, I doubt if it is just EMG (muscle) interference alone. EMG signal is ussually very high frequency (~50-500Hz). Obviously, it would be one contributing factor. But, from the time-domain plot of the EEG signal, it doesn't look to contain very high frequency noise. Rather I could see low frequency baseline wander and motion artifact. So, @blaine could be right. We could indeed be fitting low frequency noise.\r\n\r\nBut, I second Alex's call for the organizers to give us benchmark in-house AUC so we could have some clue as to what we are really doing here. Organizers? Admins?", "votes": null }, { "id": "86430", "postDate": "07/22/2015 07:16:33", "content": "[quote=Elena Cuoco;86425]
\n\nHi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?
\n\n[/quote]\nMy submission of 0.92 AUC (0.9 validation AUC) is in essence your code with some minor tricks added (nothing really fancy - no tuned labelling or anything like that) and exploiting low frequency content. The latter part is the main source of >0.9 AUC in my case.
", "rawMarkdown": "[quote=Elena Cuoco;86425]\r\n\r\nHi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?\r\n\r\n[/quote]\r\nMy submission of 0.92 AUC (0.9 validation AUC) is in essence your code with some minor tricks added (nothing really fancy - no tuned labelling or anything like that) and exploiting low frequency content. The latter part is the main source of >0.9 AUC in my case.", "votes": null }, { "id": "86432", "postDate": "07/22/2015 07:29:06", "content": "@blaine, thanks for being honest and confirming that! Now, I am pretty convinced that it is indeed the low frequency baseline wander and motion artifact that is coming out prominent.
\n\nSo, I wonder in this case then if we can say we are predicting from EEG rather from the motion pattern (meaning motion artifact). Ussually you'd remove/reject motion artifact and low frequency non-signal components and very high frequency noise using bandpass so you would concentrate on the real electrophysiological signal.
", "rawMarkdown": "blaine, thanks for being honest and confirming that! Now, I am pretty convinced that it is indeed the low frequency baseline wander and motion artifact that is coming out prominent. \r\n\r\nSo, I wonder in this case then if we can say we are predicting from EEG rather from the motion pattern (meaning motion artifact). Ussually you'd remove/reject motion artifact and low frequency non-signal components and very high frequency noise using bandpass so you would concentrate on the real electrophysiological signal.", "votes": null }, { "id": "86435", "postDate": "07/22/2015 08:02:54", "content": "I use band-pass filter 1-35 Hz in my current solution, so at least "classical" EMG frequencies are suppressed. AUC of this solution is 0.92 though.
\n\nI suppose that there are a few reasons of such improbable scores:
\n\n[quote=mibel;86311]
\n\nI've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.
\n\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :)
\n\n[/quote]
\n\nI was thinking of trying the same thing. I'm surprised the effect is so drastic.
", "rawMarkdown": "[quote=mibel;86311]\r\n\r\nI've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.\r\n\r\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :) \r\n\r\n[/quote]\r\n\r\nI was thinking of trying the same thing. I'm surprised the effect is so drastic.", "votes": null }, { "id": "86785", "postDate": "07/24/2015 10:35:33", "content": "In Santas Little Helper the data wasn't thought through properly and the competition was reset but the current leader prior to the reset was rewarded - is the same thing going to happen here? The leader deserves at least some free swag?!
", "rawMarkdown": "In Santas Little Helper the data wasn't thought through properly and the competition was reset but the current leader prior to the reset was rewarded - is the same thing going to happen here? The leader deserves at least some free swag?!", "votes": null }, { "id": "90513", "postDate": "08/27/2015 09:26:59", "content": "@Klaus Greff Hi, my latest submission is a such good result surprises me, after I check my code, I think I used wrong data index, so the submission may be predicted by using future feature. please help me remove it from LB, really sorry for inconvenience.
", "rawMarkdown": "Klaus Greff Hi, my latest submission is a such good result surprises me, after I check my code, I think I used wrong data index, so the submission may be predicted by using future feature. please help me remove it from LB, really sorry for inconvenience.", "votes": null }, { "id": "90514", "postDate": "08/27/2015 09:44:21", "content": "Hi @Bowen-Yuan. I'm sorry, but I don't know how to do that. But I'll ask William Cukierski from Kaggle to look into it.
", "rawMarkdown": "Hi @Bowen-Yuan. I'm sorry, but I don't know how to do that. But I'll ask William Cukierski from Kaggle to look into it.", "votes": null }, { "id": "90523", "postDate": "08/27/2015 12:15:58", "content": "It's Ok Bowen! You can just not select that submission in the end. Thank you for reporting! We tried so hard to get any future leak out of the system. You warm our heart : )
", "rawMarkdown": "It's Ok Bowen! You can just not select that submission in the end. Thank you for reporting! We tried so hard to get any future leak out of the system. You warm our heart : )", "votes": null }, { "id": "90537", "postDate": "08/27/2015 15:23:33", "content": "Out of curiosity, i once ran part of our pipeline with shifted labels (like in the original dataset). According to our CV, should have scored near to 0.987. i think with the original dataset, LB score would have end up higher than 0.99.
\n\nIn BCI, it is a common practice to introduce a small latency on the prediction. The reason for that is that brain pattern related to hand movement usually occurs slightly after the execution of the movement. From a user perspective, it is well tolerated to have a small latency on the output as long as it improve classification accuracy.\nI made this kind of test during PhD thesis (the topic was online detection of imagined Hand movement from EEG), and it turns out that a latency up to 0.5s is well tolerated by the user. More than that, and is start to be confusing.
\n\nSaying that, it really depends on the application, and you have to find a good tradeoff between latency and accuracy. For real time continuous feedback, like 1D /2D cursor control, latency is unbearable (imagine you have a 0.5s delay when you move your mouse), and you will prefer a noisy output rather than a delayed output. For a simple brain switch, you can go up to 2s, because you really want to minimize the false positive rather than having a fast acting switch.
\n\nSorry for the off topic :)
", "rawMarkdown": "Out of curiosity, i once ran part of our pipeline with shifted labels (like in the original dataset). According to our CV, should have scored near to 0.987. i think with the original dataset, LB score would have end up higher than 0.99. \r\n\r\nIn BCI, it is a common practice to introduce a small latency on the prediction. The reason for that is that brain pattern related to hand movement usually occurs slightly after the execution of the movement. From a user perspective, it is well tolerated to have a small latency on the output as long as it improve classification accuracy.\r\nI made this kind of test during PhD thesis (the topic was online detection of imagined Hand movement from EEG), and it turns out that a latency up to 0.5s is well tolerated by the user. More than that, and is start to be confusing.\r\n\r\nSaying that, it really depends on the application, and you have to find a good tradeoff between latency and accuracy. For real time continuous feedback, like 1D /2D cursor control, latency is unbearable (imagine you have a 0.5s delay when you move your mouse), and you will prefer a noisy output rather than a delayed output. For a simple brain switch, you can go up to 2s, because you really want to minimize the false positive rather than having a fast acting switch.\r\n\r\nSorry for the off topic :)", "votes": null } ], "comments": [ { "id": 86297, "author_name": "jlachance", "author_url": "", "post_date": "07/21/2015 13:38:06", "content": "Hi Klaus,
\n\nCan you give us more information about the nature of the bug? It might help us to understand how close our current solutions might be. Thanks in advance.
\n\nAll the best,\nJulie
", "votes": null, "replies": [] }, { "id": 86302, "author_name": "wcukierski", "author_url": "", "post_date": "07/21/2015 14:11:42", "content": "The new data is published and scoring has been reset. The data will shortly be updated in Scripts as well.
\n\n@Julie Your current methodology will likely still work (the bug was a time offset error, small but it affected the whole data set).
", "votes": null, "replies": [] }, { "id": 86306, "author_name": "jasiekl", "author_url": "", "post_date": "07/21/2015 14:44:57", "content": "So only the *_events.csv files has been changed, right?
", "votes": null, "replies": [] }, { "id": 86310, "author_name": "wcukierski", "author_url": "", "post_date": "07/21/2015 14:59:50", "content": "[quote=Jasiek;86306]
\n\nSo only the *_events.csv files has been changed, right?
\n\n[/quote]
\n\nCorrect
", "votes": null, "replies": [] }, { "id": 86311, "author_name": "mibell", "author_url": "", "post_date": "07/21/2015 15:04:15", "content": "I've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.
\n\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :)
", "votes": null, "replies": [] }, { "id": 86381, "author_name": "mibell", "author_url": "", "post_date": "07/21/2015 20:40:23", "content": "@William One small problem still exists in the data:
\n\nThe same thing happens for subj7_series8_63084 - subj7_series8_63074.
\n\nAs far as I see there are only few instances of such problem, so its influence is probably negligible.
\n\n@Everyone It seems that all events were moved with almost constant shift (-1002, -1001 or -1000 ticks).
", "votes": null, "replies": [] }, { "id": 86382, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/21/2015 20:56:34", "content": "@mibel : Thanks for reporting that. 1000 ticks is two second ! \nThis simple fact make me doubt even more that our solutions are actually decoding brain signals.\nAn AUC ~0.95 is already too good to be true (for EEG classification)
", "votes": null, "replies": [] }, { "id": 86385, "author_name": "deepcnn", "author_url": "", "post_date": "07/21/2015 21:59:16", "content": "I was very surprised in the previous dataset when people get 98% and when simple rescaled time-domain, unfiltered raw signal gave 70%+. I still doubt this dataset. It is still too early and yet single submission trial by #1 is 92%. It is still too early to get such a hugh AUC.
\n\nI reran Elena's code (raw rescaled data) and it gave me 71%. Previous one was 70%. It actually improved. 2 seconds is so huge for so many other events to affect the 'elicited' response. I work with physiological signals (EKG, PPG, EDA, EMG, ...) and my colleague works with EEG and I know that these signals are very noisy. They can be corrupted with so many sources of artifacts: movement, EOG, EMG, EKG, powerline interference, ... I still don't believe this very high AUC.
", "votes": null, "replies": [] }, { "id": 86387, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/21/2015 22:14:31", "content": "Yep,\nwe are probably triggering on some residual EMG activity + learning the experimental constraints.
\n\nAsynchronous BCI are considered as the hardest challenge to solve. This one is even more difficult since the event duration are really short (150ms).
\n\nI'm sure the organizer already tried to run a classifier on this data. i will be really interested to know the value of their in-house AUC. This is probably the crazy AUC we got that make them look for possible Leakage.
", "votes": null, "replies": [] }, { "id": 86388, "author_name": "songgc", "author_url": "", "post_date": "07/21/2015 22:24:48", "content": "@Deep @Alex: I have a hypothesis for why Elena's code (raw rescaled data) has better AUC than Alex's CSP code. This may be because the finger and arm sensors interfered the EEG sensors. The finger and arm sensors generated a lot high-frequency (HF) signals, which easily modulated with EEG sensors (it could be regarded as leakage). Alex's CSP code firstly removes the HF signals, which removes the leakage equivalently .
\n\nDo you think the EEG here having too much HF signals compared to other EEG signals?
\n\nBTW, my education background was wireless and signal processing.
", "votes": null, "replies": [] }, { "id": 86390, "author_name": "rafalcycon", "author_url": "", "post_date": "07/21/2015 22:36:36", "content": "There seems to be a lot of useful predictive information contained in very low frequencies (like < 3Hz). Elena's code does not filter low frequencies too. My wild guess is that head/body movements (i.e. not the related brain activity, but movements themselves) got somehow leaked into EEG signals. But I'm not an expert though.
", "votes": null, "replies": [] }, { "id": 86391, "author_name": "alexandrebarachant", "author_url": "", "post_date": "07/21/2015 22:39:28", "content": "@ blaine, Indeed, this is also my main hypothesis.
\n\nIf you plot the coefs of the LR on a topomap, you will see it picks the electrodes on the frontal or on the occipital part, known to be corrupted by many source of artefact (movement, muscle, EOG)
", "votes": null, "replies": [] }, { "id": 86416, "author_name": "pierregutierrez", "author_url": "", "post_date": "07/22/2015 03:54:21", "content": "Also, how is that possible that each target action has exactly 150 points ? Fo each series, you have either 28 or 34 sub series for each target of exactly 150 value 1. \nHow was the data labelled ? Actions cant be always exactly the same length ...
", "votes": null, "replies": [] }, { "id": 86425, "author_name": "elenacuoco", "author_url": "", "post_date": "07/22/2015 06:42:16", "content": "Hi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?
", "votes": null, "replies": [] }, { "id": 86427, "author_name": "deepcnn", "author_url": "", "post_date": "07/22/2015 06:56:38", "content": "@Guocong, you might be right that the absense of the bandpass filter could be the reason the raw rescaled signal is performing well. However, I doubt if it is just EMG (muscle) interference alone. EMG signal is ussually very high frequency (~50-500Hz). Obviously, it would be one contributing factor. But, from the time-domain plot of the EEG signal, it doesn't look to contain very high frequency noise. Rather I could see low frequency baseline wander and motion artifact. So, @blaine could be right. We could indeed be fitting low frequency noise.
\n\nBut, I second Alex's call for the organizers to give us benchmark in-house AUC so we could have some clue as to what we are really doing here. Organizers? Admins?
", "votes": null, "replies": [] }, { "id": 86430, "author_name": "rafalcycon", "author_url": "", "post_date": "07/22/2015 07:16:33", "content": "[quote=Elena Cuoco;86425]
\n\nHi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?
\n\n[/quote]\nMy submission of 0.92 AUC (0.9 validation AUC) is in essence your code with some minor tricks added (nothing really fancy - no tuned labelling or anything like that) and exploiting low frequency content. The latter part is the main source of >0.9 AUC in my case.
", "votes": null, "replies": [] }, { "id": 86432, "author_name": "deepcnn", "author_url": "", "post_date": "07/22/2015 07:29:06", "content": "@blaine, thanks for being honest and confirming that! Now, I am pretty convinced that it is indeed the low frequency baseline wander and motion artifact that is coming out prominent.
\n\nSo, I wonder in this case then if we can say we are predicting from EEG rather from the motion pattern (meaning motion artifact). Ussually you'd remove/reject motion artifact and low frequency non-signal components and very high frequency noise using bandpass so you would concentrate on the real electrophysiological signal.
", "votes": null, "replies": [] }, { "id": 86435, "author_name": "mibell", "author_url": "", "post_date": "07/22/2015 08:02:54", "content": "I use band-pass filter 1-35 Hz in my current solution, so at least "classical" EMG frequencies are suppressed. AUC of this solution is 0.92 though.
\n\nI suppose that there are a few reasons of such improbable scores:
\n\n[quote=mibel;86311]
\n\nI've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.
\n\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :)
\n\n[/quote]
\n\nI was thinking of trying the same thing. I'm surprised the effect is so drastic.
", "votes": null, "replies": [] }, { "id": 86785, "author_name": "scirpus", "author_url": "", "post_date": "07/24/2015 10:35:33", "content": "In Santas Little Helper the data wasn't thought through properly and the competition was reset but the current leader prior to the reset was rewarded - is the same thing going to happen here? The leader deserves at least some free swag?!
", "votes": null, "replies": [] }, { "id": 90513, "author_name": "bowenyuan", "author_url": "", "post_date": "08/27/2015 09:26:59", "content": "@Klaus Greff Hi, my latest submission is a such good result surprises me, after I check my code, I think I used wrong data index, so the submission may be predicted by using future feature. please help me remove it from LB, really sorry for inconvenience.
", "votes": null, "replies": [] }, { "id": 90514, "author_name": "qwlouse", "author_url": "", "post_date": "08/27/2015 09:44:21", "content": "Hi @Bowen-Yuan. I'm sorry, but I don't know how to do that. But I'll ask William Cukierski from Kaggle to look into it.
", "votes": null, "replies": [] }, { "id": 90523, "author_name": "jiweiliu", "author_url": "", "post_date": "08/27/2015 12:15:58", "content": "It's Ok Bowen! You can just not select that submission in the end. Thank you for reporting! We tried so hard to get any future leak out of the system. You warm our heart : )
", "votes": null, "replies": [] }, { "id": 90537, "author_name": "alexandrebarachant", "author_url": "", "post_date": "08/27/2015 15:23:33", "content": "Out of curiosity, i once ran part of our pipeline with shifted labels (like in the original dataset). According to our CV, should have scored near to 0.987. i think with the original dataset, LB score would have end up higher than 0.99.
\n\nIn BCI, it is a common practice to introduce a small latency on the prediction. The reason for that is that brain pattern related to hand movement usually occurs slightly after the execution of the movement. From a user perspective, it is well tolerated to have a small latency on the output as long as it improve classification accuracy.\nI made this kind of test during PhD thesis (the topic was online detection of imagined Hand movement from EEG), and it turns out that a latency up to 0.5s is well tolerated by the user. More than that, and is start to be confusing.
\n\nSaying that, it really depends on the application, and you have to find a good tradeoff between latency and accuracy. For real time continuous feedback, like 1D /2D cursor control, latency is unbearable (imagine you have a 0.5s delay when you move your mouse), and you will prefer a noisy output rather than a delayed output. For a simple brain switch, you can go up to 2s, because you really want to minimize the false positive rather than having a fast acting switch.
\n\nSorry for the off topic :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86282": "Hi all,\r\n\r\nIn reviewing the competition dataset, we have discovered a preparation bug that unfortunately affects all event labels (both in training and testing). We therefore must swap the data with a corrected version and invalidate all previous submissions on the leaderboard. Once the new data is live, please download the new files and resubmit your submissions.\r\n\r\nWe are very sorry for this inconvenience are both grateful and impressed with your work so far. As an attempt to make up for the lost time, we will be increasing the daily submission limit to 4/day.\r\n\r\nKeep up the good work!\r\n\r\nBest Regards,\r\nThe WAY Project Team", "86297": "Hi Klaus,\r\n\r\nCan you give us more information about the nature of the bug? It might help us to understand how close our current solutions might be. Thanks in advance.\r\n\r\nAll the best,\r\nJulie", "86302": "The new data is published and scoring has been reset. The data will shortly be updated in Scripts as well.\r\n\r\n@Julie Your current methodology will likely still work (the bug was a time offset error, small but it affected the whole data set).", "86306": "So only the *_events.csv files has been changed, right?", "86310": "[quote=Jasiek;86306]\r\n\r\nSo only the *_events.csv files has been changed, right?\r\n\r\n[/quote]\r\n\r\nCorrect", "86311": "I've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.\r\n\r\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :)", "86381": "William One small problem still exists in the data:\r\n\r\n - subj7_series7_40520 - end of FirstDigitTouch;\r\n - subj7_series7_40596 - start of LiftOff;\r\n - and there is no BothStartLoadPhase in the middle.\r\n\r\nThe same thing happens for subj7_series8_63084 - subj7_series8_63074.\r\n\r\nAs far as I see there are only few instances of such problem, so its influence is probably negligible.\r\n\r\n@Everyone It seems that all events were moved with almost constant shift (-1002, -1001 or -1000 ticks).", "86382": "mibel : Thanks for reporting that. 1000 ticks is two second ! \r\nThis simple fact make me doubt even more that our solutions are actually decoding brain signals.\r\nAn AUC ~0.95 is already too good to be true (for EEG classification)", "86385": "I was very surprised in the previous dataset when people get 98% and when simple rescaled time-domain, unfiltered raw signal gave 70%+. I still doubt this dataset. It is still too early and yet single submission trial by #1 is 92%. It is still too early to get such a hugh AUC.\r\n\r\nI reran Elena's code (raw rescaled data) and it gave me 71%. Previous one was 70%. It actually improved. 2 seconds is so huge for so many other events to affect the 'elicited' response. I work with physiological signals (EKG, PPG, EDA, EMG, ...) and my colleague works with EEG and I know that these signals are very noisy. They can be corrupted with so many sources of artifacts: movement, EOG, EMG, EKG, powerline interference, ... I still don't believe this very high AUC.", "86387": "Yep,\r\nwe are probably triggering on some residual EMG activity + learning the experimental constraints.\r\n\r\nAsynchronous BCI are considered as the hardest challenge to solve. This one is even more difficult since the event duration are really short (150ms). \r\n\r\nI'm sure the organizer already tried to run a classifier on this data. i will be really interested to know the value of their in-house AUC. This is probably the crazy AUC we got that make them look for possible Leakage.", "86388": "Deep @Alex: I have a hypothesis for why Elena's code (raw rescaled data) has better AUC than Alex's CSP code. This may be because the finger and arm sensors interfered the EEG sensors. The finger and arm sensors generated a lot high-frequency (HF) signals, which easily modulated with EEG sensors (it could be regarded as leakage). Alex's CSP code firstly removes the HF signals, which removes the leakage equivalently . \r\n\r\nDo you think the EEG here having too much HF signals compared to other EEG signals?\r\n\r\nBTW, my education background was wireless and signal processing.", "86390": "There seems to be a lot of useful predictive information contained in very low frequencies (like < 3Hz). Elena's code does not filter low frequencies too. My wild guess is that head/body movements (i.e. not the related brain activity, but movements themselves) got somehow leaked into EEG signals. But I'm not an expert though.", "86391": "blaine, Indeed, this is also my main hypothesis.\r\n\r\nIf you plot the coefs of the LR on a topomap, you will see it picks the electrodes on the frontal or on the occipital part, known to be corrupted by many source of artefact (movement, muscle, EOG)", "86416": "Also, how is that possible that each target action has exactly 150 points ? Fo each series, you have either 28 or 34 sub series for each target of exactly 150 value 1. \r\nHow was the data labelled ? Actions cant be always exactly the same length ...", "86425": "Hi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?", "86427": "Guocong, you might be right that the absense of the bandpass filter could be the reason the raw rescaled signal is performing well. However, I doubt if it is just EMG (muscle) interference alone. EMG signal is ussually very high frequency (~50-500Hz). Obviously, it would be one contributing factor. But, from the time-domain plot of the EEG signal, it doesn't look to contain very high frequency noise. Rather I could see low frequency baseline wander and motion artifact. So, @blaine could be right. We could indeed be fitting low frequency noise.\r\n\r\nBut, I second Alex's call for the organizers to give us benchmark in-house AUC so we could have some clue as to what we are really doing here. Organizers? Admins?", "86430": "[quote=Elena Cuoco;86425]\r\n\r\nHi all, I wrote my simple script just as starting point to work on data set. I work on time series signal processing too, and I wanted to use my own software. But when I tried some cleaning procedure on the data,the results were worste than the simple starting script. Now I'm curious to check again,as soon as I can.I was wondering if the ones with highest accuracy,are using some kind of tuned labelling,that is once found the first trigger they labeled for 150 frames,as it is for train data.is it so or you obtain such good results with a ' black box' prediction?\r\n\r\n[/quote]\r\nMy submission of 0.92 AUC (0.9 validation AUC) is in essence your code with some minor tricks added (nothing really fancy - no tuned labelling or anything like that) and exploiting low frequency content. The latter part is the main source of >0.9 AUC in my case.", "86432": "blaine, thanks for being honest and confirming that! Now, I am pretty convinced that it is indeed the low frequency baseline wander and motion artifact that is coming out prominent. \r\n\r\nSo, I wonder in this case then if we can say we are predicting from EEG rather from the motion pattern (meaning motion artifact). Ussually you'd remove/reject motion artifact and low frequency non-signal components and very high frequency noise using bandpass so you would concentrate on the real electrophysiological signal.", "86435": "I use band-pass filter 1-35 Hz in my current solution, so at least \"classical\" EMG frequencies are suppressed. AUC of this solution is 0.92 though.\r\n\r\nI suppose that there are a few reasons of such improbable scores: \r\n\r\n- Column-wise auc is a slightly inflated metric. If the objective were accuracy of 6-class classification, the results wouldn't be so impressive.\r\n- This experiment is only quasi-asynchronous since there are more or less stable sequence of events. Our model can (even implicitly) learn & reproduce this sequence. It would be great to have a test session with a random delays between, for example, BothStartLoadPhas and LiftOff. Such session would allow to check this hypothesis.\r\n- Also we can probably catch P300 related to visual stimulus (start & release signals).", "86444": "[quote=mibel;86311]\r\n\r\nI've just resubmitted my best entry for the previous version of data (auc > 0.95) and received 0.32, so events were moved significantly. It certainly doesn't mean that our previous approaches won't work, this commit was just a quick&dirty check.\r\n\r\nProbably this bug is the reason of some strange behavior I observed earlier, so things started falling into place :) \r\n\r\n[/quote]\r\n\r\nI was thinking of trying the same thing. I'm surprised the effect is so drastic.", "86785": "In Santas Little Helper the data wasn't thought through properly and the competition was reset but the current leader prior to the reset was rewarded - is the same thing going to happen here? The leader deserves at least some free swag?!", "90513": "Klaus Greff Hi, my latest submission is a such good result surprises me, after I check my code, I think I used wrong data index, so the submission may be predicted by using future feature. please help me remove it from LB, really sorry for inconvenience.", "90514": "Hi @Bowen-Yuan. I'm sorry, but I don't know how to do that. But I'll ask William Cukierski from Kaggle to look into it.", "90523": "It's Ok Bowen! You can just not select that submission in the end. Thank you for reporting! We tried so hard to get any future leak out of the system. You warm our heart : )", "90537": "Out of curiosity, i once ran part of our pipeline with shifted labels (like in the original dataset). According to our CV, should have scored near to 0.987. i think with the original dataset, LB score would have end up higher than 0.99. \r\n\r\nIn BCI, it is a common practice to introduce a small latency on the prediction. The reason for that is that brain pattern related to hand movement usually occurs slightly after the execution of the movement. From a user perspective, it is well tolerated to have a small latency on the output as long as it improve classification accuracy.\r\nI made this kind of test during PhD thesis (the topic was online detection of imagined Hand movement from EEG), and it turns out that a latency up to 0.5s is well tolerated by the user. More than that, and is start to be confusing.\r\n\r\nSaying that, it really depends on the application, and you have to find a good tradeoff between latency and accuracy. For real time continuous feedback, like 1D /2D cursor control, latency is unbearable (imagine you have a 0.5s delay when you move your mouse), and you will prefer a noisy output rather than a delayed output. For a simple brain switch, you can go up to 2s, because you really want to minimize the false positive rather than having a fast acting switch.\r\n\r\nSorry for the off topic :)" }, "source": "meta" }