{
  "id": 15961,
  "title": "Is per subject model allowed?",
  "url": "/competitions/grasp-and-lift-eeg-detection/discussion/15961",
  "author_name": "",
  "post_date": "2015-08-16T02:26:17.683Z",
  "votes": null,
  "comment_count": 5,
  "views": 1277,
  "content": "<p>if subject 1:\n    then ...\nelse:\n    then ...</p>\n\n<p>Is it allowed? Thank you.</p>",
  "messages": [
    {
      "id": "89484",
      "postDate": "08/16/2015 02:26:17",
      "content": "<p>if subject 1:\n    then ...\nelse:\n    then ...</p>\n\n<p>Is it allowed? Thank you.</p>",
      "rawMarkdown": "if subject 1:\r\n    then ...\r\nelse:\r\n    then ...\r\n\r\nIs it allowed? Thank you.",
      "votes": null
    },
    {
      "id": "89493",
      "postDate": "08/16/2015 09:51:11",
      "content": "<p>Absolutely! In fact, we would be surprised if you could generalize well over subjects.</p>",
      "rawMarkdown": "Absolutely! In fact, we would be surprised if you could generalize well over subjects.",
      "votes": null
    },
    {
      "id": "89507",
      "postDate": "08/16/2015 13:07:56",
      "content": "<p>[quote=Klaus Greff;89493]</p>\n\n<p>Absolutely! In fact, we would be surprised if you could generalize well over subjects.</p>\n\n<p>[/quote]\nThank you for reply. Sorry I probably didn't make myself clear. so far I think the common strategy is </p>\n\n<pre><code>for each subject:\n\n      train a model with the same hyper parameters\n</code></pre>\n\n<p>This is legal.</p>\n\n<p>I am asking</p>\n\n<pre><code>if subject 1:\n\n     train model 1\n\nif subject 2:\n\n     train model 2 (different hyper parameters)\n</code></pre>\n\n<p>....</p>\n\n<p>I am happy it is legal, but the generalizability of this approach is limited because for a new subject, we may not be able to find a new model with similar performance.</p>\n\n<p>Thank you very much.</p>\n\n<p>p.s. this is not allowed in another eeg related challenge, <a href=\"https://www.kaggle.com/c/seizure-detection/rules\">https://www.kaggle.com/c/seizure-detection/rules</a></p>",
      "rawMarkdown": "[quote=Klaus Greff;89493]\r\n\r\nAbsolutely! In fact, we would be surprised if you could generalize well over subjects.\r\n\r\n[/quote]\r\nThank you for reply. Sorry I probably didn't make myself clear. so far I think the common strategy is \r\n\r\n    for each subject:\r\n\r\n          train a model with the same hyper parameters\r\n\r\nThis is legal.\r\n\r\nI am asking\r\n\r\n    if subject 1:\r\n\r\n         train model 1\r\n\r\n    if subject 2:\r\n\r\n         train model 2 (different hyper parameters)\r\n\r\n....\r\n\r\nI am happy it is legal, but the generalizability of this approach is limited because for a new subject, we may not be able to find a new model with similar performance.\r\n\r\nThank you very much.\r\n\r\np.s. this is not allowed in another eeg related challenge, https://www.kaggle.com/c/seizure-detection/rules",
      "votes": null
    },
    {
      "id": "89959",
      "postDate": "08/21/2015 00:08:14",
      "content": "<p>If you aren't finding good generalization across models, doesn't that limit the usefulness of any model to future real world people ? </p>\n\n<p>I have been working under the assumption that a model had to generalize across subjects...</p>",
      "rawMarkdown": "If you aren't finding good generalization across models, doesn't that limit the usefulness of any model to future real world people ? \r\n\r\nI have been working under the assumption that a model had to generalize across subjects...",
      "votes": null
    },
    {
      "id": "89998",
      "postDate": "08/21/2015 11:25:25",
      "content": "<p>It would of course be much better if we could generalize between subjects. But from our experience with EEG we don't see this happening on a level that would be usable for prosthetics control. So to make this work at all, our choice is to have a system that has to be trained separately for each patient. </p>\n\n<p>@rcason: Yes that is legal. </p>",
      "rawMarkdown": "It would of course be much better if we could generalize between subjects. But from our experience with EEG we don't see this happening on a level that would be usable for prosthetics control. So to make this work at all, our choice is to have a system that has to be trained separately for each patient. \r\n\r\n@rcason: Yes that is legal.",
      "votes": null
    },
    {
      "id": "89999",
      "postDate": "08/21/2015 12:05:58",
      "content": "<p>If you're interested in per-subject, why is the evaluation metric not mean AUC subject-wise? Calibrating the AUCs is a whole separate problem but it sounds like it's unnecessary for the real-world and only necessary for the competition? </p>",
      "rawMarkdown": "If you're interested in per-subject, why is the evaluation metric not mean AUC subject-wise? Calibrating the AUCs is a whole separate problem but it sounds like it's unnecessary for the real-world and only necessary for the competition?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 89493,
      "author_name": "qwlouse",
      "author_url": "",
      "post_date": "08/16/2015 09:51:11",
      "content": "<p>Absolutely! In fact, we would be surprised if you could generalize well over subjects.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 89507,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "08/16/2015 13:07:56",
      "content": "<p>[quote=Klaus Greff;89493]</p>\n\n<p>Absolutely! In fact, we would be surprised if you could generalize well over subjects.</p>\n\n<p>[/quote]\nThank you for reply. Sorry I probably didn't make myself clear. so far I think the common strategy is </p>\n\n<pre><code>for each subject:\n\n      train a model with the same hyper parameters\n</code></pre>\n\n<p>This is legal.</p>\n\n<p>I am asking</p>\n\n<pre><code>if subject 1:\n\n     train model 1\n\nif subject 2:\n\n     train model 2 (different hyper parameters)\n</code></pre>\n\n<p>....</p>\n\n<p>I am happy it is legal, but the generalizability of this approach is limited because for a new subject, we may not be able to find a new model with similar performance.</p>\n\n<p>Thank you very much.</p>\n\n<p>p.s. this is not allowed in another eeg related challenge, <a href=\"https://www.kaggle.com/c/seizure-detection/rules\">https://www.kaggle.com/c/seizure-detection/rules</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 89959,
      "author_name": "chrisfs",
      "author_url": "",
      "post_date": "08/21/2015 00:08:14",
      "content": "<p>If you aren't finding good generalization across models, doesn't that limit the usefulness of any model to future real world people ? </p>\n\n<p>I have been working under the assumption that a model had to generalize across subjects...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 89998,
      "author_name": "qwlouse",
      "author_url": "",
      "post_date": "08/21/2015 11:25:25",
      "content": "<p>It would of course be much better if we could generalize between subjects. But from our experience with EEG we don't see this happening on a level that would be usable for prosthetics control. So to make this work at all, our choice is to have a system that has to be trained separately for each patient. </p>\n\n<p>@rcason: Yes that is legal. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 89999,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "08/21/2015 12:05:58",
      "content": "<p>If you're interested in per-subject, why is the evaluation metric not mean AUC subject-wise? Calibrating the AUCs is a whole separate problem but it sounds like it's unnecessary for the real-world and only necessary for the competition? </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "89484": "if subject 1:\r\n    then ...\r\nelse:\r\n    then ...\r\n\r\nIs it allowed? Thank you.",
    "89493": "Absolutely! In fact, we would be surprised if you could generalize well over subjects.",
    "89507": "[quote=Klaus Greff;89493]\r\n\r\nAbsolutely! In fact, we would be surprised if you could generalize well over subjects.\r\n\r\n[/quote]\r\nThank you for reply. Sorry I probably didn't make myself clear. so far I think the common strategy is \r\n\r\n    for each subject:\r\n\r\n          train a model with the same hyper parameters\r\n\r\nThis is legal.\r\n\r\nI am asking\r\n\r\n    if subject 1:\r\n\r\n         train model 1\r\n\r\n    if subject 2:\r\n\r\n         train model 2 (different hyper parameters)\r\n\r\n....\r\n\r\nI am happy it is legal, but the generalizability of this approach is limited because for a new subject, we may not be able to find a new model with similar performance.\r\n\r\nThank you very much.\r\n\r\np.s. this is not allowed in another eeg related challenge, https://www.kaggle.com/c/seizure-detection/rules",
    "89959": "If you aren't finding good generalization across models, doesn't that limit the usefulness of any model to future real world people ? \r\n\r\nI have been working under the assumption that a model had to generalize across subjects...",
    "89998": "It would of course be much better if we could generalize between subjects. But from our experience with EEG we don't see this happening on a level that would be usable for prosthetics control. So to make this work at all, our choice is to have a system that has to be trained separately for each patient. \r\n\r\n@rcason: Yes that is legal.",
    "89999": "If you're interested in per-subject, why is the evaluation metric not mean AUC subject-wise? Calibrating the AUCs is a whole separate problem but it sounds like it's unnecessary for the real-world and only necessary for the competition?"
  },
  "source": "meta"
}