{
  "id": 15416,
  "title": "Understanding and preprocessing data",
  "url": "/competitions/grasp-and-lift-eeg-detection/discussion/15416",
  "author_name": "Alexander Rosenberg Johansen",
  "post_date": "2015-07-21T13:55:46.680000",
  "votes": 0,
  "comment_count": 0,
  "views": 409,
  "content": "<p>So far I've understood that this is a time-series challenge, thus shape of the data should look as:</p>\n\n<p><strong>Data observed</strong></p>\n\n<p>data.shape = [n_samples, sequence_length, d_features]</p>\n\n<p>Loading one of the 12*8 training files, I get the shape:</p>\n\n<p>[~200000 , 33]</p>\n\n<p>I assume the ~200000 is the sequence_length and the 33 is the d_features(first being the 'id').</p>\n\n<p>Reason for ~ is that sequence_length varies between the subjects and series.</p>\n\n<p>This then results in a shape of:\n[12*8, ~200000, 33] if the data files are concatenated.</p>\n\n<p><strong>Discussion</strong></p>\n\n<p>Is it possible to reduce the ~200,000 sequence by grouping the 16 different trails? Such a long sequence_length seems extreme.</p>\n\n<p>I have noticed that only 4200*6=25,200 positive labels exists in each of the dataset responding to only about 10%. Should we mask some of the 90% sequence with zero labels? or is it a part of the assignment to understand when a task is begun?</p>",
  "messages": [
    {
      "id": 86300,
      "postDate": "2015-07-21T13:55:46.680Z",
      "content": "<p>So far I've understood that this is a time-series challenge, thus shape of the data should look as:</p>\n\n<p><strong>Data observed</strong></p>\n\n<p>data.shape = [n_samples, sequence_length, d_features]</p>\n\n<p>Loading one of the 12*8 training files, I get the shape:</p>\n\n<p>[~200000 , 33]</p>\n\n<p>I assume the ~200000 is the sequence_length and the 33 is the d_features(first being the 'id').</p>\n\n<p>Reason for ~ is that sequence_length varies between the subjects and series.</p>\n\n<p>This then results in a shape of:\n[12*8, ~200000, 33] if the data files are concatenated.</p>\n\n<p><strong>Discussion</strong></p>\n\n<p>Is it possible to reduce the ~200,000 sequence by grouping the 16 different trails? Such a long sequence_length seems extreme.</p>\n\n<p>I have noticed that only 4200*6=25,200 positive labels exists in each of the dataset responding to only about 10%. Should we mask some of the 90% sequence with zero labels? or is it a part of the assignment to understand when a task is begun?</p>",
      "rawMarkdown": "So far I've understood that this is a time-series challenge, thus shape of the data should look as:\r\n\r\n**Data observed**\r\n\r\ndata.shape = [n_samples, sequence_length, d_features]\r\n\r\nLoading one of the 12*8 training files, I get the shape:\r\n\r\n[~200000 , 33]\r\n\r\nI assume the ~200000 is the sequence_length and the 33 is the d_features(first being the 'id').\r\n\r\nReason for ~ is that sequence_length varies between the subjects and series.\r\n\r\nThis then results in a shape of:\r\n[12*8, ~200000, 33] if the data files are concatenated.\r\n\r\n**Discussion**\r\n\r\nIs it possible to reduce the ~200,000 sequence by grouping the 16 different trails? Such a long sequence_length seems extreme.\r\n\r\nI have noticed that only 4200*6=25,200 positive labels exists in each of the dataset responding to only about 10%. Should we mask some of the 90% sequence with zero labels? or is it a part of the assignment to understand when a task is begun?"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "86300": "So far I've understood that this is a time-series challenge, thus shape of the data should look as:\r\n\r\n**Data observed**\r\n\r\ndata.shape = [n_samples, sequence_length, d_features]\r\n\r\nLoading one of the 12*8 training files, I get the shape:\r\n\r\n[~200000 , 33]\r\n\r\nI assume the ~200000 is the sequence_length and the 33 is the d_features(first being the 'id').\r\n\r\nReason for ~ is that sequence_length varies between the subjects and series.\r\n\r\nThis then results in a shape of:\r\n[12*8, ~200000, 33] if the data files are concatenated.\r\n\r\n**Discussion**\r\n\r\nIs it possible to reduce the ~200,000 sequence by grouping the 16 different trails? Such a long sequence_length seems extreme.\r\n\r\nI have noticed that only 4200*6=25,200 positive labels exists in each of the dataset responding to only about 10%. Should we mask some of the 90% sequence with zero labels? or is it a part of the assignment to understand when a task is begun?"
  }
}