{
  "id": 459754,
  "title": "Simple ideas",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/459754",
  "author_name": "DongYK",
  "post_date": "2023-12-06T14:04:10.609000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I didn't get enough score, but I'd like to share a couple of very simple but effective things.</p>\n<p><strong>1. Patch Embedding</strong></p>\n<p>I hadn't used any transformer models. So, I wanted to use it this competition. However, the problem was huge computational costs. I reduced the sequences with 720 * 8 length by 12 of the patch size (batch_size, 720*8 // 12, 2) and can use simply four squeezeformerblocks with the same sequence length and d_model. This simple embedding shows the performance gains not harming the performance.</p>\n<p>I didn't any preprocessing except for standardization. I only used these two features (anglez, enmo).</p>\n<pre><code> :\n    def :\n        super.\n        self.projection = nn.\n        self.ln = nn.\n\n    def forward(self, x: Tensor) -&gt; Tensor:\n        x = x.transpose(, )\n        x = self.projection(x)\n        x = x.transpose(, )\n        x = self.ln(x)\n\n        return x\n</code></pre>\n<p><strong>2. Simple post processing</strong></p>\n<p>I used 4 chunks. The steps of the first chunk are [(0, 8000), (8000, 16000), (16000, 24000), …] and the steps of the second are [(2000, 10000), (10000, 18000), (18000, 26000)…]. I averaged over these four chunks.<br>\nThe inputs are reduced by 12 of the factor. The outputs are filled in the 5th and 6th position in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11].</p>\n<pre><code>               (, seq_len, seq_len \n\n                te_dataset = (series_id, space)\n                len_series = (te_dataset.this_series)\n                preds = torch.(((len_series - space) + (seq_len - (len_series - space) % seq_len) % seq_len, ), torch.nan).(-, seq_len, )\n\n                for batch_idx, i in ((, (te_dataset), self.batch_size)):\n                    x_batch = te_dataset[:i+self.batch_size]\n\n                    with ():\n                        pred = (x_batch)[:, :, :]\n                    preds[:i+self.batch_size, patch_size\n                    preds[:i+self.batch_size, patch_size\n\n                preds = torch.(\n                    [torch.((space, ), np.nan), preds.(-, )[:len_series - space]]\n                )\n                predss.(preds.())\n\n            predss = np.(predss, axis=)\n            predss[np.(predss)] = \n            predsss.(predss)\n</code></pre>\n<p><code>find_peaks</code> is from the tubo's notebook and I added two lines.<br>\nThe parameters are window=11, threshold=0.01, distance=720*10.</p>\n<pre><code>   ((predsss)):\n        pred = predsss()\n        pred = np(pred, v=*window, mode=)\n        pred = np(pred, v=*window, mode=)\n        wakeup_peaks = (pred, =threshold, distance=distance)\n        onset_peaks = (pred, =threshold, distance=distance)\n</code></pre>",
  "messages": [
    {
      "id": 2551106,
      "postDate": "2023-12-06T14:04:10.610Z",
      "content": "<p>I didn't get enough score, but I'd like to share a couple of very simple but effective things.</p>\n<p><strong>1. Patch Embedding</strong></p>\n<p>I hadn't used any transformer models. So, I wanted to use it this competition. However, the problem was huge computational costs. I reduced the sequences with 720 * 8 length by 12 of the patch size (batch_size, 720*8 // 12, 2) and can use simply four squeezeformerblocks with the same sequence length and d_model. This simple embedding shows the performance gains not harming the performance.</p>\n<p>I didn't any preprocessing except for standardization. I only used these two features (anglez, enmo).</p>\n<pre><code> :\n    def :\n        super.\n        self.projection = nn.\n        self.ln = nn.\n\n    def forward(self, x: Tensor) -&gt; Tensor:\n        x = x.transpose(, )\n        x = self.projection(x)\n        x = x.transpose(, )\n        x = self.ln(x)\n\n        return x\n</code></pre>\n<p><strong>2. Simple post processing</strong></p>\n<p>I used 4 chunks. The steps of the first chunk are [(0, 8000), (8000, 16000), (16000, 24000), …] and the steps of the second are [(2000, 10000), (10000, 18000), (18000, 26000)…]. I averaged over these four chunks.<br>\nThe inputs are reduced by 12 of the factor. The outputs are filled in the 5th and 6th position in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11].</p>\n<pre><code>               (, seq_len, seq_len \n\n                te_dataset = (series_id, space)\n                len_series = (te_dataset.this_series)\n                preds = torch.(((len_series - space) + (seq_len - (len_series - space) % seq_len) % seq_len, ), torch.nan).(-, seq_len, )\n\n                for batch_idx, i in ((, (te_dataset), self.batch_size)):\n                    x_batch = te_dataset[:i+self.batch_size]\n\n                    with ():\n                        pred = (x_batch)[:, :, :]\n                    preds[:i+self.batch_size, patch_size\n                    preds[:i+self.batch_size, patch_size\n\n                preds = torch.(\n                    [torch.((space, ), np.nan), preds.(-, )[:len_series - space]]\n                )\n                predss.(preds.())\n\n            predss = np.(predss, axis=)\n            predss[np.(predss)] = \n            predsss.(predss)\n</code></pre>\n<p><code>find_peaks</code> is from the tubo's notebook and I added two lines.<br>\nThe parameters are window=11, threshold=0.01, distance=720*10.</p>\n<pre><code>   ((predsss)):\n        pred = predsss()\n        pred = np(pred, v=*window, mode=)\n        pred = np(pred, v=*window, mode=)\n        wakeup_peaks = (pred, =threshold, distance=distance)\n        onset_peaks = (pred, =threshold, distance=distance)\n</code></pre>",
      "rawMarkdown": "I didn't get enough score, but I'd like to share a couple of very simple but effective things.\n\n**1. Patch Embedding**\n\nI hadn't used any transformer models. So, I wanted to use it this competition. However, the problem was huge computational costs. I reduced the sequences with 720 * 8 length by 12 of the patch size (batch_size, 720*8 // 12, 2) and can use simply four squeezeformerblocks with the same sequence length and d_model. This simple embedding shows the performance gains not harming the performance.\n\nI didn't any preprocessing except for standardization. I only used these two features (anglez, enmo).\n\n```\nclass PatchEmbedding(nn.Module):\n    def __init__(self, patch_size: int = 12, n_features: int = 2, d_model: int = 128):\n        super().__init__()\n        self.projection = nn.Conv1d(n_features, d_model, kernel_size=patch_size, stride=patch_size)\n        self.ln = nn.LayerNorm(d_model)\n                \n    def forward(self, x: Tensor) -> Tensor:\n        x = x.transpose(1, 2)\n        x = self.projection(x)\n        x = x.transpose(1, 2)\n        x = self.ln(x)\n        \n        return x\n```\n\n**2. Simple post processing**\n\nI used 4 chunks. The steps of the first chunk are [(0, 8000), (8000, 16000), (16000, 24000), ...] and the steps of the second are [(2000, 10000), (10000, 18000), (18000, 26000)...]. I averaged over these four chunks.\nThe inputs are reduced by 12 of the factor. The outputs are filled in the 5th and 6th position in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11].\n```\n            for space in range(0, seq_len, seq_len // n_chunks):\n\n                te_dataset = TestDataGenerator(series_id, space)\n                len_series = len(te_dataset.this_series)\n                preds = torch.full(((len_series - space) + (seq_len - (len_series - space) % seq_len) % seq_len, 2), torch.nan).reshape(-1, seq_len, 2)\n\n                for batch_idx, i in enumerate(range(0, len(te_dataset), self.batch_size)):\n                    x_batch = te_dataset[i:i+self.batch_size]\n\n                    with autocast():\n                        pred = model(x_batch)[:, :, 1:]\n                    preds[i:i+self.batch_size, patch_size//2-1::patch_size, :] = pred\n                    preds[i:i+self.batch_size, patch_size//2::patch_size, :] = pred\n\n                preds = torch.concatenate(\n                    [torch.full((space, 2), np.nan), preds.reshape(-1, 2)[:len_series - space]]\n                )\n                predss.append(preds.numpy())\n\n            predss = np.nanmean(predss, axis=0)\n            predss[np.isnan(predss)] = 0\n            predsss.append(predss)\n```\n`find_peaks` is from the tubo's notebook and I added two lines.\nThe parameters are window=11, threshold=0.01, distance=720*10.\n```\nfor i in range(len(predsss)):\n        pred = predsss[i].copy()\n        pred[:, 0] = np.convolve(pred[:, 0], v=[1]*window, mode='same')\n        pred[:, 1] = np.convolve(pred[:, 1], v=[1]*window, mode='same')\n        wakeup_peaks = find_peaks(pred[:, 0], height=threshold, distance=distance)\n        onset_peaks = find_peaks(pred[:, 1], height=threshold, distance=distance)\n```"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2551106": "I didn't get enough score, but I'd like to share a couple of very simple but effective things.\n\n**1. Patch Embedding**\n\nI hadn't used any transformer models. So, I wanted to use it this competition. However, the problem was huge computational costs. I reduced the sequences with 720 * 8 length by 12 of the patch size (batch_size, 720*8 // 12, 2) and can use simply four squeezeformerblocks with the same sequence length and d_model. This simple embedding shows the performance gains not harming the performance.\n\nI didn't any preprocessing except for standardization. I only used these two features (anglez, enmo).\n\n```\nclass PatchEmbedding(nn.Module):\n    def __init__(self, patch_size: int = 12, n_features: int = 2, d_model: int = 128):\n        super().__init__()\n        self.projection = nn.Conv1d(n_features, d_model, kernel_size=patch_size, stride=patch_size)\n        self.ln = nn.LayerNorm(d_model)\n                \n    def forward(self, x: Tensor) -> Tensor:\n        x = x.transpose(1, 2)\n        x = self.projection(x)\n        x = x.transpose(1, 2)\n        x = self.ln(x)\n        \n        return x\n```\n\n**2. Simple post processing**\n\nI used 4 chunks. The steps of the first chunk are [(0, 8000), (8000, 16000), (16000, 24000), ...] and the steps of the second are [(2000, 10000), (10000, 18000), (18000, 26000)...]. I averaged over these four chunks.\nThe inputs are reduced by 12 of the factor. The outputs are filled in the 5th and 6th position in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11].\n```\n            for space in range(0, seq_len, seq_len // n_chunks):\n\n                te_dataset = TestDataGenerator(series_id, space)\n                len_series = len(te_dataset.this_series)\n                preds = torch.full(((len_series - space) + (seq_len - (len_series - space) % seq_len) % seq_len, 2), torch.nan).reshape(-1, seq_len, 2)\n\n                for batch_idx, i in enumerate(range(0, len(te_dataset), self.batch_size)):\n                    x_batch = te_dataset[i:i+self.batch_size]\n\n                    with autocast():\n                        pred = model(x_batch)[:, :, 1:]\n                    preds[i:i+self.batch_size, patch_size//2-1::patch_size, :] = pred\n                    preds[i:i+self.batch_size, patch_size//2::patch_size, :] = pred\n\n                preds = torch.concatenate(\n                    [torch.full((space, 2), np.nan), preds.reshape(-1, 2)[:len_series - space]]\n                )\n                predss.append(preds.numpy())\n\n            predss = np.nanmean(predss, axis=0)\n            predss[np.isnan(predss)] = 0\n            predsss.append(predss)\n```\n`find_peaks` is from the tubo's notebook and I added two lines.\nThe parameters are window=11, threshold=0.01, distance=720*10.\n```\nfor i in range(len(predsss)):\n        pred = predsss[i].copy()\n        pred[:, 0] = np.convolve(pred[:, 0], v=[1]*window, mode='same')\n        pred[:, 1] = np.convolve(pred[:, 1], v=[1]*window, mode='same')\n        wakeup_peaks = find_peaks(pred[:, 0], height=threshold, distance=distance)\n        onset_peaks = find_peaks(pred[:, 1], height=threshold, distance=distance)\n```"
  }
}