{
  "id": 285176,
  "title": "Brute-Force Multi-Task RNN?",
  "url": "/competitions/g-research-crypto-forecasting/discussion/285176",
  "author_name": "",
  "post_date": "2021-11-03T16:39:20.855320500Z",
  "votes": 14,
  "comment_count": 5,
  "views": 0,
  "content": "<p>We have 14 assets in total and 7 basic features (i.e, close, open, etc.). A 'brute-force' and quick idea is to first pivot the data in each timestamp to make 14 * 7 = 98 features and 14 targets, then apply a sliding-window training with multi-task RNN/Transformer/TCN… Looking forward to seeing such methods.</p>",
  "messages": [
    {
      "id": "1569675",
      "postDate": "11/03/2021 16:39:20",
      "content": "<p>We have 14 assets in total and 7 basic features (i.e, close, open, etc.). A 'brute-force' and quick idea is to first pivot the data in each timestamp to make 14 * 7 = 98 features and 14 targets, then apply a sliding-window training with multi-task RNN/Transformer/TCN… Looking forward to seeing such methods.</p>",
      "rawMarkdown": "We have 14 assets in total and 7 basic features (i.e, close, open, etc.). A 'brute-force' and quick idea is to first pivot the data in each timestamp to make 14 * 7 = 98 features and 14 targets, then apply a sliding-window training with multi-task RNN/Transformer/TCN... Looking forward to seeing such methods.",
      "votes": null
    },
    {
      "id": "1570045",
      "postDate": "11/03/2021 23:35:33",
      "content": "<p>Might be helpfull to avoid too much redundant features. Main problem I see is some asset missing a lot of timestamps. See:</p>\n<pre><code>train_df['timestamp'].hist(by=train_df['Asset_ID']);\n</code></pre>",
      "rawMarkdown": "Might be helpfull to avoid too much redundant features. Main problem I see is some asset missing a lot of timestamps. See:\n\n```\ntrain_df['timestamp'].hist(by=train_df['Asset_ID']);\n```",
      "votes": null
    },
    {
      "id": "1570675",
      "postDate": "11/04/2021 09:45:27",
      "content": "<p>We can just ffill them</p>",
      "rawMarkdown": "We can just ffill them",
      "votes": null
    },
    {
      "id": "1570916",
      "postDate": "11/04/2021 13:34:17",
      "content": "<p>The missing timestamps problem is taken care of in the Tutorial notebook with ffill as <a href=\"https://www.kaggle.com/yamqwe\" target=\"_blank\">@yamqwe</a> suggested.</p>",
      "rawMarkdown": "The missing timestamps problem is taken care of in the Tutorial notebook with ffill as @yamqwe suggested.",
      "votes": null
    },
    {
      "id": "1570920",
      "postDate": "11/04/2021 13:35:58",
      "content": "<p>I am exploring this possibility, the biggest problem I am facing is RAM. The training tensor ensuing from this is simply too big for my machine and even some online instances. If you find a workaround or obtain any results with this, I'd be glad to hear about it!</p>",
      "rawMarkdown": "I am exploring this possibility, the biggest problem I am facing is RAM. The training tensor ensuing from this is simply too big for my machine and even some online instances. If you find a workaround or obtain any results with this, I'd be glad to hear about it!",
      "votes": null
    },
    {
      "id": "1570936",
      "postDate": "11/04/2021 13:47:52",
      "content": "<p>I am working on Google Colab Pro+. Its 50GB RAM is enough for most of the cases.</p>",
      "rawMarkdown": "I am working on Google Colab Pro+. Its 50GB RAM is enough for most of the cases.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1570045,
      "author_name": "lucasmorin",
      "author_url": "",
      "post_date": "11/03/2021 23:35:33",
      "content": "<p>Might be helpfull to avoid too much redundant features. Main problem I see is some asset missing a lot of timestamps. See:</p>\n<pre><code>train_df['timestamp'].hist(by=train_df['Asset_ID']);\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 1570675,
          "author_name": "",
          "author_url": "",
          "post_date": "11/04/2021 09:45:27",
          "content": "<p>We can just ffill them</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1570916,
          "author_name": "propanon",
          "author_url": "",
          "post_date": "11/04/2021 13:34:17",
          "content": "<p>The missing timestamps problem is taken care of in the Tutorial notebook with ffill as <a href=\"https://www.kaggle.com/yamqwe\" target=\"_blank\">@yamqwe</a> suggested.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1570920,
      "author_name": "propanon",
      "author_url": "",
      "post_date": "11/04/2021 13:35:58",
      "content": "<p>I am exploring this possibility, the biggest problem I am facing is RAM. The training tensor ensuing from this is simply too big for my machine and even some online instances. If you find a workaround or obtain any results with this, I'd be glad to hear about it!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1570936,
          "author_name": "gogo827jz",
          "author_url": "",
          "post_date": "11/04/2021 13:47:52",
          "content": "<p>I am working on Google Colab Pro+. Its 50GB RAM is enough for most of the cases.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1569675": "We have 14 assets in total and 7 basic features (i.e, close, open, etc.). A 'brute-force' and quick idea is to first pivot the data in each timestamp to make 14 * 7 = 98 features and 14 targets, then apply a sliding-window training with multi-task RNN/Transformer/TCN... Looking forward to seeing such methods.",
    "1570045": "Might be helpfull to avoid too much redundant features. Main problem I see is some asset missing a lot of timestamps. See:\n\n```\ntrain_df['timestamp'].hist(by=train_df['Asset_ID']);\n```",
    "1570675": "We can just ffill them",
    "1570916": "The missing timestamps problem is taken care of in the Tutorial notebook with ffill as @yamqwe suggested.",
    "1570920": "I am exploring this possibility, the biggest problem I am facing is RAM. The training tensor ensuing from this is simply too big for my machine and even some online instances. If you find a workaround or obtain any results with this, I'd be glad to hear about it!",
    "1570936": "I am working on Google Colab Pro+. Its 50GB RAM is enough for most of the cases."
  },
  "source": "meta"
}