{
  "id": 302543,
  "title": "Submission - OutofMemory or ValueError: Input contains NaN, infinity or ",
  "url": "/competitions/g-research-crypto-forecasting/discussion/302543",
  "author_name": "",
  "post_date": "2022-01-23T02:58:42.455083300Z",
  "votes": 4,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi all!</p>\n<p>When training my models for every crypto asset one by one and then submit I get either:</p>\n<ul>\n<li>Out of Memory when not reducing the memory with  reduce_mem_usage() function</li>\n<li>ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..</li>\n</ul>\n<p>This is extremely frustrating and disappointing.. I see some notebooks online that have no problems with this. I wonder why it then happens with me when I'm working with the same data.. </p>\n<p>Any tips or solutions for this? <br>\nThx :)!</p>",
  "messages": [
    {
      "id": "1660850",
      "postDate": "01/23/2022 02:58:42",
      "content": "<p>Hi all!</p>\n<p>When training my models for every crypto asset one by one and then submit I get either:</p>\n<ul>\n<li>Out of Memory when not reducing the memory with  reduce_mem_usage() function</li>\n<li>ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..</li>\n</ul>\n<p>This is extremely frustrating and disappointing.. I see some notebooks online that have no problems with this. I wonder why it then happens with me when I'm working with the same data.. </p>\n<p>Any tips or solutions for this? <br>\nThx :)!</p>",
      "rawMarkdown": "Hi all!\n\nWhen training my models for every crypto asset one by one and then submit I get either:\n\n* Out of Memory when not reducing the memory with  reduce_mem_usage() function\n* ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..\n\nThis is extremely frustrating and disappointing.. I see some notebooks online that have no problems with this. I wonder why it then happens with me when I'm working with the same data.. \n\nAny tips or solutions for this? \nThx :)!",
      "votes": null
    },
    {
      "id": "1660856",
      "postDate": "01/23/2022 03:03:24",
      "content": "<blockquote>\n  <p>ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..</p>\n</blockquote>\n<p>There are some inf values in VWAP column, so you need to deal with them before modeling.</p>",
      "rawMarkdown": "> ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..\n\nThere are some inf values in VWAP column, so you need to deal with them before modeling.",
      "votes": null
    },
    {
      "id": "1660863",
      "postDate": "01/23/2022 03:17:41",
      "content": "<p>Great thx will take a look!</p>",
      "rawMarkdown": "Great thx will take a look!",
      "votes": null
    },
    {
      "id": "1661118",
      "postDate": "01/23/2022 08:33:39",
      "content": "<p>I also encountered this problem, just in the process of using XGBoost.</p>\n<p>I solved it with some preprocessing and manipulation!</p>",
      "rawMarkdown": "I also encountered this problem, just in the process of using XGBoost.\n\nI solved it with some preprocessing and manipulation!",
      "votes": null
    },
    {
      "id": "1661691",
      "postDate": "01/23/2022 16:08:15",
      "content": "<p><a href=\"https://www.kaggle.com/xinyangkabuda\" target=\"_blank\">@xinyangkabuda</a> could you maybe explain a bit what did you manipulate? Only the VWAP? Thank you! </p>",
      "rawMarkdown": "xinyangkabuda could you maybe explain a bit what did you manipulate? Only the VWAP? Thank you!",
      "votes": null
    },
    {
      "id": "1662285",
      "postDate": "01/24/2022 06:56:37",
      "content": "<p>I mainly refer to <a href=\"https://www.kaggle.com/tarlannazarov/g-research-crypto-starter-xgb-pipeline\" target=\"_blank\">this notebook</a>, which solves a problem similar to yours in XGBoost</p>\n<p>I'm so sorry, I don't know if this will help you! 😂</p>",
      "rawMarkdown": "I mainly refer to [this notebook](https://www.kaggle.com/tarlannazarov/g-research-crypto-starter-xgb-pipeline), which solves a problem similar to yours in XGBoost\n\nI'm so sorry, I don't know if this will help you! 😂",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1660856,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "01/23/2022 03:03:24",
      "content": "<blockquote>\n  <p>ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..</p>\n</blockquote>\n<p>There are some inf values in VWAP column, so you need to deal with them before modeling.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1660863,
      "author_name": "vweimer",
      "author_url": "",
      "post_date": "01/23/2022 03:17:41",
      "content": "<p>Great thx will take a look!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1661118,
      "author_name": "",
      "author_url": "",
      "post_date": "01/23/2022 08:33:39",
      "content": "<p>I also encountered this problem, just in the process of using XGBoost.</p>\n<p>I solved it with some preprocessing and manipulation!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1661691,
          "author_name": "vweimer",
          "author_url": "",
          "post_date": "01/23/2022 16:08:15",
          "content": "<p><a href=\"https://www.kaggle.com/xinyangkabuda\" target=\"_blank\">@xinyangkabuda</a> could you maybe explain a bit what did you manipulate? Only the VWAP? Thank you! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1662285,
          "author_name": "",
          "author_url": "",
          "post_date": "01/24/2022 06:56:37",
          "content": "<p>I mainly refer to <a href=\"https://www.kaggle.com/tarlannazarov/g-research-crypto-starter-xgb-pipeline\" target=\"_blank\">this notebook</a>, which solves a problem similar to yours in XGBoost</p>\n<p>I'm so sorry, I don't know if this will help you! 😂</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1660850": "Hi all!\n\nWhen training my models for every crypto asset one by one and then submit I get either:\n\n* Out of Memory when not reducing the memory with  reduce_mem_usage() function\n* ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..\n\nThis is extremely frustrating and disappointing.. I see some notebooks online that have no problems with this. I wonder why it then happens with me when I'm working with the same data.. \n\nAny tips or solutions for this? \nThx :)!",
    "1660856": "> ValueError: Input contains NaN, infinity or a value too large for dtype('float64) when I DO use the reduce_mem_usage() function, which also results into failure..\n\nThere are some inf values in VWAP column, so you need to deal with them before modeling.",
    "1660863": "Great thx will take a look!",
    "1661118": "I also encountered this problem, just in the process of using XGBoost.\n\nI solved it with some preprocessing and manipulation!",
    "1661691": "xinyangkabuda could you maybe explain a bit what did you manipulate? Only the VWAP? Thank you!",
    "1662285": "I mainly refer to [this notebook](https://www.kaggle.com/tarlannazarov/g-research-crypto-starter-xgb-pipeline), which solves a problem similar to yours in XGBoost\n\nI'm so sorry, I don't know if this will help you! 😂"
  },
  "source": "meta"
}