{
  "id": 43405,
  "title": "How large memory is enough?",
  "url": "/competitions/favorita-grocery-sales-forecasting/discussion/43405",
  "author_name": "",
  "post_date": "2017-11-14T08:51:45.695792600Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi Kagglers,</p>\n\n<p>I might not be doing this in a right way, but the size of the data gave me a big headache. I went on AWS with 64gb RAM and used data after 2016-08 only. I still get memory error when trying to fit a Random Forest model with 10 trees. </p>\n\n<p>Since I haven't try xgboost yest, would xgboost require less memory usage? What are your solutions?</p>\n\n<p>I googled a little bit, it came to the solutions:</p>\n\n<ol>\n<li>use online learning method like vowpal wabbit</li>\n<li>external memory version of xgboost</li>\n</ol>\n\n<p>There must be other ways. How do you guys tackle data with this size?</p>\n\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "243529",
      "postDate": "11/14/2017 08:51:45",
      "content": "<p>Hi Kagglers,</p>\n\n<p>I might not be doing this in a right way, but the size of the data gave me a big headache. I went on AWS with 64gb RAM and used data after 2016-08 only. I still get memory error when trying to fit a Random Forest model with 10 trees. </p>\n\n<p>Since I haven't try xgboost yest, would xgboost require less memory usage? What are your solutions?</p>\n\n<p>I googled a little bit, it came to the solutions:</p>\n\n<ol>\n<li>use online learning method like vowpal wabbit</li>\n<li>external memory version of xgboost</li>\n</ol>\n\n<p>There must be other ways. How do you guys tackle data with this size?</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "Hi Kagglers,\n\n I might not be doing this in a right way, but the size of the data gave me a big headache. I went on AWS with 64gb RAM and used data after 2016-08 only. I still get memory error when trying to fit a Random Forest model with 10 trees. \n\nSince I haven't try xgboost yest, would xgboost require less memory usage? What are your solutions?\n\nI googled a little bit, it came to the solutions:\n\n 1. use online learning method like vowpal wabbit\n 2. external memory version of xgboost\n\nThere must be other ways. How do you guys tackle data with this size?\n\nThanks!",
      "votes": null
    },
    {
      "id": "243780",
      "postDate": "11/14/2017 21:44:50",
      "content": "<p>This is rather strange - can you provide more details, starting with language you used (R/Python/sth else maybe) and preprocessing you used? I did local preprocessing in R (similar data range to yours) and fit a lightgbm with no problems on my 16gb laptop.</p>",
      "rawMarkdown": "This is rather strange - can you provide more details, starting with language you used (R/Python/sth else maybe) and preprocessing you used? I did local preprocessing in R (similar data range to yours) and fit a lightgbm with no problems on my 16gb laptop.",
      "votes": null
    },
    {
      "id": "255588",
      "postDate": "12/09/2017 13:21:05",
      "content": "<p>I do not think we need much RAM. 16 or 32GB is sufficient, however it requires significant processing power. </p>",
      "rawMarkdown": "I do not think we need much RAM. 16 or 32GB is sufficient, however it requires significant processing power.",
      "votes": null
    },
    {
      "id": "256885",
      "postDate": "12/12/2017 22:01:04",
      "content": "<p>Have you tried confirming what the KPI and features should be at least for the un-aggregated data using AWS server?  I am not sure how much I can help because my group is new to kaggle.  I was recently going to try AWS myself because little 8 GB computer cannot handle the unaggreggated KPI and feature analysis.</p>\n\n<p>^These are all things my group has tried, but we are stuck:</p>\n\n<ol>\n<li>Have you tried submitting that data without AWS but scaling for just\nthe first year to just test if your output it working? </li>\n<li>Have you tested the same submission, but separately aggregating by\nweek?</li>\n<li>(*)Have you separated each of the items into cluster and family/class\nand then put them in new .csv or .numpy files.  Then you would write a giant for loop made\npredictions just for them with data.  </li>\n</ol>\n\n<p>The fundamental problem with the data size is that it is forcing us to make assumptions about TIME itself.  Time is a KPI.  For when the data is unaggregated evening knowing how things correlate, analysis of variance, the accuracy of the clusters (by comparing to actual lazy algorithm like K-means). <br>\nFor example a lot of people either last week or the week before were assuming objectively that 1 year or 1.5 years was a good cut-off.  However, I don't like making assumptions I cannot confirm.  The same thing goes with aggregating by week, day, month or holiday.</p>\n\n<p>Basically we don't have a true baseline measure of KPIs (though the top 2-rated kernels do provide very good plots that hint at those KPIs).  I really wish there was an introduction to Kaggle thread that just talked about dealing with large data sets in Kaggle and costs.  Does anyone know of such a thread?</p>\n\n<p>This problem along with a few others is why I think moving average forecasting is currently performing best, which is odd considering the data set we have.</p>",
      "rawMarkdown": "Have you tried confirming what the KPI and features should be at least for the un-aggregated data using AWS server?  I am not sure how much I can help because my group is new to kaggle.  I was recently going to try AWS myself because little 8 GB computer cannot handle the unaggreggated KPI and feature analysis.\n\n^These are all things my group has tried, but we are stuck:\n\n 1. Have you tried submitting that data without AWS but scaling for just\n   the first year to just test if your output it working? \n 2. Have you tested the same submission, but separately aggregating by\n    week?\n 3. (*)Have you separated each of the items into cluster and family/class\n   and then put them in new .csv or .numpy files.  Then you would write a giant for loop made\n   predictions just for them with data.  \n\nThe fundamental problem with the data size is that it is forcing us to make assumptions about TIME itself.  Time is a KPI.  For when the data is unaggregated evening knowing how things correlate, analysis of variance, the accuracy of the clusters (by comparing to actual lazy algorithm like K-means).   \nFor example a lot of people either last week or the week before were assuming objectively that 1 year or 1.5 years was a good cut-off.  However, I don't like making assumptions I cannot confirm.  The same thing goes with aggregating by week, day, month or holiday.\n\nBasically we don't have a true baseline measure of KPIs (though the top 2-rated kernels do provide very good plots that hint at those KPIs).  I really wish there was an introduction to Kaggle thread that just talked about dealing with large data sets in Kaggle and costs.  Does anyone know of such a thread?\n\nThis problem along with a few others is why I think moving average forecasting is currently performing best, which is odd considering the data set we have.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 243780,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "11/14/2017 21:44:50",
      "content": "<p>This is rather strange - can you provide more details, starting with language you used (R/Python/sth else maybe) and preprocessing you used? I did local preprocessing in R (similar data range to yours) and fit a lightgbm with no problems on my 16gb laptop.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 255588,
      "author_name": "gowrishankarin",
      "author_url": "",
      "post_date": "12/09/2017 13:21:05",
      "content": "<p>I do not think we need much RAM. 16 or 32GB is sufficient, however it requires significant processing power. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 256885,
      "author_name": "skyrmion",
      "author_url": "",
      "post_date": "12/12/2017 22:01:04",
      "content": "<p>Have you tried confirming what the KPI and features should be at least for the un-aggregated data using AWS server?  I am not sure how much I can help because my group is new to kaggle.  I was recently going to try AWS myself because little 8 GB computer cannot handle the unaggreggated KPI and feature analysis.</p>\n\n<p>^These are all things my group has tried, but we are stuck:</p>\n\n<ol>\n<li>Have you tried submitting that data without AWS but scaling for just\nthe first year to just test if your output it working? </li>\n<li>Have you tested the same submission, but separately aggregating by\nweek?</li>\n<li>(*)Have you separated each of the items into cluster and family/class\nand then put them in new .csv or .numpy files.  Then you would write a giant for loop made\npredictions just for them with data.  </li>\n</ol>\n\n<p>The fundamental problem with the data size is that it is forcing us to make assumptions about TIME itself.  Time is a KPI.  For when the data is unaggregated evening knowing how things correlate, analysis of variance, the accuracy of the clusters (by comparing to actual lazy algorithm like K-means). <br>\nFor example a lot of people either last week or the week before were assuming objectively that 1 year or 1.5 years was a good cut-off.  However, I don't like making assumptions I cannot confirm.  The same thing goes with aggregating by week, day, month or holiday.</p>\n\n<p>Basically we don't have a true baseline measure of KPIs (though the top 2-rated kernels do provide very good plots that hint at those KPIs).  I really wish there was an introduction to Kaggle thread that just talked about dealing with large data sets in Kaggle and costs.  Does anyone know of such a thread?</p>\n\n<p>This problem along with a few others is why I think moving average forecasting is currently performing best, which is odd considering the data set we have.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "243529": "Hi Kagglers,\n\n I might not be doing this in a right way, but the size of the data gave me a big headache. I went on AWS with 64gb RAM and used data after 2016-08 only. I still get memory error when trying to fit a Random Forest model with 10 trees. \n\nSince I haven't try xgboost yest, would xgboost require less memory usage? What are your solutions?\n\nI googled a little bit, it came to the solutions:\n\n 1. use online learning method like vowpal wabbit\n 2. external memory version of xgboost\n\nThere must be other ways. How do you guys tackle data with this size?\n\nThanks!",
    "243780": "This is rather strange - can you provide more details, starting with language you used (R/Python/sth else maybe) and preprocessing you used? I did local preprocessing in R (similar data range to yours) and fit a lightgbm with no problems on my 16gb laptop.",
    "255588": "I do not think we need much RAM. 16 or 32GB is sufficient, however it requires significant processing power.",
    "256885": "Have you tried confirming what the KPI and features should be at least for the un-aggregated data using AWS server?  I am not sure how much I can help because my group is new to kaggle.  I was recently going to try AWS myself because little 8 GB computer cannot handle the unaggreggated KPI and feature analysis.\n\n^These are all things my group has tried, but we are stuck:\n\n 1. Have you tried submitting that data without AWS but scaling for just\n   the first year to just test if your output it working? \n 2. Have you tested the same submission, but separately aggregating by\n    week?\n 3. (*)Have you separated each of the items into cluster and family/class\n   and then put them in new .csv or .numpy files.  Then you would write a giant for loop made\n   predictions just for them with data.  \n\nThe fundamental problem with the data size is that it is forcing us to make assumptions about TIME itself.  Time is a KPI.  For when the data is unaggregated evening knowing how things correlate, analysis of variance, the accuracy of the clusters (by comparing to actual lazy algorithm like K-means).   \nFor example a lot of people either last week or the week before were assuming objectively that 1 year or 1.5 years was a good cut-off.  However, I don't like making assumptions I cannot confirm.  The same thing goes with aggregating by week, day, month or holiday.\n\nBasically we don't have a true baseline measure of KPIs (though the top 2-rated kernels do provide very good plots that hint at those KPIs).  I really wish there was an introduction to Kaggle thread that just talked about dealing with large data sets in Kaggle and costs.  Does anyone know of such a thread?\n\nThis problem along with a few others is why I think moving average forecasting is currently performing best, which is odd considering the data set we have."
  },
  "source": "meta"
}