{
  "id": 75713,
  "title": "My computer cannot handle such big data ",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/75713",
  "author_name": "",
  "post_date": "2018-12-25T09:34:04.347167100Z",
  "votes": 7,
  "comment_count": 20,
  "views": 0,
  "content": "<p>How can I handle such big data in my computer? </p>",
  "messages": [
    {
      "id": "444990",
      "postDate": "12/25/2018 09:34:04",
      "content": "<p>How can I handle such big data in my computer? </p>",
      "rawMarkdown": "How can I handle such big data in my computer?",
      "votes": null
    },
    {
      "id": "445000",
      "postDate": "12/25/2018 10:24:01",
      "content": "<p><code>reduce_mem_usage</code> function is so effective for dealing with such large size data, which is found on many public kernels like <a href=\"https://www.kaggle.com/fabiendaniel/elo-world\">this</a> (the cell containing the function is hiden).</p>",
      "rawMarkdown": "`reduce_mem_usage` function is so effective for dealing with such large size data, which is found on many public kernels like [this](https://www.kaggle.com/fabiendaniel/elo-world) (the cell containing the function is hiden).",
      "votes": null
    },
    {
      "id": "445056",
      "postDate": "12/25/2018 14:05:26",
      "content": "<p>if aggregation on this dataset is proving to a problem for your hardware, I would suggest to  have a look at Dask</p>",
      "rawMarkdown": "if aggregation on this dataset is proving to a problem for your hardware, I would suggest to  have a look at Dask",
      "votes": null
    },
    {
      "id": "445098",
      "postDate": "12/25/2018 16:21:01",
      "content": "<p>@JinMa Have a look <a href=\"https://www.kaggle.com/questions-and-answers/75495#post443758\">https://www.kaggle.com/questions-and-answers/75495#post443758</a></p>",
      "rawMarkdown": "JinMa Have a look https://www.kaggle.com/questions-and-answers/75495#post443758",
      "votes": null
    },
    {
      "id": "445099",
      "postDate": "12/25/2018 16:21:32",
      "content": "<p>Hi! Jin Ma, like Tak mentioned the reduce_men_usage is a good option, also I will consider to run a VM on Google Cloud with more ram, how much memory you have ?</p>",
      "rawMarkdown": "Hi! Jin Ma, like Tak mentioned the reduce_men_usage is a good option, also I will consider to run a VM on Google Cloud with more ram, how much memory you have ?",
      "votes": null
    },
    {
      "id": "445173",
      "postDate": "12/25/2018 20:16:49",
      "content": "<p>i hope you find this useful. i faced the same issue as well. too huge data. jupyter/ local python everything runs out of memory.\nusing dask didnt solve it too well..i found myself getting too many dask specific errors and didnt wnt to go down another modules rabbit hole.</p>\n\n<p>very rustic, bt these are the steps i undertook:</p>\n\n<ol>\n<li>treated new transaction and past transaction separately.</li>\n<li>once i have done my data engineering/cleaning on each data,wrote it to csv on disk.</li>\n<li>next iterations were made on the written files. and once a logical end has been reached i would write it back to disk using to_csv</li>\n<li>i reduced the num of merges on panda dataframes. used concat along axis =1 more often.</li>\n</ol>\n\n<p>took some manual effort...bt worked around to reach a place where i cud play with the data</p>",
      "rawMarkdown": "i hope you find this useful. i faced the same issue as well. too huge data. jupyter/ local python everything runs out of memory.\nusing dask didnt solve it too well..i found myself getting too many dask specific errors and didnt wnt to go down another modules rabbit hole.\n\nvery rustic, bt these are the steps i undertook:\n\n1. treated new transaction and past transaction separately.\n2. once i have done my data engineering/cleaning on each data,wrote it to csv on disk.\n3. next iterations were made on the written files. and once a logical end has been reached i would write it back to disk using to_csv\n4. i reduced the num of merges on panda dataframes. used concat along axis =1 more often.\n\ntook some manual effort...bt worked around to reach a place where i cud play with the data",
      "votes": null
    },
    {
      "id": "445886",
      "postDate": "12/27/2018 07:02:24",
      "content": "<p>Hi. I havw 16g ram. </p>",
      "rawMarkdown": "Hi. I havw 16g ram.",
      "votes": null
    },
    {
      "id": "446172",
      "postDate": "12/27/2018 16:40:08",
      "content": "<p>Followed v similar process, broke down the historical transactions into smaller chunks, 3 monthly or even monthly, got the features, grouped by card_id and then summed up across the chunks, grouped by card_id / took max / mean depending on the feature.</p>",
      "rawMarkdown": "Followed v similar process, broke down the historical transactions into smaller chunks, 3 monthly or even monthly, got the features, grouped by card_id and then summed up across the chunks, grouped by card_id / took max / mean depending on the feature.",
      "votes": null
    },
    {
      "id": "446193",
      "postDate": "12/27/2018 17:23:47",
      "content": "<p>This was useful for me in the past\n<a href=\"https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows\">https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows</a></p>",
      "rawMarkdown": "This was useful for me in the past\nhttps://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows",
      "votes": null
    },
    {
      "id": "447785",
      "postDate": "12/30/2018 15:14:30",
      "content": "<p>You can use the <code>reducemen</code> function. Aditionally, you can delete the dataframes you don't need anymore and free some memory with <code>gc.collect()</code>. For example, after merging hist_trans and new_merchant_tans with train and test, you no longer need them and you can delete them.</p>",
      "rawMarkdown": "You can use the `reducemen` function. Aditionally, you can delete the dataframes you don't need anymore and free some memory with `gc.collect()`. For example, after merging hist_trans and new_merchant_tans with train and test, you no longer need them and you can delete them.",
      "votes": null
    },
    {
      "id": "447811",
      "postDate": "12/30/2018 16:02:41",
      "content": "<p>I feel for you a couple of years ago you could just about compete with 6Gb.</p>\n\n<p>Now I have 128 Gb of RAM.</p>\n\n<p>I am rather disturbed by the way Kaggle comps are going as it encourages hardware wars - one competition recently had .25 Tb of data - so I didn't bother</p>",
      "rawMarkdown": "I feel for you a couple of years ago you could just about compete with 6Gb.\n\nNow I have 128 Gb of RAM.\n\nI am rather disturbed by the way Kaggle comps are going as it encourages hardware wars - one competition recently had .25 Tb of data - so I didn't bother",
      "votes": null
    },
    {
      "id": "447964",
      "postDate": "12/30/2018 23:33:59",
      "content": "<p>I also don't like the fact that competition with smaller dataset seems to be a thing of the past. \nBut then again, perhaps companies are starting to have larger and larger dataset, perhaps that is just an industrial trend.</p>\n\n<p>I do think they need to do more to make it easier for new starters without big hardware to get involved</p>",
      "rawMarkdown": "I also don't like the fact that competition with smaller dataset seems to be a thing of the past. \nBut then again, perhaps companies are starting to have larger and larger dataset, perhaps that is just an industrial trend.\n\nI do think they need to do more to make it easier for new starters without big hardware to get involved",
      "votes": null
    },
    {
      "id": "448745",
      "postDate": "01/02/2019 02:15:22",
      "content": "<p>You can see function from <a href=\"https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\">https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65</a> (it helped me) but I recommend to increase RAM size anyway (and I also planning to to it).</p>\n\n<p>p.s. also, I modified such function to avoid crashes on datetime obejcts and replacing NaN-contained columns with integers (with -1 instead of NaN) - it still helps a lot, so I prefferer to avoid such hacks. You can see my version here: <a href=\"https://gist.github.com/alex4321/a93dcb1f712fdb4e9bceec0ddedc3e1b\">https://gist.github.com/alex4321/a93dcb1f712fdb4e9bceec0ddedc3e1b</a></p>",
      "rawMarkdown": "You can see function from https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65 (it helped me) but I recommend to increase RAM size anyway (and I also planning to to it).\n\np.s. also, I modified such function to avoid crashes on datetime obejcts and replacing NaN-contained columns with integers (with -1 instead of NaN) - it still helps a lot, so I prefferer to avoid such hacks. You can see my version here: https://gist.github.com/alex4321/a93dcb1f712fdb4e9bceec0ddedc3e1b",
      "votes": null
    },
    {
      "id": "448823",
      "postDate": "01/02/2019 07:03:27",
      "content": "<p>Can't we upload the data set on amazon cloud and run your algorithm model from there  ? Is there any policy of Kaggle for not uploading the dataset ?</p>",
      "rawMarkdown": "Can't we upload the data set on amazon cloud and run your algorithm model from there  ? Is there any policy of Kaggle for not uploading the dataset ?",
      "votes": null
    },
    {
      "id": "448974",
      "postDate": "01/02/2019 13:09:31",
      "content": "<p>I don't think there is any problem using a Cloud service but is always nice to learn new approaches in how to deal with this type of issues in a more efficient way, rather than use more hardware.</p>",
      "rawMarkdown": "I don't think there is any problem using a Cloud service but is always nice to learn new approaches in how to deal with this type of issues in a more efficient way, rather than use more hardware.",
      "votes": null
    },
    {
      "id": "450106",
      "postDate": "01/04/2019 10:12:54",
      "content": "<p>The suggestion I have is : you might not need the entire historical data to do your initial data exploration or prediction. Last 3 months data will give you good enough results to start with. And then you can add more data as you go.</p>",
      "rawMarkdown": "The suggestion I have is : you might not need the entire historical data to do your initial data exploration or prediction. Last 3 months data will give you good enough results to start with. And then you can add more data as you go.",
      "votes": null
    },
    {
      "id": "452412",
      "postDate": "01/08/2019 17:31:37",
      "content": "<p>If you are running Linux and you happen to have an SSD drive, then you can setup a swap file of a reasonable size and it should give you memory for running memory hungry operations.  My motherboard has limitation of 32GB of RAM and it is maxed out. I set up a 64GB swap file and it is like having 96 GB of RAM. It is a bit slower when swap is in use, but at least the code does not quit on out of memory exceptions. </p>",
      "rawMarkdown": "If you are running Linux and you happen to have an SSD drive, then you can setup a swap file of a reasonable size and it should give you memory for running memory hungry operations.  My motherboard has limitation of 32GB of RAM and it is maxed out. I set up a 64GB swap file and it is like having 96 GB of RAM. It is a bit slower when swap is in use, but at least the code does not quit on out of memory exceptions.",
      "votes": null
    },
    {
      "id": "452774",
      "postDate": "01/09/2019 06:24:46",
      "content": "<p>hi, is it sure that using more data gives better results ?</p>",
      "rawMarkdown": "hi, is it sure that using more data gives better results ?",
      "votes": null
    },
    {
      "id": "455485",
      "postDate": "01/14/2019 02:53:30",
      "content": "<p>There are some good suggestions here. Before upgrading your own laptop, consider google cloud platform. An instance with ~50 GB RAM would not cost much. I using it for this competition. Charges = ~ $3-4 for ~ 2hrs.</p>\n\n<p>Plus for a new account you get $300 free credit. more than enough to try things with. </p>",
      "rawMarkdown": "There are some good suggestions here. Before upgrading your own laptop, consider google cloud platform. An instance with ~50 GB RAM would not cost much. I using it for this competition. Charges = ~ $3-4 for ~ 2hrs.\n\nPlus for a new account you get $300 free credit. more than enough to try things with.",
      "votes": null
    },
    {
      "id": "455496",
      "postDate": "01/14/2019 03:50:24",
      "content": "<p>If Python is the prefered language used for this competition, then I believe Google colab can be used instead of using our Local Machines. This would reduce the load on our machines. The only problem that we might encounter while using this is we might need to connect to the Server everytime , to initialize the process(will take couple of seconds.)</p>",
      "rawMarkdown": "If Python is the prefered language used for this competition, then I believe Google colab can be used instead of using our Local Machines. This would reduce the load on our machines. The only problem that we might encounter while using this is we might need to connect to the Server everytime , to initialize the process(will take couple of seconds.)",
      "votes": null
    },
    {
      "id": "459545",
      "postDate": "01/22/2019 00:44:35",
      "content": "<p>Take a look at the datatypes in use - I got big savings by changing data types to their smallest possible types. I.e. reduce a binary integer to an int8 rather than an int64. </p>",
      "rawMarkdown": "Take a look at the datatypes in use - I got big savings by changing data types to their smallest possible types. I.e. reduce a binary integer to an int8 rather than an int64.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 445000,
      "author_name": "takaishikawa",
      "author_url": "",
      "post_date": "12/25/2018 10:24:01",
      "content": "<p><code>reduce_mem_usage</code> function is so effective for dealing with such large size data, which is found on many public kernels like <a href=\"https://www.kaggle.com/fabiendaniel/elo-world\">this</a> (the cell containing the function is hiden).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 445056,
      "author_name": "yifanxie",
      "author_url": "",
      "post_date": "12/25/2018 14:05:26",
      "content": "<p>if aggregation on this dataset is proving to a problem for your hardware, I would suggest to  have a look at Dask</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 445098,
      "author_name": "cyberia",
      "author_url": "",
      "post_date": "12/25/2018 16:21:01",
      "content": "<p>@JinMa Have a look <a href=\"https://www.kaggle.com/questions-and-answers/75495#post443758\">https://www.kaggle.com/questions-and-answers/75495#post443758</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 445099,
      "author_name": "cv13j0",
      "author_url": "",
      "post_date": "12/25/2018 16:21:32",
      "content": "<p>Hi! Jin Ma, like Tak mentioned the reduce_men_usage is a good option, also I will consider to run a VM on Google Cloud with more ram, how much memory you have ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 445886,
          "author_name": "jinmahkust",
          "author_url": "",
          "post_date": "12/27/2018 07:02:24",
          "content": "<p>Hi. I havw 16g ram. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 446193,
          "author_name": "cv13j0",
          "author_url": "",
          "post_date": "12/27/2018 17:23:47",
          "content": "<p>This was useful for me in the past\n<a href=\"https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows\">https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 445173,
      "author_name": "somaktukai",
      "author_url": "",
      "post_date": "12/25/2018 20:16:49",
      "content": "<p>i hope you find this useful. i faced the same issue as well. too huge data. jupyter/ local python everything runs out of memory.\nusing dask didnt solve it too well..i found myself getting too many dask specific errors and didnt wnt to go down another modules rabbit hole.</p>\n\n<p>very rustic, bt these are the steps i undertook:</p>\n\n<ol>\n<li>treated new transaction and past transaction separately.</li>\n<li>once i have done my data engineering/cleaning on each data,wrote it to csv on disk.</li>\n<li>next iterations were made on the written files. and once a logical end has been reached i would write it back to disk using to_csv</li>\n<li>i reduced the num of merges on panda dataframes. used concat along axis =1 more often.</li>\n</ol>\n\n<p>took some manual effort...bt worked around to reach a place where i cud play with the data</p>",
      "votes": null,
      "replies": [
        {
          "id": 446172,
          "author_name": "anirbank",
          "author_url": "",
          "post_date": "12/27/2018 16:40:08",
          "content": "<p>Followed v similar process, broke down the historical transactions into smaller chunks, 3 monthly or even monthly, got the features, grouped by card_id and then summed up across the chunks, grouped by card_id / took max / mean depending on the feature.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 447785,
      "author_name": "luhego",
      "author_url": "",
      "post_date": "12/30/2018 15:14:30",
      "content": "<p>You can use the <code>reducemen</code> function. Aditionally, you can delete the dataframes you don't need anymore and free some memory with <code>gc.collect()</code>. For example, after merging hist_trans and new_merchant_tans with train and test, you no longer need them and you can delete them.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 447811,
      "author_name": "scirpus",
      "author_url": "",
      "post_date": "12/30/2018 16:02:41",
      "content": "<p>I feel for you a couple of years ago you could just about compete with 6Gb.</p>\n\n<p>Now I have 128 Gb of RAM.</p>\n\n<p>I am rather disturbed by the way Kaggle comps are going as it encourages hardware wars - one competition recently had .25 Tb of data - so I didn't bother</p>",
      "votes": null,
      "replies": [
        {
          "id": 447964,
          "author_name": "yifanxie",
          "author_url": "",
          "post_date": "12/30/2018 23:33:59",
          "content": "<p>I also don't like the fact that competition with smaller dataset seems to be a thing of the past. \nBut then again, perhaps companies are starting to have larger and larger dataset, perhaps that is just an industrial trend.</p>\n\n<p>I do think they need to do more to make it easier for new starters without big hardware to get involved</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 448745,
      "author_name": "gaussmake1994",
      "author_url": "",
      "post_date": "01/02/2019 02:15:22",
      "content": "<p>You can see function from <a href=\"https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\">https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65</a> (it helped me) but I recommend to increase RAM size anyway (and I also planning to to it).</p>\n\n<p>p.s. also, I modified such function to avoid crashes on datetime obejcts and replacing NaN-contained columns with integers (with -1 instead of NaN) - it still helps a lot, so I prefferer to avoid such hacks. You can see my version here: <a href=\"https://gist.github.com/alex4321/a93dcb1f712fdb4e9bceec0ddedc3e1b\">https://gist.github.com/alex4321/a93dcb1f712fdb4e9bceec0ddedc3e1b</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 448823,
      "author_name": "pulkit21aug",
      "author_url": "",
      "post_date": "01/02/2019 07:03:27",
      "content": "<p>Can't we upload the data set on amazon cloud and run your algorithm model from there  ? Is there any policy of Kaggle for not uploading the dataset ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 448974,
          "author_name": "cv13j0",
          "author_url": "",
          "post_date": "01/02/2019 13:09:31",
          "content": "<p>I don't think there is any problem using a Cloud service but is always nice to learn new approaches in how to deal with this type of issues in a more efficient way, rather than use more hardware.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 450106,
      "author_name": "anirbank",
      "author_url": "",
      "post_date": "01/04/2019 10:12:54",
      "content": "<p>The suggestion I have is : you might not need the entire historical data to do your initial data exploration or prediction. Last 3 months data will give you good enough results to start with. And then you can add more data as you go.</p>",
      "votes": null,
      "replies": [
        {
          "id": 452774,
          "author_name": "jinmahkust",
          "author_url": "",
          "post_date": "01/09/2019 06:24:46",
          "content": "<p>hi, is it sure that using more data gives better results ?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 452412,
      "author_name": "calabist",
      "author_url": "",
      "post_date": "01/08/2019 17:31:37",
      "content": "<p>If you are running Linux and you happen to have an SSD drive, then you can setup a swap file of a reasonable size and it should give you memory for running memory hungry operations.  My motherboard has limitation of 32GB of RAM and it is maxed out. I set up a 64GB swap file and it is like having 96 GB of RAM. It is a bit slower when swap is in use, but at least the code does not quit on out of memory exceptions. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 455485,
      "author_name": "apoorvg",
      "author_url": "",
      "post_date": "01/14/2019 02:53:30",
      "content": "<p>There are some good suggestions here. Before upgrading your own laptop, consider google cloud platform. An instance with ~50 GB RAM would not cost much. I using it for this competition. Charges = ~ $3-4 for ~ 2hrs.</p>\n\n<p>Plus for a new account you get $300 free credit. more than enough to try things with. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 455496,
      "author_name": "damnboy",
      "author_url": "",
      "post_date": "01/14/2019 03:50:24",
      "content": "<p>If Python is the prefered language used for this competition, then I believe Google colab can be used instead of using our Local Machines. This would reduce the load on our machines. The only problem that we might encounter while using this is we might need to connect to the Server everytime , to initialize the process(will take couple of seconds.)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 459545,
      "author_name": "drtuff",
      "author_url": "",
      "post_date": "01/22/2019 00:44:35",
      "content": "<p>Take a look at the datatypes in use - I got big savings by changing data types to their smallest possible types. I.e. reduce a binary integer to an int8 rather than an int64. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "444990": "How can I handle such big data in my computer?",
    "445000": "`reduce_mem_usage` function is so effective for dealing with such large size data, which is found on many public kernels like [this](https://www.kaggle.com/fabiendaniel/elo-world) (the cell containing the function is hiden).",
    "445056": "if aggregation on this dataset is proving to a problem for your hardware, I would suggest to  have a look at Dask",
    "445098": "JinMa Have a look https://www.kaggle.com/questions-and-answers/75495#post443758",
    "445099": "Hi! Jin Ma, like Tak mentioned the reduce_men_usage is a good option, also I will consider to run a VM on Google Cloud with more ram, how much memory you have ?",
    "445173": "i hope you find this useful. i faced the same issue as well. too huge data. jupyter/ local python everything runs out of memory.\nusing dask didnt solve it too well..i found myself getting too many dask specific errors and didnt wnt to go down another modules rabbit hole.\n\nvery rustic, bt these are the steps i undertook:\n\n1. treated new transaction and past transaction separately.\n2. once i have done my data engineering/cleaning on each data,wrote it to csv on disk.\n3. next iterations were made on the written files. and once a logical end has been reached i would write it back to disk using to_csv\n4. i reduced the num of merges on panda dataframes. used concat along axis =1 more often.\n\ntook some manual effort...bt worked around to reach a place where i cud play with the data",
    "445886": "Hi. I havw 16g ram.",
    "446172": "Followed v similar process, broke down the historical transactions into smaller chunks, 3 monthly or even monthly, got the features, grouped by card_id and then summed up across the chunks, grouped by card_id / took max / mean depending on the feature.",
    "446193": "This was useful for me in the past\nhttps://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows",
    "447785": "You can use the `reducemen` function. Aditionally, you can delete the dataframes you don't need anymore and free some memory with `gc.collect()`. For example, after merging hist_trans and new_merchant_tans with train and test, you no longer need them and you can delete them.",
    "447811": "I feel for you a couple of years ago you could just about compete with 6Gb.\n\nNow I have 128 Gb of RAM.\n\nI am rather disturbed by the way Kaggle comps are going as it encourages hardware wars - one competition recently had .25 Tb of data - so I didn't bother",
    "447964": "I also don't like the fact that competition with smaller dataset seems to be a thing of the past. \nBut then again, perhaps companies are starting to have larger and larger dataset, perhaps that is just an industrial trend.\n\nI do think they need to do more to make it easier for new starters without big hardware to get involved",
    "448745": "You can see function from https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65 (it helped me) but I recommend to increase RAM size anyway (and I also planning to to it).\n\np.s. also, I modified such function to avoid crashes on datetime obejcts and replacing NaN-contained columns with integers (with -1 instead of NaN) - it still helps a lot, so I prefferer to avoid such hacks. You can see my version here: https://gist.github.com/alex4321/a93dcb1f712fdb4e9bceec0ddedc3e1b",
    "448823": "Can't we upload the data set on amazon cloud and run your algorithm model from there  ? Is there any policy of Kaggle for not uploading the dataset ?",
    "448974": "I don't think there is any problem using a Cloud service but is always nice to learn new approaches in how to deal with this type of issues in a more efficient way, rather than use more hardware.",
    "450106": "The suggestion I have is : you might not need the entire historical data to do your initial data exploration or prediction. Last 3 months data will give you good enough results to start with. And then you can add more data as you go.",
    "452412": "If you are running Linux and you happen to have an SSD drive, then you can setup a swap file of a reasonable size and it should give you memory for running memory hungry operations.  My motherboard has limitation of 32GB of RAM and it is maxed out. I set up a 64GB swap file and it is like having 96 GB of RAM. It is a bit slower when swap is in use, but at least the code does not quit on out of memory exceptions.",
    "452774": "hi, is it sure that using more data gives better results ?",
    "455485": "There are some good suggestions here. Before upgrading your own laptop, consider google cloud platform. An instance with ~50 GB RAM would not cost much. I using it for this competition. Charges = ~ $3-4 for ~ 2hrs.\n\nPlus for a new account you get $300 free credit. more than enough to try things with.",
    "455496": "If Python is the prefered language used for this competition, then I believe Google colab can be used instead of using our Local Machines. This would reduce the load on our machines. The only problem that we might encounter while using this is we might need to connect to the Server everytime , to initialize the process(will take couple of seconds.)",
    "459545": "Take a look at the datatypes in use - I got big savings by changing data types to their smallest possible types. I.e. reduce a binary integer to an int8 rather than an int64."
  },
  "source": "meta"
}