I am using 8gb ram and with using python hy5 how can i load all the file like test_numeric,train_numeric, test_date, train_date, and categorical also...
\n\nthank you
", "messages": [ { "id": "133461", "postDate": "09/02/2016 11:23:24", "content": "I am using 8gb ram and with using python hy5 how can i load all the file like test_numeric,train_numeric, test_date, train_date, and categorical also...
\n\nthank you
", "rawMarkdown": "I am using 8gb ram and with using python hy5 how can i load all the file like test_numeric,train_numeric, test_date, train_date, and categorical also...\r\n\r\nthank you", "votes": null }, { "id": "133913", "postDate": "09/04/2016 13:15:31", "content": "You have a 2 options:
\n1. Use sframe library, but yes that would cost a lot of time
\n2. You could partition each file, into smaller chunks by using split -b <file size>, for example you want to split into 20MB chunks for train_numeric data: split -b 20m train_numeric.csv. Then you can load each chunks using pythons library for example pandas.
1) You could also load the data entirely in memory by using scipy.sparse matrix. You will have to build it by your own. \n2) I think that by default, the majority of the libraries in python utilizes Double (np.float64) to represent the data values. One thing that you can do to save lots of memory is to force the representation to be Float (usually np.float32). The data values don't require too much precision to use Double float point representation. You can further improve the memory usage by using np.float16, but may not work for some classifiers of the scikit that require Double (such as KNN).
", "rawMarkdown": "1) You could also load the data entirely in memory by using scipy.sparse matrix. You will have to build it by your own. \r\n2) I think that by default, the majority of the libraries in python utilizes Double (np.float64) to represent the data values. One thing that you can do to save lots of memory is to force the representation to be Float (usually np.float32). The data values don't require too much precision to use Double float point representation. You can further improve the memory usage by using np.float16, but may not work for some classifiers of the scikit that require Double (such as KNN).", "votes": null }, { "id": "133960", "postDate": "09/04/2016 15:32:15", "content": "I managed to load all data by using pandas sparse data frames on a 16GB memory machine. But you need to read data by chunks first as you cannot read directly into a sparse data frame. @Laurae has described how to do it using R. You can do something similar in Python.
", "rawMarkdown": "I managed to load all data by using pandas sparse data frames on a 16GB memory machine. But you need to read data by chunks first as you cannot read directly into a sparse data frame. @Laurae has described how to do it using R. You can do something similar in Python.", "votes": null }, { "id": "134062", "postDate": "09/04/2016 22:16:28", "content": "If you are like me and short on memory but long on hard drive base, you might want to setup your data in a postgresql database (its free). I dunno how far I will get, but that was my first thought (That and I really need a new computer or a dedicated data science rig!).
\n\nImporting the numeric data didn't take long and it was pretty easy to create the table ahead of time and then just import the table.
\n\nAlways good to have options.
\n\nSome other tricks in the past that have worked for me is to import with defined data types (on my machine, it defaults to 64 bit floats, which is huge--dropping that down helps, also storing a "1" as a 64 bit float, is bad--smallints! With data this size, I doubt that gets us enough space though in memory. SOmething to hold in your pocket for the future though.
\n\nWorks something like this (I was using unsigned ints for this particular dataset--my apologies--the category names are in spanish):
\n\n train_df = pd.read_csv("C:/users/Robert/kaggle/Bimbo/data/train.csv",\n dtype= \n {'Semana':np.uint16,\n 'Agencia_ID':np.uint16,\n 'Canal_ID':np.uint16,\n 'Ruta_SAK':np.uint16, \n ... }) \nHere's the command in python to do memory chunks ('chunksize, along with how to only import certain columns')
\n\ntrain = pd.read_csv("./kaggle/expedia/data/train.csv",\n dtype={'is_booking':bool,\n 'srch_destination_id':np.int32, \n 'hotel_cluster':np.int32},\n usecols=['srch_destination_id','is_booking','hotel_cluster'],\n chunksize=1000000)\nI looked at the Python script and remembered some of the sparse DataFrame stuff I was looking at briefly on redhat, and I whipped this up just now and it pretty much matches CPMP's description. It processed date_chunks in ~13:34 and ~6.6/5.6GB on my W5590 with 48GB RAM and SSD, so this should work for even 8GB systems. The output file is 2,561,720,271 bytes long.
\n\nI haven't tried numeric and categorical files yet, but it looks like a two pass model over categorical, recording all unique values then remapping them, would make memory usage reasonable there too. On a larger memory system it should be possible to merge all three .csv files into one huge sparse dataframe.
\n\nAlso it looks like the numeric data might fit into int16's when multiplied by 1000, but I haven't done anything in depth on it yet.
\n\nimport numpy as np\nimport pandas as pd\n\nfrom scipy.sparse import csr_matrix, hstack\n\ndate_chunks = pd.read_csv("../input/train_date.csv", index_col=0, chunksize=10000, dtype=np.float32)\nnum_chunks = pd.read_csv("../input/train_numeric.csv", index_col=0, chunksize=10000, dtype=np.float32)\n\ndchunks_sparse = []\ni = 0\nfor dchunk in date_chunks:\n dchunks_sparse.append(dchunk.to_sparse())\n print(i, dchunks_sparse[-1].memory_usage().sum(), dchunk.memory_usage().sum())\n i += 1\n\n\ntrain_date_sparse = pd.concat(dchunks_sparse)\n\nprint(train_date_sparse.memory_usage().sum())\n\ntrain_date_sparse.to_pickle('train_date_sparse.pkl')\n@happycube,\nRight, that's what I was describing indeed.
", "rawMarkdown": "happycube,\r\nRight, that's what I was describing indeed.", "votes": null }, { "id": "137274", "postDate": "09/29/2016 21:14:44", "content": "@happy cube
\n\nHi \nI tried above mentioned code . It ran for a while and then crashed outputting 'memory error'.\nI am using windows 10 64 bits with 32.0 GB ram.
\n\nKindly refer attached image and suggest a way to get around this.
", "rawMarkdown": "happy cube\r\n\r\nHi \r\nI tried above mentioned code . It ran for a while and then crashed outputting 'memory error'.\r\nI am using windows 10 64 bits with 32.0 GB ram.\r\n\r\nKindly refer attached image and suggest a way to get around this.", "votes": null }, { "id": "137307", "postDate": "09/30/2016 08:24:19", "content": "@BhanuKiran
\n\nTry running for lesser value of chunk size. Probably chunk size is too big to load the data in memory.
", "rawMarkdown": "BhanuKiran \r\n\r\nTry running for lesser value of chunk size. Probably chunk size is too big to load the data in memory.", "votes": null } ], "comments": [ { "id": 133913, "author_name": "zscansorted", "author_url": "", "post_date": "09/04/2016 13:15:31", "content": "You have a 2 options:
\n1. Use sframe library, but yes that would cost a lot of time
\n2. You could partition each file, into smaller chunks by using split -b <file size>, for example you want to split into 20MB chunks for train_numeric data: split -b 20m train_numeric.csv. Then you can load each chunks using pythons library for example pandas.
1) You could also load the data entirely in memory by using scipy.sparse matrix. You will have to build it by your own. \n2) I think that by default, the majority of the libraries in python utilizes Double (np.float64) to represent the data values. One thing that you can do to save lots of memory is to force the representation to be Float (usually np.float32). The data values don't require too much precision to use Double float point representation. You can further improve the memory usage by using np.float16, but may not work for some classifiers of the scikit that require Double (such as KNN).
", "votes": null, "replies": [] }, { "id": 133960, "author_name": "cpmpml", "author_url": "", "post_date": "09/04/2016 15:32:15", "content": "I managed to load all data by using pandas sparse data frames on a 16GB memory machine. But you need to read data by chunks first as you cannot read directly into a sparse data frame. @Laurae has described how to do it using R. You can do something similar in Python.
", "votes": null, "replies": [] }, { "id": 134062, "author_name": "rdslater", "author_url": "", "post_date": "09/04/2016 22:16:28", "content": "If you are like me and short on memory but long on hard drive base, you might want to setup your data in a postgresql database (its free). I dunno how far I will get, but that was my first thought (That and I really need a new computer or a dedicated data science rig!).
\n\nImporting the numeric data didn't take long and it was pretty easy to create the table ahead of time and then just import the table.
\n\nAlways good to have options.
\n\nSome other tricks in the past that have worked for me is to import with defined data types (on my machine, it defaults to 64 bit floats, which is huge--dropping that down helps, also storing a "1" as a 64 bit float, is bad--smallints! With data this size, I doubt that gets us enough space though in memory. SOmething to hold in your pocket for the future though.
\n\nWorks something like this (I was using unsigned ints for this particular dataset--my apologies--the category names are in spanish):
\n\n train_df = pd.read_csv("C:/users/Robert/kaggle/Bimbo/data/train.csv",\n dtype= \n {'Semana':np.uint16,\n 'Agencia_ID':np.uint16,\n 'Canal_ID':np.uint16,\n 'Ruta_SAK':np.uint16, \n ... }) \nHere's the command in python to do memory chunks ('chunksize, along with how to only import certain columns')
\n\ntrain = pd.read_csv("./kaggle/expedia/data/train.csv",\n dtype={'is_booking':bool,\n 'srch_destination_id':np.int32, \n 'hotel_cluster':np.int32},\n usecols=['srch_destination_id','is_booking','hotel_cluster'],\n chunksize=1000000)\nI looked at the Python script and remembered some of the sparse DataFrame stuff I was looking at briefly on redhat, and I whipped this up just now and it pretty much matches CPMP's description. It processed date_chunks in ~13:34 and ~6.6/5.6GB on my W5590 with 48GB RAM and SSD, so this should work for even 8GB systems. The output file is 2,561,720,271 bytes long.
\n\nI haven't tried numeric and categorical files yet, but it looks like a two pass model over categorical, recording all unique values then remapping them, would make memory usage reasonable there too. On a larger memory system it should be possible to merge all three .csv files into one huge sparse dataframe.
\n\nAlso it looks like the numeric data might fit into int16's when multiplied by 1000, but I haven't done anything in depth on it yet.
\n\nimport numpy as np\nimport pandas as pd\n\nfrom scipy.sparse import csr_matrix, hstack\n\ndate_chunks = pd.read_csv("../input/train_date.csv", index_col=0, chunksize=10000, dtype=np.float32)\nnum_chunks = pd.read_csv("../input/train_numeric.csv", index_col=0, chunksize=10000, dtype=np.float32)\n\ndchunks_sparse = []\ni = 0\nfor dchunk in date_chunks:\n dchunks_sparse.append(dchunk.to_sparse())\n print(i, dchunks_sparse[-1].memory_usage().sum(), dchunk.memory_usage().sum())\n i += 1\n\n\ntrain_date_sparse = pd.concat(dchunks_sparse)\n\nprint(train_date_sparse.memory_usage().sum())\n\ntrain_date_sparse.to_pickle('train_date_sparse.pkl')\n@happycube,\nRight, that's what I was describing indeed.
", "votes": null, "replies": [] }, { "id": 137274, "author_name": "bhanukiranchaluvadi", "author_url": "", "post_date": "09/29/2016 21:14:44", "content": "@happy cube
\n\nHi \nI tried above mentioned code . It ran for a while and then crashed outputting 'memory error'.\nI am using windows 10 64 bits with 32.0 GB ram.
\n\nKindly refer attached image and suggest a way to get around this.
", "votes": null, "replies": [] }, { "id": 137307, "author_name": "vishsangale", "author_url": "", "post_date": "09/30/2016 08:24:19", "content": "@BhanuKiran
\n\nTry running for lesser value of chunk size. Probably chunk size is too big to load the data in memory.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "133461": "I am using 8gb ram and with using python hy5 how can i load all the file like test_numeric,train_numeric, test_date, train_date, and categorical also...\r\n\r\nthank you", "133913": "You have a 2 options: