I'm splitting the data by columns to create manageable datasets for pandas. First I read 5 rows of data and extract+keep all columns per dataset. Then for each dataset, I select groups of 200 columns that go together in a batch of columns (9-21 subsets of data per dataset). With pandas, you can filter out this list of columns per batch using the "usecols" parameter and write that out as a subset. It then won't load all the others. Reduce 200 to 100 for much smaller memory.
\n\nIf you want to hot-encode the categorical vars, then consider appending the specific value type to the column name just as the pandas.get_dummies function does. The reason why this is useful is that you no longer need to keep an expensive vectorizer in memory that contains all types and keep that around for both train/test, making things more complicated. You can just work on each file individually to work that out and as a final step grab the column order from train and enforce that on test (usecols again).
\n\nIn the end, I combine the final subsets using standard python 'line-by-line' processing, basically concatenating columns by hand. If rows and columns are ordered and consistent, then you can use a very simple python script to do this. If you don't import pandas/numpy, you can use pypy which is much faster for small bits of code (2-10x or so), because it compiles certain frequent loops to native code.
\n\nThe end result is probably a 25G file, compressed to 500M in size for both train and test. Then convert this to formats that are used in online learning, etc.
", "messages": [ { "id": 134222, "postDate": "2016-09-05T21:38:13.477Z", "content": "I'm splitting the data by columns to create manageable datasets for pandas. First I read 5 rows of data and extract+keep all columns per dataset. Then for each dataset, I select groups of 200 columns that go together in a batch of columns (9-21 subsets of data per dataset). With pandas, you can filter out this list of columns per batch using the "usecols" parameter and write that out as a subset. It then won't load all the others. Reduce 200 to 100 for much smaller memory.
\n\nIf you want to hot-encode the categorical vars, then consider appending the specific value type to the column name just as the pandas.get_dummies function does. The reason why this is useful is that you no longer need to keep an expensive vectorizer in memory that contains all types and keep that around for both train/test, making things more complicated. You can just work on each file individually to work that out and as a final step grab the column order from train and enforce that on test (usecols again).
\n\nIn the end, I combine the final subsets using standard python 'line-by-line' processing, basically concatenating columns by hand. If rows and columns are ordered and consistent, then you can use a very simple python script to do this. If you don't import pandas/numpy, you can use pypy which is much faster for small bits of code (2-10x or so), because it compiles certain frequent loops to native code.
\n\nThe end result is probably a 25G file, compressed to 500M in size for both train and test. Then convert this to formats that are used in online learning, etc.
", "rawMarkdown": "I'm splitting the data by columns to create manageable datasets for pandas. First I read 5 rows of data and extract+keep all columns per dataset. Then for each dataset, I select groups of 200 columns that go together in a batch of columns (9-21 subsets of data per dataset). With pandas, you can filter out this list of columns per batch using the \"usecols\" parameter and write that out as a subset. It then won't load all the others. Reduce 200 to 100 for much smaller memory.\r\n\r\nIf you want to hot-encode the categorical vars, then consider appending the specific value type to the column name just as the pandas.get_dummies function does. The reason why this is useful is that you no longer need to keep an expensive vectorizer in memory that contains all types and keep that around for both train/test, making things more complicated. You can just work on each file individually to work that out and as a final step grab the column order from train and enforce that on test (usecols again).\r\n\r\nIn the end, I combine the final subsets using standard python 'line-by-line' processing, basically concatenating columns by hand. If rows and columns are ordered and consistent, then you can use a very simple python script to do this. If you don't import pandas/numpy, you can use pypy which is much faster for small bits of code (2-10x or so), because it compiles certain frequent loops to native code. \r\n\r\nThe end result is probably a 25G file, compressed to 500M in size for both train and test. Then convert this to formats that are used in online learning, etc." } ], "comments": [], "raw_markdown_by_id": { "134222": "I'm splitting the data by columns to create manageable datasets for pandas. First I read 5 rows of data and extract+keep all columns per dataset. Then for each dataset, I select groups of 200 columns that go together in a batch of columns (9-21 subsets of data per dataset). With pandas, you can filter out this list of columns per batch using the \"usecols\" parameter and write that out as a subset. It then won't load all the others. Reduce 200 to 100 for much smaller memory.\r\n\r\nIf you want to hot-encode the categorical vars, then consider appending the specific value type to the column name just as the pandas.get_dummies function does. The reason why this is useful is that you no longer need to keep an expensive vectorizer in memory that contains all types and keep that around for both train/test, making things more complicated. You can just work on each file individually to work that out and as a final step grab the column order from train and enforce that on test (usecols again).\r\n\r\nIn the end, I combine the final subsets using standard python 'line-by-line' processing, basically concatenating columns by hand. If rows and columns are ordered and consistent, then you can use a very simple python script to do this. If you don't import pandas/numpy, you can use pypy which is much faster for small bits of code (2-10x or so), because it compiles certain frequent loops to native code. \r\n\r\nThe end result is probably a 25G file, compressed to 500M in size for both train and test. Then convert this to formats that are used in online learning, etc." } }