Just for getting started, to get the data down to a more manageable size, I extracted only transactions where the category was a category on at least one of the offers. This got the transactions down from about 22GB to about 1GB.
\nObviously, as the competition progresses, I'd recommend delving into the "other 21GB" but this might help getting started.
\nCode is below. It is a combo of grep and R.
\n-----------------
\nUnix script for getting transactions where category is in at least one offer. WARNING, this will pick up some junk rows and maybe even duplicate rows because of grep matching other parts of the row.
\ngrep ",706," transactions > trans_cat.csv
grep ",799," transactions >> trans_cat.csv
grep ",1703," transactions >> trans_cat.csv
grep ",1726," transactions >> trans_cat.csv
grep ",2119," transactions >> trans_cat.csv
grep ",2202," transactions >> trans_cat.csv
grep ",3203," transactions >> trans_cat.csv
grep ",3504," transactions >> trans_cat.csv
grep ",3509," transactions >> trans_cat.csv
grep ",4401," transactions >> trans_cat.csv
grep ",4517," transactions >> trans_cat.csv
grep ",5122," transactions >> trans_cat.csv
grep ",5558," transactions >> trans_cat.csv
grep ",5616," transactions >> trans_cat.csv
grep ",5619," transactions >> trans_cat.csv
grep ",5824," transactions >> trans_cat.csv
grep ",6202," transactions >> trans_cat.csv
grep ",7205," transactions >> trans_cat.csv
grep ",9115," transactions >> trans_cat.csv
grep ",9909," transactions >> trans_cat.csv
# R code to weed out junk rows that were accidentally picked up by grep.
# If duplicates exist, getting rid of them is left as an exercise to the reader.
\ntrans_cat <- read.csv("trans_cat.csv",header=TRUE)
nrow(trans_cat)
table(trans_cat$category)
trans_cat <- trans_cat[which(trans_cat$category %in% c(706,799,1703,1726,2119,2202,3203,3504,3509,4401,4517,5122,5558,5616,5619,5824,6202,7205,9115,9909)),]
nrow(trans_cat)
table(trans_cat$category)
Just for getting started, to get the data down to a more manageable size, I extracted only transactions where the category was a category on at least one of the offers. This got the transactions down from about 22GB to about 1GB.
\nObviously, as the competition progresses, I'd recommend delving into the "other 21GB" but this might help getting started.
\nCode is below. It is a combo of grep and R.
\n-----------------
\nUnix script for getting transactions where category is in at least one offer. WARNING, this will pick up some junk rows and maybe even duplicate rows because of grep matching other parts of the row.
\ngrep ",706," transactions > trans_cat.csv
grep ",799," transactions >> trans_cat.csv
grep ",1703," transactions >> trans_cat.csv
grep ",1726," transactions >> trans_cat.csv
grep ",2119," transactions >> trans_cat.csv
grep ",2202," transactions >> trans_cat.csv
grep ",3203," transactions >> trans_cat.csv
grep ",3504," transactions >> trans_cat.csv
grep ",3509," transactions >> trans_cat.csv
grep ",4401," transactions >> trans_cat.csv
grep ",4517," transactions >> trans_cat.csv
grep ",5122," transactions >> trans_cat.csv
grep ",5558," transactions >> trans_cat.csv
grep ",5616," transactions >> trans_cat.csv
grep ",5619," transactions >> trans_cat.csv
grep ",5824," transactions >> trans_cat.csv
grep ",6202," transactions >> trans_cat.csv
grep ",7205," transactions >> trans_cat.csv
grep ",9115," transactions >> trans_cat.csv
grep ",9909," transactions >> trans_cat.csv
# R code to weed out junk rows that were accidentally picked up by grep.
# If duplicates exist, getting rid of them is left as an exercise to the reader.
\ntrans_cat <- read.csv("trans_cat.csv",header=TRUE)
nrow(trans_cat)
table(trans_cat$category)
trans_cat <- trans_cat[which(trans_cat$category %in% c(706,799,1703,1726,2119,2202,3203,3504,3509,4401,4517,5122,5558,5616,5619,5824,6202,7205,9115,9909)),]
nrow(trans_cat)
table(trans_cat$category)
Use, awk -F, '$4 = 706' transactions > trans_cat.csv and so on to avoid junk rows
", "rawMarkdown": "", "votes": null }, { "id": "41752", "postDate": "04/11/2014 20:03:51", "content": "My machine can't read more than about 50 MB, at once. Just curious what kind of rigs professionals have in place? Do you guys use clusters/hadoop?
", "rawMarkdown": "", "votes": null }, { "id": "41758", "postDate": "04/11/2014 22:10:53", "content": "You can do most challenges on Kaggle on a budget laptop. SSD helps. 16GB memory helps. Some professionals which I envy have access to clusters and servers, but that is not needed to win a contest. If nothing else, you learn how to be more resourceful (using less resources) and care more about speed.
", "rawMarkdown": "", "votes": null }, { "id": "41764", "postDate": "04/11/2014 22:34:09", "content": "I agree with Triskelion. I am able to use my 8GB laptop for most Kaggle contests. If there is a contest that requires a large amount of memory, I sometimes rent an Amazon EC2 instance with 32GB for a couple weeks.
", "rawMarkdown": "", "votes": null }, { "id": "41774", "postDate": "04/12/2014 02:29:31", "content": "I've also found that another useful subset is: only transactions where the company was a company on at least one of the offers. This was also about 1GB. There is obviously some overlap between this and the category subset. What would be best would be to have the union of these without duplicates.
", "rawMarkdown": "", "votes": null }, { "id": "41820", "postDate": "04/12/2014 22:56:44", "content": "In Python to reduce from 20GB to about 1GB (349.655.789 lines to 15.349.956 lines) or "the category subset" as BreakfastPirate calls it:
\nfrom datetime import datetime
loc_offers = "kaggle_shop\\\\offers.csv"
loc_transactions = "kaggle_shop\\\\transactions.csv"
loc_reduced = "kaggle_shop\\\\reduced2.csv" # will be created
def reduce_data(loc_offers, loc_transactions, loc_reduced):
start = datetime.now()
#get all categories on offer in a dict
offers = {}
for e, line in enumerate( open(loc_offers) ):
offers[ line.split(",")[1] ] = 1
#open output file
with open(loc_reduced, "wb") as outfile:
#go through transactions file and reduce
reduced = 0
for e, line in enumerate( open(loc_transactions) ):
if e == 0:
outfile.write( line ) #print header
else:
#only write when category in offers dict
if line.split(",")[3] in offers:
outfile.write( line )
reduced += 1
#progress
if e % 5000000 == 0:
print e, reduced, datetime.now() - start
print e, reduced, datetime.now() - start
reduce_data(loc_offers, loc_transactions, loc_reduced)
if you want to reduce the data with company, change:
\noffers[ line.split(",")[1] ] = 1
to:
\noffers[ line.split(",")[3] ] = 1
and:
\nif line.split(",")[3] in offers
to:
\nif line.split(",")[4] in offers
The category+company union is about 27 million lines (1.6Gb).
", "rawMarkdown": "", "votes": null }, { "id": "41821", "postDate": "04/12/2014 23:11:24", "content": "Python is good for these kind of tasks - the standard CPython is pretty slow, but if your script uses only standard libraries and doesn't use numpy, sklearn, etc. then you should be able to run it with pypy, which is really fast, can be easily an order of magnitude faster than CPython. I think Triskelion's script would take about 10 minutes to process the transactions data on an average PC (I've written something similar to reduce the amount of data).
", "rawMarkdown": "", "votes": null }, { "id": "41822", "postDate": "04/12/2014 23:20:24", "content": "Yup, about 10 minutes to write, about 10 minutes to run. Not the fastest, but should work on any PC (takes max 6mb memory).
", "rawMarkdown": "", "votes": null }, { "id": "41842", "postDate": "04/13/2014 11:32:55", "content": "It takes exactly 3:29 with both PyPy and CPython to run that filtering script on my laptop.
", "rawMarkdown": "", "votes": null }, { "id": "41850", "postDate": "04/13/2014 16:44:47", "content": "Transaction file summary statistics:
\n\n349655789 records
\ntotal purchasequantity 584540514
\ntotal purchaseamount 1568488398
", "rawMarkdown": "", "votes": null }, { "id": "42660", "postDate": "04/22/2014 01:09:09", "content": "My recommendation - Download Datameer and get 15 days trial edition.
\nCreate one workbook in datameer that merges the offers + trainhistory + transactions.
\nYou are ready to go
", "rawMarkdown": "", "votes": null }, { "id": "43083", "postDate": "04/26/2014 16:03:13", "content": "hi! and how did you do that?
\nwhich variables did you use to merge the 3 tables??? the relation is 1to1??
", "rawMarkdown": "", "votes": null }, { "id": "43097", "postDate": "04/26/2014 19:04:49", "content": "[quote=neeraj;42660]
\nMy recommendation - Download Datameer and get 15 days trial edition.
\nCreate one workbook in datameer that merges the offers + trainhistory + transactions.
\nYou are ready to go
\n[/quote]
\nWhy pay for commercial software when R and python are free?
", "rawMarkdown": "", "votes": null }, { "id": "43098", "postDate": "04/26/2014 19:21:12", "content": "Data sets of this size, might be larger than the memory of the computer could hold. Open source R will may fail in such scenarios. (Unless you use packages like ff/bigmemory/bigglm). Revolution R Enterprise, uses external memory algorithms have no data limits, and is easier to use than ff/bigmemory/bigglm.
\nIt is available for Kaggle users for free for participating in Kaggle contests. Download here
\nhttp://info.revolutionanalytics.com/Kaggle.html
\n[post edited to incorporate comments from Zach]
", "rawMarkdown": "", "votes": null }, { "id": "43103", "postDate": "04/26/2014 20:18:18", "content": "[quote=James Paul;43098]
\nData sets of this size, might be larger than the memory of the computer could hold. Open source R will fail in such scenarios. Revolution R Enteprise, uses external memory algorithms have no data limits.
\nIt is available for Kaggle users for free for participating in Kaggle contests. Download here
\nhttp://info.revolutionanalytics.com/Kaggle.html
\n[/quote]
\nThe dataset for this competition will fit in memory on a computer with 16 gigs of RAM. Furthermore, open source R can use out-of-memory algorithms, for example with the bigmemory and ff packages.
\nDon't get me wrong, I love Revolution R: I have the t-shirt and foreach is one of my favorite packages. But open-source R is definitely up for this challenge.
", "rawMarkdown": "", "votes": null }, { "id": "43416", "postDate": "05/01/2014 01:20:18", "content": "Thank you for referring to revolutionanalytics (RA). I downloaded their software and attempted to import the entire transactions file. But ran out of memory as the import was being handled locally. I have a windows xp 4GB RAM laptop. Does anyone know if we can leverage RA's remote computing capability for kaggle comps? and if so, will appreciate any info on how to do this. On the side I am also giving the ff package a go as well but again limited by memory issues. And I am also aware of other opportunities to scale down data size from previous forum contributions.
", "rawMarkdown": "", "votes": null }, { "id": "43417", "postDate": "05/01/2014 01:40:26", "content": "Please contact support@revolutionanalytics.com with your specific error. I have a 16GB laptop and I imported the data easily. I used the following commands to import to the Extended Data Frame format.
\noffers<-rxImport(inData="offers.csv", outFile="offers.xdf", stringsAsFactors=TRUE, missingValueString="M")
trainHistory<-rxImport(inData="trainHistory.csv", outFile="trainHistory.xdf", stringsAsFactors=TRUE, missingValueString="M")
tx<-rxImport(inData="transactions.csv", outFile="tx.xdf", stringsAsFactors=TRUE, missingValueString="M") (It does take time to import this one)
You can merge/combine the .xdf files to proceed with the next steps. (Use rxDataStep command)
\nThe user guide has a number of commonly used scenarios in manipulating data.
\nYou can also use Revolution R Enterprise in AWS. (You can get upto 64GB RAM machines)
", "rawMarkdown": "", "votes": null }, { "id": "43489", "postDate": "05/02/2014 01:57:55", "content": ""fread" from the data.table package is also sufficient to load the full dataset in open source R (at least on a mac).
", "rawMarkdown": "", "votes": null }, { "id": "43522", "postDate": "05/02/2014 18:21:10", "content": "I have a windows xp laptop 2GB RAM, and I get good results in Kaggle competitions ... even a winning price ... I use speed (Kaggle user license) and python ...
\nI think this enough for this competition, but needs a careful treatment of data ...
", "rawMarkdown": "", "votes": null }, { "id": "43626", "postDate": "05/03/2014 19:34:08", "content": "@Triskelion
\nGreat reduction code sir!
\nI ran it in a little over 6 minutes and reduced down to 1GB. I was wondering if you can you provide an explanation, line by line what is going on for the group? I am new to python but I understand some of the basics.
\nBasically, I wanted to know why this works and something like just usingn pandas and dataframes crashes. Is it because you're only doing a pivot table and filtering on 5million records at a time?
", "rawMarkdown": "", "votes": null }, { "id": "43646", "postDate": "05/03/2014 23:32:04", "content": "[quote=Neil Patel;43626]
\n@Triskelion
\nGreat reduction code sir!
\nI ran it in a little over 6 minutes and reduced down to 1GB. I was wondering if you can you provide an explanation, line by line what is going on for the group? I am new to python but I understand some of the basics.
\nBasically, I wanted to know why this works and something like just usingn pandas and dataframes crashes. Is it because you're only doing a pivot table and filtering on 5million records at a time?
\n[/quote]
\nTy sir. I will provide an explanation, line by line, but do keep in mind that I am new to Python too, you may just be a tad newer.
\nWhy this works and not using pandas, I think is because you try to create a dataframe of the entire dataset (before reducing). With ~8GB mem you probably want to read the data line by line or in chunks. It is possible that Pandas is faster than my code if you make smart use of 'chunksize' and possibly an iterator.
\nBTW: Are you Neil Patel from Seattle? If so, I am a big fan, sir!
", "rawMarkdown": "", "votes": null }, { "id": "44373", "postDate": "05/11/2014 13:37:20", "content": "Thanks
", "rawMarkdown": "", "votes": null }, { "id": "44563", "postDate": "05/14/2014 23:37:18", "content": "Triskelion awesome starter code. Interestingly I was implementing my own version of this code but noticed it was running substantially slower than original. Turns out, I was defining the "offer" as an array rather than a dictionary which made a huge difference in the speed of the code. I am relatively new to python and was wondering if anyone could comment as to why this occurs.
\n\noffer = [] #runs slow
\noffer = {} # runs quickly
", "rawMarkdown": "", "votes": null }, { "id": "44567", "postDate": "05/15/2014 01:55:50", "content": "How do you figure out that the performance issue of your program is due to list? (Did you use a profiler or ...)
\nSince offer.csv is not large, I doubt whether using list to store the offers will slow down the program substantially
", "rawMarkdown": "", "votes": null }, { "id": "44575", "postDate": "05/15/2014 04:11:33", "content": "There is a time stamp within the code. It also reports the row which it is on. It clearly was taking longer to run the script based on observation. You are correct, offer.csv is small; however transactions.csv is very large. I believe the issue to be how the created dictionary from offer.csv is used to cross reference against the data in the transaction file when the for loop to check to see if some value from offer is found within the transactions file.
", "rawMarkdown": "", "votes": null }, { "id": "44577", "postDate": "05/15/2014 06:35:57", "content": "Hmm, maybe you are right.
\nBut why not just using dict? Don't waste your time on list and dict, after all, it's not the major part of the competition
", "rawMarkdown": "", "votes": null }, { "id": "44619", "postDate": "05/15/2014 15:54:19", "content": "I ended up using a dict for the code, but it took me a little while to figure out that was the issue. I am mostly wondering why the code performs differently since the data within the dict or list is the same size.
", "rawMarkdown": "", "votes": null }, { "id": "44621", "postDate": "05/15/2014 16:04:00", "content": "You need to provide more information (i.e. link to your code) if you want an answer.
\nYou can not generally replace a list with a dict (and vice versa), since their construction will differ. You might be using:
\nif something in offers:
\nThis will be O(1) for dict, and O(n) for list. That is, to answer if some key is in a dictionary takes 1 unit of time, and to answer if some key is in a list takes some amount of time proportional to the amount of elements in the list.
", "rawMarkdown": "", "votes": null }, { "id": "44677", "postDate": "05/16/2014 18:42:04", "content": "[quote=Nolan Wilson;44563]
\nI am relatively new to python and was wondering if anyone could comment as to why this occurs.
\noffer = [] #runs slow
\noffer = {} # runs quickly
\n[/quote]
\n"If key in dictionary"
\nis faster than:
\n"If key in list".
\nFrom https://wiki.python.org/moin/PythonSpeed:
\nMembership testing with sets and dictionaries is much faster, O(1), than searching sequences, O(n). When testing "a in b", b should be a set or dictionary instead of a list or tuple.
\nWhy this is? I think this is (not an expert) because Python dictionaries are hash tables or hash mappings. Notes on the original implementation here. Like the above quote on Python speed best practices, checking if an item is a member of a hash table is near instant, in the case of a list, it has to check every member if it is the same, and so has to traverse the entire list in the worst case scenario (when the item is not in the list).
\nSource for DictObject "with comments" at: http://svn.python.org/view/python/trunk/Objects/dictobject.c?view=markup
\nSince the membership check occurs in the loop that scans every transaction record, it is run ~350million times. That extra 0.00001 second it takes for a check may then turn into an hour.
", "rawMarkdown": "", "votes": null }, { "id": "44678", "postDate": "05/16/2014 18:48:27", "content": "dict is always faster than the list in python (comparable for small data)
", "rawMarkdown": "", "votes": null }, { "id": "44771", "postDate": "05/18/2014 07:22:25", "content": "Hello, do you know if it is possible to reduce the file size only using R? I don't have Unix. :(
\nI tried to read the R transactions file, however it failed as the file is too big to read. :(
\n\nThanks
", "rawMarkdown": "", "votes": null }, { "id": "44793", "postDate": "05/18/2014 19:47:20", "content": "There are Unix commands available for Windows. Here is one source: http://unxutils.sourceforge.net/
", "rawMarkdown": "", "votes": null }, { "id": "45982", "postDate": "05/21/2014 10:33:07", "content": "[quote=Nubee;44771]
\nHello, do you know if it is possible to reduce the file size only using R? I don't have Unix. :(
\nI tried to read the R transactions file, however it failed as the file is too big to read. :(
\nThanks
\n[/quote]
\nOption 1, split the file and process them in R and merge the results
\nOption 2, use read.table with skip and nrows multiple times and merger the results.
", "rawMarkdown": "", "votes": null }, { "id": "49266", "postDate": "06/18/2014 03:20:48", "content": "I think the R code below does the same thing as the data reduction being discussed in this thread, i.e. create a subset of transactions containing only the category corresponding to the offer which was received. But the result has 987,246 rows, considerably less than the 15.349.956 that Triskelion got with his Python code. As far as I know this code does not pick up categories corresponding to offers received by other customers. Perhaps the Python code does? Either that or this code is not working like I think it should. This is my first time using the data.table package.
\n\nlibrary(data.table)
\ntransactions <- fread("transactions.csv",integer64="character")
offers <- fread("offers.csv",integer64="character")
train <- fread("trainHistory.csv",integer64="character")
setkey(offers,offer,category)
setkey(train,offer)
train <- merge(train,offers,all.x=TRUE)
setkey(transactions,id,category)
setkey(train,id,category)
train.transcat <- transactions[train]
dim(train.transcat); head(train.transcat)
#[1] 987246 21
", "rawMarkdown": "", "votes": null }, { "id": "49345", "postDate": "06/20/2014 18:37:17", "content": "
I have a laptop with 4 gigs of RAM and I am trying to read the transaction.csv file using the read.csv.ffdf command.... "read.csv.ffdf(file="transactions.csv", header=TRUE, VERBOSE=TRUE, nrows=20000, next.rows=500000, colClasses=NA)".... I can read rows starting from 1 itself to whatever no. I wish which is not a problem but I am not able to set specific no. of rows as arguments in this command... like I want to read no. of rows starting from 10000 to 20000... please suggest some other command where I can set the no. of row to start with... Also I dont I have an access to UNIX...!
", "rawMarkdown": "", "votes": null }, { "id": "50211", "postDate": "07/09/2014 16:00:58", "content": "you can also rent an EC2 HANA system (in-memory database from SAP), load the data and do everything via SQL.
\nQueries on the 300Mio rows run in sub-second response times... :-)
\n\nbr, WM
", "rawMarkdown": "", "votes": null }, { "id": "50258", "postDate": "07/10/2014 09:41:17", "content": "[quote=Sarah Hardy;49266]
\nI think the R code below does the same thing as the data reduction being discussed in this thread, i.e. create a subset of transactions containing only the category corresponding to the offer which was received. But the result has 987,246 rows, considerably less than the 15.349.956 that Triskelion got with his Python code. As far as I know this code does not pick up categories corresponding to offers received by other customers. Perhaps the Python code does? Either that or this code is not working like I think it should. This is my first time using the data.table package.
\nlibrary(data.table)
\ntransactions <- fread("transactions.csv",integer64="character")
offers <- fread("offers.csv",integer64="character")
train <- fread("trainHistory.csv",integer64="character")
setkey(offers,offer,category)
setkey(train,offer)
train <- merge(train,offers,all.x=TRUE)
setkey(transactions,id,category)
setkey(train,id,category)
train.transcat <- transactions[train]
dim(train.transcat); head(train.transcat)
#[1] 987246 21
\n
[/quote]
\n\nI think the data.table merge doesn't work with integer64
\nsee for instance: http://r.789695.n4.nabble.com/integer64-group-by-doesn-t-find-all-groups-td4684906.html
\nI found it does seem to work if you convert the integer64's to double before the merge.
", "rawMarkdown": "", "votes": null }, { "id": "62675", "postDate": "01/13/2015 04:29:40", "content": "use this :
\nawk -F',' '$4 == 9115 || $4 == 9909 || $4 == 3203 || $4 == 5558 || $4 == 4401 || $4 == 1703 || $4 == 1726 || $4 == 3504 || $4 == 3509 || $4 == 9909 || $4 == 5122 || $4 == 5616 || $4 == 5619 || $4 == 2202 || $4 == 2119 || $4 == 6202 || $4 == 5824 || $4 == 799 || $4 == 4517 || $4 == 7205 || $4 == 706' transactions.csv > trans_cat.csv
", "rawMarkdown": "", "votes": null }, { "id": "69479", "postDate": "04/02/2015 22:22:32", "content": "Hi there,
\nI starting the project just for practice. But I'm having troubles I dont how to start. Can someone give me some hints.
\nI'll appreciate it!
\nThanks,
", "rawMarkdown": "", "votes": null }, { "id": "69961", "postDate": "04/08/2015 00:36:22", "content": "[quote=Daniel Medina;69479]
\nHi there,
\nI starting the project just for practice. But I'm having troubles I dont how to start. Can someone give me some hints.
\nI'll appreciate it!
\nThanks,
\n[/quote]
\nYou can download Python (http://continuum.io/downloads) and run BreakfastPirate's reduction code to get the transaction dataset down to a ~1GB size.
\nTo reduce from 1GB down to an even smaller size, you can use the following R code:
\nnrow_trans_smaller <- round(nrow(trans)*.05)
set.seed(201)
rowIndicies_trans_smaller <- sample(1:15349956, size = nrow_trans_smaller, replace = FALSE)
trans_smaller <- trans[rowIndicies_trans_smaller, ]
write.csv(trans_smaller, "trans_smaller-5%.csv")
Hope that helps you get started!
", "rawMarkdown": "", "votes": null }, { "id": "116222", "postDate": "04/22/2016 15:25:20", "content": "mark!
", "rawMarkdown": "mark!", "votes": null }, { "id": "653454", "postDate": "10/20/2019 13:42:22", "content": "I am using the following commands . It is throwing the following error \"grep: transactions: No such file or directory\".\nAnd extracted csv was 14.7MB. I am using google colab.
", "rawMarkdown": "I am using the following commands . It is throwing the following error \"grep: transactions: No such file or directory\".\nAnd extracted csv was 14.7MB. I am using google colab.", "votes": null } ], "comments": [ { "id": 41735, "author_name": "abhishek", "author_url": "", "post_date": "04/11/2014 16:10:42", "content": "Use, awk -F, '$4 = 706' transactions > trans_cat.csv and so on to avoid junk rows
", "votes": null, "replies": [] }, { "id": 41752, "author_name": "shuvrobiswas", "author_url": "", "post_date": "04/11/2014 20:03:51", "content": "My machine can't read more than about 50 MB, at once. Just curious what kind of rigs professionals have in place? Do you guys use clusters/hadoop?
", "votes": null, "replies": [] }, { "id": 41758, "author_name": "triskelion", "author_url": "", "post_date": "04/11/2014 22:10:53", "content": "You can do most challenges on Kaggle on a budget laptop. SSD helps. 16GB memory helps. Some professionals which I envy have access to clusters and servers, but that is not needed to win a contest. If nothing else, you learn how to be more resourceful (using less resources) and care more about speed.
", "votes": null, "replies": [] }, { "id": 41764, "author_name": "breakfastpirate", "author_url": "", "post_date": "04/11/2014 22:34:09", "content": "I agree with Triskelion. I am able to use my 8GB laptop for most Kaggle contests. If there is a contest that requires a large amount of memory, I sometimes rent an Amazon EC2 instance with 32GB for a couple weeks.
", "votes": null, "replies": [] }, { "id": 41774, "author_name": "breakfastpirate", "author_url": "", "post_date": "04/12/2014 02:29:31", "content": "I've also found that another useful subset is: only transactions where the company was a company on at least one of the offers. This was also about 1GB. There is obviously some overlap between this and the category subset. What would be best would be to have the union of these without duplicates.
", "votes": null, "replies": [] }, { "id": 41820, "author_name": "triskelion", "author_url": "", "post_date": "04/12/2014 22:56:44", "content": "In Python to reduce from 20GB to about 1GB (349.655.789 lines to 15.349.956 lines) or "the category subset" as BreakfastPirate calls it:
\nfrom datetime import datetime
loc_offers = "kaggle_shop\\\\offers.csv"
loc_transactions = "kaggle_shop\\\\transactions.csv"
loc_reduced = "kaggle_shop\\\\reduced2.csv" # will be created
def reduce_data(loc_offers, loc_transactions, loc_reduced):
start = datetime.now()
#get all categories on offer in a dict
offers = {}
for e, line in enumerate( open(loc_offers) ):
offers[ line.split(",")[1] ] = 1
#open output file
with open(loc_reduced, "wb") as outfile:
#go through transactions file and reduce
reduced = 0
for e, line in enumerate( open(loc_transactions) ):
if e == 0:
outfile.write( line ) #print header
else:
#only write when category in offers dict
if line.split(",")[3] in offers:
outfile.write( line )
reduced += 1
#progress
if e % 5000000 == 0:
print e, reduced, datetime.now() - start
print e, reduced, datetime.now() - start
reduce_data(loc_offers, loc_transactions, loc_reduced)
if you want to reduce the data with company, change:
\noffers[ line.split(",")[1] ] = 1
to:
\noffers[ line.split(",")[3] ] = 1
and:
\nif line.split(",")[3] in offers
to:
\nif line.split(",")[4] in offers
The category+company union is about 27 million lines (1.6Gb).
", "votes": null, "replies": [] }, { "id": 41821, "author_name": "piotrkuchta", "author_url": "", "post_date": "04/12/2014 23:11:24", "content": "Python is good for these kind of tasks - the standard CPython is pretty slow, but if your script uses only standard libraries and doesn't use numpy, sklearn, etc. then you should be able to run it with pypy, which is really fast, can be easily an order of magnitude faster than CPython. I think Triskelion's script would take about 10 minutes to process the transactions data on an average PC (I've written something similar to reduce the amount of data).
", "votes": null, "replies": [] }, { "id": 41822, "author_name": "triskelion", "author_url": "", "post_date": "04/12/2014 23:20:24", "content": "Yup, about 10 minutes to write, about 10 minutes to run. Not the fastest, but should work on any PC (takes max 6mb memory).
", "votes": null, "replies": [] }, { "id": 41842, "author_name": "nizhibitsky", "author_url": "", "post_date": "04/13/2014 11:32:55", "content": "It takes exactly 3:29 with both PyPy and CPython to run that filtering script on my laptop.
", "votes": null, "replies": [] }, { "id": 41850, "author_name": "jfkingiii", "author_url": "", "post_date": "04/13/2014 16:44:47", "content": "Transaction file summary statistics:
\n\n349655789 records
\ntotal purchasequantity 584540514
\ntotal purchaseamount 1568488398
", "votes": null, "replies": [] }, { "id": 42660, "author_name": "webscale", "author_url": "", "post_date": "04/22/2014 01:09:09", "content": "My recommendation - Download Datameer and get 15 days trial edition.
\nCreate one workbook in datameer that merges the offers + trainhistory + transactions.
\nYou are ready to go
", "votes": null, "replies": [] }, { "id": 43083, "author_name": "tonidm", "author_url": "", "post_date": "04/26/2014 16:03:13", "content": "hi! and how did you do that?
\nwhich variables did you use to merge the 3 tables??? the relation is 1to1??
", "votes": null, "replies": [] }, { "id": 43097, "author_name": "zachmayer", "author_url": "", "post_date": "04/26/2014 19:04:49", "content": "[quote=neeraj;42660]
\nMy recommendation - Download Datameer and get 15 days trial edition.
\nCreate one workbook in datameer that merges the offers + trainhistory + transactions.
\nYou are ready to go
\n[/quote]
\nWhy pay for commercial software when R and python are free?
", "votes": null, "replies": [] }, { "id": 43098, "author_name": "jamespaul", "author_url": "", "post_date": "04/26/2014 19:21:12", "content": "Data sets of this size, might be larger than the memory of the computer could hold. Open source R will may fail in such scenarios. (Unless you use packages like ff/bigmemory/bigglm). Revolution R Enterprise, uses external memory algorithms have no data limits, and is easier to use than ff/bigmemory/bigglm.
\nIt is available for Kaggle users for free for participating in Kaggle contests. Download here
\nhttp://info.revolutionanalytics.com/Kaggle.html
\n[post edited to incorporate comments from Zach]
", "votes": null, "replies": [] }, { "id": 43103, "author_name": "zachmayer", "author_url": "", "post_date": "04/26/2014 20:18:18", "content": "[quote=James Paul;43098]
\nData sets of this size, might be larger than the memory of the computer could hold. Open source R will fail in such scenarios. Revolution R Enteprise, uses external memory algorithms have no data limits.
\nIt is available for Kaggle users for free for participating in Kaggle contests. Download here
\nhttp://info.revolutionanalytics.com/Kaggle.html
\n[/quote]
\nThe dataset for this competition will fit in memory on a computer with 16 gigs of RAM. Furthermore, open source R can use out-of-memory algorithms, for example with the bigmemory and ff packages.
\nDon't get me wrong, I love Revolution R: I have the t-shirt and foreach is one of my favorite packages. But open-source R is definitely up for this challenge.
", "votes": null, "replies": [] }, { "id": 43416, "author_name": "vateeshc", "author_url": "", "post_date": "05/01/2014 01:20:18", "content": "Thank you for referring to revolutionanalytics (RA). I downloaded their software and attempted to import the entire transactions file. But ran out of memory as the import was being handled locally. I have a windows xp 4GB RAM laptop. Does anyone know if we can leverage RA's remote computing capability for kaggle comps? and if so, will appreciate any info on how to do this. On the side I am also giving the ff package a go as well but again limited by memory issues. And I am also aware of other opportunities to scale down data size from previous forum contributions.
", "votes": null, "replies": [] }, { "id": 43417, "author_name": "jamespaul", "author_url": "", "post_date": "05/01/2014 01:40:26", "content": "Please contact support@revolutionanalytics.com with your specific error. I have a 16GB laptop and I imported the data easily. I used the following commands to import to the Extended Data Frame format.
\noffers<-rxImport(inData="offers.csv", outFile="offers.xdf", stringsAsFactors=TRUE, missingValueString="M")
trainHistory<-rxImport(inData="trainHistory.csv", outFile="trainHistory.xdf", stringsAsFactors=TRUE, missingValueString="M")
tx<-rxImport(inData="transactions.csv", outFile="tx.xdf", stringsAsFactors=TRUE, missingValueString="M") (It does take time to import this one)
You can merge/combine the .xdf files to proceed with the next steps. (Use rxDataStep command)
\nThe user guide has a number of commonly used scenarios in manipulating data.
\nYou can also use Revolution R Enterprise in AWS. (You can get upto 64GB RAM machines)
", "votes": null, "replies": [] }, { "id": 43489, "author_name": "zachmayer", "author_url": "", "post_date": "05/02/2014 01:57:55", "content": ""fread" from the data.table package is also sufficient to load the full dataset in open source R (at least on a mac).
", "votes": null, "replies": [] }, { "id": 43522, "author_name": "manueldiaz", "author_url": "", "post_date": "05/02/2014 18:21:10", "content": "I have a windows xp laptop 2GB RAM, and I get good results in Kaggle competitions ... even a winning price ... I use speed (Kaggle user license) and python ...
\nI think this enough for this competition, but needs a careful treatment of data ...
", "votes": null, "replies": [] }, { "id": 43626, "author_name": "neilpatel", "author_url": "", "post_date": "05/03/2014 19:34:08", "content": "@Triskelion
\nGreat reduction code sir!
\nI ran it in a little over 6 minutes and reduced down to 1GB. I was wondering if you can you provide an explanation, line by line what is going on for the group? I am new to python but I understand some of the basics.
\nBasically, I wanted to know why this works and something like just usingn pandas and dataframes crashes. Is it because you're only doing a pivot table and filtering on 5million records at a time?
", "votes": null, "replies": [] }, { "id": 43646, "author_name": "triskelion", "author_url": "", "post_date": "05/03/2014 23:32:04", "content": "[quote=Neil Patel;43626]
\n@Triskelion
\nGreat reduction code sir!
\nI ran it in a little over 6 minutes and reduced down to 1GB. I was wondering if you can you provide an explanation, line by line what is going on for the group? I am new to python but I understand some of the basics.
\nBasically, I wanted to know why this works and something like just usingn pandas and dataframes crashes. Is it because you're only doing a pivot table and filtering on 5million records at a time?
\n[/quote]
\nTy sir. I will provide an explanation, line by line, but do keep in mind that I am new to Python too, you may just be a tad newer.
\nWhy this works and not using pandas, I think is because you try to create a dataframe of the entire dataset (before reducing). With ~8GB mem you probably want to read the data line by line or in chunks. It is possible that Pandas is faster than my code if you make smart use of 'chunksize' and possibly an iterator.
\nBTW: Are you Neil Patel from Seattle? If so, I am a big fan, sir!
", "votes": null, "replies": [] }, { "id": 44373, "author_name": "user191111", "author_url": "", "post_date": "05/11/2014 13:37:20", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 44563, "author_name": "nolanwilson", "author_url": "", "post_date": "05/14/2014 23:37:18", "content": "Triskelion awesome starter code. Interestingly I was implementing my own version of this code but noticed it was running substantially slower than original. Turns out, I was defining the "offer" as an array rather than a dictionary which made a huge difference in the speed of the code. I am relatively new to python and was wondering if anyone could comment as to why this occurs.
\n\noffer = [] #runs slow
\noffer = {} # runs quickly
", "votes": null, "replies": [] }, { "id": 44567, "author_name": "user191111", "author_url": "", "post_date": "05/15/2014 01:55:50", "content": "How do you figure out that the performance issue of your program is due to list? (Did you use a profiler or ...)
\nSince offer.csv is not large, I doubt whether using list to store the offers will slow down the program substantially
", "votes": null, "replies": [] }, { "id": 44575, "author_name": "nolanwilson", "author_url": "", "post_date": "05/15/2014 04:11:33", "content": "There is a time stamp within the code. It also reports the row which it is on. It clearly was taking longer to run the script based on observation. You are correct, offer.csv is small; however transactions.csv is very large. I believe the issue to be how the created dictionary from offer.csv is used to cross reference against the data in the transaction file when the for loop to check to see if some value from offer is found within the transactions file.
", "votes": null, "replies": [] }, { "id": 44577, "author_name": "user191111", "author_url": "", "post_date": "05/15/2014 06:35:57", "content": "Hmm, maybe you are right.
\nBut why not just using dict? Don't waste your time on list and dict, after all, it's not the major part of the competition
", "votes": null, "replies": [] }, { "id": 44619, "author_name": "nolanwilson", "author_url": "", "post_date": "05/15/2014 15:54:19", "content": "I ended up using a dict for the code, but it took me a little while to figure out that was the issue. I am mostly wondering why the code performs differently since the data within the dict or list is the same size.
", "votes": null, "replies": [] }, { "id": 44621, "author_name": "jkalmeij", "author_url": "", "post_date": "05/15/2014 16:04:00", "content": "You need to provide more information (i.e. link to your code) if you want an answer.
\nYou can not generally replace a list with a dict (and vice versa), since their construction will differ. You might be using:
\nif something in offers:
\nThis will be O(1) for dict, and O(n) for list. That is, to answer if some key is in a dictionary takes 1 unit of time, and to answer if some key is in a list takes some amount of time proportional to the amount of elements in the list.
", "votes": null, "replies": [] }, { "id": 44677, "author_name": "triskelion", "author_url": "", "post_date": "05/16/2014 18:42:04", "content": "[quote=Nolan Wilson;44563]
\nI am relatively new to python and was wondering if anyone could comment as to why this occurs.
\noffer = [] #runs slow
\noffer = {} # runs quickly
\n[/quote]
\n"If key in dictionary"
\nis faster than:
\n"If key in list".
\nFrom https://wiki.python.org/moin/PythonSpeed:
\nMembership testing with sets and dictionaries is much faster, O(1), than searching sequences, O(n). When testing "a in b", b should be a set or dictionary instead of a list or tuple.
\nWhy this is? I think this is (not an expert) because Python dictionaries are hash tables or hash mappings. Notes on the original implementation here. Like the above quote on Python speed best practices, checking if an item is a member of a hash table is near instant, in the case of a list, it has to check every member if it is the same, and so has to traverse the entire list in the worst case scenario (when the item is not in the list).
\nSource for DictObject "with comments" at: http://svn.python.org/view/python/trunk/Objects/dictobject.c?view=markup
\nSince the membership check occurs in the loop that scans every transaction record, it is run ~350million times. That extra 0.00001 second it takes for a check may then turn into an hour.
", "votes": null, "replies": [] }, { "id": 44678, "author_name": "abhishek", "author_url": "", "post_date": "05/16/2014 18:48:27", "content": "dict is always faster than the list in python (comparable for small data)
", "votes": null, "replies": [] }, { "id": 44771, "author_name": "nubeee", "author_url": "", "post_date": "05/18/2014 07:22:25", "content": "Hello, do you know if it is possible to reduce the file size only using R? I don't have Unix. :(
\nI tried to read the R transactions file, however it failed as the file is too big to read. :(
\n\nThanks
", "votes": null, "replies": [] }, { "id": 44793, "author_name": "breakfastpirate", "author_url": "", "post_date": "05/18/2014 19:47:20", "content": "There are Unix commands available for Windows. Here is one source: http://unxutils.sourceforge.net/
", "votes": null, "replies": [] }, { "id": 45982, "author_name": "chitrasen", "author_url": "", "post_date": "05/21/2014 10:33:07", "content": "[quote=Nubee;44771]
\nHello, do you know if it is possible to reduce the file size only using R? I don't have Unix. :(
\nI tried to read the R transactions file, however it failed as the file is too big to read. :(
\nThanks
\n[/quote]
\nOption 1, split the file and process them in R and merge the results
\nOption 2, use read.table with skip and nrows multiple times and merger the results.
", "votes": null, "replies": [] }, { "id": 49266, "author_name": "swhardy", "author_url": "", "post_date": "06/18/2014 03:20:48", "content": "I think the R code below does the same thing as the data reduction being discussed in this thread, i.e. create a subset of transactions containing only the category corresponding to the offer which was received. But the result has 987,246 rows, considerably less than the 15.349.956 that Triskelion got with his Python code. As far as I know this code does not pick up categories corresponding to offers received by other customers. Perhaps the Python code does? Either that or this code is not working like I think it should. This is my first time using the data.table package.
\n\nlibrary(data.table)
\ntransactions <- fread("transactions.csv",integer64="character")
offers <- fread("offers.csv",integer64="character")
train <- fread("trainHistory.csv",integer64="character")
setkey(offers,offer,category)
setkey(train,offer)
train <- merge(train,offers,all.x=TRUE)
setkey(transactions,id,category)
setkey(train,id,category)
train.transcat <- transactions[train]
dim(train.transcat); head(train.transcat)
#[1] 987246 21
", "votes": null, "replies": [] }, { "id": 49345, "author_name": "shreedharpawar", "author_url": "", "post_date": "06/20/2014 18:37:17", "content": "
I have a laptop with 4 gigs of RAM and I am trying to read the transaction.csv file using the read.csv.ffdf command.... "read.csv.ffdf(file="transactions.csv", header=TRUE, VERBOSE=TRUE, nrows=20000, next.rows=500000, colClasses=NA)".... I can read rows starting from 1 itself to whatever no. I wish which is not a problem but I am not able to set specific no. of rows as arguments in this command... like I want to read no. of rows starting from 10000 to 20000... please suggest some other command where I can set the no. of row to start with... Also I dont I have an access to UNIX...!
", "votes": null, "replies": [] }, { "id": 50211, "author_name": "analyticsanimal", "author_url": "", "post_date": "07/09/2014 16:00:58", "content": "you can also rent an EC2 HANA system (in-memory database from SAP), load the data and do everything via SQL.
\nQueries on the 300Mio rows run in sub-second response times... :-)
\n\nbr, WM
", "votes": null, "replies": [] }, { "id": 50258, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "07/10/2014 09:41:17", "content": "[quote=Sarah Hardy;49266]
\nI think the R code below does the same thing as the data reduction being discussed in this thread, i.e. create a subset of transactions containing only the category corresponding to the offer which was received. But the result has 987,246 rows, considerably less than the 15.349.956 that Triskelion got with his Python code. As far as I know this code does not pick up categories corresponding to offers received by other customers. Perhaps the Python code does? Either that or this code is not working like I think it should. This is my first time using the data.table package.
\nlibrary(data.table)
\ntransactions <- fread("transactions.csv",integer64="character")
offers <- fread("offers.csv",integer64="character")
train <- fread("trainHistory.csv",integer64="character")
setkey(offers,offer,category)
setkey(train,offer)
train <- merge(train,offers,all.x=TRUE)
setkey(transactions,id,category)
setkey(train,id,category)
train.transcat <- transactions[train]
dim(train.transcat); head(train.transcat)
#[1] 987246 21
\n
[/quote]
\n\nI think the data.table merge doesn't work with integer64
\nsee for instance: http://r.789695.n4.nabble.com/integer64-group-by-doesn-t-find-all-groups-td4684906.html
\nI found it does seem to work if you convert the integer64's to double before the merge.
", "votes": null, "replies": [] }, { "id": 62675, "author_name": "vikasveshishth", "author_url": "", "post_date": "01/13/2015 04:29:40", "content": "use this :
\nawk -F',' '$4 == 9115 || $4 == 9909 || $4 == 3203 || $4 == 5558 || $4 == 4401 || $4 == 1703 || $4 == 1726 || $4 == 3504 || $4 == 3509 || $4 == 9909 || $4 == 5122 || $4 == 5616 || $4 == 5619 || $4 == 2202 || $4 == 2119 || $4 == 6202 || $4 == 5824 || $4 == 799 || $4 == 4517 || $4 == 7205 || $4 == 706' transactions.csv > trans_cat.csv
", "votes": null, "replies": [] }, { "id": 69479, "author_name": "danielmedina", "author_url": "", "post_date": "04/02/2015 22:22:32", "content": "Hi there,
\nI starting the project just for practice. But I'm having troubles I dont how to start. Can someone give me some hints.
\nI'll appreciate it!
\nThanks,
", "votes": null, "replies": [] }, { "id": 69961, "author_name": "dluzansky", "author_url": "", "post_date": "04/08/2015 00:36:22", "content": "[quote=Daniel Medina;69479]
\nHi there,
\nI starting the project just for practice. But I'm having troubles I dont how to start. Can someone give me some hints.
\nI'll appreciate it!
\nThanks,
\n[/quote]
\nYou can download Python (http://continuum.io/downloads) and run BreakfastPirate's reduction code to get the transaction dataset down to a ~1GB size.
\nTo reduce from 1GB down to an even smaller size, you can use the following R code:
\nnrow_trans_smaller <- round(nrow(trans)*.05)
set.seed(201)
rowIndicies_trans_smaller <- sample(1:15349956, size = nrow_trans_smaller, replace = FALSE)
trans_smaller <- trans[rowIndicies_trans_smaller, ]
write.csv(trans_smaller, "trans_smaller-5%.csv")
Hope that helps you get started!
", "votes": null, "replies": [] }, { "id": 116222, "author_name": "leyue260351", "author_url": "", "post_date": "04/22/2016 15:25:20", "content": "mark!
", "votes": null, "replies": [] }, { "id": 653454, "author_name": "subho22", "author_url": "", "post_date": "10/20/2019 13:42:22", "content": "I am using the following commands . It is throwing the following error \"grep: transactions: No such file or directory\".\nAnd extracted csv was 14.7MB. I am using google colab.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "41729": "", "41735": "", "41752": "", "41758": "", "41764": "", "41774": "", "41820": "", "41821": "", "41822": "", "41842": "", "41850": "", "42660": "", "43083": "", "43097": "", "43098": "", "43103": "", "43416": "", "43417": "", "43489": "", "43522": "", "43626": "", "43646": "", "44373": "", "44563": "", "44567": "", "44575": "", "44577": "", "44619": "", "44621": "", "44677": "", "44678": "", "44771": "", "44793": "", "45982": "", "49266": "", "49345": "", "50211": "", "50258": "", "62675": "", "69479": "", "69961": "", "116222": "mark!", "653454": "I am using the following commands . It is throwing the following error \"grep: transactions: No such file or directory\".\nAnd extracted csv was 14.7MB. I am using google colab." }, "source": "meta" }