{ "id": 10757, "title": "What's the size of data?", "url": "/competitions/avazu-ctr-prediction/discussion/10757", "author_name": "", "post_date": "2014-10-27T02:21:44.740Z", "votes": null, "comment_count": 26, "views": 7575, "content": "

Hello guys, I want to know the data size, but since the host is preparing the data, I don't see any information about the size.

\n

Does anyone know how much will be the data size?

\n\n

By the way, I am a TA of a Data Mining course, I am looking for interesting competition like this one for student's final project. Do you guys have any advices?

", "messages": [ { "id": "56815", "postDate": "10/27/2014 02:21:44", "content": "

Hello guys, I want to know the data size, but since the host is preparing the data, I don't see any information about the size.

\n

Does anyone know how much will be the data size?

\n\n

By the way, I am a TA of a Data Mining course, I am looking for interesting competition like this one for student's final project. Do you guys have any advices?

", "rawMarkdown": "", "votes": null }, { "id": "56816", "postDate": "10/27/2014 02:36:14", "content": "

[quote=Chun-Hao Chang;56815]

\n

Hello guys, I want to know the data size, but since the host is preparing the data, I don't see any information about the size.

\n

Does anyone know how much will be the data size?

\n

By the way, I am a TA of a Data Mining course, I am looking for interesting competition like this one for student's final project. Do you guys have any advices?

\n

[/quote]

\n\n

Hi Chun Hao,

\n\n

The training data set will be around 1.6GB, and test data set will about 170MB.

\n

The correct data set will be ready soon in one or two days.

\n\n

Thanks for your support and patience!

", "rawMarkdown": "", "votes": null }, { "id": "56819", "postDate": "10/27/2014 03:32:10", "content": "

Thanks for the super quick response！

\n

It helps a lot.

", "rawMarkdown": "", "votes": null }, { "id": "56824", "postDate": "10/27/2014 05:30:18", "content": "", "rawMarkdown": "", "votes": null }, { "id": "56826", "postDate": "10/27/2014 05:54:35", "content": "

Dear all,

\n

I'm using open source R language for modelling purpose. R doesn't work with large data. I don't know what software to use in order to take part in this competition which has more than a GB of data. Could anyone please help me out by suggesting some ideas.

\n

Thank you

", "rawMarkdown": "", "votes": null }, { "id": "56828", "postDate": "10/27/2014 06:00:59", "content": "

Hey asho_sats:

\n

R loads every things into memory, and it is not a efficient language. 

\n

You might need a general purposed language like Python, it's quite famous in Data Science community. 

\n

As for data size issue, if the size is too big to store in memory. You can either store data in a Database or store in a file 

", "rawMarkdown": "", "votes": null }, { "id": "56829", "postDate": "10/27/2014 06:06:51", "content": "

there are 48 millian records in the train data, about 8.2G after decompression

\n

and 4.8 millian in the test data, about 831M after decompression

", "rawMarkdown": "", "votes": null }, { "id": "56830", "postDate": "10/27/2014 06:13:08", "content": "

wow, that's huge, I am more excited right now :D!!!!

", "rawMarkdown": "", "votes": null }, { "id": "56831", "postDate": "10/27/2014 06:17:54", "content": "

The data size is a bit of a problem for me too. I think there are a couple reasonable ways to proceed, but I guess we'll see what works best. 

\n

One way is to use exclusively 'online learners' that can just take new data rows one at a time. In that case, you never have to load the entire thing into memory at once. This is a little limiting as far as what kinds of algorithms you can use, but this seems to be a pretty active area of development so you can find a bunch of research papers making online versions of stuff. The downside here is that this could mean a lot of custom implementation, which slows down one's ability to try a bunch of possibilities without committing too much time until they pan out.

\n

The other way, which seemed like a better bet to me, is to split the data up into smaller chunks and then ensemble the results. Since there's a good chunk of data, it means you can do a lot more with hold-out sets/etc than you could in a more data-starved case, so some techniques that'd otherwise be prone to over-fitting might be feasible here. If nothing else, this is probably a good first thing to do just to get a feeling for what class of algorithms are performing best with this data-set, before looking into writing up online versions of those things.

", "rawMarkdown": "", "votes": null }, { "id": "56832", "postDate": "10/27/2014 06:24:28", "content": "

Hi, I have been looking into packages that allow R to efficiently manage large data such as ff or bigmemory. However if you just want to have a quick look at the data and start playing with them you may want to try (from a shell):

\n

shuf -n 100000 train.csv > small.tmp

\n

head -n 1 train.csv > h

\n

cat h small.tmp > small.csv

\n

and you have a 100000 line subsample. I wonder whether the sampling is actually random, though.

", "rawMarkdown": "", "votes": null }, { "id": "57058", "postDate": "10/30/2014 18:19:18", "content": "

I am new to competitions with such large data set. I intend to use scikit-learn algos to begin with. As such I want to know whether I should proceed with this 8GB data on a mac with 4GB ram and core i5? Or should I look for a more powerful mac?

", "rawMarkdown": "", "votes": null }, { "id": "57084", "postDate": "10/31/2014 02:10:22", "content": "

To Pronojit:

\n

I think not all the algorithms on Scikit-learn require you to load all data into memory at once.

\n

For example, you can try to load 10% of data and use that 10% to train your model. Repeat it ten times then you can get a fully trained model without crashing your memory. 

", "rawMarkdown": "", "votes": null }, { "id": "57096", "postDate": "10/31/2014 07:12:20", "content": "

To Chun-Hao Chang,

\n\n

"I think not all the algorithms on Scikit-learn require you to load all data into memory at once.

\n

For example, you can try to load 10% of data and use that 10% to train your model. Repeat it ten times then you can get a fully trained model without crashing your memory."

\n

Can you please explain me if I want to do prediction for any test data row (any test sample) out of these 10 models  which model  I should choose? Or I should use all models and check which prediction (either 1 or o) is coming most number of time and use that?

", "rawMarkdown": "", "votes": null }, { "id": "57098", "postDate": "10/31/2014 07:32:06", "content": "

Dear Aniket:

\n

I think you misunderstood my point, you don't have to build multiple models just because you can not load data into memory at once.

", "rawMarkdown": "", "votes": null }, { "id": "57099", "postDate": "10/31/2014 08:49:37", "content": "

To Chun-Hao Chang,

\n

 Thanks for reply, I again gone through your answer to Pronojit and as far as my understanding your are suggesting to iteratively update the model by  taking 10% data in each pass.Can you please tell me by which classifier or by which package this can be done.Thanks in advance.

", "rawMarkdown": "", "votes": null }, { "id": "57100", "postDate": "10/31/2014 09:15:48", "content": "

Here is an example using Python Scikit-learn

\n

https://gist.github.com/GBJim/2c69fc444d1f0c740ace

\n

In this example you can see that I can train my GNB model multiple times and then predict the test data. Similarly, you can load part of your data into memory and repeat it until you get a fully trained model.  

\n

But in this contest, don't use any classifier like the one I showed you. Because you need to compute a probability instead of giving a discrete tag.

", "rawMarkdown": "", "votes": null }, { "id": "57101", "postDate": "10/31/2014 09:30:36", "content": "

Thanks Again Chun-Hao Chang!!!!!

", "rawMarkdown": "", "votes": null }, { "id": "57111", "postDate": "10/31/2014 14:49:36", "content": "

You can get somehwere with R just using the built in utilities, but I agree it's not ideal. For example, this code fragment will create a series of tables of click against hour, which you can then merge yourself. It reads rows in 500k at a time (but that can be changed to suit your requirements)

\n\n

tl <- list()
n <- 1L
fin <- file("train_rev2.csv",open="r")
trainHeader <- readLines(fin,n=1L)
nchunk <- 500000L
while(TRUE){
read.csv(fin,nrows=nchunk,header=FALSE,colClasses="character") -> df.tmp
tl[[n]] <- table(df.tmp[,c("V2","V3")])
n <- n + 1L
}
close(fin)

\n

## tl now contains a list of tables with "hour" as the col names

\n

This should run fine as is with 512MB ram (probably less)

", "rawMarkdown": "", "votes": null }, { "id": "57117", "postDate": "10/31/2014 16:20:02", "content": "

Chun-Hao Chang, maybe there is something I'm not understanding from your post. There are indeed plenty of models you can imagine updating or ensembling, but as far as I know, if you call fit multiple times on a single instance of an sklearn class, it will overwrite the previous values of the model. If you look at the code (it's in sklearn/naive_bayes.py for your example), it's pretty unambiguous. The steps are:

\n\n

So, unless there are specific models which implement fit differently (I'm not aware of any), or you've patched sklearn, the code you've provided trains three models, throws out the first two, and then makes predictions based on the last model you trained.

\n

Have I missed something obvious?

", "rawMarkdown": "", "votes": null }, { "id": "57127", "postDate": "10/31/2014 18:49:25", "content": "

jdl37, I think you might be missing something in the code fragment... read.csv should include argument skip = n*500000, no?  Otherwise it'll just keep reading the first 500k lines?

", "rawMarkdown": "", "votes": null }, { "id": "57128", "postDate": "10/31/2014 18:50:26", "content": "

kinnskogr: Thank you for you pointed it out! I am really grateful about you check through my idea. 

\n

I have to say I didn't carefully check through the training methodology in scikit-learn and simply assumed GNB has incremental learning features. I am wrong, I hope you guys didn't go to the wrong direction because of my misleading post.

\n

Thankfully, Sklearn still has solutions in this kind of big data situation:

\n

Please check this document on Scikit-Learn

\n

http://scikit-learn.org/stable/modules/scaling_strategies.html

", "rawMarkdown": "", "votes": null }, { "id": "57131", "postDate": "10/31/2014 20:22:29", "content": "

[quote=Chun-Hao Chang;56815]

\n

Hello guys, I want to know the data size, but since the host is preparing the data, I don't see any information about the size.

\n

Does anyone know how much will be the data size?

\n

By the way, I am a TA of a Data Mining course, I am looking for interesting competition like this one for student's final project. Do you guys have any advices?

\n

[/quote]

\n

There have been some issues in some earlier competitions with student groups competing (e.g., Partly Sunny with a chance of hashtags), so I'd be sure to remind your students of the competition rules, especially these:

\n

One account per participant

\n

You cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.

\n\n

No private sharing outside teams

\n

Privately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.

", "rawMarkdown": "", "votes": null }, { "id": "57136", "postDate": "10/31/2014 21:16:06", "content": "

It has 47686351 observations and 27 variables. Is that right?

", "rawMarkdown": "", "votes": null }, { "id": "57141", "postDate": "10/31/2014 22:47:52", "content": "

[quote=kinnskogr;57117]

\n

Chun-Hao Chang, maybe there is something I'm not understanding from your post. There are indeed plenty of models you can imagine updating or ensembling, but as far as I know, if you call fit multiple times on a single instance of an sklearn class, it will overwrite the previous values of the model. If you look at the code (it's in sklearn/naive_bayes.py for your example), it's pretty unambiguous. The steps are:

\n\n

So, unless there are specific models which implement fit differently (I'm not aware of any), or you've patched sklearn, the code you've provided trains three models, throws out the first two, and then makes predictions based on the last model you trained.

\n

Have I missed something obvious?

\n

[/quote]

\n

Several sklearn classes have a partial fit method for mini-batch learning. The classes that I can recall are: SGDRegressor/Classifier, perceptron, all of the naive Bayes models and passive-aggressive models.

\n

I think you can call each with fit() the first time, and then partial_fit() subsequently, or you can call it with partial_fit() straight through, in which case you need to call it with the classes parameter, at least the first time. Here's some more info: sklearn incremental training.

", "rawMarkdown": "", "votes": null }, { "id": "57143", "postDate": "10/31/2014 23:37:58", "content": "

Hey David,

\n

   That's right. My initial post wasn't quite complete without mentioning those. Chun-Hao has also pointed to the methods under the Incremental Learning section of http://scikit-learn.org/stable/modules/scaling_strategies.html. The point I was raising was that by convention, the fit() method will overwrite the model parameters. The partial_fit() method, again by convention, allows updating (and thankfully, the sklearn maintainers are great about enforcing conventions).

\n

Also, you don't need to call fit() the first time, there's a check in partial_fit() to handle newly initialized instances.

\n

Cheers,

\n

- Emanuel

", "rawMarkdown": "", "votes": null }, { "id": "57234", "postDate": "11/03/2014 09:18:58", "content": "

@skwalas, not if the connection is opened in mode "r". If it was opened in another mode, that would be true.

", "rawMarkdown": "", "votes": null }, { "id": "57385", "postDate": "11/06/2014 05:52:06", "content": "

[quote=Nicholas Guttenberg;56831]

\n

The data size is a bit of a problem for me too. I think there are a couple reasonable ways to proceed, but I guess we'll see what works best. 

\n

One way is to use exclusively 'online learners' that can just take new data rows one at a time. In that case, you never have to load the entire thing into memory at once. This is a little limiting as far as what kinds of algorithms you can use, but this seems to be a pretty active area of development so you can find a bunch of research papers making online versions of stuff. The downside here is that this could mean a lot of custom implementation, which slows down one's ability to try a bunch of possibilities without committing too much time until they pan out.

\n

The other way, which seemed like a better bet to me, is to split the data up into smaller chunks and then ensemble the results. Since there's a good chunk of data, it means you can do a lot more with hold-out sets/etc than you could in a more data-starved case, so some techniques that'd otherwise be prone to over-fitting might be feasible here. If nothing else, this is probably a good first thing to do just to get a feeling for what class of algorithms are performing best with this data-set, before looking into writing up online versions of those things.

\n

[/quote]

\n

Yet another way, is to buy* a giant server with 32 gigabytes memory.

\n\n

*Or rent one on AWS.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 56816, "author_name": "stevewang0", "author_url": "", "post_date": "10/27/2014 02:36:14", "content": "

[quote=Chun-Hao Chang;56815]

\n

Hello guys, I want to know the data size, but since the host is preparing the data, I don't see any information about the size.

\n

Does anyone know how much will be the data size?

\n

By the way, I am a TA of a Data Mining course, I am looking for interesting competition like this one for student's final project. Do you guys have any advices?

\n

[/quote]

\n\n

Hi Chun Hao,

\n\n

The training data set will be around 1.6GB, and test data set will about 170MB.

\n

The correct data set will be ready soon in one or two days.

\n\n

Thanks for your support and patience!

", "votes": null, "replies": [] }, { "id": 56819, "author_name": "ccha97u", "author_url": "", "post_date": "10/27/2014 03:32:10", "content": "

Thanks for the super quick response！

\n

It helps a lot.

", "votes": null, "replies": [] }, { "id": 56824, "author_name": "asio555", "author_url": "", "post_date": "10/27/2014 05:30:18", "content": "", "votes": null, "replies": [] }, { "id": 56826, "author_name": "asio555", "author_url": "", "post_date": "10/27/2014 05:54:35", "content": "

Dear all,

\n

I'm using open source R language for modelling purpose. R doesn't work with large data. I don't know what software to use in order to take part in this competition which has more than a GB of data. Could anyone please help me out by suggesting some ideas.

\n

Thank you

", "votes": null, "replies": [] }, { "id": 56828, "author_name": "ccha97u", "author_url": "", "post_date": "10/27/2014 06:00:59", "content": "

Hey asho_sats:

\n

R loads every things into memory, and it is not a efficient language. 

\n

You might need a general purposed language like Python, it's quite famous in Data Science community. 

\n

As for data size issue, if the size is too big to store in memory. You can either store data in a Database or store in a file 

", "votes": null, "replies": [] }, { "id": 56829, "author_name": "stevewang0", "author_url": "", "post_date": "10/27/2014 06:06:51", "content": "

there are 48 millian records in the train data, about 8.2G after decompression

\n

and 4.8 millian in the test data, about 831M after decompression

", "votes": null, "replies": [] }, { "id": 56830, "author_name": "ccha97u", "author_url": "", "post_date": "10/27/2014 06:13:08", "content": "

wow, that's huge, I am more excited right now :D!!!!

", "votes": null, "replies": [] }, { "id": 56831, "author_name": "ngutten", "author_url": "", "post_date": "10/27/2014 06:17:54", "content": "

The data size is a bit of a problem for me too. I think there are a couple reasonable ways to proceed, but I guess we'll see what works best. 

\n

One way is to use exclusively 'online learners' that can just take new data rows one at a time. In that case, you never have to load the entire thing into memory at once. This is a little limiting as far as what kinds of algorithms you can use, but this seems to be a pretty active area of development so you can find a bunch of research papers making online versions of stuff. The downside here is that this could mean a lot of custom implementation, which slows down one's ability to try a bunch of possibilities without committing too much time until they pan out.

\n

The other way, which seemed like a better bet to me, is to split the data up into smaller chunks and then ensemble the results. Since there's a good chunk of data, it means you can do a lot more with hold-out sets/etc than you could in a more data-starved case, so some techniques that'd otherwise be prone to over-fitting might be feasible here. If nothing else, this is probably a good first thing to do just to get a feeling for what class of algorithms are performing best with this data-set, before looking into writing up online versions of those things.

", "votes": null, "replies": [] }, { "id": 56832, "author_name": "mariopasquato", "author_url": "", "post_date": "10/27/2014 06:24:28", "content": "

Hi, I have been looking into packages that allow R to efficiently manage large data such as ff or bigmemory. However if you just want to have a quick look at the data and start playing with them you may want to try (from a shell):

\n

shuf -n 100000 train.csv > small.tmp

\n

head -n 1 train.csv > h

\n

cat h small.tmp > small.csv

\n

and you have a 100000 line subsample. I wonder whether the sampling is actually random, though.

", "votes": null, "replies": [] }, { "id": 57058, "author_name": "pronojit", "author_url": "", "post_date": "10/30/2014 18:19:18", "content": "

I am new to competitions with such large data set. I intend to use scikit-learn algos to begin with. As such I want to know whether I should proceed with this 8GB data on a mac with 4GB ram and core i5? Or should I look for a more powerful mac?

", "votes": null, "replies": [] }, { "id": 57084, "author_name": "ccha97u", "author_url": "", "post_date": "10/31/2014 02:10:22", "content": "

To Pronojit:

\n

I think not all the algorithms on Scikit-learn require you to load all data into memory at once.

\n

For example, you can try to load 10% of data and use that 10% to train your model. Repeat it ten times then you can get a fully trained model without crashing your memory. 

", "votes": null, "replies": [] }, { "id": 57096, "author_name": "kapitsa2811", "author_url": "", "post_date": "10/31/2014 07:12:20", "content": "

To Chun-Hao Chang,

\n\n

"I think not all the algorithms on Scikit-learn require you to load all data into memory at once.

\n

For example, you can try to load 10% of data and use that 10% to train your model. Repeat it ten times then you can get a fully trained model without crashing your memory."

\n

Can you please explain me if I want to do prediction for any test data row (any test sample) out of these 10 models  which model  I should choose? Or I should use all models and check which prediction (either 1 or o) is coming most number of time and use that?

", "votes": null, "replies": [] }, { "id": 57098, "author_name": "ccha97u", "author_url": "", "post_date": "10/31/2014 07:32:06", "content": "

Dear Aniket:

\n

I think you misunderstood my point, you don't have to build multiple models just because you can not load data into memory at once.

", "votes": null, "replies": [] }, { "id": 57099, "author_name": "kapitsa2811", "author_url": "", "post_date": "10/31/2014 08:49:37", "content": "

To Chun-Hao Chang,

\n

 Thanks for reply, I again gone through your answer to Pronojit and as far as my understanding your are suggesting to iteratively update the model by  taking 10% data in each pass.Can you please tell me by which classifier or by which package this can be done.Thanks in advance.

", "votes": null, "replies": [] }, { "id": 57100, "author_name": "ccha97u", "author_url": "", "post_date": "10/31/2014 09:15:48", "content": "

Here is an example using Python Scikit-learn

\n

https://gist.github.com/GBJim/2c69fc444d1f0c740ace

\n

In this example you can see that I can train my GNB model multiple times and then predict the test data. Similarly, you can load part of your data into memory and repeat it until you get a fully trained model.  

\n

But in this contest, don't use any classifier like the one I showed you. Because you need to compute a probability instead of giving a discrete tag.

", "votes": null, "replies": [] }, { "id": 57101, "author_name": "kapitsa2811", "author_url": "", "post_date": "10/31/2014 09:30:36", "content": "

Thanks Again Chun-Hao Chang!!!!!

", "votes": null, "replies": [] }, { "id": 57111, "author_name": "jamesdlawrence", "author_url": "", "post_date": "10/31/2014 14:49:36", "content": "

You can get somehwere with R just using the built in utilities, but I agree it's not ideal. For example, this code fragment will create a series of tables of click against hour, which you can then merge yourself. It reads rows in 500k at a time (but that can be changed to suit your requirements)

\n\n

tl <- list()
n <- 1L
fin <- file("train_rev2.csv",open="r")
trainHeader <- readLines(fin,n=1L)
nchunk <- 500000L
while(TRUE){
read.csv(fin,nrows=nchunk,header=FALSE,colClasses="character") -> df.tmp
tl[[n]] <- table(df.tmp[,c("V2","V3")])
n <- n + 1L
}
close(fin)

\n

## tl now contains a list of tables with "hour" as the col names

\n

This should run fine as is with 512MB ram (probably less)

", "votes": null, "replies": [] }, { "id": 57117, "author_name": "kinnskogr", "author_url": "", "post_date": "10/31/2014 16:20:02", "content": "

Chun-Hao Chang, maybe there is something I'm not understanding from your post. There are indeed plenty of models you can imagine updating or ensembling, but as far as I know, if you call fit multiple times on a single instance of an sklearn class, it will overwrite the previous values of the model. If you look at the code (it's in sklearn/naive_bayes.py for your example), it's pretty unambiguous. The steps are:

\n\n

So, unless there are specific models which implement fit differently (I'm not aware of any), or you've patched sklearn, the code you've provided trains three models, throws out the first two, and then makes predictions based on the last model you trained.

\n

Have I missed something obvious?

", "votes": null, "replies": [] }, { "id": 57127, "author_name": "taffeylewis", "author_url": "", "post_date": "10/31/2014 18:49:25", "content": "

jdl37, I think you might be missing something in the code fragment... read.csv should include argument skip = n*500000, no?  Otherwise it'll just keep reading the first 500k lines?

", "votes": null, "replies": [] }, { "id": 57128, "author_name": "ccha97u", "author_url": "", "post_date": "10/31/2014 18:50:26", "content": "

kinnskogr: Thank you for you pointed it out! I am really grateful about you check through my idea. 

\n

I have to say I didn't carefully check through the training methodology in scikit-learn and simply assumed GNB has incremental learning features. I am wrong, I hope you guys didn't go to the wrong direction because of my misleading post.

\n

Thankfully, Sklearn still has solutions in this kind of big data situation:

\n

Please check this document on Scikit-Learn

\n

http://scikit-learn.org/stable/modules/scaling_strategies.html

", "votes": null, "replies": [] }, { "id": 57131, "author_name": "telser", "author_url": "", "post_date": "10/31/2014 20:22:29", "content": "

[quote=Chun-Hao Chang;56815]

\n

Hello guys, I want to know the data size, but since the host is preparing the data, I don't see any information about the size.

\n

Does anyone know how much will be the data size?

\n

By the way, I am a TA of a Data Mining course, I am looking for interesting competition like this one for student's final project. Do you guys have any advices?

\n

[/quote]

\n

There have been some issues in some earlier competitions with student groups competing (e.g., Partly Sunny with a chance of hashtags), so I'd be sure to remind your students of the competition rules, especially these:

\n

One account per participant

\n

You cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.

\n\n

No private sharing outside teams

\n

Privately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.

", "votes": null, "replies": [] }, { "id": 57136, "author_name": "lucasqi", "author_url": "", "post_date": "10/31/2014 21:16:06", "content": "

It has 47686351 observations and 27 variables. Is that right?

", "votes": null, "replies": [] }, { "id": 57141, "author_name": "davidthaler", "author_url": "", "post_date": "10/31/2014 22:47:52", "content": "

[quote=kinnskogr;57117]

\n

Chun-Hao Chang, maybe there is something I'm not understanding from your post. There are indeed plenty of models you can imagine updating or ensembling, but as far as I know, if you call fit multiple times on a single instance of an sklearn class, it will overwrite the previous values of the model. If you look at the code (it's in sklearn/naive_bayes.py for your example), it's pretty unambiguous. The steps are:

\n\n

So, unless there are specific models which implement fit differently (I'm not aware of any), or you've patched sklearn, the code you've provided trains three models, throws out the first two, and then makes predictions based on the last model you trained.

\n

Have I missed something obvious?

\n

[/quote]

\n

Several sklearn classes have a partial fit method for mini-batch learning. The classes that I can recall are: SGDRegressor/Classifier, perceptron, all of the naive Bayes models and passive-aggressive models.

\n

I think you can call each with fit() the first time, and then partial_fit() subsequently, or you can call it with partial_fit() straight through, in which case you need to call it with the classes parameter, at least the first time. Here's some more info: sklearn incremental training.

", "votes": null, "replies": [] }, { "id": 57143, "author_name": "kinnskogr", "author_url": "", "post_date": "10/31/2014 23:37:58", "content": "

Hey David,

\n

   That's right. My initial post wasn't quite complete without mentioning those. Chun-Hao has also pointed to the methods under the Incremental Learning section of http://scikit-learn.org/stable/modules/scaling_strategies.html. The point I was raising was that by convention, the fit() method will overwrite the model parameters. The partial_fit() method, again by convention, allows updating (and thankfully, the sklearn maintainers are great about enforcing conventions).

\n

Also, you don't need to call fit() the first time, there's a check in partial_fit() to handle newly initialized instances.

\n

Cheers,

\n

- Emanuel

", "votes": null, "replies": [] }, { "id": 57234, "author_name": "jamesdlawrence", "author_url": "", "post_date": "11/03/2014 09:18:58", "content": "

@skwalas, not if the connection is opened in mode "r". If it was opened in another mode, that would be true.

", "votes": null, "replies": [] }, { "id": 57385, "author_name": "", "author_url": "", "post_date": "11/06/2014 05:52:06", "content": "

[quote=Nicholas Guttenberg;56831]

\n

The data size is a bit of a problem for me too. I think there are a couple reasonable ways to proceed, but I guess we'll see what works best. 

\n

One way is to use exclusively 'online learners' that can just take new data rows one at a time. In that case, you never have to load the entire thing into memory at once. This is a little limiting as far as what kinds of algorithms you can use, but this seems to be a pretty active area of development so you can find a bunch of research papers making online versions of stuff. The downside here is that this could mean a lot of custom implementation, which slows down one's ability to try a bunch of possibilities without committing too much time until they pan out.

\n

The other way, which seemed like a better bet to me, is to split the data up into smaller chunks and then ensemble the results. Since there's a good chunk of data, it means you can do a lot more with hold-out sets/etc than you could in a more data-starved case, so some techniques that'd otherwise be prone to over-fitting might be feasible here. If nothing else, this is probably a good first thing to do just to get a feeling for what class of algorithms are performing best with this data-set, before looking into writing up online versions of those things.

\n

[/quote]

\n

Yet another way, is to buy* a giant server with 32 gigabytes memory.

\n\n

*Or rent one on AWS.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "56815": "", "56816": "", "56819": "", "56824": "", "56826": "", "56828": "", "56829": "", "56830": "", "56831": "", "56832": "", "57058": "", "57084": "", "57096": "", "57098": "", "57099": "", "57100": "", "57101": "", "57111": "", "57117": "", "57127": "", "57128": "", "57131": "", "57136": "", "57141": "", "57143": "", "57234": "", "57385": "" }, "source": "meta" }