{ "id": 11118, "title": "Are your top solutions already very complex?", "url": "/competitions/avazu-ctr-prediction/discussion/11118", "author_name": "", "post_date": "2014-12-02T17:56:22.353Z", "votes": 6, "comment_count": 19, "views": 5218, "content": "

Hi fellow Kagglers!

\n

My actual solution (LB 0.3944) is already very complex, with a high number of models. I would like to know if the guys in the Top 100 are noticing this too, or are you doing well with simpler alternatives?

\n

Thanks!

", "messages": [ { "id": "59407", "postDate": "12/02/2014 17:56:22", "content": "

Hi fellow Kagglers!

\n

My actual solution (LB 0.3944) is already very complex, with a high number of models. I would like to know if the guys in the Top 100 are noticing this too, or are you doing well with simpler alternatives?

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "59408", "postDate": "12/02/2014 18:01:04", "content": "

Mine's simple. Takes 2 hours maximum and 8gb memory.

", "rawMarkdown": "", "votes": null }, { "id": "59409", "postDate": "12/02/2014 18:06:37", "content": "

Has anyone had success doing variable selection? I tried greedy forward and backwards, using VW as my base model, and cannot really get much out of it.

", "rawMarkdown": "", "votes": null }, { "id": "59413", "postDate": "12/02/2014 19:13:53", "content": "

[quote=Domcastro;59408]

\n

Mine's simple. Takes 2 hours maximum and 8gb memory.

\n

[/quote]

\n

Nice!

\n

I find amazing what experienced Kagglers can do with simple models. Do you know any papers about using/optimizing simple models, beyond hyperparameters, that you can recommend?

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "59414", "postDate": "12/02/2014 19:15:30", "content": "

For my current model, it takes less than 30mins. I'm currently using a very simple model.

\n

EDIT: no feature selection and no subsampling.

", "rawMarkdown": "", "votes": null }, { "id": "59415", "postDate": "12/02/2014 19:16:01", "content": "

No - I'm blagging my way through. But I think it's data prep not algorithms that's more important

", "rawMarkdown": "", "votes": null }, { "id": "59416", "postDate": "12/02/2014 19:16:34", "content": "

Mine is very simple -- single model based on tinrtgu's code.

", "rawMarkdown": "", "votes": null }, { "id": "59420", "postDate": "12/02/2014 20:06:41", "content": "

I haven't tried any very complex models yet. Still thinking about data preparation.

", "rawMarkdown": "", "votes": null }, { "id": "59430", "postDate": "12/03/2014 01:32:52", "content": "

I'm bouncing back and forth between improving the individual models and improving the way in which I'm combining them.

\n

The 'complex' parts of the process helps with a few 0.001 here and there, but there are clearly lynchpin models which control most of the overall performance, and which will need to be improved to close any large gaps in LB score. As the lynchpin gets better, the ensemble step actually becomes less useful unless I also improve the other components of the ensemble too.

\n

Even if its not necessary for high scores, I think the complexity is quite psychologically beneficial. If I have an idea to improve any element of the mix, that stands a good chance of improving the whole, so I stay in a mindset of trying lots of different things. Whereas if I picked one model and pushed hard on it, I might get stuck and get frustrated. So when one thing stops paying off, I can jump to other things and still feel like I'm making progress. 

", "rawMarkdown": "", "votes": null }, { "id": "59432", "postDate": "12/03/2014 01:46:12", "content": "

Simple feature engineering with tinrtgu's code gave me 0.3918 (top 20 yesterday), running in 10-15 min with about 5GB RAM.

", "rawMarkdown": "", "votes": null }, { "id": "59439", "postDate": "12/03/2014 05:12:15", "content": "

My current LB score 0.3914 is also obtained by using my benchmark code. All in all I'm still trying to figure out a way to reliable do validation. After I build up my validation framework, I'll then work on more complex models.

", "rawMarkdown": "", "votes": null }, { "id": "59442", "postDate": "12/03/2014 05:23:28", "content": "

Simplified and slightly modified tinrtgu code, no ensembles. I would say that most of the improvement was from working with data and optimizing hyperparameters.

", "rawMarkdown": "", "votes": null }, { "id": "59455", "postDate": "12/03/2014 12:04:07", "content": "

Thanks for all the answers!

\n

I am trying to figure out ways to work with the data, and feature engineering, but still haven't seen any progress.

\n

I suspected my ensemble mess was too much for this stage of the competition, and it's the case, so now I will try a different approach.

", "rawMarkdown": "", "votes": null }, { "id": "59457", "postDate": "12/03/2014 12:43:01", "content": "

Thank you guys! Our best solution is an average of one online model and one fm model.

", "rawMarkdown": "", "votes": null }, { "id": "59470", "postDate": "12/03/2014 16:01:19", "content": "

Well, My score is not high but all I've done is playing with tinrtgu's code. No feature engineering or ensemble.

", "rawMarkdown": "", "votes": null }, { "id": "59559", "postDate": "12/04/2014 18:31:42", "content": "

[quote=rcarson;59457]

\n

Thank you guys! Our best solution is an average of one online model and one fm model.

\n

[/quote]

\n

I am fairly new to FM models. Could you give us a few pointers as to why would you want to use FM models for this kind of a problem setting? I am missing something which probably might be very intuitive to you. Space my ignorance.

\n

EDIT: Spare*

", "rawMarkdown": "", "votes": null }, { "id": "59560", "postDate": "12/04/2014 18:41:05", "content": "

[quote=binga;59559]

\n

[quote=rcarson;59457]

\n

Thank you guys! Our best solution is an average of one online model and one fm model.

\n

[/quote]

\n

I am fairly new to FM models. Could you give us a few pointers as to why would you want to use FM models for this kind of a problem setting? I am missing something which probably might be very intuitive to you. Space my ignorance.

\n

[/quote]

\n

Hi, I'm no expert. I simply follow the previous contests:

\n

https://storage.googleapis.com/kaggle-competitions/kddcup2012/2748/media/Rendle.pdf

\n

https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10555/3-idiots-solution

", "rawMarkdown": "", "votes": null }, { "id": "59561", "postDate": "12/04/2014 18:49:42", "content": "

I did have a look at the 3 Idiots' solution from Criteo but I missed the intuition behind their approach. Thank you for the resources anyways. A quick question. On this size of the data, how much memory did your FM model take?

", "rawMarkdown": "", "votes": null }, { "id": "59567", "postDate": "12/04/2014 19:24:36", "content": "

8 GB I think, depending on the number of latent factor.

", "rawMarkdown": "", "votes": null }, { "id": "59745", "postDate": "12/07/2014 21:16:44", "content": "

[quote=Little Boat;59470]

\n

Well, My score is not high but all I've done is playing with tinrtgu's code. No feature engineering or ensemble.

\n

[/quote]

\n\n

Where can I find tinrtgu's code? Thanks! 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 59408, "author_name": "domcastro", "author_url": "", "post_date": "12/02/2014 18:01:04", "content": "

Mine's simple. Takes 2 hours maximum and 8gb memory.

", "votes": null, "replies": [] }, { "id": 59409, "author_name": "adjgiulio", "author_url": "", "post_date": "12/02/2014 18:06:37", "content": "

Has anyone had success doing variable selection? I tried greedy forward and backwards, using VW as my base model, and cannot really get much out of it.

", "votes": null, "replies": [] }, { "id": 59413, "author_name": "mariofilho", "author_url": "", "post_date": "12/02/2014 19:13:53", "content": "

[quote=Domcastro;59408]

\n

Mine's simple. Takes 2 hours maximum and 8gb memory.

\n

[/quote]

\n

Nice!

\n

I find amazing what experienced Kagglers can do with simple models. Do you know any papers about using/optimizing simple models, beyond hyperparameters, that you can recommend?

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 59414, "author_name": "abhishek", "author_url": "", "post_date": "12/02/2014 19:15:30", "content": "

For my current model, it takes less than 30mins. I'm currently using a very simple model.

\n

EDIT: no feature selection and no subsampling.

", "votes": null, "replies": [] }, { "id": 59415, "author_name": "domcastro", "author_url": "", "post_date": "12/02/2014 19:16:01", "content": "

No - I'm blagging my way through. But I think it's data prep not algorithms that's more important

", "votes": null, "replies": [] }, { "id": 59416, "author_name": "psilogram", "author_url": "", "post_date": "12/02/2014 19:16:34", "content": "

Mine is very simple -- single model based on tinrtgu's code.

", "votes": null, "replies": [] }, { "id": 59420, "author_name": "jwjohnson314", "author_url": "", "post_date": "12/02/2014 20:06:41", "content": "

I haven't tried any very complex models yet. Still thinking about data preparation.

", "votes": null, "replies": [] }, { "id": 59430, "author_name": "ngutten", "author_url": "", "post_date": "12/03/2014 01:32:52", "content": "

I'm bouncing back and forth between improving the individual models and improving the way in which I'm combining them.

\n

The 'complex' parts of the process helps with a few 0.001 here and there, but there are clearly lynchpin models which control most of the overall performance, and which will need to be improved to close any large gaps in LB score. As the lynchpin gets better, the ensemble step actually becomes less useful unless I also improve the other components of the ensemble too.

\n

Even if its not necessary for high scores, I think the complexity is quite psychologically beneficial. If I have an idea to improve any element of the mix, that stands a good chance of improving the whole, so I stay in a mindset of trying lots of different things. Whereas if I picked one model and pushed hard on it, I might get stuck and get frustrated. So when one thing stops paying off, I can jump to other things and still feel like I'm making progress. 

", "votes": null, "replies": [] }, { "id": 59432, "author_name": "alexistaugeron", "author_url": "", "post_date": "12/03/2014 01:46:12", "content": "

Simple feature engineering with tinrtgu's code gave me 0.3918 (top 20 yesterday), running in 10-15 min with about 5GB RAM.

", "votes": null, "replies": [] }, { "id": 59439, "author_name": "ggglhf", "author_url": "", "post_date": "12/03/2014 05:12:15", "content": "

My current LB score 0.3914 is also obtained by using my benchmark code. All in all I'm still trying to figure out a way to reliable do validation. After I build up my validation framework, I'll then work on more complex models.

", "votes": null, "replies": [] }, { "id": 59442, "author_name": "ivanlobov", "author_url": "", "post_date": "12/03/2014 05:23:28", "content": "

Simplified and slightly modified tinrtgu code, no ensembles. I would say that most of the improvement was from working with data and optimizing hyperparameters.

", "votes": null, "replies": [] }, { "id": 59455, "author_name": "mariofilho", "author_url": "", "post_date": "12/03/2014 12:04:07", "content": "

Thanks for all the answers!

\n

I am trying to figure out ways to work with the data, and feature engineering, but still haven't seen any progress.

\n

I suspected my ensemble mess was too much for this stage of the competition, and it's the case, so now I will try a different approach.

", "votes": null, "replies": [] }, { "id": 59457, "author_name": "jiweiliu", "author_url": "", "post_date": "12/03/2014 12:43:01", "content": "

Thank you guys! Our best solution is an average of one online model and one fm model.

", "votes": null, "replies": [] }, { "id": 59470, "author_name": "xiaozhouwang", "author_url": "", "post_date": "12/03/2014 16:01:19", "content": "

Well, My score is not high but all I've done is playing with tinrtgu's code. No feature engineering or ensemble.

", "votes": null, "replies": [] }, { "id": 59559, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/04/2014 18:31:42", "content": "

[quote=rcarson;59457]

\n

Thank you guys! Our best solution is an average of one online model and one fm model.

\n

[/quote]

\n

I am fairly new to FM models. Could you give us a few pointers as to why would you want to use FM models for this kind of a problem setting? I am missing something which probably might be very intuitive to you. Space my ignorance.

\n

EDIT: Spare*

", "votes": null, "replies": [] }, { "id": 59560, "author_name": "jiweiliu", "author_url": "", "post_date": "12/04/2014 18:41:05", "content": "

[quote=binga;59559]

\n

[quote=rcarson;59457]

\n

Thank you guys! Our best solution is an average of one online model and one fm model.

\n

[/quote]

\n

I am fairly new to FM models. Could you give us a few pointers as to why would you want to use FM models for this kind of a problem setting? I am missing something which probably might be very intuitive to you. Space my ignorance.

\n

[/quote]

\n

Hi, I'm no expert. I simply follow the previous contests:

\n

https://storage.googleapis.com/kaggle-competitions/kddcup2012/2748/media/Rendle.pdf

\n

https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10555/3-idiots-solution

", "votes": null, "replies": [] }, { "id": 59561, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/04/2014 18:49:42", "content": "

I did have a look at the 3 Idiots' solution from Criteo but I missed the intuition behind their approach. Thank you for the resources anyways. A quick question. On this size of the data, how much memory did your FM model take?

", "votes": null, "replies": [] }, { "id": 59567, "author_name": "jiweiliu", "author_url": "", "post_date": "12/04/2014 19:24:36", "content": "

8 GB I think, depending on the number of latent factor.

", "votes": null, "replies": [] }, { "id": 59745, "author_name": "arpitgupta", "author_url": "", "post_date": "12/07/2014 21:16:44", "content": "

[quote=Little Boat;59470]

\n

Well, My score is not high but all I've done is playing with tinrtgu's code. No feature engineering or ensemble.

\n

[/quote]

\n\n

Where can I find tinrtgu's code? Thanks! 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "59407": "", "59408": "", "59409": "", "59413": "", "59414": "", "59415": "", "59416": "", "59420": "", "59430": "", "59432": "", "59439": "", "59442": "", "59455": "", "59457": "", "59470": "", "59559": "", "59560": "", "59561": "", "59567": "", "59745": "" }, "source": "meta" }