{ "id": 11292, "title": "Neural Network Ensembles", "url": "/competitions/avazu-ctr-prediction/discussion/11292", "author_name": "", "post_date": "2014-12-18T12:49:40.440Z", "votes": 2, "comment_count": 5, "views": 3856, "content": "

Hi guys,

\n

i have a question regarding the ensembling of neural networks.
I tried a few things (different preprocessing/ architectures / weight initialization) but my ensembles seem to
saturate very quickly (only the first 3-4 Nets contribute).

\n

So my question is what is generally the most important point ot go for when ensembling many different neural networks?

\n

Thanks for your help!

", "messages": [ { "id": "61478", "postDate": "12/18/2014 12:49:40", "content": "

Hi guys,

\n

i have a question regarding the ensembling of neural networks.
I tried a few things (different preprocessing/ architectures / weight initialization) but my ensembles seem to
saturate very quickly (only the first 3-4 Nets contribute).

\n

So my question is what is generally the most important point ot go for when ensembling many different neural networks?

\n

Thanks for your help!

", "rawMarkdown": "", "votes": null }, { "id": "61479", "postDate": "12/18/2014 13:30:16", "content": "

Hello,

\n

Training on different parts of the trainset. (for instance leave 1-2 days out)

\n

Training on different parts of the feature set.(for instance leave device ip out)

\n

It's my belief that having a dropout layer also gives more varied nets..

\n

Overfitting a little doesn't hurt..

\n\n

This dataset is less "rich" than for instance Criteo so perhaps the NN has a hard time finding something "new".

", "rawMarkdown": "", "votes": null }, { "id": "61480", "postDate": "12/18/2014 13:50:41", "content": "

Thank you very much for your advise!

", "rawMarkdown": "", "votes": null }, { "id": "61559", "postDate": "12/19/2014 23:39:54", "content": "

I think your best ensembling strategy might depend on your ensembling goals. Are you trying to reduce model bias or trying to reduce variance (overfitting)? Someone more mathematical than I could probably elaborate on this, but my intuition is that:

\n\n

Also, how are you ensembling the different networks? That can make a fairly large difference.

", "rawMarkdown": "", "votes": null }, { "id": "61946", "postDate": "12/27/2014 18:54:33", "content": "

I'd be interested in how much improvement you're able to obtain in an ensemble compared to the single model performance specific to this competition (if that doesn't gives away too much of your solution of course). My experience is about ~0.005. I'm using the tipps from Julian and simply average the predictions. Is that a realistic value or do you guys get even stronger ensemble boost?

", "rawMarkdown": "", "votes": null }, { "id": "61957", "postDate": "12/27/2014 22:19:30", "content": "

You might want to try saving some data for creating a weighted ensemble. One approach is to use a simple regression model to create weights for ensembles on the left out data. Simple averaging often isn't a bad approach, but it ignores that some models may have more or less correlation in their errors.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 61479, "author_name": "juliandewit", "author_url": "", "post_date": "12/18/2014 13:30:16", "content": "

Hello,

\n

Training on different parts of the trainset. (for instance leave 1-2 days out)

\n

Training on different parts of the feature set.(for instance leave device ip out)

\n

It's my belief that having a dropout layer also gives more varied nets..

\n

Overfitting a little doesn't hurt..

\n\n

This dataset is less "rich" than for instance Criteo so perhaps the NN has a hard time finding something "new".

", "votes": null, "replies": [] }, { "id": 61480, "author_name": "niklaskoehler", "author_url": "", "post_date": "12/18/2014 13:50:41", "content": "

Thank you very much for your advise!

", "votes": null, "replies": [] }, { "id": 61559, "author_name": "danielyoo", "author_url": "", "post_date": "12/19/2014 23:39:54", "content": "

I think your best ensembling strategy might depend on your ensembling goals. Are you trying to reduce model bias or trying to reduce variance (overfitting)? Someone more mathematical than I could probably elaborate on this, but my intuition is that:

\n\n

Also, how are you ensembling the different networks? That can make a fairly large difference.

", "votes": null, "replies": [] }, { "id": 61946, "author_name": "niklaskoehler", "author_url": "", "post_date": "12/27/2014 18:54:33", "content": "

I'd be interested in how much improvement you're able to obtain in an ensemble compared to the single model performance specific to this competition (if that doesn't gives away too much of your solution of course). My experience is about ~0.005. I'm using the tipps from Julian and simply average the predictions. Is that a realistic value or do you guys get even stronger ensemble boost?

", "votes": null, "replies": [] }, { "id": 61957, "author_name": "danielyoo", "author_url": "", "post_date": "12/27/2014 22:19:30", "content": "

You might want to try saving some data for creating a weighted ensemble. One approach is to use a simple regression model to create weights for ensembles on the left out data. Simple averaging often isn't a bad approach, but it ignores that some models may have more or less correlation in their errors.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "61478": "", "61479": "", "61480": "", "61559": "", "61946": "", "61957": "" }, "source": "meta" }