{ "id": 12461, "title": "Tiny LB shakeup", "url": "/competitions/avazu-ctr-prediction/discussion/12461", "author_name": "", "post_date": "2015-02-10T00:33:42.220Z", "votes": 6, "comment_count": 7, "views": 1982, "content": "

I'm surprised at how small the LB shakeup was in this one. According to the forums, people were having trouble getting reliable validation scores, and that is often a sign of a big LB shakeup. BreakfastPirate introduced a 'Leaderboard Shakeup Metric' in this post: LB shakeup thread. By that measure, this competition had one of the smallest shakeups ever. From that thread:

\n

Competition             Shake-up    Shake-up (Top 10%)
See Click Predict Fix     0.004           0.005
Genentech                 0.006           0.000
Walmart                   0.007           0.006
Yelp                      0.007           0.007

\n

and now...

\n

Avazu                     0.005           0.006

\n

There are some results for reference down-thread there: more shakeup results.

", "messages": [ { "id": "63851", "postDate": "02/10/2015 00:33:42", "content": "

I'm surprised at how small the LB shakeup was in this one. According to the forums, people were having trouble getting reliable validation scores, and that is often a sign of a big LB shakeup. BreakfastPirate introduced a 'Leaderboard Shakeup Metric' in this post: LB shakeup thread. By that measure, this competition had one of the smallest shakeups ever. From that thread:

\n

Competition             Shake-up    Shake-up (Top 10%)
See Click Predict Fix     0.004           0.005
Genentech                 0.006           0.000
Walmart                   0.007           0.006
Yelp                      0.007           0.007

\n

and now...

\n

Avazu                     0.005           0.006

\n

There are some results for reference down-thread there: more shakeup results.

", "rawMarkdown": "", "votes": null }, { "id": "63853", "postDate": "02/10/2015 00:44:17", "content": "

I think with test data of 900,000 rows, it is very believable. IMHO the problem with getting your cv right in this one had to do with the fact that you were scoring on a new day in the future and the fact that this day ("31") appeared to be quite different so that tiny improvements in cv were not being reflected- Still the results were consistent along that day, hence the tiny shake-up

", "rawMarkdown": "", "votes": null }, { "id": "63854", "postDate": "02/10/2015 00:54:36", "content": "

[quote=KazAnova;63853]

\n

... the problem with getting your cv right in this one had to do with the fact that you were scoring on a new day in the future and the fact that this day ("31") appeared to be quite different...

\n

[/quote]

\n

Actually, I abandoned that method in favor of ordinary holdout (every 5th or 10th example). It seemed a little more stable, and was mostly directionally consistent. I don't think time was as important of a predictor here as most people think it was. Clearly there was temporal variation in CTR, but didn't look periodic, and it had no trend. My best single model (0.3905) didn't use time directly as a variable.

", "rawMarkdown": "", "votes": null }, { "id": "63858", "postDate": "02/10/2015 01:22:01", "content": "

You can theoretically bound the max likely possible difference in losses given an iid train and test set, right? I'm referring to this: https://www.youtube.com/watch?v=puDzy2XmR5c#t=279 albeit maybe I interpreted it incorrectly. It should be quite small in that first case given the sample sizes. In this case, it's not iid but forecasting, and the test dataset is quite large, but the public private split should be random. Furthermore unlike the video example, in this case you don't have direct access to the public test which can somewhat mitigate the direct overtraining issues. So it seems there's no reason not to overfit to the public leaderboard.

", "rawMarkdown": "", "votes": null }, { "id": "63886", "postDate": "02/10/2015 07:21:49", "content": "

[quote=David Thaler;63854]

\n

Actually, I abandoned that method in favor of ordinary holdout (every 5th or 10th example). It seemed a little more stable, and was mostly directionally consistent. I don't think time was as important of a predictor here as most people think it was. Clearly there was temporal variation in CTR, but didn't look periodic, and it had no trend. My best single model (0.3905) didn't use time directly as a variable.

\n

[/quote]

\n

David, your best single model .3905 and you got .3895 on LB, so .0010 improvement from blending!

\n

i'm curious on your blending method, and how many models contributed to it.

", "rawMarkdown": "", "votes": null }, { "id": "63891", "postDate": "02/10/2015 08:11:33", "content": "

[quote=laserwolf;63886]

\n

David, your best single model .3905 and you got .3895 on LB, so .0010 improvement from blending!

\n

i'm curious on your blending method, and how many models contributed to it.

\n

[/quote]

\n

I just used a straight average (on the log-odds scale) from 3 VW models and a modified version of tingrtu's model.

", "rawMarkdown": "", "votes": null }, { "id": "63893", "postDate": "02/10/2015 08:15:33", "content": "

[quote=David Thaler;63854]

\n

 I don't think time was as important of a predictor here as most people think it was.

\n

[/quote]

\n

Interesting you say that. I recall removing day 30 from my training at some point and was getting much worse results on leaderboard...

", "rawMarkdown": "", "votes": null }, { "id": "63897", "postDate": "02/10/2015 08:40:22", "content": "

[quote=KazAnova;63893]

\n

[quote=David Thaler;63854]

\n

 I don't think time was as important of a predictor here as most people think it was.

\n

[/quote]

\n

Interesting you say that. I recall removing day 30 from my training at some point and was getting much worse results on leaderboard...

\n

[/quote]

\n

What I meant by that was that when I tried models with time (hour of day) in or out of the model, at least in local validation, out was slightly better.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 63853, "author_name": "kazanova", "author_url": "", "post_date": "02/10/2015 00:44:17", "content": "

I think with test data of 900,000 rows, it is very believable. IMHO the problem with getting your cv right in this one had to do with the fact that you were scoring on a new day in the future and the fact that this day ("31") appeared to be quite different so that tiny improvements in cv were not being reflected- Still the results were consistent along that day, hence the tiny shake-up

", "votes": null, "replies": [] }, { "id": 63854, "author_name": "davidthaler", "author_url": "", "post_date": "02/10/2015 00:54:36", "content": "

[quote=KazAnova;63853]

\n

... the problem with getting your cv right in this one had to do with the fact that you were scoring on a new day in the future and the fact that this day ("31") appeared to be quite different...

\n

[/quote]

\n

Actually, I abandoned that method in favor of ordinary holdout (every 5th or 10th example). It seemed a little more stable, and was mostly directionally consistent. I don't think time was as important of a predictor here as most people think it was. Clearly there was temporal variation in CTR, but didn't look periodic, and it had no trend. My best single model (0.3905) didn't use time directly as a variable.

", "votes": null, "replies": [] }, { "id": 63858, "author_name": "mikeskim", "author_url": "", "post_date": "02/10/2015 01:22:01", "content": "

You can theoretically bound the max likely possible difference in losses given an iid train and test set, right? I'm referring to this: https://www.youtube.com/watch?v=puDzy2XmR5c#t=279 albeit maybe I interpreted it incorrectly. It should be quite small in that first case given the sample sizes. In this case, it's not iid but forecasting, and the test dataset is quite large, but the public private split should be random. Furthermore unlike the video example, in this case you don't have direct access to the public test which can somewhat mitigate the direct overtraining issues. So it seems there's no reason not to overfit to the public leaderboard.

", "votes": null, "replies": [] }, { "id": 63886, "author_name": "vvarkey", "author_url": "", "post_date": "02/10/2015 07:21:49", "content": "

[quote=David Thaler;63854]

\n

Actually, I abandoned that method in favor of ordinary holdout (every 5th or 10th example). It seemed a little more stable, and was mostly directionally consistent. I don't think time was as important of a predictor here as most people think it was. Clearly there was temporal variation in CTR, but didn't look periodic, and it had no trend. My best single model (0.3905) didn't use time directly as a variable.

\n

[/quote]

\n

David, your best single model .3905 and you got .3895 on LB, so .0010 improvement from blending!

\n

i'm curious on your blending method, and how many models contributed to it.

", "votes": null, "replies": [] }, { "id": 63891, "author_name": "davidthaler", "author_url": "", "post_date": "02/10/2015 08:11:33", "content": "

[quote=laserwolf;63886]

\n

David, your best single model .3905 and you got .3895 on LB, so .0010 improvement from blending!

\n

i'm curious on your blending method, and how many models contributed to it.

\n

[/quote]

\n

I just used a straight average (on the log-odds scale) from 3 VW models and a modified version of tingrtu's model.

", "votes": null, "replies": [] }, { "id": 63893, "author_name": "kazanova", "author_url": "", "post_date": "02/10/2015 08:15:33", "content": "

[quote=David Thaler;63854]

\n

 I don't think time was as important of a predictor here as most people think it was.

\n

[/quote]

\n

Interesting you say that. I recall removing day 30 from my training at some point and was getting much worse results on leaderboard...

", "votes": null, "replies": [] }, { "id": 63897, "author_name": "davidthaler", "author_url": "", "post_date": "02/10/2015 08:40:22", "content": "

[quote=KazAnova;63893]

\n

[quote=David Thaler;63854]

\n

 I don't think time was as important of a predictor here as most people think it was.

\n

[/quote]

\n

Interesting you say that. I recall removing day 30 from my training at some point and was getting much worse results on leaderboard...

\n

[/quote]

\n

What I meant by that was that when I tried models with time (hour of day) in or out of the model, at least in local validation, out was slightly better.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "63851": "", "63853": "", "63854": "", "63858": "", "63886": "", "63891": "", "63893": "", "63897": "" }, "source": "meta" }