{ "id": 11020, "title": "Constant click-rate in data", "url": "/competitions/avazu-ctr-prediction/discussion/11020", "author_name": "", "post_date": "2014-11-22T20:33:49.733Z", "votes": 4, "comment_count": 2, "views": 1662, "content": "

This is my first competition and I'm seeing a very big difference between my trained model (0.38) and my submission (0.5); I'm training on 20% of the data and when I test against unseen data I get 0.38 howsoever I chop up the remaining 80% of the data. The only way I can rationalise this is to assume that the test data has something quite different about it compared to the training data and so I'm investigating what is different.

\n

What I think I have discovered is that for every combination of site_id and device_id that exists in the test data that the average click rate in the training data is constant by site_id.

\n

For example:

\n

> setwd("~/Downloads/Project")
> load(file="df_raw")
> load(file="tf_raw")
>
> temp<-tapply(df$click,df$site_id,mean)
> df$q_site_id<-as.numeric(temp[df$site_id])
> quantile(df$q_site_id)
0% 25% 50% 75% 100%
0.0000000 0.1188264 0.1188264 0.2056326 1.0000000
>
> o<-unique(tf$site_id)
> df$temp <- o[df$site_id]
> df<-df[!is.na(df$temp),]
>
> temp<-tapply(df$click,df$site_id,mean)
> df$q_site_id<-as.numeric(temp[df$site_id])
> quantile(df$q_site_id)
0% 25% 50% 75% 100%
0.0000000 0.1188264 0.1188264 0.2056326 1.0000000
>
> o<-unique(tf$app_id)
> df$temp <- o[df$app_id]
> df<-df[!is.na(df$temp),]
>
> temp<-tapply(df$click,df$site_id,mean)
> df$q_site_id<-as.numeric(temp[df$site_id])
> quantile(df$q_site_id)
0% 25% 50% 75% 100%
0.1166539 0.1166539 0.1166539 0.1166539 0.1166539

\n

And, I'm thinking that this is pretty surprising.

", "messages": [ { "id": "58700", "postDate": "11/22/2014 20:33:49", "content": "

This is my first competition and I'm seeing a very big difference between my trained model (0.38) and my submission (0.5); I'm training on 20% of the data and when I test against unseen data I get 0.38 howsoever I chop up the remaining 80% of the data. The only way I can rationalise this is to assume that the test data has something quite different about it compared to the training data and so I'm investigating what is different.

\n

What I think I have discovered is that for every combination of site_id and device_id that exists in the test data that the average click rate in the training data is constant by site_id.

\n

For example:

\n

> setwd("~/Downloads/Project")
> load(file="df_raw")
> load(file="tf_raw")
>
> temp<-tapply(df$click,df$site_id,mean)
> df$q_site_id<-as.numeric(temp[df$site_id])
> quantile(df$q_site_id)
0% 25% 50% 75% 100%
0.0000000 0.1188264 0.1188264 0.2056326 1.0000000
>
> o<-unique(tf$site_id)
> df$temp <- o[df$site_id]
> df<-df[!is.na(df$temp),]
>
> temp<-tapply(df$click,df$site_id,mean)
> df$q_site_id<-as.numeric(temp[df$site_id])
> quantile(df$q_site_id)
0% 25% 50% 75% 100%
0.0000000 0.1188264 0.1188264 0.2056326 1.0000000
>
> o<-unique(tf$app_id)
> df$temp <- o[df$app_id]
> df<-df[!is.na(df$temp),]
>
> temp<-tapply(df$click,df$site_id,mean)
> df$q_site_id<-as.numeric(temp[df$site_id])
> quantile(df$q_site_id)
0% 25% 50% 75% 100%
0.1166539 0.1166539 0.1166539 0.1166539 0.1166539

\n

And, I'm thinking that this is pretty surprising.

", "rawMarkdown": "", "votes": null }, { "id": "58702", "postDate": "11/22/2014 21:12:10", "content": "

Well, the data page states that "Non-clicks and clicks are subsampled according to different strategies.". You might have figured out part of those strategies...

", "rawMarkdown": "", "votes": null }, { "id": "60487", "postDate": "12/17/2014 14:43:12", "content": "

I don't think this is correct. what are you doing with app_id exactly? i believe you're looking at the mean of only 1 row.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 58702, "author_name": "adjgiulio", "author_url": "", "post_date": "11/22/2014 21:12:10", "content": "

Well, the data page states that "Non-clicks and clicks are subsampled according to different strategies.". You might have figured out part of those strategies...

", "votes": null, "replies": [] }, { "id": 60487, "author_name": "vvarkey", "author_url": "", "post_date": "12/17/2014 14:43:12", "content": "

I don't think this is correct. what are you doing with app_id exactly? i believe you're looking at the mean of only 1 row.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "58700": "", "58702": "", "60487": "" }, "source": "meta" }