{ "id": 11433, "title": "High number of levels in some features", "url": "/competitions/avazu-ctr-prediction/discussion/11433", "author_name": "", "post_date": "2015-01-06T00:37:58.577Z", "votes": null, "comment_count": 26, "views": 9339, "content": "

Hi,

\n


Beginner getting some experience. I want to try logistic regression and SVM, but there are some features that have lots of levels, the option I know is to use dummy features, is there any other option?, because I end up with a high dimensionality dataset.

\n

Any suggestions?

\n

Thanks in advance

", "messages": [ { "id": "62321", "postDate": "01/06/2015 00:37:58", "content": "

Hi,

\n


Beginner getting some experience. I want to try logistic regression and SVM, but there are some features that have lots of levels, the option I know is to use dummy features, is there any other option?, because I end up with a high dimensionality dataset.

\n

Any suggestions?

\n

Thanks in advance

", "rawMarkdown": "", "votes": null }, { "id": "62322", "postDate": "01/06/2015 01:56:43", "content": "

A few of thoughts you might consider:

\n

1) Start without those features

\n

2) Convert a column of features into a single, continuous feature by replacing each individual feature with the overall average of that feature.

\n

3) Figure out which features have the most leverage, and just use those.

", "rawMarkdown": "", "votes": null }, { "id": "62340", "postDate": "01/06/2015 12:13:58", "content": "

Thank you very much, inversion.

\n

In the case of the point 2. Did you mean "columns of features" right?, and could you give an example of how do I have to get those averages?. Thanks again.

", "rawMarkdown": "", "votes": null }, { "id": "62355", "postDate": "01/06/2015 15:41:33", "content": "

Yup. Something like this:

\n

\n

You can do it using "brute force" by writing a loop that counts the number of clicks for each feature, divide by the occurrence of each feature, and then replace each feature with that average.

\n

There are, of course, more elegant ways to do this, which depend on whether you are using Python, R, etc.

\n

A google search should be able to get you what you need for your specific setup.

", "rawMarkdown": "", "votes": null }, { "id": "62356", "postDate": "01/06/2015 15:43:37", "content": "

@inversion: i really like the idea (and I've seen it used in other contests), but don't you run a risk of overfitting? after all, you are building your features using values from the target you are trying to predict...

", "rawMarkdown": "", "votes": null }, { "id": "62358", "postDate": "01/06/2015 16:02:02", "content": "

[quote=Konrad Banachewicz;62356]

\n

@inversion: i really like the idea (and I've seen it used in other contests), but don't you run a risk of overfitting? after all, you are building your features using values from the target you are trying to predict...

\n

[/quote]

\n

Thanks for making that point, which is definitely a consideration.

\n

I was approaching the question not from the perspective of "what is the best method to try" but from the perspective of "how can I easily get started".

\n

In other words, for a beginner (well, perhaps even in general), it's good to see if you can get your model to work with a few columns and features. How well did it do? How long did it take? If I can't get my model to work in a simplified case, chances are I won't get it to work in the more complicated case.

\n

For a beginner (and perhaps even generally), it's a simple thing to try and see what happens. But, to your point, there is risk that should be assessed.

\n

(My opinion: I'm not sure over-fitting is going to be much of a problem for this contest. I've tried very hard to over-fit, without much success. I think there is just too much probabilistic noise in the data. But I could be wrong.)

", "rawMarkdown": "", "votes": null }, { "id": "62359", "postDate": "01/06/2015 16:14:37", "content": "

[quote=Konrad Banachewicz;62356]

\n

@inversion: i really like the idea (and I've seen it used in other contests), but don't you run a risk of overfitting? after all, you are building your features using values from the target you are trying to predict...

\n

[/quote]

\n

It might cause overfitting, especially if decision tree based learners would be used. I think it would be a better idea to do some kind of smoothing, ie. instead of sum(click)/N, you could use something like (sum(click)+alpha)/(N+2*alpha), where alpha>0. (I think it's called Laplace smoothing, which correspond to certain kind of Bayesian prior&likelihood). Then, at least infrequent features would be closer together which would prevent at least some forms of overfitting.

", "rawMarkdown": "", "votes": null }, { "id": "62362", "postDate": "01/06/2015 16:32:23", "content": "

@inversion: not sure if I agree about this being difficult to overfit: 1. i am consistently seeing a discrepancy between my validation vs lb score (in the wrong direction :-) 2. the issue might be related to the fact that the test day (= Halloween) seems to be somewhat different qualitatively different from the training set (there is a discussion about that in the forum).

", "rawMarkdown": "", "votes": null }, { "id": "62364", "postDate": "01/06/2015 16:34:31", "content": "

[quote=Herra Huu;62359]

\n

 I think it would be a better idea to do some kind of smoothing, ie. instead of sum(click)/N, you could use something like (sum(click)+alpha)/(N+2*alpha), where alpha>0. (I think it's called Laplace smoothing, which correspond to certain kind of Bayesian prior&likelihood). Then, at least infrequent features would be closer together which would prevent at least some forms of overfitting.

\n

[/quote]

\n

That is certainly a good approach, I am wondering if that's enough (or, in light of the Helloween thing, is that even treating the core issue). Due to my not-that-modern hardware, I have some optimizing to do before I can attempt it :-)

", "rawMarkdown": "", "votes": null }, { "id": "62366", "postDate": "01/06/2015 16:59:08", "content": "

[quote=Konrad Banachewicz;62364]

\n

That is certainly a good approach, I am wondering if that's enough (or, in light of the Helloween thing, is that even treating the core issue).

\n

[/quote]

\n

Yeah, it's not necessary enough (depending on what kind of model/distributions of values etc). For example even with smoothing, all features with count==1 would be presented by some values x_0 (click==0) and x_1 (click==1). Then decision tree based algorithms could separate those rows from everything else with just a couple of splits => possibly huge overfitting problem (if there is many such features). I think another good idea would be at least remove rare features (say count<5). But as usual, it depends.

", "rawMarkdown": "", "votes": null }, { "id": "62367", "postDate": "01/06/2015 16:59:21", "content": "

[quote=Konrad Banachewicz;62362]

\n

@inversion: not sure if I agree about this being difficult to overfit: 1. i am consistently seeing a discrepancy between my validation vs lb score (in the wrong direction :-) 2. the issue might be related to the fact that the test day (= Halloween) seems to be somewhat different qualitatively different from the training set (there is a discussion about that in the forum).

\n

[/quote]

\n

As I've posted earlier, my validation scores have tracked nicely with LB score.

\n

It's like the old joke:

\n", "rawMarkdown": "", "votes": null }, { "id": "62368", "postDate": "01/06/2015 17:02:17", "content": "

P.S. Here's the reference for those who have not seen it.

\n

", "rawMarkdown": "", "votes": null }, { "id": "62380", "postDate": "01/06/2015 23:02:02", "content": "

[quote=Herra Huu;62366]

\n

Yeah, it's not necessary enough (depending on what kind of model/distributions of values etc). For example even with smoothing, all features with count==1 would be presented by some values x_0 (click==0) and x_1 (click==1). Then decision tree based algorithms could separate those rows from everything else with just a couple of splits => possibly huge overfitting problem (if there is many such features). I think another good idea would be at least remove rare features (say count<5). But as usual, it depends.

\n

[/quote]

\n

For smaller data sets, I have fit a random effects model to each categorical variable. Making the category a random effect shrinks the estimate towards the mean. I wonder for these other purposes, perhaps averaging across a few bootstrap samples could help reduce overfiting. 

", "rawMarkdown": "", "votes": null }, { "id": "62397", "postDate": "01/07/2015 12:14:35", "content": "

[quote=Ofd;62321]

\n

[/quote]

\n

attached R code would give you a head start .  :-)

\n

cheers

", "rawMarkdown": "", "votes": null }, { "id": "62399", "postDate": "01/07/2015 14:56:03", "content": "

Thanks to all that who gave his approach in this post. it's a great learning journey with this community :-)

", "rawMarkdown": "", "votes": null }, { "id": "62406", "postDate": "01/07/2015 16:07:17", "content": "

It is possible to get a reasonable score via the CTR histogram approach but my first efforts couldn't quite break the 0.40 level, from a 5m sample. If you are using R you might want to consider:

\n

- the combine.levels function from Hmisc to easily drop infrequent categories

\n

- the ave function to easily cacluate the CTR for each level you retain (though without any smoothing)

", "rawMarkdown": "", "votes": null }, { "id": "62413", "postDate": "01/07/2015 18:09:28", "content": "

[quote=lewis ml;62406]

\n

It is possible to get a reasonable score via the CTR histogram approach but my first efforts couldn't quite break the 0.40 level, from a 5m sample. If you are using R you might want to consider:

\n

- the combine.levels function from Hmisc to easily drop infrequent categories

\n

- the ave function to easily cacluate the CTR for each level you retain (though without any smoothing)

\n

[/quote]

\n\n

@Lewis ML, would you please elaborate on the CTR histogram approach a bit more?  I saw it mentioned from the previous competition, but couldn't actually get the gist of it... thank you in advance!

", "rawMarkdown": "", "votes": null }, { "id": "62415", "postDate": "01/07/2015 18:35:03", "content": "

Sure, though I'm no expert on it. As inversion described above, you:

\n

- remove infrequent levels from each variable

\n

- calculate the ctr for each level of each variable

\n

- use those new numeric features as the inputs to a gradient boosted tree

\n

As with other methods you can add features by combining levels across variables.

\n

I'm not sure if the small sample was the limiting factor on my effort. I tried both gbm and xgb via R.

\n

fchollet and the xgb authors deserve the credit for the idea. 

", "rawMarkdown": "", "votes": null }, { "id": "62485", "postDate": "01/09/2015 08:21:52", "content": "

Hi all,

\n

My last unsuccessful attempt with CTR is as follows:

\n

For each day calculate CTR per variable and per hour excepted for the last day (10th day). At the end, I had a bunch of features from all 9 days (around 150) which I merged to the 10th day for training using hour as key.

\n

Good Luck,

", "rawMarkdown": "", "votes": null }, { "id": "62504", "postDate": "01/09/2015 15:12:56", "content": "

[quote=lewis ml;62415]

\n

Sure, though I'm no expert on it. As inversion described above, you:

\n

- remove infrequent levels from each variable

\n

- calculate the ctr for each level of each variable

\n

- use those new numeric features as the inputs to a gradient boosted tree

\n

As with other methods you can add features by combining levels across variables.

\n

I'm not sure if the small sample was the limiting factor on my effort. I tried both gbm and xgb via R.

\n

fchollet and the xgb authors deserve the credit for the idea.

\n

[/quote]

\n

Thank you for the explanation, Lewis ML. What you explained, Inversion's post, and the previous comments now all connects. I tried to replace each variable by the mean CTR at the beginning of the competition, but didn't see much result. Maybe because I didn't remove rare features, or didn't do Laplace smoothing as Herra Huu suggested.  I will give it another try and let you guys know if I see any improvements. 

", "rawMarkdown": "", "votes": null }, { "id": "62505", "postDate": "01/09/2015 15:41:16", "content": "

@DerekZH -

\n

Also note, this method doesn't account for the interactions between features. So, if this data set has strong interactions, you wouldn't expect it to do as well as methods that account for the interactions.

", "rawMarkdown": "", "votes": null }, { "id": "62580", "postDate": "01/10/2015 18:59:36", "content": "

[quote=inversion;62505]

\n

@DerekZH -

\n

Also note, this method doesn't account for the interactions between features. So, if this data set has strong interactions, you wouldn't expect it to do as well as methods that account for the interactions.

\n

[/quote]

\n

I appreciate this insight, Inversion.  I will try to give the CTR histogram approach another try, just to learn the approach, and make it a candidate for later ensemble.  And I will give more thoughts to accounting for feature interactions.  

\n

Thanks again :)

", "rawMarkdown": "", "votes": null }, { "id": "62853", "postDate": "01/18/2015 06:28:50", "content": "

In this case, How would you prepare the test set for prediction as It does not contain 'click'

\n

[quote=inversion;62355]

\n

Yup. Something like this:

\n

\n

You can do it using "brute force" by writing a loop that counts the number of clicks for each feature, divide by the occurrence of each feature, and then replace each feature with that average.

\n

There are, of course, more elegant ways to do this, which depend on whether you are using Python, R, etc.

\n

A google search should be able to get you what you need for your specific setup.

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "62854", "postDate": "01/18/2015 08:05:16", "content": "

@inversion  @lewis ml

\n

Thank lewis and inversion for bringing us the idea and explanation.  

\n

There is one thing I'm still not clear is how do we deal with the null value in dataset? I think it is obviously not a good idea to consider those null values as unseen features in test dataset and allocate a smoothing value to them (eg, 0.25).

\n

So should we just conclude these null values as a new category and calculate the average CTR for this category, or do we just keep them as null values in dataset?

\n

Thanks :)

", "rawMarkdown": "", "votes": null }, { "id": "62855", "postDate": "01/18/2015 08:51:51", "content": "

Herimanitra: you have to build the test set by looking up the relevant ctr from the train set. So with inversion's example, any test case with A gets a 0.75 for that ctr feature.

\n

Ivan: you raise a good point but I'm not sure there's an  obvious 'right' way to deal with the issue. Might be worth trying different treatments.

\n

inversion: I feel like I'm missing something on your point about interactions. You can add interactions by creating a new categorical feature from two existing features, then calculating the ctr for it. But are saying you wouldn't expect that to be effective?

", "rawMarkdown": "", "votes": null }, { "id": "62860", "postDate": "01/18/2015 15:04:38", "content": "

[quote=lewis ml;62855]

\n

inversion: I feel like I'm missing something on your point about interactions. You can add interactions by creating a new categorical feature from two existing features, then calculating the ctr for it. But are saying you wouldn't expect that to be effective?

\n

[/quote]

\n

You're absolutely right . . . what I was trying to convey was that if you just reduced the feature set as is to average CTRs, you shouldn't necessarily expect a decent score. But there's no reason you couldn't build interactions as you suggest.

", "rawMarkdown": "", "votes": null }, { "id": "62861", "postDate": "01/18/2015 15:07:18", "content": "

[quote=Ivan Liu;62854]

\n

@inversion  @lewis ml

\n

So should we just conclude these null values as a new category and calculate the average CTR for this category, or do we just keep them as null values in dataset?

\n

Thanks :)

\n

[/quote]

\n

I've tried both ways . . . it's far less a hassle to change nulls to a new categorical feature.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 62322, "author_name": "inversion", "author_url": "", "post_date": "01/06/2015 01:56:43", "content": "

A few of thoughts you might consider:

\n

1) Start without those features

\n

2) Convert a column of features into a single, continuous feature by replacing each individual feature with the overall average of that feature.

\n

3) Figure out which features have the most leverage, and just use those.

", "votes": null, "replies": [] }, { "id": 62340, "author_name": "ofdomejean", "author_url": "", "post_date": "01/06/2015 12:13:58", "content": "

Thank you very much, inversion.

\n

In the case of the point 2. Did you mean "columns of features" right?, and could you give an example of how do I have to get those averages?. Thanks again.

", "votes": null, "replies": [] }, { "id": 62355, "author_name": "inversion", "author_url": "", "post_date": "01/06/2015 15:41:33", "content": "

Yup. Something like this:

\n

\n

You can do it using "brute force" by writing a loop that counts the number of clicks for each feature, divide by the occurrence of each feature, and then replace each feature with that average.

\n

There are, of course, more elegant ways to do this, which depend on whether you are using Python, R, etc.

\n

A google search should be able to get you what you need for your specific setup.

", "votes": null, "replies": [] }, { "id": 62356, "author_name": "konradb", "author_url": "", "post_date": "01/06/2015 15:43:37", "content": "

@inversion: i really like the idea (and I've seen it used in other contests), but don't you run a risk of overfitting? after all, you are building your features using values from the target you are trying to predict...

", "votes": null, "replies": [] }, { "id": 62358, "author_name": "inversion", "author_url": "", "post_date": "01/06/2015 16:02:02", "content": "

[quote=Konrad Banachewicz;62356]

\n

@inversion: i really like the idea (and I've seen it used in other contests), but don't you run a risk of overfitting? after all, you are building your features using values from the target you are trying to predict...

\n

[/quote]

\n

Thanks for making that point, which is definitely a consideration.

\n

I was approaching the question not from the perspective of "what is the best method to try" but from the perspective of "how can I easily get started".

\n

In other words, for a beginner (well, perhaps even in general), it's good to see if you can get your model to work with a few columns and features. How well did it do? How long did it take? If I can't get my model to work in a simplified case, chances are I won't get it to work in the more complicated case.

\n

For a beginner (and perhaps even generally), it's a simple thing to try and see what happens. But, to your point, there is risk that should be assessed.

\n

(My opinion: I'm not sure over-fitting is going to be much of a problem for this contest. I've tried very hard to over-fit, without much success. I think there is just too much probabilistic noise in the data. But I could be wrong.)

", "votes": null, "replies": [] }, { "id": 62359, "author_name": "herrahuu", "author_url": "", "post_date": "01/06/2015 16:14:37", "content": "

[quote=Konrad Banachewicz;62356]

\n

@inversion: i really like the idea (and I've seen it used in other contests), but don't you run a risk of overfitting? after all, you are building your features using values from the target you are trying to predict...

\n

[/quote]

\n

It might cause overfitting, especially if decision tree based learners would be used. I think it would be a better idea to do some kind of smoothing, ie. instead of sum(click)/N, you could use something like (sum(click)+alpha)/(N+2*alpha), where alpha>0. (I think it's called Laplace smoothing, which correspond to certain kind of Bayesian prior&likelihood). Then, at least infrequent features would be closer together which would prevent at least some forms of overfitting.

", "votes": null, "replies": [] }, { "id": 62362, "author_name": "konradb", "author_url": "", "post_date": "01/06/2015 16:32:23", "content": "

@inversion: not sure if I agree about this being difficult to overfit: 1. i am consistently seeing a discrepancy between my validation vs lb score (in the wrong direction :-) 2. the issue might be related to the fact that the test day (= Halloween) seems to be somewhat different qualitatively different from the training set (there is a discussion about that in the forum).

", "votes": null, "replies": [] }, { "id": 62364, "author_name": "konradb", "author_url": "", "post_date": "01/06/2015 16:34:31", "content": "

[quote=Herra Huu;62359]

\n

 I think it would be a better idea to do some kind of smoothing, ie. instead of sum(click)/N, you could use something like (sum(click)+alpha)/(N+2*alpha), where alpha>0. (I think it's called Laplace smoothing, which correspond to certain kind of Bayesian prior&likelihood). Then, at least infrequent features would be closer together which would prevent at least some forms of overfitting.

\n

[/quote]

\n

That is certainly a good approach, I am wondering if that's enough (or, in light of the Helloween thing, is that even treating the core issue). Due to my not-that-modern hardware, I have some optimizing to do before I can attempt it :-)

", "votes": null, "replies": [] }, { "id": 62366, "author_name": "herrahuu", "author_url": "", "post_date": "01/06/2015 16:59:08", "content": "

[quote=Konrad Banachewicz;62364]

\n

That is certainly a good approach, I am wondering if that's enough (or, in light of the Helloween thing, is that even treating the core issue).

\n

[/quote]

\n

Yeah, it's not necessary enough (depending on what kind of model/distributions of values etc). For example even with smoothing, all features with count==1 would be presented by some values x_0 (click==0) and x_1 (click==1). Then decision tree based algorithms could separate those rows from everything else with just a couple of splits => possibly huge overfitting problem (if there is many such features). I think another good idea would be at least remove rare features (say count<5). But as usual, it depends.

", "votes": null, "replies": [] }, { "id": 62367, "author_name": "inversion", "author_url": "", "post_date": "01/06/2015 16:59:21", "content": "

[quote=Konrad Banachewicz;62362]

\n

@inversion: not sure if I agree about this being difficult to overfit: 1. i am consistently seeing a discrepancy between my validation vs lb score (in the wrong direction :-) 2. the issue might be related to the fact that the test day (= Halloween) seems to be somewhat different qualitatively different from the training set (there is a discussion about that in the forum).

\n

[/quote]

\n

As I've posted earlier, my validation scores have tracked nicely with LB score.

\n

It's like the old joke:

\n", "votes": null, "replies": [] }, { "id": 62368, "author_name": "inversion", "author_url": "", "post_date": "01/06/2015 17:02:17", "content": "

P.S. Here's the reference for those who have not seen it.

\n

", "votes": null, "replies": [] }, { "id": 62380, "author_name": "inspector", "author_url": "", "post_date": "01/06/2015 23:02:02", "content": "

[quote=Herra Huu;62366]

\n

Yeah, it's not necessary enough (depending on what kind of model/distributions of values etc). For example even with smoothing, all features with count==1 would be presented by some values x_0 (click==0) and x_1 (click==1). Then decision tree based algorithms could separate those rows from everything else with just a couple of splits => possibly huge overfitting problem (if there is many such features). I think another good idea would be at least remove rare features (say count<5). But as usual, it depends.

\n

[/quote]

\n

For smaller data sets, I have fit a random effects model to each categorical variable. Making the category a random effect shrinks the estimate towards the mean. I wonder for these other purposes, perhaps averaging across a few bootstrap samples could help reduce overfiting. 

", "votes": null, "replies": [] }, { "id": 62397, "author_name": "namesj", "author_url": "", "post_date": "01/07/2015 12:14:35", "content": "

[quote=Ofd;62321]

\n

[/quote]

\n

attached R code would give you a head start .  :-)

\n

cheers

", "votes": null, "replies": [] }, { "id": 62399, "author_name": "ofdomejean", "author_url": "", "post_date": "01/07/2015 14:56:03", "content": "

Thanks to all that who gave his approach in this post. it's a great learning journey with this community :-)

", "votes": null, "replies": [] }, { "id": 62406, "author_name": "cartographic", "author_url": "", "post_date": "01/07/2015 16:07:17", "content": "

It is possible to get a reasonable score via the CTR histogram approach but my first efforts couldn't quite break the 0.40 level, from a 5m sample. If you are using R you might want to consider:

\n

- the combine.levels function from Hmisc to easily drop infrequent categories

\n

- the ave function to easily cacluate the CTR for each level you retain (though without any smoothing)

", "votes": null, "replies": [] }, { "id": 62413, "author_name": "derekzh", "author_url": "", "post_date": "01/07/2015 18:09:28", "content": "

[quote=lewis ml;62406]

\n

It is possible to get a reasonable score via the CTR histogram approach but my first efforts couldn't quite break the 0.40 level, from a 5m sample. If you are using R you might want to consider:

\n

- the combine.levels function from Hmisc to easily drop infrequent categories

\n

- the ave function to easily cacluate the CTR for each level you retain (though without any smoothing)

\n

[/quote]

\n\n

@Lewis ML, would you please elaborate on the CTR histogram approach a bit more?  I saw it mentioned from the previous competition, but couldn't actually get the gist of it... thank you in advance!

", "votes": null, "replies": [] }, { "id": 62415, "author_name": "cartographic", "author_url": "", "post_date": "01/07/2015 18:35:03", "content": "

Sure, though I'm no expert on it. As inversion described above, you:

\n

- remove infrequent levels from each variable

\n

- calculate the ctr for each level of each variable

\n

- use those new numeric features as the inputs to a gradient boosted tree

\n

As with other methods you can add features by combining levels across variables.

\n

I'm not sure if the small sample was the limiting factor on my effort. I tried both gbm and xgb via R.

\n

fchollet and the xgb authors deserve the credit for the idea. 

", "votes": null, "replies": [] }, { "id": 62485, "author_name": "herimanitra", "author_url": "", "post_date": "01/09/2015 08:21:52", "content": "

Hi all,

\n

My last unsuccessful attempt with CTR is as follows:

\n

For each day calculate CTR per variable and per hour excepted for the last day (10th day). At the end, I had a bunch of features from all 9 days (around 150) which I merged to the 10th day for training using hour as key.

\n

Good Luck,

", "votes": null, "replies": [] }, { "id": 62504, "author_name": "derekzh", "author_url": "", "post_date": "01/09/2015 15:12:56", "content": "

[quote=lewis ml;62415]

\n

Sure, though I'm no expert on it. As inversion described above, you:

\n

- remove infrequent levels from each variable

\n

- calculate the ctr for each level of each variable

\n

- use those new numeric features as the inputs to a gradient boosted tree

\n

As with other methods you can add features by combining levels across variables.

\n

I'm not sure if the small sample was the limiting factor on my effort. I tried both gbm and xgb via R.

\n

fchollet and the xgb authors deserve the credit for the idea.

\n

[/quote]

\n

Thank you for the explanation, Lewis ML. What you explained, Inversion's post, and the previous comments now all connects. I tried to replace each variable by the mean CTR at the beginning of the competition, but didn't see much result. Maybe because I didn't remove rare features, or didn't do Laplace smoothing as Herra Huu suggested.  I will give it another try and let you guys know if I see any improvements. 

", "votes": null, "replies": [] }, { "id": 62505, "author_name": "inversion", "author_url": "", "post_date": "01/09/2015 15:41:16", "content": "

@DerekZH -

\n

Also note, this method doesn't account for the interactions between features. So, if this data set has strong interactions, you wouldn't expect it to do as well as methods that account for the interactions.

", "votes": null, "replies": [] }, { "id": 62580, "author_name": "derekzh", "author_url": "", "post_date": "01/10/2015 18:59:36", "content": "

[quote=inversion;62505]

\n

@DerekZH -

\n

Also note, this method doesn't account for the interactions between features. So, if this data set has strong interactions, you wouldn't expect it to do as well as methods that account for the interactions.

\n

[/quote]

\n

I appreciate this insight, Inversion.  I will try to give the CTR histogram approach another try, just to learn the approach, and make it a candidate for later ensemble.  And I will give more thoughts to accounting for feature interactions.  

\n

Thanks again :)

", "votes": null, "replies": [] }, { "id": 62853, "author_name": "herimanitra", "author_url": "", "post_date": "01/18/2015 06:28:50", "content": "

In this case, How would you prepare the test set for prediction as It does not contain 'click'

\n

[quote=inversion;62355]

\n

Yup. Something like this:

\n

\n

You can do it using "brute force" by writing a loop that counts the number of clicks for each feature, divide by the occurrence of each feature, and then replace each feature with that average.

\n

There are, of course, more elegant ways to do this, which depend on whether you are using Python, R, etc.

\n

A google search should be able to get you what you need for your specific setup.

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 62854, "author_name": "ivanliu1989", "author_url": "", "post_date": "01/18/2015 08:05:16", "content": "

@inversion  @lewis ml

\n

Thank lewis and inversion for bringing us the idea and explanation.  

\n

There is one thing I'm still not clear is how do we deal with the null value in dataset? I think it is obviously not a good idea to consider those null values as unseen features in test dataset and allocate a smoothing value to them (eg, 0.25).

\n

So should we just conclude these null values as a new category and calculate the average CTR for this category, or do we just keep them as null values in dataset?

\n

Thanks :)

", "votes": null, "replies": [] }, { "id": 62855, "author_name": "cartographic", "author_url": "", "post_date": "01/18/2015 08:51:51", "content": "

Herimanitra: you have to build the test set by looking up the relevant ctr from the train set. So with inversion's example, any test case with A gets a 0.75 for that ctr feature.

\n

Ivan: you raise a good point but I'm not sure there's an  obvious 'right' way to deal with the issue. Might be worth trying different treatments.

\n

inversion: I feel like I'm missing something on your point about interactions. You can add interactions by creating a new categorical feature from two existing features, then calculating the ctr for it. But are saying you wouldn't expect that to be effective?

", "votes": null, "replies": [] }, { "id": 62860, "author_name": "inversion", "author_url": "", "post_date": "01/18/2015 15:04:38", "content": "

[quote=lewis ml;62855]

\n

inversion: I feel like I'm missing something on your point about interactions. You can add interactions by creating a new categorical feature from two existing features, then calculating the ctr for it. But are saying you wouldn't expect that to be effective?

\n

[/quote]

\n

You're absolutely right . . . what I was trying to convey was that if you just reduced the feature set as is to average CTRs, you shouldn't necessarily expect a decent score. But there's no reason you couldn't build interactions as you suggest.

", "votes": null, "replies": [] }, { "id": 62861, "author_name": "inversion", "author_url": "", "post_date": "01/18/2015 15:07:18", "content": "

[quote=Ivan Liu;62854]

\n

@inversion  @lewis ml

\n

So should we just conclude these null values as a new category and calculate the average CTR for this category, or do we just keep them as null values in dataset?

\n

Thanks :)

\n

[/quote]

\n

I've tried both ways . . . it's far less a hassle to change nulls to a new categorical feature.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "62321": "", "62322": "", "62340": "", "62355": "", "62356": "", "62358": "", "62359": "", "62362": "", "62364": "", "62366": "", "62367": "", "62368": "", "62380": "", "62397": "", "62399": "", "62406": "", "62413": "", "62415": "", "62485": "", "62504": "", "62505": "", "62580": "", "62853": "", "62854": "", "62855": "", "62860": "", "62861": "" }, "source": "meta" }