{ "id": 7671, "title": "Evaluation", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7671", "author_name": "", "post_date": "2014-04-11T23:16:03.290Z", "votes": null, "comment_count": 15, "views": 5726, "content": "

I understand evaluation is done via AUC. I am familiar with AUC when predictions are binary, but in this case predictions are probabilities. Just saying AUC is also a little vague definition, since I've seen (in the literature) at least two different set of measurements being used in the XY axes.

\n

Is it possible (for the sake of clarity) to provide the specific formula under which the evaluation is being done? I think many people will appreciate this.

\n

Many thanks in advance,

", "messages": [ { "id": "41767", "postDate": "04/11/2014 23:16:03", "content": "

I understand evaluation is done via AUC. I am familiar with AUC when predictions are binary, but in this case predictions are probabilities. Just saying AUC is also a little vague definition, since I've seen (in the literature) at least two different set of measurements being used in the XY axes.

\n

Is it possible (for the sake of clarity) to provide the specific formula under which the evaluation is being done? I think many people will appreciate this.

\n

Many thanks in advance,

", "rawMarkdown": "", "votes": null }, { "id": "41770", "postDate": "04/12/2014 00:19:22", "content": "

Not sure what you mean by that. It's Area under the ROC Curve. Which is meant to be probabilities since it tries to evaluate a classifier as its discrimination threshold is varied so using that metric to evaluate binary predictions defeat the purpose of using the metric in the first place [1].

\n

Can you elaborate on the 2 or more Area under the ROC Curve measurement you're referring to?

\n

Regardless here's some R code/formula to what kaggle is referring to:

\n

EDIT: It came out weird when I posted it, but you can look up the verification package in R's implementation. [2]

\n

Hope that helps.

\n

[1] http://en.wikipedia.org/wiki/Receiver_operating_characteristic

\n

[2] http://cran.r-project.org/web/packages/verification/verification.pdf

", "rawMarkdown": "", "votes": null }, { "id": "41772", "postDate": "04/12/2014 01:13:13", "content": "

What I wanted to say (in short) is that every contest I've seen so far, explicitly states the formula for the evaluation. For instance, if prediction for sample i is y_i, and true value for sample i is t_i, then the "evaluation" page provides some specific formula that depends on y_i and t_i for i=1,...,#samples.

\n

I dont see this formula (or algorithm) for this contest. That's why I brought this to your attention. Are you guys using specifically "roc.area" in R, or following the reference in the file verification.pdf that you just sent me (Mason and Graham 2002)?

", "rawMarkdown": "", "votes": null }, { "id": "41784", "postDate": "04/12/2014 13:51:29", "content": "

I'm sorry, but I'm having trouble understanding your confusion. Can you elaborate on the other Area under the ROC curves you know of?

\n\n

I linked the roc.area function from the verification package because it was previously referenced by Kaggle on the amazon competition. That said, there's only one area under the ROC Curve that I know of so I don't think I can adequately respond without knowing what other metrics or differences you're referring to. The comment about using Area under the ROC Curve to evaluate binary predictions is a bit confusing too since it defeats the purpose of the metric I am referring to. As far as I know, the pdf referencing the metric I sent you,  the implementations in R, and every other "area under the ROC" are the same thing so I'm a bit confused by what you're differentiating.

\n\n

If it helps, here are some other references for the exact metric.:

\n\n

[1] http://gim.unmc.edu/dxtests/roc3.htm

\n

[2] https://www.kaggle.com/wiki/AreaUnderCurve

\n\n

Good luck! Sorry I couldn't help much.

", "rawMarkdown": "", "votes": null }, { "id": "41786", "postDate": "04/12/2014 14:35:14", "content": "

Suppose that we have x (binary) and y (continuous) values.

\n

y is sorted, and accordingly, x also.

\n

Using the first value of y (sorted) as a threshold, sensitivity and specificity are calculated.

\n

Again, with the second value of y as a threshold, until the last value of y....

\n

Now we can draw a roc curve.

", "rawMarkdown": "", "votes": null }, { "id": "41916", "postDate": "04/15/2014 06:57:40", "content": "

Hi all, 

\n

Did you manage yet to reproduce your score on the public board with a independent testsample of the training data? 

\n

I am struggling with this. To make sure its not just due to a fluctuation in the testsample used for online evaluation I split my private testsample into junks of the same size but found variations in the AUC of about 1%... definitely much lower than the ~15% difference I observe with respect to the online score. 

\n

My ROC curve follows in principle the explanation of Jung with one difference being that I use to bin my samples into equidistant parts.. lets say p ordered in 50 bins between 0-1. 

\n

Cheers,

\n

TT

", "rawMarkdown": "", "votes": null }, { "id": "41977", "postDate": "04/15/2014 22:56:15", "content": "

[quote=Tautau;41916]

\n

Hi all, 

\n

Did you manage yet to reproduce your score on the public board with a independent testsample of the training data? 

\n

I am struggling with this.

\n

[/quote]

\n

Me too. I think it is due to noise or outliers. Have you tried removing outliers from your train set or tried normalization by any other means?

", "rawMarkdown": "", "votes": null }, { "id": "42002", "postDate": "04/16/2014 05:34:51", "content": "

Hi Triskelion, 

\n\n

thanks for your answer. 

\n

I don't think that outliers in the part of trainingdata I use for evaluation are a problem since the ROC ruve is astonishing smooth. But you might be right when it comes to the online test sample. 

\n

I should try reducing the size of the evaluation dataset until I see large fluctuation appearing. 

\n\n

There is also a second thread about this, so it seems we are not the only ones.. 

\n

Thanks!

\n

TT

", "rawMarkdown": "", "votes": null }, { "id": "42033", "postDate": "04/16/2014 13:33:40", "content": "

[quote=Tautau;42002]

\n

There is also a second thread about this, so it seems we are not the only ones.

\n

Thanks!

\n

TT

\n

[/quote]

\n\n

You are not the only ones!  I think the issue is that the test set is actually very different from the train set.  There are lots of chains, categories, companies, and brands in the test set that are not present in the training set. It's hard to make your own validation set with a similar split.

", "rawMarkdown": "", "votes": null }, { "id": "42041", "postDate": "04/16/2014 14:05:03", "content": "

[quote=Triskelion;41977]

\n

[quote=Tautau;41916]

\n

Hi all, 

\n

Did you manage yet to reproduce your score on the public board with a independent testsample of the training data? 

\n

I am struggling with this.

\n

[/quote]

\n

Me too. I think it is due to noise or outliers. Have you tried removing outliers from your train set or tried normalization by any other means?

\n

[/quote]

\n

I gave this competition a shot and got a random sampled CV score of .66 AUC and I knew something was wrong because top kagglers would have easily beaten it so I made some submissions yesterday. It confirmed my suspicion something was wrong when I got .503 AUC. I figured it was due to temporal distortions and when I split the training set to March and CV set to April, my AUC dropped to .53. I did a few more tests with the leaderboard and it seemed to match my hypothesis that it's mainly due to temporal issues, especially considering the fact that the training set is March-April and test set is the months afterwards. When I focused only on the single offervalue variable, the LB AUC slightly increased after dropping the market and brand variables.

\n

It makes sense since company success and brand success has shown correlation with sales, but not much causation (at least according to my last semester marketing professor, lol). The split based on temporal lines would force one to find variables with actual causation.

\n

EDIT: Also note that March/April CV split isn't that good either since the months in the test set are further apart from the training set.

\n

EDIT2: Actually, I think it's better to say a random CV split causes leakage and the temporal train/test split prevents it. Making a good CV set will be a challenge.

", "rawMarkdown": "", "votes": null }, { "id": "42046", "postDate": "04/16/2014 14:45:21", "content": "

Good catch! 

\n

We are looking at a time dependence one might wanna try to use in some smart way.. Lets try this ; )

", "rawMarkdown": "", "votes": null }, { "id": "42049", "postDate": "04/16/2014 14:53:06", "content": "

Leave AUC aloneee....

", "rawMarkdown": "", "votes": null }, { "id": "42199", "postDate": "04/17/2014 13:10:53", "content": "

I'm also finding that my public leaderboard scores come out lower than my CV scores.  My best leaderboard score is currently a non-learning solution.  It was simply an attempt to build a "improved benchmark."  It is about 4 lines of code using some of the features that others have described in other forum topics.

", "rawMarkdown": "", "votes": null }, { "id": "42258", "postDate": "04/17/2014 20:41:06", "content": "

Getting validation right is a challenge in itself in this competition. I think it has to do with test offers being mostly absent from the training set.

\n

import pandas as pd

\n

train = pd.read_csv( train_file )
test = pd.read_csv( test_file )

\n

for col in ( 'offer', 'chain', 'market', 'id' ):
    print "train in test:", col, sum( train[col].isin( test[col] ))
    print "test in train:", col, sum( test[col].isin( train[col] ))


train in test: offer 71672
test in train: offer 5568
train in test: chain 159966
test in train: chain 151183
train in test: market 160057
test in train: market 151484
train in test: id 0
test in train: id 0

", "rawMarkdown": "", "votes": null }, { "id": "42263", "postDate": "04/17/2014 21:33:42", "content": "

I'm a total newbie at ROC and AUC, so, I apologize in advance, but here's what I'd like: I'm pretty sure that the evaluation method takes as input two vectors, one with predicted values and one with the observed values, both ordered by customer, and calculates a single number, and I'm pretty sure that there's an algorithm for that. I would find it very helpful to have someone provide that algorithm, exactly, preferable written in standard mathematical notation. (Yes, I know, someone published a link to some C++ code, which may or may not be the algorithm used in this contest, but I would prefer not to have to learn yet another language just to figure out the evaluation method.)

", "rawMarkdown": "", "votes": null }, { "id": "42277", "postDate": "04/18/2014 02:11:49", "content": "

[quote=rks13;42263]

\n

... I would find it very helpful to have someone provide that algorithm, exactly, preferable written in standard mathematical notation ...

\n

[/quote]

\n

If you use python see: sklearn model evaluation and sklearn.metrics.roc_auc_score:

\n

import numpy as np
from sklearn.metrics import roc_auc_score
y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])
score = roc_auc_score(y_true, y_scores)
print score # 0.75

\n

If you use Matlab:

\n

[X,Y,T,AUC] = perfcurve(class,y_pred,posclass);

\n

For algorithmic C# code see this post. That may be the code in use at Kaggle.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 41770, "author_name": "duffman", "author_url": "", "post_date": "04/12/2014 00:19:22", "content": "

Not sure what you mean by that. It's Area under the ROC Curve. Which is meant to be probabilities since it tries to evaluate a classifier as its discrimination threshold is varied so using that metric to evaluate binary predictions defeat the purpose of using the metric in the first place [1].

\n

Can you elaborate on the 2 or more Area under the ROC Curve measurement you're referring to?

\n

Regardless here's some R code/formula to what kaggle is referring to:

\n

EDIT: It came out weird when I posted it, but you can look up the verification package in R's implementation. [2]

\n

Hope that helps.

\n

[1] http://en.wikipedia.org/wiki/Receiver_operating_characteristic

\n

[2] http://cran.r-project.org/web/packages/verification/verification.pdf

", "votes": null, "replies": [] }, { "id": 41772, "author_name": "somniumvenator", "author_url": "", "post_date": "04/12/2014 01:13:13", "content": "

What I wanted to say (in short) is that every contest I've seen so far, explicitly states the formula for the evaluation. For instance, if prediction for sample i is y_i, and true value for sample i is t_i, then the "evaluation" page provides some specific formula that depends on y_i and t_i for i=1,...,#samples.

\n

I dont see this formula (or algorithm) for this contest. That's why I brought this to your attention. Are you guys using specifically "roc.area" in R, or following the reference in the file verification.pdf that you just sent me (Mason and Graham 2002)?

", "votes": null, "replies": [] }, { "id": 41784, "author_name": "duffman", "author_url": "", "post_date": "04/12/2014 13:51:29", "content": "

I'm sorry, but I'm having trouble understanding your confusion. Can you elaborate on the other Area under the ROC curves you know of?

\n\n

I linked the roc.area function from the verification package because it was previously referenced by Kaggle on the amazon competition. That said, there's only one area under the ROC Curve that I know of so I don't think I can adequately respond without knowing what other metrics or differences you're referring to. The comment about using Area under the ROC Curve to evaluate binary predictions is a bit confusing too since it defeats the purpose of the metric I am referring to. As far as I know, the pdf referencing the metric I sent you,  the implementations in R, and every other "area under the ROC" are the same thing so I'm a bit confused by what you're differentiating.

\n\n

If it helps, here are some other references for the exact metric.:

\n\n

[1] http://gim.unmc.edu/dxtests/roc3.htm

\n

[2] https://www.kaggle.com/wiki/AreaUnderCurve

\n\n

Good luck! Sorry I couldn't help much.

", "votes": null, "replies": [] }, { "id": 41786, "author_name": "jung299056", "author_url": "", "post_date": "04/12/2014 14:35:14", "content": "

Suppose that we have x (binary) and y (continuous) values.

\n

y is sorted, and accordingly, x also.

\n

Using the first value of y (sorted) as a threshold, sensitivity and specificity are calculated.

\n

Again, with the second value of y as a threshold, until the last value of y....

\n

Now we can draw a roc curve.

", "votes": null, "replies": [] }, { "id": 41916, "author_name": "waldi32", "author_url": "", "post_date": "04/15/2014 06:57:40", "content": "

Hi all, 

\n

Did you manage yet to reproduce your score on the public board with a independent testsample of the training data? 

\n

I am struggling with this. To make sure its not just due to a fluctuation in the testsample used for online evaluation I split my private testsample into junks of the same size but found variations in the AUC of about 1%... definitely much lower than the ~15% difference I observe with respect to the online score. 

\n

My ROC curve follows in principle the explanation of Jung with one difference being that I use to bin my samples into equidistant parts.. lets say p ordered in 50 bins between 0-1. 

\n

Cheers,

\n

TT

", "votes": null, "replies": [] }, { "id": 41977, "author_name": "triskelion", "author_url": "", "post_date": "04/15/2014 22:56:15", "content": "

[quote=Tautau;41916]

\n

Hi all, 

\n

Did you manage yet to reproduce your score on the public board with a independent testsample of the training data? 

\n

I am struggling with this.

\n

[/quote]

\n

Me too. I think it is due to noise or outliers. Have you tried removing outliers from your train set or tried normalization by any other means?

", "votes": null, "replies": [] }, { "id": 42002, "author_name": "waldi32", "author_url": "", "post_date": "04/16/2014 05:34:51", "content": "

Hi Triskelion, 

\n\n

thanks for your answer. 

\n

I don't think that outliers in the part of trainingdata I use for evaluation are a problem since the ROC ruve is astonishing smooth. But you might be right when it comes to the online test sample. 

\n

I should try reducing the size of the evaluation dataset until I see large fluctuation appearing. 

\n\n

There is also a second thread about this, so it seems we are not the only ones.. 

\n

Thanks!

\n

TT

", "votes": null, "replies": [] }, { "id": 42033, "author_name": "zachmayer", "author_url": "", "post_date": "04/16/2014 13:33:40", "content": "

[quote=Tautau;42002]

\n

There is also a second thread about this, so it seems we are not the only ones.

\n

Thanks!

\n

TT

\n

[/quote]

\n\n

You are not the only ones!  I think the issue is that the test set is actually very different from the train set.  There are lots of chains, categories, companies, and brands in the test set that are not present in the training set. It's hard to make your own validation set with a similar split.

", "votes": null, "replies": [] }, { "id": 42041, "author_name": "duffman", "author_url": "", "post_date": "04/16/2014 14:05:03", "content": "

[quote=Triskelion;41977]

\n

[quote=Tautau;41916]

\n

Hi all, 

\n

Did you manage yet to reproduce your score on the public board with a independent testsample of the training data? 

\n

I am struggling with this.

\n

[/quote]

\n

Me too. I think it is due to noise or outliers. Have you tried removing outliers from your train set or tried normalization by any other means?

\n

[/quote]

\n

I gave this competition a shot and got a random sampled CV score of .66 AUC and I knew something was wrong because top kagglers would have easily beaten it so I made some submissions yesterday. It confirmed my suspicion something was wrong when I got .503 AUC. I figured it was due to temporal distortions and when I split the training set to March and CV set to April, my AUC dropped to .53. I did a few more tests with the leaderboard and it seemed to match my hypothesis that it's mainly due to temporal issues, especially considering the fact that the training set is March-April and test set is the months afterwards. When I focused only on the single offervalue variable, the LB AUC slightly increased after dropping the market and brand variables.

\n

It makes sense since company success and brand success has shown correlation with sales, but not much causation (at least according to my last semester marketing professor, lol). The split based on temporal lines would force one to find variables with actual causation.

\n

EDIT: Also note that March/April CV split isn't that good either since the months in the test set are further apart from the training set.

\n

EDIT2: Actually, I think it's better to say a random CV split causes leakage and the temporal train/test split prevents it. Making a good CV set will be a challenge.

", "votes": null, "replies": [] }, { "id": 42046, "author_name": "waldi32", "author_url": "", "post_date": "04/16/2014 14:45:21", "content": "

Good catch! 

\n

We are looking at a time dependence one might wanna try to use in some smart way.. Lets try this ; )

", "votes": null, "replies": [] }, { "id": 42049, "author_name": "abhishek", "author_url": "", "post_date": "04/16/2014 14:53:06", "content": "

Leave AUC aloneee....

", "votes": null, "replies": [] }, { "id": 42199, "author_name": "breakfastpirate", "author_url": "", "post_date": "04/17/2014 13:10:53", "content": "

I'm also finding that my public leaderboard scores come out lower than my CV scores.  My best leaderboard score is currently a non-learning solution.  It was simply an attempt to build a "improved benchmark."  It is about 4 lines of code using some of the features that others have described in other forum topics.

", "votes": null, "replies": [] }, { "id": 42258, "author_name": "zygmunt", "author_url": "", "post_date": "04/17/2014 20:41:06", "content": "

Getting validation right is a challenge in itself in this competition. I think it has to do with test offers being mostly absent from the training set.

\n

import pandas as pd

\n

train = pd.read_csv( train_file )
test = pd.read_csv( test_file )

\n

for col in ( 'offer', 'chain', 'market', 'id' ):
    print "train in test:", col, sum( train[col].isin( test[col] ))
    print "test in train:", col, sum( test[col].isin( train[col] ))


train in test: offer 71672
test in train: offer 5568
train in test: chain 159966
test in train: chain 151183
train in test: market 160057
test in train: market 151484
train in test: id 0
test in train: id 0

", "votes": null, "replies": [] }, { "id": 42263, "author_name": "rks138062", "author_url": "", "post_date": "04/17/2014 21:33:42", "content": "

I'm a total newbie at ROC and AUC, so, I apologize in advance, but here's what I'd like: I'm pretty sure that the evaluation method takes as input two vectors, one with predicted values and one with the observed values, both ordered by customer, and calculates a single number, and I'm pretty sure that there's an algorithm for that. I would find it very helpful to have someone provide that algorithm, exactly, preferable written in standard mathematical notation. (Yes, I know, someone published a link to some C++ code, which may or may not be the algorithm used in this contest, but I would prefer not to have to learn yet another language just to figure out the evaluation method.)

", "votes": null, "replies": [] }, { "id": 42277, "author_name": "triskelion", "author_url": "", "post_date": "04/18/2014 02:11:49", "content": "

[quote=rks13;42263]

\n

... I would find it very helpful to have someone provide that algorithm, exactly, preferable written in standard mathematical notation ...

\n

[/quote]

\n

If you use python see: sklearn model evaluation and sklearn.metrics.roc_auc_score:

\n

import numpy as np
from sklearn.metrics import roc_auc_score
y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])
score = roc_auc_score(y_true, y_scores)
print score # 0.75

\n

If you use Matlab:

\n

[X,Y,T,AUC] = perfcurve(class,y_pred,posclass);

\n

For algorithmic C# code see this post. That may be the code in use at Kaggle.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "41767": "", "41770": "", "41772": "", "41784": "", "41786": "", "41916": "", "41977": "", "42002": "", "42033": "", "42041": "", "42046": "", "42049": "", "42199": "", "42258": "", "42263": "", "42277": "" }, "source": "meta" }