{ "id": 5684, "title": "Evaluation metric", "url": "/competitions/expedia-personalized-sort/discussion/5684", "author_name": "", "post_date": "2013-09-06T22:56:02.763Z", "votes": null, "comment_count": 23, "views": 10178, "content": "

Hello,

\n

 

\n

The Evaluation page says:

\n

Hotels for each user query are assigned relevance grades as follows:

\n

5 - The user purchased a room at this hotel
1 - The user clicked through to see more information on this hotel
0 - The user neither clicked on this hotel nor purchased a room at this hotel

\n

 

\n

Does a hotel that gets clicked AND booked get 5 or 6? (Is the numerator of the DCG formula 31 or 63?)

\n

 

\n

Thank you.

", "messages": [ { "id": "30379", "postDate": "09/06/2013 22:56:02", "content": "

Hello,

\n

 

\n

The Evaluation page says:

\n

Hotels for each user query are assigned relevance grades as follows:

\n

5 - The user purchased a room at this hotel
1 - The user clicked through to see more information on this hotel
0 - The user neither clicked on this hotel nor purchased a room at this hotel

\n

 

\n

Does a hotel that gets clicked AND booked get 5 or 6? (Is the numerator of the DCG formula 31 or 63?)

\n

 

\n

Thank you.

", "rawMarkdown": "", "votes": null }, { "id": "30404", "postDate": "09/07/2013 04:00:54", "content": "

Bookings override clickings, so 5

", "rawMarkdown": "", "votes": null }, { "id": "30970", "postDate": "09/16/2013 13:43:06", "content": "

I want to make sure that I am computing the evaluation metric correctly. I have a set of questions:

\n", "rawMarkdown": "", "votes": null }, { "id": "30984", "postDate": "09/16/2013 16:23:39", "content": "

All of those are correct.

", "rawMarkdown": "", "votes": null }, { "id": "30993", "postDate": "09/16/2013 18:35:39", "content": "

One of the reasons I was asking is because there is a rather large discrepancy between my scores on the leaderboard and the scores I get when predicting on the training set. A rather simple approach/baseline gives me 0.378 on the leaderboard and around 0.267 when I use the first 1,000,000 lines of the training set. I would expect something a little closer than this - and as far as I can see, I am computing the evaluation metric correctly.

\n

Is anybody else experiencing trouble with getting realistic/trustworthy scores on the training set?

", "rawMarkdown": "", "votes": null }, { "id": "31013", "postDate": "09/16/2013 21:37:28", "content": "

My CV scores are pretty close to the leaderboard. You could try calculating NDCG manually in excel and checking that against your code. My NDCG function is about 40 lines long, it would be easy to screw something up in there.

", "rawMarkdown": "", "votes": null }, { "id": "31169", "postDate": "09/19/2013 10:23:26", "content": "

Here the code for NDCG in R (2Ben: I could not find this metric in Metrics package :)

\n

#x is a vector of relevance scores
ndcg <- function(x) {
   ideal_x <- rev(sort(x))
   dcg <- function(y) sum((2^y - 1)/log(2:(length(y)+1), base = 2))
   dcg(x)/dcg(ideal_x)
}

", "rawMarkdown": "", "votes": null }, { "id": "31329", "postDate": "09/22/2013 04:39:36", "content": "

I would like to make sure that my understanding is correct - what do you get for the train set in the original sort order? I have 0.3393.

", "rawMarkdown": "", "votes": null }, { "id": "31453", "postDate": "09/24/2013 05:28:30", "content": "

so close to leadboard!   can you share how you do cv?

", "rawMarkdown": "", "votes": null }, { "id": "31468", "postDate": "09/24/2013 10:57:28", "content": "

This leader board is calculated on approximately 25% of the test data.

\n

 

\n

Is this 25% selected randomly for each submission?

", "rawMarkdown": "", "votes": null }, { "id": "31472", "postDate": "09/24/2013 11:13:18", "content": "

No, it is fixed for all submissions.

", "rawMarkdown": "", "votes": null }, { "id": "31473", "postDate": "09/24/2013 11:14:45", "content": "

[quote=Vikas Bhargava (LatentView);31468]

\n

This leader board is calculated on approximately 25% of the test data.

\n

 

\n

Is this 25% selected randomly for each submission?

\n

[/quote]

\n

The random sample is the same for everybody. All submissions are evaluated against the same sample.

", "rawMarkdown": "", "votes": null }, { "id": "31916", "postDate": "10/02/2013 11:19:03", "content": "

Hi all

\n

Computing NDCG for single search result set is quite straingtforward, having in mind this forum so far.

\n

Still, it is not clear to me how all NDCG values for each search are combined into score for the submission with many searches?

\n

X

", "rawMarkdown": "", "votes": null }, { "id": "31932", "postDate": "10/02/2013 13:35:35", "content": "

I think that if there are 100 searches, you'd add up the individual ndcg scores and then divide by 100. Your score is the mean of your individual search scores

", "rawMarkdown": "", "votes": null }, { "id": "31956", "postDate": "10/02/2013 22:47:23", "content": "

My Code in Python With Multiprocesser 

\n

 

\n

import math

\n

import multiprocessing

\n

CPU = 4

\n

def deg(rel):
    # rel: list of relevance
    d = rel[0]
    length = len(rel)
    for i in range(1, length):
        d += rel[i] / math.log(i + 1, 2)
    return d

\n

def ndcg(rel):
    # rel: list of relevances
    irel = sorted(rel, reverse = True)
    return dcg(rel) / dcg(irel)

\n

def _single_ndcg(sid, rel):
    return sid, ndcg(rel)

\n

def batch_ndcg(rels):
    # rels: touples of (key, rel), rel is a list
    pool = multiprocessing.Pool(CPU)
    ndcgs = {}
    for sid, value in pool.imap_unordered(_single_ndcg, rels, chunksize=1000):
        ndcgs[sid] = value
    score = 0
    keys = ndcgs.keys()
    for sid in keys:
        score += ndcgs[sid]
    return score * 1.0 / len(keys)

\n

 

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31972", "postDate": "10/03/2013 06:33:36", "content": "

How to evaluate NDCG if the user didn't book or click any of the hotel? In this case, both DCG and IDCG will be 0, so I am wondering are we going to let NDCG=1 since any rank will be the same as best rank, or remove this search when averaging individual NDCGs, or do something else?

", "rawMarkdown": "", "votes": null }, { "id": "31983", "postDate": "10/03/2013 12:25:37", "content": "

I have the same problem with different scores over train data and official score for my submissions.

\n

Did you find solution/reason for this?

", "rawMarkdown": "", "votes": null }, { "id": "32035", "postDate": "10/04/2013 19:25:16", "content": "

Based on my own observation, all srch_ids have at least one click_bool==1. So there won't be 0/0 situation.

\n

For those interested, here is my R code. It gets about .49xx on the training data.

\n

my_ndcg<-function(rel, idx, orderby, k=1000){
  d<-data.frame(rel=rel, idx=idx, one=1, o=orderby)
  d<-d[order(d$idx, -d$rel),]
  d$co<-with(d, unlist(tapply(one, idx, cumsum)))
  d$v<-with(d, (2^rel-1)/(log(co+1, 2)))
  #print(head(d, 20))
  dcg0<-with(d[d$co<=k,], unlist(tapply(v, idx, sum)))
  d<-d[order(d$idx, d$o),]
  d$co<-with(d, unlist(tapply(one, idx, cumsum)))
  d$v<-with(d, (2^rel-1)/(log(co+1, 2)))
  #print(head(d, 20))
  #dcg<-sum(d$v[d$co<=k])
  dcg<-with(d[d$co<=k,], unlist(tapply(v, idx, sum)))
  return(mean(dcg/dcg0))
}

\n

#sample call

\n

with(train, my_ndcg(rel, srch_id, -pred, 38))

\n

XXX

", "rawMarkdown": "", "votes": null }, { "id": "32066", "postDate": "10/06/2013 15:13:34", "content": "

Can anyone enlighten me why the K needs to be set 38? 

", "rawMarkdown": "", "votes": null }, { "id": "32068", "postDate": "10/07/2013 00:23:02", "content": "

Sorry for the dumb question. I found it. "http://www.kaggle.com/c/expedia-personalized-sort/details/evaluation"

\n

"NDCG@38 is per query and averaged over all queries."

", "rawMarkdown": "", "votes": null }, { "id": "32238", "postDate": "10/11/2013 02:01:16", "content": "

does anybody have this metric implemented in c++?

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "32299", "postDate": "10/13/2013 00:23:57", "content": "

Owen, thanks for posting your code. Apologies if I'm being dense, but a couple of clarifying questions for you. In this instance, rel is an rank ordered list of property id's, idx is the srch_id corresponding to each rank ordered list, correct? What does the orderby parameter mean and why is it used with "-pred" in the call?

\n

Not familiar with this metric and just trying to get my head around this conceptually...really appreciate any help, thanks!

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "32302", "postDate": "10/13/2013 04:27:54", "content": "

@swaller:

\n

rel is the actual value, i.e., rel=5 for booking=1, and rel=1 for booking=0,click=1

\n

orderby parameter is the "prediction", i.e., the value that determines the ranking of the observations within each srch_id. The negative sign is there because my "pred" is the probability of events, thus taking negative will rank the most likely positive events first.

\n

The better "orderby" ranks "rel", the higher the ndcg would be. If you use pred=-rel, you would get ndcg=1.

\n

Hope this answers your questions.

", "rawMarkdown": "", "votes": null }, { "id": "32532", "postDate": "10/22/2013 15:21:18", "content": "

A few conceptual questions:

\n

1) When submitting, we don't give our predicted relevancy scores, only the order (pairs, of srch_id, prop_id) which are sorted within srch_id by our predicted relevancy.  I understand this makes sense because if the real data was 5, 1, 0, and we predict 5,1,1 the order is the same even though we predicted the third would be clicked even if it wasn't actually.

\n

2) I have training data which I will split into a smaller training dataset and a validation dataset (the validation dataset would obviously have the actual relevancy scores).  If I understand correctly, I am using the validation dataset to verify for myself how I am doing.  I think the following is true but correct me if I'm wrong: the validation dataset should be sorted in order of actual relevancy, and then when I predict, my order may or may not follow the actual order.  So then I use the ideal order of my predictions to calculate the NCDG.  

\n

Basically when they evaluate our order, do they then combine our predictions with the test data, sort the dataset by srch_id then by actual relevancy, then compare our predicted order to the ideal order of our predictions?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30404, "author_name": "benhamner", "author_url": "", "post_date": "09/07/2013 04:00:54", "content": "

Bookings override clickings, so 5

", "votes": null, "replies": [] }, { "id": 30970, "author_name": "utdiscant", "author_url": "", "post_date": "09/16/2013 13:43:06", "content": "

I want to make sure that I am computing the evaluation metric correctly. I have a set of questions:

\n", "votes": null, "replies": [] }, { "id": 30984, "author_name": "benhamner", "author_url": "", "post_date": "09/16/2013 16:23:39", "content": "

All of those are correct.

", "votes": null, "replies": [] }, { "id": 30993, "author_name": "utdiscant", "author_url": "", "post_date": "09/16/2013 18:35:39", "content": "

One of the reasons I was asking is because there is a rather large discrepancy between my scores on the leaderboard and the scores I get when predicting on the training set. A rather simple approach/baseline gives me 0.378 on the leaderboard and around 0.267 when I use the first 1,000,000 lines of the training set. I would expect something a little closer than this - and as far as I can see, I am computing the evaluation metric correctly.

\n

Is anybody else experiencing trouble with getting realistic/trustworthy scores on the training set?

", "votes": null, "replies": [] }, { "id": 31013, "author_name": "bensolucky", "author_url": "", "post_date": "09/16/2013 21:37:28", "content": "

My CV scores are pretty close to the leaderboard. You could try calculating NDCG manually in excel and checking that against your code. My NDCG function is about 40 lines long, it would be easy to screw something up in there.

", "votes": null, "replies": [] }, { "id": 31169, "author_name": "efimov", "author_url": "", "post_date": "09/19/2013 10:23:26", "content": "

Here the code for NDCG in R (2Ben: I could not find this metric in Metrics package :)

\n

#x is a vector of relevance scores
ndcg <- function(x) {
   ideal_x <- rev(sort(x))
   dcg <- function(y) sum((2^y - 1)/log(2:(length(y)+1), base = 2))
   dcg(x)/dcg(ideal_x)
}

", "votes": null, "replies": [] }, { "id": 31329, "author_name": "bebolino", "author_url": "", "post_date": "09/22/2013 04:39:36", "content": "

I would like to make sure that my understanding is correct - what do you get for the train set in the original sort order? I have 0.3393.

", "votes": null, "replies": [] }, { "id": 31453, "author_name": "dylanfan1", "author_url": "", "post_date": "09/24/2013 05:28:30", "content": "

so close to leadboard!   can you share how you do cv?

", "votes": null, "replies": [] }, { "id": 31468, "author_name": "vikasbhargava", "author_url": "", "post_date": "09/24/2013 10:57:28", "content": "

This leader board is calculated on approximately 25% of the test data.

\n

 

\n

Is this 25% selected randomly for each submission?

", "votes": null, "replies": [] }, { "id": 31472, "author_name": "adamwoz", "author_url": "", "post_date": "09/24/2013 11:13:18", "content": "

No, it is fixed for all submissions.

", "votes": null, "replies": [] }, { "id": 31473, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "09/24/2013 11:14:45", "content": "

[quote=Vikas Bhargava (LatentView);31468]

\n

This leader board is calculated on approximately 25% of the test data.

\n

 

\n

Is this 25% selected randomly for each submission?

\n

[/quote]

\n

The random sample is the same for everybody. All submissions are evaluated against the same sample.

", "votes": null, "replies": [] }, { "id": 31916, "author_name": "striki", "author_url": "", "post_date": "10/02/2013 11:19:03", "content": "

Hi all

\n

Computing NDCG for single search result set is quite straingtforward, having in mind this forum so far.

\n

Still, it is not clear to me how all NDCG values for each search are combined into score for the submission with many searches?

\n

X

", "votes": null, "replies": [] }, { "id": 31932, "author_name": "totalnovice", "author_url": "", "post_date": "10/02/2013 13:35:35", "content": "

I think that if there are 100 searches, you'd add up the individual ndcg scores and then divide by 100. Your score is the mean of your individual search scores

", "votes": null, "replies": [] }, { "id": 31956, "author_name": "binghsu", "author_url": "", "post_date": "10/02/2013 22:47:23", "content": "

My Code in Python With Multiprocesser 

\n

 

\n

import math

\n

import multiprocessing

\n

CPU = 4

\n

def deg(rel):
    # rel: list of relevance
    d = rel[0]
    length = len(rel)
    for i in range(1, length):
        d += rel[i] / math.log(i + 1, 2)
    return d

\n

def ndcg(rel):
    # rel: list of relevances
    irel = sorted(rel, reverse = True)
    return dcg(rel) / dcg(irel)

\n

def _single_ndcg(sid, rel):
    return sid, ndcg(rel)

\n

def batch_ndcg(rels):
    # rels: touples of (key, rel), rel is a list
    pool = multiprocessing.Pool(CPU)
    ndcgs = {}
    for sid, value in pool.imap_unordered(_single_ndcg, rels, chunksize=1000):
        ndcgs[sid] = value
    score = 0
    keys = ndcgs.keys()
    for sid in keys:
        score += ndcgs[sid]
    return score * 1.0 / len(keys)

\n

 

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 31972, "author_name": "jingweiwu", "author_url": "", "post_date": "10/03/2013 06:33:36", "content": "

How to evaluate NDCG if the user didn't book or click any of the hotel? In this case, both DCG and IDCG will be 0, so I am wondering are we going to let NDCG=1 since any rank will be the same as best rank, or remove this search when averaging individual NDCGs, or do something else?

", "votes": null, "replies": [] }, { "id": 31983, "author_name": "striki", "author_url": "", "post_date": "10/03/2013 12:25:37", "content": "

I have the same problem with different scores over train data and official score for my submissions.

\n

Did you find solution/reason for this?

", "votes": null, "replies": [] }, { "id": 32035, "author_name": "owenzhang1", "author_url": "", "post_date": "10/04/2013 19:25:16", "content": "

Based on my own observation, all srch_ids have at least one click_bool==1. So there won't be 0/0 situation.

\n

For those interested, here is my R code. It gets about .49xx on the training data.

\n

my_ndcg<-function(rel, idx, orderby, k=1000){
  d<-data.frame(rel=rel, idx=idx, one=1, o=orderby)
  d<-d[order(d$idx, -d$rel),]
  d$co<-with(d, unlist(tapply(one, idx, cumsum)))
  d$v<-with(d, (2^rel-1)/(log(co+1, 2)))
  #print(head(d, 20))
  dcg0<-with(d[d$co<=k,], unlist(tapply(v, idx, sum)))
  d<-d[order(d$idx, d$o),]
  d$co<-with(d, unlist(tapply(one, idx, cumsum)))
  d$v<-with(d, (2^rel-1)/(log(co+1, 2)))
  #print(head(d, 20))
  #dcg<-sum(d$v[d$co<=k])
  dcg<-with(d[d$co<=k,], unlist(tapply(v, idx, sum)))
  return(mean(dcg/dcg0))
}

\n

#sample call

\n

with(train, my_ndcg(rel, srch_id, -pred, 38))

\n

XXX

", "votes": null, "replies": [] }, { "id": 32066, "author_name": "ericwu0", "author_url": "", "post_date": "10/06/2013 15:13:34", "content": "

Can anyone enlighten me why the K needs to be set 38? 

", "votes": null, "replies": [] }, { "id": 32068, "author_name": "ericwu0", "author_url": "", "post_date": "10/07/2013 00:23:02", "content": "

Sorry for the dumb question. I found it. "http://www.kaggle.com/c/expedia-personalized-sort/details/evaluation"

\n

"NDCG@38 is per query and averaged over all queries."

", "votes": null, "replies": [] }, { "id": 32238, "author_name": "leustagos", "author_url": "", "post_date": "10/11/2013 02:01:16", "content": "

does anybody have this metric implemented in c++?

\n

 

", "votes": null, "replies": [] }, { "id": 32299, "author_name": "swaller", "author_url": "", "post_date": "10/13/2013 00:23:57", "content": "

Owen, thanks for posting your code. Apologies if I'm being dense, but a couple of clarifying questions for you. In this instance, rel is an rank ordered list of property id's, idx is the srch_id corresponding to each rank ordered list, correct? What does the orderby parameter mean and why is it used with "-pred" in the call?

\n

Not familiar with this metric and just trying to get my head around this conceptually...really appreciate any help, thanks!

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 32302, "author_name": "owenzhang1", "author_url": "", "post_date": "10/13/2013 04:27:54", "content": "

@swaller:

\n

rel is the actual value, i.e., rel=5 for booking=1, and rel=1 for booking=0,click=1

\n

orderby parameter is the "prediction", i.e., the value that determines the ranking of the observations within each srch_id. The negative sign is there because my "pred" is the probability of events, thus taking negative will rank the most likely positive events first.

\n

The better "orderby" ranks "rel", the higher the ndcg would be. If you use pred=-rel, you would get ndcg=1.

\n

Hope this answers your questions.

", "votes": null, "replies": [] }, { "id": 32532, "author_name": "statsguy", "author_url": "", "post_date": "10/22/2013 15:21:18", "content": "

A few conceptual questions:

\n

1) When submitting, we don't give our predicted relevancy scores, only the order (pairs, of srch_id, prop_id) which are sorted within srch_id by our predicted relevancy.  I understand this makes sense because if the real data was 5, 1, 0, and we predict 5,1,1 the order is the same even though we predicted the third would be clicked even if it wasn't actually.

\n

2) I have training data which I will split into a smaller training dataset and a validation dataset (the validation dataset would obviously have the actual relevancy scores).  If I understand correctly, I am using the validation dataset to verify for myself how I am doing.  I think the following is true but correct me if I'm wrong: the validation dataset should be sorted in order of actual relevancy, and then when I predict, my order may or may not follow the actual order.  So then I use the ideal order of my predictions to calculate the NCDG.  

\n

Basically when they evaluate our order, do they then combine our predictions with the test data, sort the dataset by srch_id then by actual relevancy, then compare our predicted order to the ideal order of our predictions?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30379": "", "30404": "", "30970": "", "30984": "", "30993": "", "31013": "", "31169": "", "31329": "", "31453": "", "31468": "", "31472": "", "31473": "", "31916": "", "31932": "", "31956": "", "31972": "", "31983": "", "32035": "", "32066": "", "32068": "", "32238": "", "32299": "", "32302": "", "32532": "" }, "source": "meta" }