{ "id": 5667, "title": "how do I cross validate?", "url": "/competitions/expedia-personalized-sort/discussion/5667", "author_name": "", "post_date": "2013-09-05T20:52:40.177Z", "votes": null, "comment_count": 8, "views": 3461, "content": "

can anybody throw some light on this topic?

", "messages": [ { "id": "30247", "postDate": "09/05/2013 20:52:40", "content": "

can anybody throw some light on this topic?

", "rawMarkdown": "", "votes": null }, { "id": "30251", "postDate": "09/05/2013 21:39:17", "content": "

It's an interesting competition - it needs deep thinking! Bring your creative side out

", "rawMarkdown": "", "votes": null }, { "id": "30287", "postDate": "09/06/2013 07:36:43", "content": "

Hi,

\n

It think it is worth stressing that any sampling technique (such as cross validation) should not "split" data points that correspond to the same user search query (as identified by srch_id). In other words, all entries of given user search query should be in the same data sample (whether it is training or testing).

\n

Cheers,

\n

Adam

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31359", "postDate": "09/22/2013 22:13:57", "content": "

Edit: I have no idea what happened to the formatting of the last few sentences

\n

The following is my understanding of how the evaluation metric works (which is needed for cross-validation), but I'd love to be corrected if I made a mistake somewhere. After that I'll explain how I think cross-validation should work.

\n

Say we have a list of search results in the following order with the following properties:

\n

SearchId, PropertyId, click_bool, booking_bool

\n

X              , y2,                       1         , 0

\n

X              , y1,                       1         , 1

\n

X              , y4,                       0         , 0

\n

X              , y3,                       1         , 0

\n

Then as I understand it, y2 will receive a relevance score of 1 because it was clicked but not booked, y1 will have a relevance score of 5 because it was booked, y4 will have a relevance of 0 because it was not clicked and not booked, and y3 will have relevance of 1 because it too was clicked, but not booked (see http://www.kaggle.com/c/expedia-personalized-sort/details/evaluation for an explanation of where the values of 5, 1, and 0 came from). Using the notation at https://www.kaggle.com/wiki/NormalizedDiscountedCumulativeGain, k will be 4, so

\n

DCG_4 = (2^1 - 1) / (log_2(2))

\n

      + (2^5 - 1) / (log_2(3))

\n

      + (2^0 - 1) / (log_2(4))

\n

      + (2^1 - 1) / (log_2(5))

\n

= 20.9894989188(remember that here, rel_i = 1, rel_2=5, rel_3 = 0, and rel_4=1) while

\n

IDCG_4 = (2^5 - 1) / (log_2(2))

\n

       + (2^1 - 1) / (log_2(3))

\n

       + (2^1 - 1) / (log_2(4))

\n

       + (2^0 - 1) / (log_2(5))

\n

= 32.1309297536

\n

because if we'd sorted in the best possible order, we'd have ordered the booked properties first then all the clicked but not booked properties, then all the other properties.So, for this particular search X, nDCG_4 = DCG_4 / IDCG_4 = .653249.According to the evaluation page linked above, NDCG@38 is per query and averaged over all queries. I think NDCG@38 means we act as if, for each query, we keep the best 38 results. (If there were r<38 results, we pretend there are 38-r results each with score 0 for both dcg and idcg.) This way, you can't just return 1 booked result and achieve a ndcg 1. even if you happened to only return the one result that was booked, you'd have a dcg of 31 ((2^5-1)/log_2(2)), but for the denominator idcg we'd take the scores for up to the first 38 best results, so your ndcg would be less than 1. So now that we know how to compute the score ourselves, assuming we know booking_bool and click_bool (which we do for all the training data), the way to cross-validate i think is to select a fraction of the training data, compute some model, and then using this model order the remaining training data based on which results you think were clicked and booked. Then, to compute ho good your results are on the set you withheld from training, compute the ndcg_38 score fore each of the queries, average over all queries, and that's the cross-validation score.

", "rawMarkdown": "", "votes": null }, { "id": "31379", "postDate": "09/23/2013 06:46:47", "content": "

@TotalNovice it was distracting to read through the "rest of formatting" - but it was helpful. Thanks for working this out. 

", "rawMarkdown": "", "votes": null }, { "id": "31384", "postDate": "09/23/2013 08:07:48", "content": "

This is highly interesting issue.

\n

How to check our models without submitting predictions based on them?

\n

Obviously, we need to know rules how to determine which is the ideal sort order.

\n

Any guidelines for this?

\n

Can you publish rules how to determine ideal sort order, that you are using for calculation of the scores of our submissions?

\n

striki

", "rawMarkdown": "", "votes": null }, { "id": "31403", "postDate": "09/23/2013 14:30:19", "content": "

Striki, I think the ideal sort order is the order which places the booked properties (where booking_bool is true) first, then places the clicked but not booked properties after them, and finally places all of the not clicked and not booked properties after that. This is relatively easy to do with the training data because we're given click_bool and booking_bool.

\n

We're not given this data for the test set because that's what we're trying to predict. It's not that the admins don't have this information, it's just that they're not showing it to us. But behind the scenes, they still compute the ideal sort order based on which properties were booked and which were clicked.

\n

To check your models without submitting, you might try the cross-validation procedure I outlined above. (Sorry the formatting is what it is - the ="" don't show up in the editor window so I don't know where they came from)

", "rawMarkdown": "", "votes": null }, { "id": "31447", "postDate": "09/24/2013 02:09:57", "content": "

when i see the test sets, i found that all the srchid(search id) is not exsited in train datasets. so I think we can get 80% srchids as cvtrain，others as cvtest！

\n

I see you rank high, congratulation!  which method is yours now?

", "rawMarkdown": "", "votes": null }, { "id": "32471", "postDate": "10/20/2013 21:57:28", "content": "

Corrected the formatting of your post and thanks – this really helped my understanding of the problem. 

\n

"we pretend there are 38-r results each with score 0 for both dcg and idcg.)
this way, you can't just return 1 booked result and achieve a ndcg 1. even
if you happened to only return the one result that was booked, you'd have
a dcg of 31 ((2^5-1)/log_2(2)), but for the denominator idcg we'd take the
scores for up to the first 38 best results, so your ndcg would be less than 1.
so now that we know how to compute the score ourselves, assuming we know booking_bool
and click_bool (which we do for all the training data), the way to cross-validate i
think is to select a fraction of the training data, compute some model, and then using
this model order the remaining training data based on which results you think were clicked
and booked. then, to compute ho good your results are on the set you withheld from training,
compute the ndcg_38 score fore each of the queries, average over all queries, and that's the cross-validation"

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30251, "author_name": "domcastro", "author_url": "", "post_date": "09/05/2013 21:39:17", "content": "

It's an interesting competition - it needs deep thinking! Bring your creative side out

", "votes": null, "replies": [] }, { "id": 30287, "author_name": "adamwoz", "author_url": "", "post_date": "09/06/2013 07:36:43", "content": "

Hi,

\n

It think it is worth stressing that any sampling technique (such as cross validation) should not "split" data points that correspond to the same user search query (as identified by srch_id). In other words, all entries of given user search query should be in the same data sample (whether it is training or testing).

\n

Cheers,

\n

Adam

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 31359, "author_name": "totalnovice", "author_url": "", "post_date": "09/22/2013 22:13:57", "content": "

Edit: I have no idea what happened to the formatting of the last few sentences

\n

The following is my understanding of how the evaluation metric works (which is needed for cross-validation), but I'd love to be corrected if I made a mistake somewhere. After that I'll explain how I think cross-validation should work.

\n

Say we have a list of search results in the following order with the following properties:

\n

SearchId, PropertyId, click_bool, booking_bool

\n

X              , y2,                       1         , 0

\n

X              , y1,                       1         , 1

\n

X              , y4,                       0         , 0

\n

X              , y3,                       1         , 0

\n

Then as I understand it, y2 will receive a relevance score of 1 because it was clicked but not booked, y1 will have a relevance score of 5 because it was booked, y4 will have a relevance of 0 because it was not clicked and not booked, and y3 will have relevance of 1 because it too was clicked, but not booked (see http://www.kaggle.com/c/expedia-personalized-sort/details/evaluation for an explanation of where the values of 5, 1, and 0 came from). Using the notation at https://www.kaggle.com/wiki/NormalizedDiscountedCumulativeGain, k will be 4, so

\n

DCG_4 = (2^1 - 1) / (log_2(2))

\n

      + (2^5 - 1) / (log_2(3))

\n

      + (2^0 - 1) / (log_2(4))

\n

      + (2^1 - 1) / (log_2(5))

\n

= 20.9894989188(remember that here, rel_i = 1, rel_2=5, rel_3 = 0, and rel_4=1) while

\n

IDCG_4 = (2^5 - 1) / (log_2(2))

\n

       + (2^1 - 1) / (log_2(3))

\n

       + (2^1 - 1) / (log_2(4))

\n

       + (2^0 - 1) / (log_2(5))

\n

= 32.1309297536

\n

because if we'd sorted in the best possible order, we'd have ordered the booked properties first then all the clicked but not booked properties, then all the other properties.So, for this particular search X, nDCG_4 = DCG_4 / IDCG_4 = .653249.According to the evaluation page linked above, NDCG@38 is per query and averaged over all queries. I think NDCG@38 means we act as if, for each query, we keep the best 38 results. (If there were r<38 results, we pretend there are 38-r results each with score 0 for both dcg and idcg.) This way, you can't just return 1 booked result and achieve a ndcg 1. even if you happened to only return the one result that was booked, you'd have a dcg of 31 ((2^5-1)/log_2(2)), but for the denominator idcg we'd take the scores for up to the first 38 best results, so your ndcg would be less than 1. So now that we know how to compute the score ourselves, assuming we know booking_bool and click_bool (which we do for all the training data), the way to cross-validate i think is to select a fraction of the training data, compute some model, and then using this model order the remaining training data based on which results you think were clicked and booked. Then, to compute ho good your results are on the set you withheld from training, compute the ndcg_38 score fore each of the queries, average over all queries, and that's the cross-validation score.

", "votes": null, "replies": [] }, { "id": 31379, "author_name": "ektagrover1", "author_url": "", "post_date": "09/23/2013 06:46:47", "content": "

@TotalNovice it was distracting to read through the "rest of formatting" - but it was helpful. Thanks for working this out. 

", "votes": null, "replies": [] }, { "id": 31384, "author_name": "striki", "author_url": "", "post_date": "09/23/2013 08:07:48", "content": "

This is highly interesting issue.

\n

How to check our models without submitting predictions based on them?

\n

Obviously, we need to know rules how to determine which is the ideal sort order.

\n

Any guidelines for this?

\n

Can you publish rules how to determine ideal sort order, that you are using for calculation of the scores of our submissions?

\n

striki

", "votes": null, "replies": [] }, { "id": 31403, "author_name": "totalnovice", "author_url": "", "post_date": "09/23/2013 14:30:19", "content": "

Striki, I think the ideal sort order is the order which places the booked properties (where booking_bool is true) first, then places the clicked but not booked properties after them, and finally places all of the not clicked and not booked properties after that. This is relatively easy to do with the training data because we're given click_bool and booking_bool.

\n

We're not given this data for the test set because that's what we're trying to predict. It's not that the admins don't have this information, it's just that they're not showing it to us. But behind the scenes, they still compute the ideal sort order based on which properties were booked and which were clicked.

\n

To check your models without submitting, you might try the cross-validation procedure I outlined above. (Sorry the formatting is what it is - the ="" don't show up in the editor window so I don't know where they came from)

", "votes": null, "replies": [] }, { "id": 31447, "author_name": "dylanfan1", "author_url": "", "post_date": "09/24/2013 02:09:57", "content": "

when i see the test sets, i found that all the srchid(search id) is not exsited in train datasets. so I think we can get 80% srchids as cvtrain，others as cvtest！

\n

I see you rank high, congratulation!  which method is yours now?

", "votes": null, "replies": [] }, { "id": 32471, "author_name": "machinedialogue", "author_url": "", "post_date": "10/20/2013 21:57:28", "content": "

Corrected the formatting of your post and thanks – this really helped my understanding of the problem. 

\n

"we pretend there are 38-r results each with score 0 for both dcg and idcg.)
this way, you can't just return 1 booked result and achieve a ndcg 1. even
if you happened to only return the one result that was booked, you'd have
a dcg of 31 ((2^5-1)/log_2(2)), but for the denominator idcg we'd take the
scores for up to the first 38 best results, so your ndcg would be less than 1.
so now that we know how to compute the score ourselves, assuming we know booking_bool
and click_bool (which we do for all the training data), the way to cross-validate i
think is to select a fraction of the training data, compute some model, and then using
this model order the remaining training data based on which results you think were clicked
and booked. then, to compute ho good your results are on the set you withheld from training,
compute the ndcg_38 score fore each of the queries, average over all queries, and that's the cross-validation"

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30247": "", "30251": "", "30287": "", "31359": "", "31379": "", "31384": "", "31403": "", "31447": "", "32471": "" }, "source": "meta" }