Is it true that bookings and clicks used for the evaluation were made by real users and not produced by designated judges? If this statement is true, I have some concerns about the evaluation procedure being used in this competition.
\nTo see why, imagine a hypothetical universe in which the ranker used by Expedia occasionally ranks relevant hotels low. Due to an effect known as the position bias, these hotels are less likely to be clicked or booked by website users, who sometimes trust the search engine and don't look through the whole search result list. It means, in turn, that a ranker which ranks such hotels high can potentially have lower NDCG than the current Expedia ranker (since the gain assigned to such hotels will be zero). In fact, in order to win the competition one is likely to need a ranker which will preserve the top N (where N can be quite small, like 5 or 10) results produced by the Expedia ranker, but reorder them in a better way.
\nIf some sort of "gold" judgements is hard to obtain (which, I assume, is the case for this problem), a better way to pose the problem would be to ask to predict clicks and bookings conditioned on an ordering (given an ordering, guess which hotels were clicked/booked). And, if one has a good click/booking predictor, he or she can then select the ranking that minimizes some kind of measure based on predicted clicks. The advantage of this formulation is that click prediction accuracy can be measured directly.
\nIs the reasoning presented here flawed in some way? If so, I would like sometime to point to the mistake. Otherwise, it would be nice to hear from the Expedia folks why they think that the competition setup is fine.
", "messages": [ { "id": "32136", "postDate": "10/08/2013 19:48:50", "content": "Is it true that bookings and clicks used for the evaluation were made by real users and not produced by designated judges? If this statement is true, I have some concerns about the evaluation procedure being used in this competition.
\nTo see why, imagine a hypothetical universe in which the ranker used by Expedia occasionally ranks relevant hotels low. Due to an effect known as the position bias, these hotels are less likely to be clicked or booked by website users, who sometimes trust the search engine and don't look through the whole search result list. It means, in turn, that a ranker which ranks such hotels high can potentially have lower NDCG than the current Expedia ranker (since the gain assigned to such hotels will be zero). In fact, in order to win the competition one is likely to need a ranker which will preserve the top N (where N can be quite small, like 5 or 10) results produced by the Expedia ranker, but reorder them in a better way.
\nIf some sort of "gold" judgements is hard to obtain (which, I assume, is the case for this problem), a better way to pose the problem would be to ask to predict clicks and bookings conditioned on an ordering (given an ordering, guess which hotels were clicked/booked). And, if one has a good click/booking predictor, he or she can then select the ranking that minimizes some kind of measure based on predicted clicks. The advantage of this formulation is that click prediction accuracy can be measured directly.
\nIs the reasoning presented here flawed in some way? If so, I would like sometime to point to the mistake. Otherwise, it would be nice to hear from the Expedia folks why they think that the competition setup is fine.
", "rawMarkdown": "", "votes": null }, { "id": "32137", "postDate": "10/08/2013 20:24:33", "content": "Imho the reasoning is flawed in that it mistakes the purpose of the competition. We're not trying to find the best hotels according to some unknown gold standard. We're trying to find these that will get clicked or booked by real users using present Expedia searchs.
", "rawMarkdown": "", "votes": null }, { "id": "32139", "postDate": "10/08/2013 20:31:04", "content": "If the purpose of the competition is to find hotels that are likely to be clicked or booked conditioned on the fact that the Expedia ranker was used (which, apparently, it is), it would be reasonable to give the position field for the test set as well (in fact, this is a very common practice). Otherwise, we have to learn the behaviour of the Expedia ranker from the training set, which is an unnecessary complication of the problem from which no one benefits.
", "rawMarkdown": "", "votes": null }, { "id": "32140", "postDate": "10/08/2013 20:37:08", "content": "Yep, I agree on this one. However the Expedia rank alone scores very high, probably not too far the best mark we will be able to reach. I guess the admins deliberatly supressed it so that competitors can prove their ability to at least reproduce it.
", "rawMarkdown": "", "votes": null }, { "id": "32142", "postDate": "10/08/2013 20:50:56", "content": "That would be my guess as well. I would also guess that they didn't realise that suppressing this information will not just make the problem harder, but significantly more ill-posed. I can't possibly imagine why would anyone formulate the problem in this way on purpose. So, if it was a deliberate decision, I would like to hear the rationale behind it in light of the arguments above.
", "rawMarkdown": "", "votes": null }, { "id": "32148", "postDate": "10/08/2013 21:29:56", "content": "Have you tried comparing the performance of your model on srch_ids split by the random_bool variable? That should give you a good idea how much this bias is actually affecting your results.
", "rawMarkdown": "", "votes": null }, { "id": "32166", "postDate": "10/09/2013 08:13:12", "content": "That doesn't seem to be correct. The same argument applies to a completely random ranker, unless users are aware that the ranking presented to them is random and they should not expect top results to be good (which, as I understand, was not the case).
", "rawMarkdown": "", "votes": null }, { "id": "32209", "postDate": "10/10/2013 11:55:14", "content": "you train separate model in random_bool = 1 and random_bool =0 datasets?
", "rawMarkdown": "", "votes": null }, { "id": "32216", "postDate": "10/10/2013 14:20:27", "content": "That certainly an option and may be wise depending on how you build your models... but I was only saying that if you think that having search results ordered well makes a big impact then you can at least attempt to measure that impact by comparing splitting on random_bool. However what hr0nix appears to be asking for is either impossible (there is no ground truth when it comes to knowing what hotel someone should book) or impractical for the competition (allowing us to use the ordering of historical searches in a model that is intended to determine the ordering of future searches is a form of leakage).
", "rawMarkdown": "", "votes": null }, { "id": "32220", "postDate": "10/10/2013 14:54:06", "content": "We did consider releasing the position column in the test set. These are some reasons why we decided not to do this:
\n1. As already noted in https://www.kaggle.com/c/expedia-personalized-sort/forums/t/5767/position-benchmark the position benchmark is not directly comparable with users' submissions. It means that the main focus of this competition is not to beat this baseline, but more to experiment with novel learning to rank methodologies, feature engineering, segmentations, etc. These new ideas will hopefully be discussed during the ICDM workshop.
\n2. We wanted to minimize the dependency of user submissions on our existing ranking methodology. As the position bias is strong in our data, the risk was that the position information will be too dominant so that other potentially interesting and powerful ideas are discarded. At the same time, the position column was included in the training data in order to be able to remove the position bias.
\n3. Predicting clicks and booking conditioned on an ordering looks interesting. However, it is not clear for me how this approach can be deployed in production, as we do not want to run two models one after the other (to keep the ranking process simple and fast). If you were to convert this model not to be conditioned on an ordering, wouldn't it be very similar to removing a position bias in the first place? Could you please elaborate on this idea?
\n4. I think that you can assume that the hotel order in the non-random impressions is at least a decent one. This means that probably learning the behavior of the Expedia ranker from the training set in addition to learning a re-ranking model will not be very beneficial for the final performance (but I might be wrong!). Obviously, this approach will fail for random impressions (around 1/3 of the data).
\n5. We did not want this competition to be easy, we wanted it to be hard :-)
\n\n
PS. Clicks and bookings are not given by any gold standards; they are given by real users interacting with our webpage.
\n\n
\n
\n
\n
", "rawMarkdown": "", "votes": null }, { "id": "32233", "postDate": "10/10/2013 22:32:16", "content": "
My 2 cents:
\nThe given data is real, so it suffered the position bias effect. No doubt about that. The visibility effect of searchs is known, and it even shows up in the random ordered benchmark.
\nThis will always happens when you guys use your data for training and i think its right to do so.
\nSo how would i deal with the position bias effect? Well my approach is to usually embed in the model the bias removing effect. So in this case i would try a very simple approach: Use positions on training and validation, but set them all to the same value when predicting to the future. The idea behind this it that the model will assign a great importance to position, but not all the importance (otherwise the position benchmark would achieve 1). Whe you make all predictions have the same position you are removing this effect while predicting.
\nWell this is a simple and first approach that i would take initialy.
\nRegarding the format of this competition, i think now it became part of it to remove the position bias effect, so i don't think there is a point in crying about it not being present in the test set...
", "rawMarkdown": "", "votes": null }, { "id": "32309", "postDate": "10/13/2013 14:59:29", "content": "Adam, thanks for the reply.
\nIn response to your question: if you can predict clicks and bookings for each ordering, then, in principle, you can enumerate all orderings, predict clicks for each of them and compute NDCG. Then the ordering having maximum value of NDCG can be chosen as the result. In practice, of course, enumerating all orderings is infeasible. But I expect certain click models to allow for a fast dynamic programming based procedure for finding the ordering with best NDCG. Such a procedure will probably be fast enough to be used at least as a re-ranker.
\nI'm not however fully satisfied with the answer. Also, comment by Leustagos shows that my point has been misunderstood at least partially. I've never complained about the position bias in the data. Removing it is, obviously, an inherent part of the problem, which is what makes it interesting. The point I was trying to make is that the evaluation procedure is heavily biased by itself, and that bias cannot easily be removed.
\nIf, as Adam said, the existing ranker produces decent rankings most of the time, and good documents are usually seen by users, the effect will probably be less noticeable. However, it wouldn't vanish completely.
\nBy the way, for an example of a competition that seems to be unaffected by the discussed problem, take a look at the Yandex Personalized Web Search Challenge.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 32137, "author_name": "jay670790", "author_url": "", "post_date": "10/08/2013 20:24:33", "content": "Imho the reasoning is flawed in that it mistakes the purpose of the competition. We're not trying to find the best hotels according to some unknown gold standard. We're trying to find these that will get clicked or booked by real users using present Expedia searchs.
", "votes": null, "replies": [] }, { "id": 32139, "author_name": "hr0nix", "author_url": "", "post_date": "10/08/2013 20:31:04", "content": "If the purpose of the competition is to find hotels that are likely to be clicked or booked conditioned on the fact that the Expedia ranker was used (which, apparently, it is), it would be reasonable to give the position field for the test set as well (in fact, this is a very common practice). Otherwise, we have to learn the behaviour of the Expedia ranker from the training set, which is an unnecessary complication of the problem from which no one benefits.
", "votes": null, "replies": [] }, { "id": 32140, "author_name": "jay670790", "author_url": "", "post_date": "10/08/2013 20:37:08", "content": "Yep, I agree on this one. However the Expedia rank alone scores very high, probably not too far the best mark we will be able to reach. I guess the admins deliberatly supressed it so that competitors can prove their ability to at least reproduce it.
", "votes": null, "replies": [] }, { "id": 32142, "author_name": "hr0nix", "author_url": "", "post_date": "10/08/2013 20:50:56", "content": "That would be my guess as well. I would also guess that they didn't realise that suppressing this information will not just make the problem harder, but significantly more ill-posed. I can't possibly imagine why would anyone formulate the problem in this way on purpose. So, if it was a deliberate decision, I would like to hear the rationale behind it in light of the arguments above.
", "votes": null, "replies": [] }, { "id": 32148, "author_name": "dmcgarry", "author_url": "", "post_date": "10/08/2013 21:29:56", "content": "Have you tried comparing the performance of your model on srch_ids split by the random_bool variable? That should give you a good idea how much this bias is actually affecting your results.
", "votes": null, "replies": [] }, { "id": 32166, "author_name": "hr0nix", "author_url": "", "post_date": "10/09/2013 08:13:12", "content": "That doesn't seem to be correct. The same argument applies to a completely random ranker, unless users are aware that the ranking presented to them is random and they should not expect top results to be good (which, as I understand, was not the case).
", "votes": null, "replies": [] }, { "id": 32209, "author_name": "qqfan116201", "author_url": "", "post_date": "10/10/2013 11:55:14", "content": "you train separate model in random_bool = 1 and random_bool =0 datasets?
", "votes": null, "replies": [] }, { "id": 32216, "author_name": "dmcgarry", "author_url": "", "post_date": "10/10/2013 14:20:27", "content": "That certainly an option and may be wise depending on how you build your models... but I was only saying that if you think that having search results ordered well makes a big impact then you can at least attempt to measure that impact by comparing splitting on random_bool. However what hr0nix appears to be asking for is either impossible (there is no ground truth when it comes to knowing what hotel someone should book) or impractical for the competition (allowing us to use the ordering of historical searches in a model that is intended to determine the ordering of future searches is a form of leakage).
", "votes": null, "replies": [] }, { "id": 32220, "author_name": "adamwoz", "author_url": "", "post_date": "10/10/2013 14:54:06", "content": "We did consider releasing the position column in the test set. These are some reasons why we decided not to do this:
\n1. As already noted in https://www.kaggle.com/c/expedia-personalized-sort/forums/t/5767/position-benchmark the position benchmark is not directly comparable with users' submissions. It means that the main focus of this competition is not to beat this baseline, but more to experiment with novel learning to rank methodologies, feature engineering, segmentations, etc. These new ideas will hopefully be discussed during the ICDM workshop.
\n2. We wanted to minimize the dependency of user submissions on our existing ranking methodology. As the position bias is strong in our data, the risk was that the position information will be too dominant so that other potentially interesting and powerful ideas are discarded. At the same time, the position column was included in the training data in order to be able to remove the position bias.
\n3. Predicting clicks and booking conditioned on an ordering looks interesting. However, it is not clear for me how this approach can be deployed in production, as we do not want to run two models one after the other (to keep the ranking process simple and fast). If you were to convert this model not to be conditioned on an ordering, wouldn't it be very similar to removing a position bias in the first place? Could you please elaborate on this idea?
\n4. I think that you can assume that the hotel order in the non-random impressions is at least a decent one. This means that probably learning the behavior of the Expedia ranker from the training set in addition to learning a re-ranking model will not be very beneficial for the final performance (but I might be wrong!). Obviously, this approach will fail for random impressions (around 1/3 of the data).
\n5. We did not want this competition to be easy, we wanted it to be hard :-)
\n\n
PS. Clicks and bookings are not given by any gold standards; they are given by real users interacting with our webpage.
\n\n
\n
\n
\n
", "votes": null, "replies": [] }, { "id": 32233, "author_name": "leustagos", "author_url": "", "post_date": "10/10/2013 22:32:16", "content": "
My 2 cents:
\nThe given data is real, so it suffered the position bias effect. No doubt about that. The visibility effect of searchs is known, and it even shows up in the random ordered benchmark.
\nThis will always happens when you guys use your data for training and i think its right to do so.
\nSo how would i deal with the position bias effect? Well my approach is to usually embed in the model the bias removing effect. So in this case i would try a very simple approach: Use positions on training and validation, but set them all to the same value when predicting to the future. The idea behind this it that the model will assign a great importance to position, but not all the importance (otherwise the position benchmark would achieve 1). Whe you make all predictions have the same position you are removing this effect while predicting.
\nWell this is a simple and first approach that i would take initialy.
\nRegarding the format of this competition, i think now it became part of it to remove the position bias effect, so i don't think there is a point in crying about it not being present in the test set...
", "votes": null, "replies": [] }, { "id": 32309, "author_name": "hr0nix", "author_url": "", "post_date": "10/13/2013 14:59:29", "content": "Adam, thanks for the reply.
\nIn response to your question: if you can predict clicks and bookings for each ordering, then, in principle, you can enumerate all orderings, predict clicks for each of them and compute NDCG. Then the ordering having maximum value of NDCG can be chosen as the result. In practice, of course, enumerating all orderings is infeasible. But I expect certain click models to allow for a fast dynamic programming based procedure for finding the ordering with best NDCG. Such a procedure will probably be fast enough to be used at least as a re-ranker.
\nI'm not however fully satisfied with the answer. Also, comment by Leustagos shows that my point has been misunderstood at least partially. I've never complained about the position bias in the data. Removing it is, obviously, an inherent part of the problem, which is what makes it interesting. The point I was trying to make is that the evaluation procedure is heavily biased by itself, and that bias cannot easily be removed.
\nIf, as Adam said, the existing ranker produces decent rankings most of the time, and good documents are usually seen by users, the effect will probably be less noticeable. However, it wouldn't vanish completely.
\nBy the way, for an example of a competition that seems to be unaffected by the discussed problem, take a look at the Yandex Personalized Web Search Challenge.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "32136": "", "32137": "", "32139": "", "32140": "", "32142": "", "32148": "", "32166": "", "32209": "", "32216": "", "32220": "", "32233": "", "32309": "" }, "source": "meta" }