{
  "id": 6228,
  "title": "My Approach",
  "url": "/competitions/expedia-personalized-sort/writeups/bing-xu-mlrush-brickmover-my-approach",
  "author_name": "",
  "post_date": "2013-11-05T01:16:33.857Z",
  "votes": 10,
  "comment_count": 44,
  "views": 18518,
  "content": "<p>As a team member, I will share my model first.</p>\n<p>I use deep learning and random forest, on nearly 8% data.</p>\n<p>I will add more information later or in my blog</p>\n<p>&nbsp;</p>",
  "messages": [
    {
      "id": "33225",
      "postDate": "11/05/2013 01:16:33",
      "content": "<p>As a team member, I will share my model first.</p>\n<p>I use deep learning and random forest, on nearly 8% data.</p>\n<p>I will add more information later or in my blog</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33229",
      "postDate": "11/05/2013 03:44:29",
      "content": "<p>deep leaning?&nbsp;</p>\n<p>really &nbsp;I am very curious about &nbsp;the features!</p>\n<p>&nbsp;can you share some about features?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33231",
      "postDate": "11/05/2013 04:32:34",
      "content": "<p>I Used only&nbsp;</p>\n<p>Property_Location_Score2!</p>\n<p>I hope Iam Right! How about Promotion_Flag? Is Promo Significant With respect to Expedia Booking!</p>\n<p>&nbsp;</p>\n<p>Thanks!</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33233",
      "postDate": "11/05/2013 06:48:36",
      "content": "<p>Congratulations to the winners, I&#8217;m looking forward to their posts!</p>\n<p>My approach:</p>\n<p>First I selected features with logistic regression, using one booking (if available) and one negative example from every query in 3/4 of the dataset and using 1/4 for validation. A pairwise approach with difference scores did hardly improve results.</p>\n<p>Then I added some techniques: sklearn/randomforests really helped, svmrank and pybrain/neuralnetworks didn&#8217;t. Only last Thursday I added ranklib/lambdaMART and it was unbelievable, wish I started using it earlier in the competition.</p>\n<p>Special features that worked really well:<br>&#8226; the number of bookings that the combi property/destination had in other queries<br>&#8226; the mean distance to other properties in the query (distance between two properties calculated as the maximum difference in distance to the user over all queries)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33234",
      "postDate": "11/05/2013 07:05:13",
      "content": "<p>Hi Gert - thanks for sharing, and the mean distance feature seems pretty clever.</p>\n<p>I have a couple questions on what you wrote:</p>\n<p>#1 When you say lambdaMART was unbelievable, how much improvement exactly did it make over a randomforest with the same features? &nbsp;More than 0.01?</p>\n<p>#2 Was it important to look at property / destination together? &nbsp;I always figured that the properties were tied to only 1 destination, but maybe I shouldn't have assumed that. &nbsp;And did you just look at the number of bookings, and not something more like the rate of bookings per search?</p>\n<p>And congrats to the winners!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33235",
      "postDate": "11/05/2013 07:28:50",
      "content": "<p>Hi BS Man, thanks!</p>\n<p>#1 the improvement was 0.008 &quot;off the shelf&quot;, over the combi of logreg and 5 random forests. I didn't have time to optimize the learning rate and the number of leaves, and I trained twice on half the dataset because I struggled with memory settings in java</p>\n<p>#2 I also tried the rate, but in the end the (natural log of the) number performed better EDIT: now I think about it: probably because of position; maybe it would have been better to calculate a rate on top 10 positions? next time...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33238",
      "postDate": "11/05/2013 08:23:40",
      "content": "<p>Our best single model is a LambdaMART from rankLib with a score of&nbsp;0.5338. It does a really good job for us even though training time was horrible.&nbsp;</p>\n<p>As features we used:</p>\n<p>-all numeric features</p>\n<p>-average of numeric features per prop_id</p>\n<p>-stddev of numeric features per prop_id</p>\n<p>-median of numeric features per prop_id</p>\n<p>&nbsp;</p>\n<p>The last 3 can be calculated using train+test.</p>\n<p>The rest of our blend were variations of rankLib/LambdaMART, GBDTs, NNs, SGD models.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33239",
      "postDate": "11/05/2013 08:42:10",
      "content": "<p>Interesting. The longest training time for my model is 4 hours, random forest with 3200 tree on a core i5 machine. It will achieve near&nbsp;0.5228</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33242",
      "postDate": "11/05/2013 08:49:17",
      "content": "<p>[quote=Michael Jahrer;33238]</p>\n<p>-average of numeric features per prop_id</p>\n<p>-stddev of numeric features per prop_id</p>\n<p>-median of numeric features per prop_id</p>\n<p>[/quote]</p>\n<p>Did you also try to average over other ids (e.g. site_id, visitor_location_country_id, prop_country_id, srch_destination_id) ? I had some success doing so.</p>\n<p>I then subtracted the averaged feature from the actual feature e.g.&nbsp;</p>\n<p style=\"padding-left: 30px\">prop_starrating - average_starrating</p>\n<p>Unfortunately, I didn't have time enough time to finish it up. So, I am interested to know if this worked out of other people. I started only Friday to work on this competition.</p>\n<p>Another idea that I had was to cluster properties/countries/destinations. Did anyone try that?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33243",
      "postDate": "11/05/2013 08:56:07",
      "content": "<p>[quote=Maarten Bosma;33242]</p>\n<p>[quote=Michael Jahrer;33238]</p>\n<p>-average of numeric features per prop_id</p>\n<p>-stddev of numeric features per prop_id</p>\n<p>-median of numeric features per prop_id</p>\n<p>[/quote]</p>\n<p>Did you also try to average over other ids (e.g. site_id, visitor_location_country_id, prop_country_id, srch_destination_id) ? I had some success doing so.</p>\n<p>I then subtracted the averaged feature from the actual feature e.g.&nbsp;</p>\n<p style=\"padding-left: 30px\">prop_starrating - average_starrating</p>\n<p>Unfortunately, I didn't have time enough time to finish it up. So, I am interested to know if this worked out of other people. I started only Friday to work on this competition.</p>\n<p>Another idea that I had was to cluster properties/countries/destinations. Did anyone try that?</p>\n<p>[/quote]</p>\n<p>Yes, we tried to do this for other categorical features like&nbsp;srch_destination_id,&nbsp;site_id but this does not work well. We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33244",
      "postDate": "11/05/2013 08:57:30",
      "content": "<p>Dear Friends!</p>\n<p>I have applied my local search ranker which uses Feature Quality Information to Focus the Top Features! But Again I applied &nbsp;my custom algorithms and got my feature pool as just a one single feature! Ex:- Promotion_Flag!</p>\n<p>But I was Skeptical and Preferred a List of Features over a Single Feature!</p>\n<p>How shall i submit my Feature Modeling for Expedia as a research poster! Please Guide Admin!</p>\n<p>With Best Regards</p>\n<p>Kannan</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33245",
      "postDate": "11/05/2013 09:07:28",
      "content": "<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p>@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33246",
      "postDate": "11/05/2013 09:09:57",
      "content": "<p>Thanks so much Mate!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33247",
      "postDate": "11/05/2013 09:12:12",
      "content": "<p>Please do the needful Gert! Very Kind of You! Thanks Mate!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33248",
      "postDate": "11/05/2013 09:32:03",
      "content": "<p>I'm not sure if you will be interested in my approach, as the final score is not impressive at all. &nbsp;However, this could be a good example of how nice ideas could be misleading in practice, especially with low computing power.</p>\n<p>First, I've done clustering of searches, using search descriptions and averages for properties related to particular query. This gave me 16 partitions, that I've used later. I hope, that this way I will concentrate users with similar needs together.&nbsp;</p>\n<p>I've then standardized properties variables over the search, so every final column was presenting the information only in relation to other properties within the same query.</p>\n<p>Finally, I've trained 32 random forests, two for every search cluster - one for random order, and one for model based. &nbsp;</p>\n<p>I've been training using binary target coming from click_book, in proportion 1 to 1, sampling by each search_id. As I've been using old laptop with only 2GB RAM, I've had to do a lot of resampling to fit data frames in memory. That was also the reason for starting with clustering and working separately over partitions. Final order was done according to estimated probability of being clicked per search.</p>\n<p>If I had a second chance, I will switch from click_bool to booking_book and then use better hardware, as using 2GB was crazy.</p>\n<p>Congratulations for the winners!</p>\n<p>B.</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33250",
      "postDate": "11/05/2013 09:51:37",
      "content": "<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p>@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</p>\n<p>&nbsp;</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33251",
      "postDate": "11/05/2013 10:05:59",
      "content": "<p>[quote=Michael Jahrer;33250]</p>\n<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>\n<p>[/quote]</p>\n<p>Fair enough, features on the prop_id are the most relevant because we have to rank propids. But I mean how does it function,</p>\n<p>Option 1: the mean of te prop_id is useful as a &quot;cleaned value&quot; and more accurately represents the actual value (because many values are clearly wrong; I filtered out prices that are more than 5 standard deviations away from the mean price within the query)</p>\n<p>Option 2: the mean of the prop_id is useful as historical (and future) information, for example: a price that is lower than average, increases the probability of being booked</p>\n<p>&nbsp;And question, just curious: did you include clicking and booking bools as numerical features to calculate averages over propids?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33257",
      "postDate": "11/05/2013 11:43:24",
      "content": "<p>[quote=Gert;33251]</p>\n<p>[quote=Michael Jahrer;33250]</p>\n<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>\n<p>[/quote]</p>\n<p>Fair enough, features on the prop_id are the most relevant because we have to rank propids. But I mean how does it function,</p>\n<p>Option 1: the mean of te prop_id is useful as a &quot;cleaned value&quot; and more accurately represents the actual value (because many values are clearly wrong; I filtered out prices that are more than 5 standard deviations away from the mean price within the query)</p>\n<p>Option 2: the mean of the prop_id is useful as historical (and future) information, for example: a price that is lower than average, increases the probability of being booked</p>\n<p>&nbsp;And question, just curious: did you include clicking and booking bools as numerical features to calculate averages over propids?</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>I think the average of the numeric features per prop_id creates a new &quot;feature vector&quot; for each of the prop_ids. You get a better description of a prop_id in some feature space. Since rankLib can only take dense features for training it helps the LambdaMART learner at the end.</p>\n<p>&nbsp;</p>\n<p>To you second question: touching targets for feature creation is never a good idea. We used all, except&nbsp;click_bool,booking_bool,position,gross_bookings_usd</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33260",
      "postDate": "11/05/2013 13:05:58",
      "content": "<p>Hi, everybody,</p>\n<p>Congratulation to the winners!</p>\n<p>Michael, thank you very much for the description of your approach. We also used LambdaMART, but the one implemented in R (gbm with pairwise distribution). Unfortunately, we forget about such simple features as average inside prop_ids. But one more time your solution confirmed the statement: The best - the simplest :)</p>\n<p>I should not agree with you about touching targets. We used different likelihoods in our model which are calculated based on click_bool and booking_bool, and it improved a lot. Our best single model was 0.52450 without the features you described. In one of our model we used modified position as a feature with predicted values for this feature on the test set.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33261",
      "postDate": "11/05/2013 13:23:15",
      "content": "<p>[quote=Dmitry Efimov;33260]I should not agree with you about touching targets. We used different likelihoods in our model which are calculated based on click_bool and booking_bool, and it improved a lot. Our best single model was 0.52450 without the features you described. In one of our model we used modified position as a feature with predicted values for this feature on the test set.[/quote]</p>\n<p><span style=\"line-height: 1.4\">Can you tell more about these likelihoods? </span></p>\n<p><span style=\"line-height: 1.4\">I was short on time and decided for an oversimplistic model, namely to rank the hotels according to the averaged probability for each prop_id to get booked (except if a prop_id was present less than 30 times, in which case I replaced it by the grand average for all hotels in this case). It seems to reach 0.51 on the training set, but on the test set the score droped to 0.43. At first I thought this was a suggestion even a simple model can overfit on large data if you don't follow &nbsp;Michael's cautious practice, but maybe I just screw up something in the last steps (I achieved submitting in the last 10 seconds :-) ). &nbsp;</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33262",
      "postDate": "11/05/2013 13:27:06",
      "content": "<p>[quote=Jay;33261]</p>\n<p><span style=\"line-height: 1.4\">I was short on time and decided for an oversimplistic model, namely to rank the hotels according to the averaged probability for each prop_id to get booked (except if a prop_id was present less than 30 times, in which case I replaced it by the grand average for all hotels in this case). It seems to reach 0.51 on the training set, but on the test set the score droped to 0.43. At first I thought this was a suggestion even a simple model can overfit on large data if you don't follow &nbsp;Michael's cautious practice, but maybe I just screw up something in the last steps (I achieved submitting in the last 10 seconds :-) ). &nbsp;</span></p>\n<p>[/quote]</p>\n<p>I think I know what happened. If you have a lot of properties which end up with the same score, then you will rank them in the same order as they appear in the training set. In this case you will overfit quite a lot since the training set is not ordered randomly (for most of the search-queries) but according to Expedia's algorithm.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33264",
      "postDate": "11/05/2013 13:38:09",
      "content": "<p>Duh thank you! That also explains a strange behavior I noticed while trying linear combinaisons of these features (averaged click, prop_location, etc.): each combination was worst than the individual features from which it was computed. Actually some may have been better, but appeared worst as it was&nbsp;destructing the Expedia rank.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33269",
      "postDate": "11/05/2013 14:13:14",
      "content": "<p>&nbsp;I think more&#65292;but do not think about the average numeric &nbsp;features for each &nbsp;prop_ids</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33270",
      "postDate": "11/05/2013 14:33:24",
      "content": "<p>Do you test your method just on train data using the last three features as you said above?</p>\n<p>thanks !</p>\n<p>&nbsp;</p>\n<p>dylan&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33276",
      "postDate": "11/05/2013 16:19:34",
      "content": "<p>Jay, here is the link to our paper from the KDD Cup'2013:&nbsp;</p>\n<p>http://www.kdd.org/kddcup2013/sites/default/files/papers/track1place2.pdf</p>\n<p>You can read the section about likelihoods. In few words, we calculate likelihood by prop_id and shrink it by global likelihood to avoid overfitting.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33280",
      "postDate": "11/05/2013 17:50:11",
      "content": "<p>My score was not fantastic (.47497), but I thought I'd share my approach anyway. &nbsp;I'm an Expedia employee, so could not submit results until after the contest was over - my score is not on the leader board.</p>\n<p>I used a home-grown ranking algorithm. &nbsp;I assigned each property a base score based on&nbsp;prop_location_score2. &nbsp;I adjusted the value by either a factor of 10 or 100 depending on the range of values for the set of hotels identified by srch_id. &nbsp;This gave a base score of somewhere between 1 and 0.1. &nbsp;Any properties that had a NULL prop_location_score2 I gave a base score of -10.</p>\n<p>I then added the prop_starrating value to the adjusted prop_location_score2 value for an overall base score.</p>\n<p>I continued to adjust the score using a customized decision tree.</p>\n<p>Finally, sorted by the score, and output the results.</p>\n<p>To test my success rate, I sorted each set of search results based on the Expedia position only to determine how many bookings/clicks wound up at the top of the list. &nbsp;Then I sorted by my ranking value and I could compare total numbers.</p>\n<p>Pros to this approach: &nbsp;FAST. &nbsp;I was able to work with sub-sets of 10,000 searches and get results in under 2 seconds. &nbsp;The entire training set could be processed in about 40 seconds. &nbsp;This allowed for rapid adjustments to my ranking algorithm.</p>\n<p>Cons to this approach: Well, the score speaks for itself I guess. &nbsp;</p>\n<p>This was my first attempt at any kind of data science, so not knowing what kind of algorithms to apply to the data set was a major hindrance.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33286",
      "postDate": "11/05/2013 22:10:00",
      "content": "<p>To those people who used LambdaMART, how did you handle the size of the data when fitting the regression tree? &nbsp;Did you use subsampling or a parallel implementation of decision trees or did it just take a long time to run?</p>\n<p>I subsampled around 10% of the Lambda gradient, more heavily sampling from data points with larger absolute value of the gradient. &nbsp;I'd be curious to know if using all of the data helps much or not.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33298",
      "postDate": "11/06/2013 03:45:44",
      "content": "<p>[quote=dylanfan;33270]Do you test your method just on train data using the last three features as you said above?[/quote]</p>\n<p>Sorry, the question was for me? Yes, but my numbers were contaminated by the problem&nbsp;utdiscant wisely identified. I'd guess prop_loc2 and proba_click each scores slightly less than proba_book (about 0.43), and prop_loc1 significantly less, maybe 0.38.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33315",
      "postDate": "11/06/2013 14:43:57",
      "content": "<p>I have a couple of questions (especially for people in the top of the leaderboard):</p>\n<ul>\n<li>How did you handle missing values in the data?</li>\n<li>If you trained on only a sub-sample of the data, how did you choose this sub-sample?</li>\n<li>If you were training a model based on a metric different than NDCG, how did you handle the different importance values of bookings and clicks (the 5 and 1's).</li>\n<li>Did anybody get anything useful out of using the categorical variables (except for prop_id which clearly was important for the winning team)?</li>\n<li>Some of the price_usd were pr. day, and some pr. stay, how did you handle this?</li>\n<li>How many features did you discard based before training the final model?</li>\n</ul>\n<p>Personally I tried with LambdaMART, Logistic regression (with different types of regularizations) and Random Forests. I only used the first 1,500,000 samples from the training set. I added a bunch of new features, where my most important ones were the ones constructed pr. query, for example the price of a hotel compared to the cheapest one in the query (both as absolute and percentage).</p>\n<p>It looks as if my techniques were quite similar to a lot of the people in the top of the leaderboard, but I only managed to get in top 140. Consequently I am looking at where my approach fails.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33317",
      "postDate": "11/06/2013 14:59:03",
      "content": "<p>I think this may have something to do with the fact that the price is more likely to go up from the previous price if the property was booked on the previous search and vice versa if the property was not booked. Similarly the property is less likely to go on promotion (on average) if it was booked on the previous search and vice versa.&nbsp;</p>\n<p>&nbsp;</p>\n<p>[quote=Gert;33251]</p>\n<p>[quote=Michael Jahrer;33250]</p>\n<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>\n<p>[/quote]</p>\n<p>Fair enough, features on the prop_id are the most relevant because we have to rank propids. But I mean how does it function,</p>\n<p>Option 1: the mean of te prop_id is useful as a &quot;cleaned value&quot; and more accurately represents the actual value (because many values are clearly wrong; I filtered out prices that are more than 5 standard deviations away from the mean price within the query)</p>\n<p>Option 2: the mean of the prop_id is useful as historical (and future) information, for example: a price that is lower than average, increases the probability of being booked</p>\n<p>&nbsp;And question, just curious: did you include clicking and booking bools as numerical features to calculate averages over propids?</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33320",
      "postDate": "11/06/2013 15:36:58",
      "content": "<p>I Express my Congrats to the Team Commendo for the great win!&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33337",
      "postDate": "11/06/2013 21:59:52",
      "content": "<p>@SURECOMMENDERS: shut up, professor, or I'LL CRUSH YOU!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33348",
      "postDate": "11/07/2013 06:45:52",
      "content": "<p>[quote=just a guy in a garage;33317]</p>\n<p>I think this may have something to do with the fact that the price is more likely to go up from the previous price if the property was booked on the previous search and vice versa if the property was not booked. Similarly the property is less likely to go on promotion (on average) if it was booked on the previous search and vice versa.&nbsp;</p>\n<p>[/quote]</p>\n<p>That is actually a good point. Maybe it's data leakage. Similarly, if say someone with children booked a certain hotel, maybe the Expedia algorithm is more likely to show the same hotel to someone who also searches for children next time.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33350",
      "postDate": "11/07/2013 07:07:08",
      "content": "<p>fox! why u r angry on me?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33354",
      "postDate": "11/07/2013 08:08:46",
      "content": "<p>[quote=Gert;33245]</p>\n<p>@SURECOMMENDERS: &nbsp;don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</p>\n<p>&nbsp;[/quote]</p>\n<p>LOL</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33356",
      "postDate": "11/07/2013 09:01:59",
      "content": "<p>[quote=Gert;33245]</p>\n<p><span style=\"line-height: 1.4\">@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</span></p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">Is there a nomination &quot;forum spamer&quot;? :)</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33359",
      "postDate": "11/07/2013 09:17:03",
      "content": "<p>I think it is a good idea... lol</p>\n<p>[quote=Dmitry Efimov;33356]</p>\n<p>[quote=Gert;33245]</p>\n<p><span style=\"line-height: 1.4\">@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</span></p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">Is there a nomination &quot;forum spamer&quot;? :)</span></p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33361",
      "postDate": "11/07/2013 11:03:41",
      "content": "<p>i understand now fox!&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33374",
      "postDate": "11/07/2013 15:24:11",
      "content": "<p>I would love for this thread to be about solutions to the Expedia challenge, not a personal discussion between SURECOMMENDERS and others. Can you please take these irrelevant forum-posts elsewhere?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33382",
      "postDate": "11/07/2013 17:49:16",
      "content": "<p>Just a Guy! yes i used one trick! As I always click when there is a Promotion! I represented the Promo_flag in place of Click_Bool!</p>\n<p>But Still I have a Point to Ponder! What Variables Influences Booking? Is it a Single Variable like Property_Score or any other variable! Since with my algorithm &nbsp;I have obtained only one feature !</p>\n<p>I invite Top 3 Scorers on the Leader Board to clarify on It is a SIngle Feature or More than One which influences Booking!</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33384",
      "postDate": "11/07/2013 19:25:49",
      "content": "<p>I'm sorry but are you seriously asking if the winning submissions used more than one feature? Overlooking the obvious answer to the question, did you forget to read half of the posts in this thread? Please stop spamming.</p>\n<p>&nbsp;</p>\n<p>On a more on-topic note:</p>\n<p>I computed rankings based on srch_ids (ie: out of the search results who had the best X or how much lower/higher was each result from the best), but did not rollup anything based on prop_ids... I knew I was missing some obvious features.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33395",
      "postDate": "11/08/2013 02:15:34",
      "content": "<p>Yes David! I did not read all threads as i was busy and left out with little time! I participated the Expedia challenge as an opportunity for the sake of designing a better Feature Model and still iam wondering what are the best features as i have to vaildate my feature selection experiments!&nbsp;</p>\n<p>But Now after reading so many posts on the approach i could understand it is not a feature optimization problem in it's entirety!&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33564",
      "postDate": "11/11/2013 15:40:35",
      "content": "<p>I'm sorry if I missed the post somewhere in all the spam on this thread, but @binghsu, did you end up posting your more detailed description to your blog? If so, could you post the link?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33584",
      "postDate": "11/11/2013 18:37:29",
      "content": "<p>Hi,&nbsp;</p>\n<p>I am working hard on the report paper now.</p>\n<p>I think after finishing the report I will post a draft on the old link:&nbsp;<a href=\"http://antinucleon.com/?p=587\">http://antinucleon.com/?p=587</a></p>\n<p>I will add some thing deleted from the report or not suitable to add in the report in the blog.</p>\n<p>[quote=TotalNovice;33564]</p>\n<p>I'm sorry if I missed the post somewhere in all the spam on this thread, but @binghsu, did you end up posting your more detailed description to your blog? If so, could you post the link?</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "336254",
      "postDate": "05/31/2018 10:07:04",
      "content": "<p><strong>To those who used a pointwise approach like random forests</strong></p>\n\n<p>Did you use a classification or a regression setting ? If you used a classification approach, did you limit yourself to binary classification( \"booking_bool =1 OR click_bool = 1 versus booking_bool =0 AND click_bool =0). In this case it's easy to induce a score (for random forest take the probability to belong to the \"booking_bool =1 OR click_bool = 1\" class) . But if one uses multi-class classification (here with class 1 : booking_bool =1 AND click_bool = 1, class 2 : booking_bool = 0 AND click_bool = 1 and class 3 : booking_bool = 0 AND click_bool = 0 )  how one would  induce a score with this method (let's say for random forest) ?\nThanks</p>",
      "rawMarkdown": "**To those who used a pointwise approach like random forests**\n\nDid you use a classification or a regression setting ? If you used a classification approach, did you limit yourself to binary classification( \"booking_bool =1 OR click_bool = 1 versus booking_bool =0 AND click_bool =0). In this case it's easy to induce a score (for random forest take the probability to belong to the \"booking_bool =1 OR click_bool = 1\" class) . But if one uses multi-class classification (here with class 1 : booking_bool =1 AND click_bool = 1, class 2 : booking_bool = 0 AND click_bool = 1 and class 3 : booking_bool = 0 AND click_bool = 0 )  how one would  induce a score with this method (let's say for random forest) ?\nThanks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 33229,
      "author_name": "dylanfan1",
      "author_url": "",
      "post_date": "11/05/2013 03:44:29",
      "content": "<p>deep leaning?&nbsp;</p>\n<p>really &nbsp;I am very curious about &nbsp;the features!</p>\n<p>&nbsp;can you share some about features?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33231,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/05/2013 04:32:34",
      "content": "<p>I Used only&nbsp;</p>\n<p>Property_Location_Score2!</p>\n<p>I hope Iam Right! How about Promotion_Flag? Is Promo Significant With respect to Expedia Booking!</p>\n<p>&nbsp;</p>\n<p>Thanks!</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33233,
      "author_name": "gertjac",
      "author_url": "",
      "post_date": "11/05/2013 06:48:36",
      "content": "<p>Congratulations to the winners, I&#8217;m looking forward to their posts!</p>\n<p>My approach:</p>\n<p>First I selected features with logistic regression, using one booking (if available) and one negative example from every query in 3/4 of the dataset and using 1/4 for validation. A pairwise approach with difference scores did hardly improve results.</p>\n<p>Then I added some techniques: sklearn/randomforests really helped, svmrank and pybrain/neuralnetworks didn&#8217;t. Only last Thursday I added ranklib/lambdaMART and it was unbelievable, wish I started using it earlier in the competition.</p>\n<p>Special features that worked really well:<br>&#8226; the number of bookings that the combi property/destination had in other queries<br>&#8226; the mean distance to other properties in the query (distance between two properties calculated as the maximum difference in distance to the user over all queries)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33234,
      "author_name": "bensolucky",
      "author_url": "",
      "post_date": "11/05/2013 07:05:13",
      "content": "<p>Hi Gert - thanks for sharing, and the mean distance feature seems pretty clever.</p>\n<p>I have a couple questions on what you wrote:</p>\n<p>#1 When you say lambdaMART was unbelievable, how much improvement exactly did it make over a randomforest with the same features? &nbsp;More than 0.01?</p>\n<p>#2 Was it important to look at property / destination together? &nbsp;I always figured that the properties were tied to only 1 destination, but maybe I shouldn't have assumed that. &nbsp;And did you just look at the number of bookings, and not something more like the rate of bookings per search?</p>\n<p>And congrats to the winners!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33235,
      "author_name": "gertjac",
      "author_url": "",
      "post_date": "11/05/2013 07:28:50",
      "content": "<p>Hi BS Man, thanks!</p>\n<p>#1 the improvement was 0.008 &quot;off the shelf&quot;, over the combi of logreg and 5 random forests. I didn't have time to optimize the learning rate and the number of leaves, and I trained twice on half the dataset because I struggled with memory settings in java</p>\n<p>#2 I also tried the rate, but in the end the (natural log of the) number performed better EDIT: now I think about it: probably because of position; maybe it would have been better to calculate a rate on top 10 positions? next time...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33238,
      "author_name": "mjahrer",
      "author_url": "",
      "post_date": "11/05/2013 08:23:40",
      "content": "<p>Our best single model is a LambdaMART from rankLib with a score of&nbsp;0.5338. It does a really good job for us even though training time was horrible.&nbsp;</p>\n<p>As features we used:</p>\n<p>-all numeric features</p>\n<p>-average of numeric features per prop_id</p>\n<p>-stddev of numeric features per prop_id</p>\n<p>-median of numeric features per prop_id</p>\n<p>&nbsp;</p>\n<p>The last 3 can be calculated using train+test.</p>\n<p>The rest of our blend were variations of rankLib/LambdaMART, GBDTs, NNs, SGD models.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33239,
      "author_name": "binghsu",
      "author_url": "",
      "post_date": "11/05/2013 08:42:10",
      "content": "<p>Interesting. The longest training time for my model is 4 hours, random forest with 3200 tree on a core i5 machine. It will achieve near&nbsp;0.5228</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33242,
      "author_name": "ma2rten",
      "author_url": "",
      "post_date": "11/05/2013 08:49:17",
      "content": "<p>[quote=Michael Jahrer;33238]</p>\n<p>-average of numeric features per prop_id</p>\n<p>-stddev of numeric features per prop_id</p>\n<p>-median of numeric features per prop_id</p>\n<p>[/quote]</p>\n<p>Did you also try to average over other ids (e.g. site_id, visitor_location_country_id, prop_country_id, srch_destination_id) ? I had some success doing so.</p>\n<p>I then subtracted the averaged feature from the actual feature e.g.&nbsp;</p>\n<p style=\"padding-left: 30px\">prop_starrating - average_starrating</p>\n<p>Unfortunately, I didn't have time enough time to finish it up. So, I am interested to know if this worked out of other people. I started only Friday to work on this competition.</p>\n<p>Another idea that I had was to cluster properties/countries/destinations. Did anyone try that?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33243,
      "author_name": "mjahrer",
      "author_url": "",
      "post_date": "11/05/2013 08:56:07",
      "content": "<p>[quote=Maarten Bosma;33242]</p>\n<p>[quote=Michael Jahrer;33238]</p>\n<p>-average of numeric features per prop_id</p>\n<p>-stddev of numeric features per prop_id</p>\n<p>-median of numeric features per prop_id</p>\n<p>[/quote]</p>\n<p>Did you also try to average over other ids (e.g. site_id, visitor_location_country_id, prop_country_id, srch_destination_id) ? I had some success doing so.</p>\n<p>I then subtracted the averaged feature from the actual feature e.g.&nbsp;</p>\n<p style=\"padding-left: 30px\">prop_starrating - average_starrating</p>\n<p>Unfortunately, I didn't have time enough time to finish it up. So, I am interested to know if this worked out of other people. I started only Friday to work on this competition.</p>\n<p>Another idea that I had was to cluster properties/countries/destinations. Did anyone try that?</p>\n<p>[/quote]</p>\n<p>Yes, we tried to do this for other categorical features like&nbsp;srch_destination_id,&nbsp;site_id but this does not work well. We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33244,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/05/2013 08:57:30",
      "content": "<p>Dear Friends!</p>\n<p>I have applied my local search ranker which uses Feature Quality Information to Focus the Top Features! But Again I applied &nbsp;my custom algorithms and got my feature pool as just a one single feature! Ex:- Promotion_Flag!</p>\n<p>But I was Skeptical and Preferred a List of Features over a Single Feature!</p>\n<p>How shall i submit my Feature Modeling for Expedia as a research poster! Please Guide Admin!</p>\n<p>With Best Regards</p>\n<p>Kannan</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33245,
      "author_name": "gertjac",
      "author_url": "",
      "post_date": "11/05/2013 09:07:28",
      "content": "<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p>@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33246,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/05/2013 09:09:57",
      "content": "<p>Thanks so much Mate!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33247,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/05/2013 09:12:12",
      "content": "<p>Please do the needful Gert! Very Kind of You! Thanks Mate!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33248,
      "author_name": "batarat",
      "author_url": "",
      "post_date": "11/05/2013 09:32:03",
      "content": "<p>I'm not sure if you will be interested in my approach, as the final score is not impressive at all. &nbsp;However, this could be a good example of how nice ideas could be misleading in practice, especially with low computing power.</p>\n<p>First, I've done clustering of searches, using search descriptions and averages for properties related to particular query. This gave me 16 partitions, that I've used later. I hope, that this way I will concentrate users with similar needs together.&nbsp;</p>\n<p>I've then standardized properties variables over the search, so every final column was presenting the information only in relation to other properties within the same query.</p>\n<p>Finally, I've trained 32 random forests, two for every search cluster - one for random order, and one for model based. &nbsp;</p>\n<p>I've been training using binary target coming from click_book, in proportion 1 to 1, sampling by each search_id. As I've been using old laptop with only 2GB RAM, I've had to do a lot of resampling to fit data frames in memory. That was also the reason for starting with clustering and working separately over partitions. Final order was done according to estimated probability of being clicked per search.</p>\n<p>If I had a second chance, I will switch from click_bool to booking_book and then use better hardware, as using 2GB was crazy.</p>\n<p>Congratulations for the winners!</p>\n<p>B.</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33250,
      "author_name": "mjahrer",
      "author_url": "",
      "post_date": "11/05/2013 09:51:37",
      "content": "<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p>@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</p>\n<p>&nbsp;</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33251,
      "author_name": "gertjac",
      "author_url": "",
      "post_date": "11/05/2013 10:05:59",
      "content": "<p>[quote=Michael Jahrer;33250]</p>\n<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>\n<p>[/quote]</p>\n<p>Fair enough, features on the prop_id are the most relevant because we have to rank propids. But I mean how does it function,</p>\n<p>Option 1: the mean of te prop_id is useful as a &quot;cleaned value&quot; and more accurately represents the actual value (because many values are clearly wrong; I filtered out prices that are more than 5 standard deviations away from the mean price within the query)</p>\n<p>Option 2: the mean of the prop_id is useful as historical (and future) information, for example: a price that is lower than average, increases the probability of being booked</p>\n<p>&nbsp;And question, just curious: did you include clicking and booking bools as numerical features to calculate averages over propids?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33257,
      "author_name": "mjahrer",
      "author_url": "",
      "post_date": "11/05/2013 11:43:24",
      "content": "<p>[quote=Gert;33251]</p>\n<p>[quote=Michael Jahrer;33250]</p>\n<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>\n<p>[/quote]</p>\n<p>Fair enough, features on the prop_id are the most relevant because we have to rank propids. But I mean how does it function,</p>\n<p>Option 1: the mean of te prop_id is useful as a &quot;cleaned value&quot; and more accurately represents the actual value (because many values are clearly wrong; I filtered out prices that are more than 5 standard deviations away from the mean price within the query)</p>\n<p>Option 2: the mean of the prop_id is useful as historical (and future) information, for example: a price that is lower than average, increases the probability of being booked</p>\n<p>&nbsp;And question, just curious: did you include clicking and booking bools as numerical features to calculate averages over propids?</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>I think the average of the numeric features per prop_id creates a new &quot;feature vector&quot; for each of the prop_ids. You get a better description of a prop_id in some feature space. Since rankLib can only take dense features for training it helps the LambdaMART learner at the end.</p>\n<p>&nbsp;</p>\n<p>To you second question: touching targets for feature creation is never a good idea. We used all, except&nbsp;click_bool,booking_bool,position,gross_bookings_usd</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33260,
      "author_name": "efimov",
      "author_url": "",
      "post_date": "11/05/2013 13:05:58",
      "content": "<p>Hi, everybody,</p>\n<p>Congratulation to the winners!</p>\n<p>Michael, thank you very much for the description of your approach. We also used LambdaMART, but the one implemented in R (gbm with pairwise distribution). Unfortunately, we forget about such simple features as average inside prop_ids. But one more time your solution confirmed the statement: The best - the simplest :)</p>\n<p>I should not agree with you about touching targets. We used different likelihoods in our model which are calculated based on click_bool and booking_bool, and it improved a lot. Our best single model was 0.52450 without the features you described. In one of our model we used modified position as a feature with predicted values for this feature on the test set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33261,
      "author_name": "jay670790",
      "author_url": "",
      "post_date": "11/05/2013 13:23:15",
      "content": "<p>[quote=Dmitry Efimov;33260]I should not agree with you about touching targets. We used different likelihoods in our model which are calculated based on click_bool and booking_bool, and it improved a lot. Our best single model was 0.52450 without the features you described. In one of our model we used modified position as a feature with predicted values for this feature on the test set.[/quote]</p>\n<p><span style=\"line-height: 1.4\">Can you tell more about these likelihoods? </span></p>\n<p><span style=\"line-height: 1.4\">I was short on time and decided for an oversimplistic model, namely to rank the hotels according to the averaged probability for each prop_id to get booked (except if a prop_id was present less than 30 times, in which case I replaced it by the grand average for all hotels in this case). It seems to reach 0.51 on the training set, but on the test set the score droped to 0.43. At first I thought this was a suggestion even a simple model can overfit on large data if you don't follow &nbsp;Michael's cautious practice, but maybe I just screw up something in the last steps (I achieved submitting in the last 10 seconds :-) ). &nbsp;</span></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33262,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "11/05/2013 13:27:06",
      "content": "<p>[quote=Jay;33261]</p>\n<p><span style=\"line-height: 1.4\">I was short on time and decided for an oversimplistic model, namely to rank the hotels according to the averaged probability for each prop_id to get booked (except if a prop_id was present less than 30 times, in which case I replaced it by the grand average for all hotels in this case). It seems to reach 0.51 on the training set, but on the test set the score droped to 0.43. At first I thought this was a suggestion even a simple model can overfit on large data if you don't follow &nbsp;Michael's cautious practice, but maybe I just screw up something in the last steps (I achieved submitting in the last 10 seconds :-) ). &nbsp;</span></p>\n<p>[/quote]</p>\n<p>I think I know what happened. If you have a lot of properties which end up with the same score, then you will rank them in the same order as they appear in the training set. In this case you will overfit quite a lot since the training set is not ordered randomly (for most of the search-queries) but according to Expedia's algorithm.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33264,
      "author_name": "jay670790",
      "author_url": "",
      "post_date": "11/05/2013 13:38:09",
      "content": "<p>Duh thank you! That also explains a strange behavior I noticed while trying linear combinaisons of these features (averaged click, prop_location, etc.): each combination was worst than the individual features from which it was computed. Actually some may have been better, but appeared worst as it was&nbsp;destructing the Expedia rank.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33269,
      "author_name": "dylanfan1",
      "author_url": "",
      "post_date": "11/05/2013 14:13:14",
      "content": "<p>&nbsp;I think more&#65292;but do not think about the average numeric &nbsp;features for each &nbsp;prop_ids</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33270,
      "author_name": "dylanfan1",
      "author_url": "",
      "post_date": "11/05/2013 14:33:24",
      "content": "<p>Do you test your method just on train data using the last three features as you said above?</p>\n<p>thanks !</p>\n<p>&nbsp;</p>\n<p>dylan&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33276,
      "author_name": "efimov",
      "author_url": "",
      "post_date": "11/05/2013 16:19:34",
      "content": "<p>Jay, here is the link to our paper from the KDD Cup'2013:&nbsp;</p>\n<p>http://www.kdd.org/kddcup2013/sites/default/files/papers/track1place2.pdf</p>\n<p>You can read the section about likelihoods. In few words, we calculate likelihood by prop_id and shrink it by global likelihood to avoid overfitting.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33280,
      "author_name": "beaujackson",
      "author_url": "",
      "post_date": "11/05/2013 17:50:11",
      "content": "<p>My score was not fantastic (.47497), but I thought I'd share my approach anyway. &nbsp;I'm an Expedia employee, so could not submit results until after the contest was over - my score is not on the leader board.</p>\n<p>I used a home-grown ranking algorithm. &nbsp;I assigned each property a base score based on&nbsp;prop_location_score2. &nbsp;I adjusted the value by either a factor of 10 or 100 depending on the range of values for the set of hotels identified by srch_id. &nbsp;This gave a base score of somewhere between 1 and 0.1. &nbsp;Any properties that had a NULL prop_location_score2 I gave a base score of -10.</p>\n<p>I then added the prop_starrating value to the adjusted prop_location_score2 value for an overall base score.</p>\n<p>I continued to adjust the score using a customized decision tree.</p>\n<p>Finally, sorted by the score, and output the results.</p>\n<p>To test my success rate, I sorted each set of search results based on the Expedia position only to determine how many bookings/clicks wound up at the top of the list. &nbsp;Then I sorted by my ranking value and I could compare total numbers.</p>\n<p>Pros to this approach: &nbsp;FAST. &nbsp;I was able to work with sub-sets of 10,000 searches and get results in under 2 seconds. &nbsp;The entire training set could be processed in about 40 seconds. &nbsp;This allowed for rapid adjustments to my ranking algorithm.</p>\n<p>Cons to this approach: Well, the score speaks for itself I guess. &nbsp;</p>\n<p>This was my first attempt at any kind of data science, so not knowing what kind of algorithms to apply to the data set was a major hindrance.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33286,
      "author_name": "george04",
      "author_url": "",
      "post_date": "11/05/2013 22:10:00",
      "content": "<p>To those people who used LambdaMART, how did you handle the size of the data when fitting the regression tree? &nbsp;Did you use subsampling or a parallel implementation of decision trees or did it just take a long time to run?</p>\n<p>I subsampled around 10% of the Lambda gradient, more heavily sampling from data points with larger absolute value of the gradient. &nbsp;I'd be curious to know if using all of the data helps much or not.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33298,
      "author_name": "jay670790",
      "author_url": "",
      "post_date": "11/06/2013 03:45:44",
      "content": "<p>[quote=dylanfan;33270]Do you test your method just on train data using the last three features as you said above?[/quote]</p>\n<p>Sorry, the question was for me? Yes, but my numbers were contaminated by the problem&nbsp;utdiscant wisely identified. I'd guess prop_loc2 and proba_click each scores slightly less than proba_book (about 0.43), and prop_loc1 significantly less, maybe 0.38.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33315,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "11/06/2013 14:43:57",
      "content": "<p>I have a couple of questions (especially for people in the top of the leaderboard):</p>\n<ul>\n<li>How did you handle missing values in the data?</li>\n<li>If you trained on only a sub-sample of the data, how did you choose this sub-sample?</li>\n<li>If you were training a model based on a metric different than NDCG, how did you handle the different importance values of bookings and clicks (the 5 and 1's).</li>\n<li>Did anybody get anything useful out of using the categorical variables (except for prop_id which clearly was important for the winning team)?</li>\n<li>Some of the price_usd were pr. day, and some pr. stay, how did you handle this?</li>\n<li>How many features did you discard based before training the final model?</li>\n</ul>\n<p>Personally I tried with LambdaMART, Logistic regression (with different types of regularizations) and Random Forests. I only used the first 1,500,000 samples from the training set. I added a bunch of new features, where my most important ones were the ones constructed pr. query, for example the price of a hotel compared to the cheapest one in the query (both as absolute and percentage).</p>\n<p>It looks as if my techniques were quite similar to a lot of the people in the top of the leaderboard, but I only managed to get in top 140. Consequently I am looking at where my approach fails.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33317,
      "author_name": "justaguyinagarage",
      "author_url": "",
      "post_date": "11/06/2013 14:59:03",
      "content": "<p>I think this may have something to do with the fact that the price is more likely to go up from the previous price if the property was booked on the previous search and vice versa if the property was not booked. Similarly the property is less likely to go on promotion (on average) if it was booked on the previous search and vice versa.&nbsp;</p>\n<p>&nbsp;</p>\n<p>[quote=Gert;33251]</p>\n<p>[quote=Michael Jahrer;33250]</p>\n<p>[quote=Gert;33245]</p>\n<p>[quote=Michael Jahrer;33243]</p>\n<p><span style=\"line-height: 1.4\">We got the biggest improvement by adding the avg per prop_id (0.51 -&gt; 0.53)</span></p>\n<p>[/quote]</p>\n<p>I wonder why. Maybe it corrects data errors (i e the avg price is more accurate than the current price)? or does it work as a contrast to the current price?</p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">because we want to rank prop_ids</span></p>\n<p>[/quote]</p>\n<p>Fair enough, features on the prop_id are the most relevant because we have to rank propids. But I mean how does it function,</p>\n<p>Option 1: the mean of te prop_id is useful as a &quot;cleaned value&quot; and more accurately represents the actual value (because many values are clearly wrong; I filtered out prices that are more than 5 standard deviations away from the mean price within the query)</p>\n<p>Option 2: the mean of the prop_id is useful as historical (and future) information, for example: a price that is lower than average, increases the probability of being booked</p>\n<p>&nbsp;And question, just curious: did you include clicking and booking bools as numerical features to calculate averages over propids?</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33320,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/06/2013 15:36:58",
      "content": "<p>I Express my Congrats to the Team Commendo for the great win!&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33337,
      "author_name": "zygmunt",
      "author_url": "",
      "post_date": "11/06/2013 21:59:52",
      "content": "<p>@SURECOMMENDERS: shut up, professor, or I'LL CRUSH YOU!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33348,
      "author_name": "ma2rten",
      "author_url": "",
      "post_date": "11/07/2013 06:45:52",
      "content": "<p>[quote=just a guy in a garage;33317]</p>\n<p>I think this may have something to do with the fact that the price is more likely to go up from the previous price if the property was booked on the previous search and vice versa if the property was not booked. Similarly the property is less likely to go on promotion (on average) if it was booked on the previous search and vice versa.&nbsp;</p>\n<p>[/quote]</p>\n<p>That is actually a good point. Maybe it's data leakage. Similarly, if say someone with children booked a certain hotel, maybe the Expedia algorithm is more likely to show the same hotel to someone who also searches for children next time.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33350,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/07/2013 07:07:08",
      "content": "<p>fox! why u r angry on me?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33354,
      "author_name": "rohits",
      "author_url": "",
      "post_date": "11/07/2013 08:08:46",
      "content": "<p>[quote=Gert;33245]</p>\n<p>@SURECOMMENDERS: &nbsp;don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</p>\n<p>&nbsp;[/quote]</p>\n<p>LOL</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33356,
      "author_name": "efimov",
      "author_url": "",
      "post_date": "11/07/2013 09:01:59",
      "content": "<p>[quote=Gert;33245]</p>\n<p><span style=\"line-height: 1.4\">@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</span></p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">Is there a nomination &quot;forum spamer&quot;? :)</span></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33359,
      "author_name": "binghsu",
      "author_url": "",
      "post_date": "11/07/2013 09:17:03",
      "content": "<p>I think it is a good idea... lol</p>\n<p>[quote=Dmitry Efimov;33356]</p>\n<p>[quote=Gert;33245]</p>\n<p><span style=\"line-height: 1.4\">@SURECOMMENDERS: don't understand why your only achievement is team member, I'd like to nominate you for forum regular as well</span></p>\n<p><span style=\"line-height: 1.4\">[/quote]</span></p>\n<p><span style=\"line-height: 1.4\">Is there a nomination &quot;forum spamer&quot;? :)</span></p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33361,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/07/2013 11:03:41",
      "content": "<p>i understand now fox!&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33374,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "11/07/2013 15:24:11",
      "content": "<p>I would love for this thread to be about solutions to the Expedia challenge, not a personal discussion between SURECOMMENDERS and others. Can you please take these irrelevant forum-posts elsewhere?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33382,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/07/2013 17:49:16",
      "content": "<p>Just a Guy! yes i used one trick! As I always click when there is a Promotion! I represented the Promo_flag in place of Click_Bool!</p>\n<p>But Still I have a Point to Ponder! What Variables Influences Booking? Is it a Single Variable like Property_Score or any other variable! Since with my algorithm &nbsp;I have obtained only one feature !</p>\n<p>I invite Top 3 Scorers on the Leader Board to clarify on It is a SIngle Feature or More than One which influences Booking!</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33384,
      "author_name": "dmcgarry",
      "author_url": "",
      "post_date": "11/07/2013 19:25:49",
      "content": "<p>I'm sorry but are you seriously asking if the winning submissions used more than one feature? Overlooking the obvious answer to the question, did you forget to read half of the posts in this thread? Please stop spamming.</p>\n<p>&nbsp;</p>\n<p>On a more on-topic note:</p>\n<p>I computed rankings based on srch_ids (ie: out of the search results who had the best X or how much lower/higher was each result from the best), but did not rollup anything based on prop_ids... I knew I was missing some obvious features.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33395,
      "author_name": "loveall",
      "author_url": "",
      "post_date": "11/08/2013 02:15:34",
      "content": "<p>Yes David! I did not read all threads as i was busy and left out with little time! I participated the Expedia challenge as an opportunity for the sake of designing a better Feature Model and still iam wondering what are the best features as i have to vaildate my feature selection experiments!&nbsp;</p>\n<p>But Now after reading so many posts on the approach i could understand it is not a feature optimization problem in it's entirety!&nbsp;</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33564,
      "author_name": "totalnovice",
      "author_url": "",
      "post_date": "11/11/2013 15:40:35",
      "content": "<p>I'm sorry if I missed the post somewhere in all the spam on this thread, but @binghsu, did you end up posting your more detailed description to your blog? If so, could you post the link?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33584,
      "author_name": "binghsu",
      "author_url": "",
      "post_date": "11/11/2013 18:37:29",
      "content": "<p>Hi,&nbsp;</p>\n<p>I am working hard on the report paper now.</p>\n<p>I think after finishing the report I will post a draft on the old link:&nbsp;<a href=\"http://antinucleon.com/?p=587\">http://antinucleon.com/?p=587</a></p>\n<p>I will add some thing deleted from the report or not suitable to add in the report in the blog.</p>\n<p>[quote=TotalNovice;33564]</p>\n<p>I'm sorry if I missed the post somewhere in all the spam on this thread, but @binghsu, did you end up posting your more detailed description to your blog? If so, could you post the link?</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 336254,
      "author_name": "jvschreibe",
      "author_url": "",
      "post_date": "05/31/2018 10:07:04",
      "content": "<p><strong>To those who used a pointwise approach like random forests</strong></p>\n\n<p>Did you use a classification or a regression setting ? If you used a classification approach, did you limit yourself to binary classification( \"booking_bool =1 OR click_bool = 1 versus booking_bool =0 AND click_bool =0). In this case it's easy to induce a score (for random forest take the probability to belong to the \"booking_bool =1 OR click_bool = 1\" class) . But if one uses multi-class classification (here with class 1 : booking_bool =1 AND click_bool = 1, class 2 : booking_bool = 0 AND click_bool = 1 and class 3 : booking_bool = 0 AND click_bool = 0 )  how one would  induce a score with this method (let's say for random forest) ?\nThanks</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "33225": "",
    "33229": "",
    "33231": "",
    "33233": "",
    "33234": "",
    "33235": "",
    "33238": "",
    "33239": "",
    "33242": "",
    "33243": "",
    "33244": "",
    "33245": "",
    "33246": "",
    "33247": "",
    "33248": "",
    "33250": "",
    "33251": "",
    "33257": "",
    "33260": "",
    "33261": "",
    "33262": "",
    "33264": "",
    "33269": "",
    "33270": "",
    "33276": "",
    "33280": "",
    "33286": "",
    "33298": "",
    "33315": "",
    "33317": "",
    "33320": "",
    "33337": "",
    "33348": "",
    "33350": "",
    "33354": "",
    "33356": "",
    "33359": "",
    "33361": "",
    "33374": "",
    "33382": "",
    "33384": "",
    "33395": "",
    "33564": "",
    "33584": "",
    "336254": "**To those who used a pointwise approach like random forests**\n\nDid you use a classification or a regression setting ? If you used a classification approach, did you limit yourself to binary classification( \"booking_bool =1 OR click_bool = 1 versus booking_bool =0 AND click_bool =0). In this case it's easy to induce a score (for random forest take the probability to belong to the \"booking_bool =1 OR click_bool = 1\" class) . But if one uses multi-class classification (here with class 1 : booking_bool =1 AND click_bool = 1, class 2 : booking_bool = 0 AND click_bool = 1 and class 3 : booking_bool = 0 AND click_bool = 0 )  how one would  induce a score with this method (let's say for random forest) ?\nThanks"
  },
  "source": "meta"
}