Hey,
\n\nI thought I'd share with you all some of what I did for this competition. I liked this one a lot because it allowed for some creative feature engineering. My final model was an ensemble of 8 XGBs, 4NN (nolearn Lasagne), 1 GLMNET, and 1 RandomForest (using 'ranger' package in R). I used my out-of-fold CV predictions to create a second layer XGB model.
\n\nFeature Engineering:
\n\nUsers:
\nI extracted all the normal things from the dates. Then I created dummy variables for all of the categorical variables. Nothing fancy here.
Sessions:
\nFilled in blanks with "NULL" so it was a new factor level.
For each user:
\n\nThen I casted the sessions frame so each level of Action, ActionType, ActionDetail, and Device had its own column. I did this twice, once using the count and the second time using the sum of seconds_elapsed. Sometimes I would use the raw count and sec_elapsed values for the actions, actiontTypes, etc. and sometimes I would convert them to proportions. Additionally I sometimes computed similarity/distance matrices, and used nearest neighbors to diffuse the matrices. Information on this can be found here.
I said in another thread that I tried using n-grams to capture the sequence of actions and it didn't really help at all, but my single model that scored the highest on the private board used n-grams, so maybe it did help a little.
\n\nThen I created 12 "helper" columns, one for each destination. Basically what I did was find the average of each of my features for each country_destination, then found for each country_destination which features were most different based on standard deviation. For example, say the countries were [NDF, US, FR, IT] and their averages for "feature_1" are [1.04, 5.40, 0.50, 0.98], then "feature_1" would be one of the features I use in creating a helper column for the US. I find 30 of these columns for each country and sum them together and that becomes the "helper" column.
", "comments": [ { "id": 107744, "author_name": "Branden Murray", "author_url": "", "post_date": "2016-02-12T08:16:13.393000", "content": "My desktop is an 8-core Xeon, 128gb RAM, and my GPU is a Nvidia GTX 980 Ti. My laptop has a 4-core i7 and 32gb RAM. Not sure if/when I will be putting my code up yet. It will be on my GitHub if I do.
", "votes": 4, "replies": [] }, { "id": 107766, "author_name": "SK", "author_url": "", "post_date": "2016-02-12T12:32:36.527000", "content": "Thanks Branden for sharing these insights into your approach, and congrats on another top 10!
\n\nIt's always very interesting to see how solutions that are quite different from each other can perform essentially the same. I also tried n-grams to no avail, and in the end also removed about half of the session based features form my models according to marginal response dependence. But it looks like there were more important differences between our approaches.
\n\nDid all your models tackle the 12-class problem directly? or did you break it up into simplified problems? (removing low-probability classes, merging classes and so on). I found it very difficult to go beyond 0.88 or so without using the hierarchical structure of the classes (i.e., that all of the classes except NDF could be clumped into a "DF" class if you will).
\n\nAnother thing that bothered me and I couldn't figure out how to address (or if it even needed to be addressed) was the apparent discrepancy between the marginal class distributions in the train and test sets (at least according to the public LB, and I guess now also the full testset, though I haven't verified this yet). It seemed to me to be more than the expected drift, given how the latter manifested throughout the 2014 part of the trainset. One hypothesis is that the time horizon for determining the class was shorter in the testset (e.g., three months instead of a year, which could reflect on the way predictions should be made). Did you do any kind of adjustment to the classes in pre- or postprocessing? In validation?
\n\nIt also keeps amazing me how much thought and effort (and how many CPU cycles) go into these competitions.
", "votes": 1, "replies": [] }, { "id": 107737, "author_name": "Branden Murray", "author_url": "", "post_date": "2016-02-12T07:18:02.280000", "content": "@Li, Upon further review, I actually created 24 helper columns. As described above some columns had count values and others had the sum of secs_elapsed and those are on much different scales so there was one helper column summing the count columns and the other summing the secs_elapsed. Half of them were in the top 60 of feature importance, the rest were all in the top 200. I've attached a list of the column names for each country.
\n\n@Jiahuan, Yes, I included them in all of my models, I forgot about that. I sometimes imputed their values using medians and other times replaced all of their values with -1.
", "votes": 1, "replies": [] }, { "id": 107729, "author_name": "Anton Laptiev", "author_url": "", "post_date": "2016-02-12T06:41:58.867000", "content": "Nice job with feature engineering and congrats with high place!
", "votes": 1, "replies": [] }, { "id": 107878, "author_name": "David Gasquez", "author_url": "", "post_date": "2016-02-13T12:10:27.097000", "content": "Thanks! I realized I was tunning my model the wrong way setting a big learning rate and n_estimators around 20/30
", "votes": 0, "replies": [] }, { "id": 107873, "author_name": "Branden Murray", "author_url": "", "post_date": "2016-02-13T10:41:29.200000", "content": "Generally something like:
\n\neta = .005
\nmax_depth=10
\nmin_child_weight=1
\nsubsample=.5
\ncolsample_bytree=.5
@Alper, I fact I did the preprocessing in parallel and the thing that caused such a slow preprocessing was the swaping due lack of memory. In the end It only took around 2 hours thanks to several improvements. I I was curious in the preprocessing time of other participants.
\n\n@Brandem. My final file was around 180mb(train users) and took around 3 hours to fit an XGB with the problem of not being able to set a big number of trees. What where your parameters of XGB to make it run in > 6hours?
", "votes": 0, "replies": [] }, { "id": 107855, "author_name": "slimper", "author_url": "", "post_date": "2016-02-13T04:32:23.110000", "content": "@BrandonMurray
\n\nHow often your model outputs rankings different from the {other, FR, IT} for the last 3 places in top-5? Do you have any estimates, how much such rerankings add to your score?
\n\nIt would be very interesting to know, how much could be gained by correct reranking of the last 3 places, because my approach was to train a binary classifier using only users with NDF and US classes to decide which class to put on the first place, the 3rd and the 4th most likely classes were always "other" and "FR" , and some simple rules based only on age and gender were used to decide which one from the IT, GB, ES to put on the 5th place.
\n\nThe classifier for the 5th place gave me +0.0004 on the public leaderboard, but -0.00006 on the private leaderboard. However, after the change of classification threshold, the binary classifier between US and NDF plus {other, FR, IT} for the last 3 places gives me 0.88556 on the private leaderboard
", "votes": 0, "replies": [] }, { "id": 107838, "author_name": "Alper", "author_url": "", "post_date": "2016-02-12T23:47:30.037000", "content": "@David\nI think you hit your memory limit (especially due to some temporaries) which caused a lot of page-faults. Even creating one such large data structure makes your code crawl.
", "votes": 0, "replies": [] }, { "id": 107836, "author_name": "Branden Murray", "author_url": "", "post_date": "2016-02-12T23:19:35.880000", "content": "@W. Yifan, I don't think there is one, when you cast the data frame the column will just be named "Action.0" instead of "Action.NULL".
", "votes": 0, "replies": [] }, { "id": 107832, "author_name": "W. Yifan", "author_url": "", "post_date": "2016-02-12T23:04:55.537000", "content": "@Branden Murray Thanks for sharing! I have a question related to this:
\n\nIn the "session" data, you '"Filled in blanks with "NULL" so it was a new factor level.'\nSo I'm wondering:\nWhat's the difference between filling with 'NULL' and filling with 0s?
\n\nSorry for the stupid question, but it would be great to learn things from masters for a newbie =)
", "votes": 0, "replies": [] }, { "id": 107826, "author_name": "Branden Murray", "author_url": "", "post_date": "2016-02-12T22:00:42.833000", "content": "@Jiahuan, Honestly, since CV improvements in either of them didn't seem to correlate very well to the LB I kept changes as long as they both weren't obviously terrible.
\n\n@David, Preprocessing didn't take much time at all, maybe 10 minutes to run, tops. 4 days seems kind of long, even with only 4GB of RAM, hopefully someone around here in a similar boat will have some suggestions. My XGB models ran in 8-12 hours usually. I think my NNs ran in a few minutes on GPU.
", "votes": 0, "replies": [] }, { "id": 107825, "author_name": "Jared Turkewitz", "author_url": "", "post_date": "2016-02-12T21:54:56.147000", "content": "@David\nI also had memory issues in processing the sessions file using pandas get dummies on the actions. One trick that I found helpful was to sub-select the sessions file by action done before doing the pd.get_dummies one hot encoding (so do one-hot encoding on the common actions separately from the rarer actions). Still I would sometimes get kernel crashes due to memory issues on my 8GB laptop if I forgot to reset my workspace before rerunning, which was kind of frustrating. But this took only on the order of 15 minutes to do the data preprocessing.
", "votes": 0, "replies": [] }, { "id": 107823, "author_name": "David Gasquez", "author_url": "", "post_date": "2016-02-12T21:22:11.337000", "content": "Sorry, I meant a complete run. Average time per model fit and/or preprocessing methods. It took me almost 4 day to preprocess 10M rows of the sessions data in a 4Gb laptop and I was wondering the performance differences.
", "votes": 0, "replies": [] }, { "id": 107813, "author_name": "Jiahuan He", "author_url": "", "post_date": "2016-02-12T20:13:24.077000", "content": "Thanks, Branden. Just an another question, I assume you have found some inconsistency between logloss and ndcg5 when you make adjustment to your model, how do you determine whether to keep the change to your model if those two metrics tell you differently?
", "votes": 0, "replies": [] }, { "id": 107795, "author_name": "Branden Murray", "author_url": "", "post_date": "2016-02-12T17:51:12.787000", "content": "@David, What do you mean by iteration? That could mean a few things.
\n\n@SK, They all tackled the 12-class problem directly. I was a hair under 0.881 on my second submission, so I didn't feel the need to go the route of breaking it up early on, and after I got sucked deeper into the Homesite competition I ran out of time to do that kind of stuff. I also didn't make any adjustments to the classes.
\n\n@SkyLibrary, I don't know the differences between the implemenations in sklearn and R. I really only use Python for training NNs because there's no easy way (that I know of) to train NNs using GPU in R. I used Xgboost to train my second layer model, so everything got the same weight.
\n\nMy XGBs are all done using different data input, slight variations in the xgb parameters, and using mlogloss and the custom ndcg5 metrics for early stopping. Same for the NNs -- different inputs, I think 3 of them had 2 hidden layers and 1 had 3 hidden layers.
", "votes": 0, "replies": [] }, { "id": 107791, "author_name": "SkyLibrary", "author_url": "", "post_date": "2016-02-12T16:51:40.150000", "content": "Thanks for sharing @Brandon.
\n\nOur model/feature engineering have a lot in common, and we are so close in private leaderboard. Now I am wondering what the top 3 are doing.
\n\nI do have two questions about your models.\nFirst, I tried logistic regression and random forest in sklearn for ensemble with nn and xgb, but they didn't improve either local validation or public leaderboard score, are GLMNET and RandomForest (using 'ranger' package in R) different from the algorithms in skearn, how much weight did you give to GLMNET and ranger RF?
\n\nSecond, your 8 XGBs are from different data input(like different transformation of your features) or with different depth, random seeds, is the two stage xgb model one of the 8 models? Same about 4NNs, are they using different network structure or different data input?
", "votes": 0, "replies": [] }, { "id": 107762, "author_name": "David Gasquez", "author_url": "", "post_date": "2016-02-12T10:48:25.537000", "content": "[quote=Branden Murray;107744]
\n\nMy desktop is an 8-core Xeon, 128gb RAM, and my GPU is a Nvidia GTX 980 Ti. My laptop has a 4-core i7 and 32gb RAM. Not sure if/when I will be putting my code up yet. It will be on my GitHub if I do.
\n\n[/quote]
\n\nReally impressive specs. How much time did you spend per iteration in average?
", "votes": 0, "replies": [] }, { "id": 107742, "author_name": "David Gasquez", "author_url": "", "post_date": "2016-02-12T08:01:12.180000", "content": "Thanks for sharing those findings. What kind of hardware did you use? I was struggling with a 8GB machine. Also, it would be great to take a look over your code, will this be possible?
", "votes": 0, "replies": [] }, { "id": 107730, "author_name": "Jiahuan He", "author_url": "", "post_date": "2016-02-12T06:52:15.223000", "content": "Thanks for the sharing. I just wonder did you ever include those users without sessions into your model?
", "votes": 0, "replies": [] }, { "id": 107727, "author_name": "Li Li", "author_url": "", "post_date": "2016-02-12T06:31:30.600000", "content": "Congratulations on your 5th! And thank you for sharing your result.
\n\nThe way to create "helper" columns is creative. Could you share more details for this method? Like what is the columns for each country and how those helper features rank in your feature importance?
\n\nThank you!\nLi
", "votes": 0, "replies": [] }, { "id": 627461, "author_name": "mnm", "author_url": "", "post_date": "2019-09-16T02:39:09.633000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 107837, "author_name": "W. Yifan", "author_url": "", "post_date": "2016-02-12T23:31:26.110000", "content": "@Branden Murray, Thanks!
", "votes": 0, "replies": [] } ], "comment_count": 23, "messages": [ { "id": 107725, "postDate": "2016-02-12T06:08:47.300Z", "content": "Hey,
\n\nI thought I'd share with you all some of what I did for this competition. I liked this one a lot because it allowed for some creative feature engineering. My final model was an ensemble of 8 XGBs, 4NN (nolearn Lasagne), 1 GLMNET, and 1 RandomForest (using 'ranger' package in R). I used my out-of-fold CV predictions to create a second layer XGB model.
\n\nFeature Engineering:
\n\nUsers:
\nI extracted all the normal things from the dates. Then I created dummy variables for all of the categorical variables. Nothing fancy here.
Sessions:
\nFilled in blanks with "NULL" so it was a new factor level.
For each user:
\n\nThen I casted the sessions frame so each level of Action, ActionType, ActionDetail, and Device had its own column. I did this twice, once using the count and the second time using the sum of seconds_elapsed. Sometimes I would use the raw count and sec_elapsed values for the actions, actiontTypes, etc. and sometimes I would convert them to proportions. Additionally I sometimes computed similarity/distance matrices, and used nearest neighbors to diffuse the matrices. Information on this can be found here.
I said in another thread that I tried using n-grams to capture the sequence of actions and it didn't really help at all, but my single model that scored the highest on the private board used n-grams, so maybe it did help a little.
\n\nThen I created 12 "helper" columns, one for each destination. Basically what I did was find the average of each of my features for each country_destination, then found for each country_destination which features were most different based on standard deviation. For example, say the countries were [NDF, US, FR, IT] and their averages for "feature_1" are [1.04, 5.40, 0.50, 0.98], then "feature_1" would be one of the features I use in creating a helper column for the US. I find 30 of these columns for each country and sum them together and that becomes the "helper" column.
", "rawMarkdown": "Hey,\r\n\r\nI thought I'd share with you all some of what I did for this competition. I liked this one a lot because it allowed for some creative feature engineering. My final model was an ensemble of 8 XGBs, 4NN (nolearn Lasagne), 1 GLMNET, and 1 RandomForest (using 'ranger' package in R). I used my out-of-fold CV predictions to create a second layer XGB model. \r\n\r\nFeature Engineering:\r\n\r\n+ Users: \r\nI extracted all the normal things from the dates. Then I created dummy variables for all of the categorical variables. Nothing fancy here. \r\n\r\n+ Sessions: \r\nFilled in blanks with \"NULL\" so it was a new factor level. \r\n \r\n For each user:\r\n + Number of actions taken \r\n + Number of\r\n unique Actions, ActionTypes, ActionDetails, Devices \r\n + Sum, mean, min,\r\n max, median, s.d., skewness, kurtosis of seconds_elapsed \r\n + Entropy of\r\n Actions, ActionTypes, ActionDetails, Devices, Secs_elapsed of the\r\n first, second, last, second-to-last, and third-to-last Actions\r\n + Ratios of unique Actions, ActionTypes, ActionDetails, Devices\r\n + Ratios of entropies \r\n\r\n\r\n Then I casted the sessions frame so each level of Action, ActionType, ActionDetail, and Device had its own column. I did this twice, once using the count and the second time using the sum of seconds_elapsed. Sometimes I would use the raw count and sec_elapsed values for the actions, actiontTypes, etc. and sometimes I would convert them to proportions. Additionally I sometimes computed similarity/distance matrices, and used nearest neighbors to diffuse the matrices. Information on this can be found [here][1].\r\n\r\nI said in another thread that I tried using n-grams to capture the sequence of actions and it didn't really help at all, but my single model that scored the highest on the private board used n-grams, so maybe it did help a little.\r\n\r\nThen I created 12 \"helper\" columns, one for each destination. Basically what I did was find the average of each of my features for each country_destination, then found for each country_destination which features were most different based on standard deviation. For example, say the countries were [NDF, US, FR, IT] and their averages for \"feature_1\" are [1.04, 5.40, 0.50, 0.98], then \"feature_1\" would be one of the features I use in creating a helper column for the US. I find 30 of these columns for each country and sum them together and that becomes the \"helper\" column.\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/walmart-recruiting-trip-type-classification/forums/t/17632/department-features-correlation-graph", "votes": 32 }, { "id": 107744, "postDate": "2016-02-12T08:16:13.393Z", "content": "My desktop is an 8-core Xeon, 128gb RAM, and my GPU is a Nvidia GTX 980 Ti. My laptop has a 4-core i7 and 32gb RAM. Not sure if/when I will be putting my code up yet. It will be on my GitHub if I do.
", "rawMarkdown": "My desktop is an 8-core Xeon, 128gb RAM, and my GPU is a Nvidia GTX 980 Ti. My laptop has a 4-core i7 and 32gb RAM. Not sure if/when I will be putting my code up yet. It will be on my GitHub if I do.", "votes": 4 }, { "id": 107766, "postDate": "2016-02-12T12:32:36.527Z", "content": "Thanks Branden for sharing these insights into your approach, and congrats on another top 10!
\n\nIt's always very interesting to see how solutions that are quite different from each other can perform essentially the same. I also tried n-grams to no avail, and in the end also removed about half of the session based features form my models according to marginal response dependence. But it looks like there were more important differences between our approaches.
\n\nDid all your models tackle the 12-class problem directly? or did you break it up into simplified problems? (removing low-probability classes, merging classes and so on). I found it very difficult to go beyond 0.88 or so without using the hierarchical structure of the classes (i.e., that all of the classes except NDF could be clumped into a "DF" class if you will).
\n\nAnother thing that bothered me and I couldn't figure out how to address (or if it even needed to be addressed) was the apparent discrepancy between the marginal class distributions in the train and test sets (at least according to the public LB, and I guess now also the full testset, though I haven't verified this yet). It seemed to me to be more than the expected drift, given how the latter manifested throughout the 2014 part of the trainset. One hypothesis is that the time horizon for determining the class was shorter in the testset (e.g., three months instead of a year, which could reflect on the way predictions should be made). Did you do any kind of adjustment to the classes in pre- or postprocessing? In validation?
\n\nIt also keeps amazing me how much thought and effort (and how many CPU cycles) go into these competitions.
", "rawMarkdown": "Thanks Branden for sharing these insights into your approach, and congrats on another top 10!\r\n\r\nIt's always very interesting to see how solutions that are quite different from each other can perform essentially the same. I also tried n-grams to no avail, and in the end also removed about half of the session based features form my models according to marginal response dependence. But it looks like there were more important differences between our approaches.\r\n\r\nDid all your models tackle the 12-class problem directly? or did you break it up into simplified problems? (removing low-probability classes, merging classes and so on). I found it very difficult to go beyond 0.88 or so without using the hierarchical structure of the classes (i.e., that all of the classes except NDF could be clumped into a \"DF\" class if you will).\r\n\r\nAnother thing that bothered me and I couldn't figure out how to address (or if it even needed to be addressed) was the apparent discrepancy between the marginal class distributions in the train and test sets (at least according to the public LB, and I guess now also the full testset, though I haven't verified this yet). It seemed to me to be more than the expected drift, given how the latter manifested throughout the 2014 part of the trainset. One hypothesis is that the time horizon for determining the class was shorter in the testset (e.g., three months instead of a year, which could reflect on the way predictions should be made). Did you do any kind of adjustment to the classes in pre- or postprocessing? In validation?\r\n\r\nIt also keeps amazing me how much thought and effort (and how many CPU cycles) go into these competitions.", "votes": 1 }, { "id": 107737, "postDate": "2016-02-12T07:18:02.280Z", "content": "@Li, Upon further review, I actually created 24 helper columns. As described above some columns had count values and others had the sum of secs_elapsed and those are on much different scales so there was one helper column summing the count columns and the other summing the secs_elapsed. Half of them were in the top 60 of feature importance, the rest were all in the top 200. I've attached a list of the column names for each country.
\n\n@Jiahuan, Yes, I included them in all of my models, I forgot about that. I sometimes imputed their values using medians and other times replaced all of their values with -1.
", "rawMarkdown": "@Li, Upon further review, I actually created 24 helper columns. As described above some columns had count values and others had the sum of secs_elapsed and those are on much different scales so there was one helper column summing the count columns and the other summing the secs_elapsed. Half of them were in the top 60 of feature importance, the rest were all in the top 200. I've attached a list of the column names for each country.\r\n\r\n@Jiahuan, Yes, I included them in all of my models, I forgot about that. I sometimes imputed their values using medians and other times replaced all of their values with -1. ", "votes": 1 }, { "id": 107729, "postDate": "2016-02-12T06:41:58.867Z", "content": "Nice job with feature engineering and congrats with high place!
", "rawMarkdown": "Nice job with feature engineering and congrats with high place!", "votes": 1 }, { "id": 107878, "postDate": "2016-02-13T12:10:27.097Z", "content": "Thanks! I realized I was tunning my model the wrong way setting a big learning rate and n_estimators around 20/30
", "rawMarkdown": "Thanks! I realized I was tunning my model the wrong way setting a big learning rate and n_estimators around 20/30" }, { "id": 107873, "postDate": "2016-02-13T10:41:29.200Z", "content": "Generally something like:
\n\neta = .005
\nmax_depth=10
\nmin_child_weight=1
\nsubsample=.5
\ncolsample_bytree=.5
@Alper, I fact I did the preprocessing in parallel and the thing that caused such a slow preprocessing was the swaping due lack of memory. In the end It only took around 2 hours thanks to several improvements. I I was curious in the preprocessing time of other participants.
\n\n@Brandem. My final file was around 180mb(train users) and took around 3 hours to fit an XGB with the problem of not being able to set a big number of trees. What where your parameters of XGB to make it run in > 6hours?
", "rawMarkdown": "@Alper, I fact I did the preprocessing in parallel and the thing that caused such a slow preprocessing was the swaping due lack of memory. In the end It only took around 2 hours thanks to several improvements. I I was curious in the preprocessing time of other participants.\r\n\r\n@Brandem. My final file was around 180mb(train users) and took around 3 hours to fit an XGB with the problem of not being able to set a big number of trees. What where your parameters of XGB to make it run in > 6hours?\r\n\r\n\r\n" }, { "id": 107855, "postDate": "2016-02-13T04:32:23.110Z", "content": "@BrandonMurray
\n\nHow often your model outputs rankings different from the {other, FR, IT} for the last 3 places in top-5? Do you have any estimates, how much such rerankings add to your score?
\n\nIt would be very interesting to know, how much could be gained by correct reranking of the last 3 places, because my approach was to train a binary classifier using only users with NDF and US classes to decide which class to put on the first place, the 3rd and the 4th most likely classes were always "other" and "FR" , and some simple rules based only on age and gender were used to decide which one from the IT, GB, ES to put on the 5th place.
\n\nThe classifier for the 5th place gave me +0.0004 on the public leaderboard, but -0.00006 on the private leaderboard. However, after the change of classification threshold, the binary classifier between US and NDF plus {other, FR, IT} for the last 3 places gives me 0.88556 on the private leaderboard
", "rawMarkdown": "@BrandonMurray\r\n\r\nHow often your model outputs rankings different from the {other, FR, IT} for the last 3 places in top-5? Do you have any estimates, how much such rerankings add to your score?\r\n\r\nIt would be very interesting to know, how much could be gained by correct reranking of the last 3 places, because my approach was to train a binary classifier using only users with NDF and US classes to decide which class to put on the first place, the 3rd and the 4th most likely classes were always \"other\" and \"FR\" , and some simple rules based only on age and gender were used to decide which one from the IT, GB, ES to put on the 5th place.\r\n\r\nThe classifier for the 5th place gave me +0.0004 on the public leaderboard, but -0.00006 on the private leaderboard. However, after the change of classification threshold, the binary classifier between US and NDF plus {other, FR, IT} for the last 3 places gives me 0.88556 on the private leaderboard" }, { "id": 107838, "postDate": "2016-02-12T23:47:30.037Z", "content": "@David\nI think you hit your memory limit (especially due to some temporaries) which caused a lot of page-faults. Even creating one such large data structure makes your code crawl.
", "rawMarkdown": "@David\r\nI think you hit your memory limit (especially due to some temporaries) which caused a lot of page-faults. Even creating one such large data structure makes your code crawl.\r\n" }, { "id": 107836, "postDate": "2016-02-12T23:19:35.880Z", "content": "@W. Yifan, I don't think there is one, when you cast the data frame the column will just be named "Action.0" instead of "Action.NULL".
", "rawMarkdown": "@W. Yifan, I don't think there is one, when you cast the data frame the column will just be named \"Action.0\" instead of \"Action.NULL\". " }, { "id": 107832, "postDate": "2016-02-12T23:04:55.537Z", "content": "@Branden Murray Thanks for sharing! I have a question related to this:
\n\nIn the "session" data, you '"Filled in blanks with "NULL" so it was a new factor level.'\nSo I'm wondering:\nWhat's the difference between filling with 'NULL' and filling with 0s?
\n\nSorry for the stupid question, but it would be great to learn things from masters for a newbie =)
", "rawMarkdown": "@Branden Murray Thanks for sharing! I have a question related to this:\r\n\r\nIn the \"session\" data, you '\"Filled in blanks with \"NULL\" so it was a new factor level.'\r\nSo I'm wondering:\r\nWhat's the difference between filling with 'NULL' and filling with 0s?\r\n\r\nSorry for the stupid question, but it would be great to learn things from masters for a newbie =)" }, { "id": 107826, "postDate": "2016-02-12T22:00:42.833Z", "content": "@Jiahuan, Honestly, since CV improvements in either of them didn't seem to correlate very well to the LB I kept changes as long as they both weren't obviously terrible.
\n\n@David, Preprocessing didn't take much time at all, maybe 10 minutes to run, tops. 4 days seems kind of long, even with only 4GB of RAM, hopefully someone around here in a similar boat will have some suggestions. My XGB models ran in 8-12 hours usually. I think my NNs ran in a few minutes on GPU.
", "rawMarkdown": "@Jiahuan, Honestly, since CV improvements in either of them didn't seem to correlate very well to the LB I kept changes as long as they both weren't obviously terrible. \r\n\r\n@David, Preprocessing didn't take much time at all, maybe 10 minutes to run, tops. 4 days seems kind of long, even with only 4GB of RAM, hopefully someone around here in a similar boat will have some suggestions. My XGB models ran in 8-12 hours usually. I think my NNs ran in a few minutes on GPU." }, { "id": 107825, "postDate": "2016-02-12T21:54:56.147Z", "content": "@David\nI also had memory issues in processing the sessions file using pandas get dummies on the actions. One trick that I found helpful was to sub-select the sessions file by action done before doing the pd.get_dummies one hot encoding (so do one-hot encoding on the common actions separately from the rarer actions). Still I would sometimes get kernel crashes due to memory issues on my 8GB laptop if I forgot to reset my workspace before rerunning, which was kind of frustrating. But this took only on the order of 15 minutes to do the data preprocessing.
", "rawMarkdown": "@David\r\nI also had memory issues in processing the sessions file using pandas get dummies on the actions. One trick that I found helpful was to sub-select the sessions file by action done before doing the pd.get_dummies one hot encoding (so do one-hot encoding on the common actions separately from the rarer actions). Still I would sometimes get kernel crashes due to memory issues on my 8GB laptop if I forgot to reset my workspace before rerunning, which was kind of frustrating. But this took only on the order of 15 minutes to do the data preprocessing." }, { "id": 107823, "postDate": "2016-02-12T21:22:11.337Z", "content": "Sorry, I meant a complete run. Average time per model fit and/or preprocessing methods. It took me almost 4 day to preprocess 10M rows of the sessions data in a 4Gb laptop and I was wondering the performance differences.
", "rawMarkdown": "Sorry, I meant a complete run. Average time per model fit and/or preprocessing methods. It took me almost 4 day to preprocess 10M rows of the sessions data in a 4Gb laptop and I was wondering the performance differences." }, { "id": 107813, "postDate": "2016-02-12T20:13:24.077Z", "content": "Thanks, Branden. Just an another question, I assume you have found some inconsistency between logloss and ndcg5 when you make adjustment to your model, how do you determine whether to keep the change to your model if those two metrics tell you differently?
", "rawMarkdown": "Thanks, Branden. Just an another question, I assume you have found some inconsistency between logloss and ndcg5 when you make adjustment to your model, how do you determine whether to keep the change to your model if those two metrics tell you differently?" }, { "id": 107795, "postDate": "2016-02-12T17:51:12.787Z", "content": "@David, What do you mean by iteration? That could mean a few things.
\n\n@SK, They all tackled the 12-class problem directly. I was a hair under 0.881 on my second submission, so I didn't feel the need to go the route of breaking it up early on, and after I got sucked deeper into the Homesite competition I ran out of time to do that kind of stuff. I also didn't make any adjustments to the classes.
\n\n@SkyLibrary, I don't know the differences between the implemenations in sklearn and R. I really only use Python for training NNs because there's no easy way (that I know of) to train NNs using GPU in R. I used Xgboost to train my second layer model, so everything got the same weight.
\n\nMy XGBs are all done using different data input, slight variations in the xgb parameters, and using mlogloss and the custom ndcg5 metrics for early stopping. Same for the NNs -- different inputs, I think 3 of them had 2 hidden layers and 1 had 3 hidden layers.
", "rawMarkdown": "@David, What do you mean by iteration? That could mean a few things. \r\n\r\n@SK, They all tackled the 12-class problem directly. I was a hair under 0.881 on my second submission, so I didn't feel the need to go the route of breaking it up early on, and after I got sucked deeper into the Homesite competition I ran out of time to do that kind of stuff. I also didn't make any adjustments to the classes. \r\n\r\n@SkyLibrary, I don't know the differences between the implemenations in sklearn and R. I really only use Python for training NNs because there's no easy way (that I know of) to train NNs using GPU in R. I used Xgboost to train my second layer model, so everything got the same weight.\r\n\r\nMy XGBs are all done using different data input, slight variations in the xgb parameters, and using mlogloss and the custom ndcg5 metrics for early stopping. Same for the NNs -- different inputs, I think 3 of them had 2 hidden layers and 1 had 3 hidden layers." }, { "id": 107791, "postDate": "2016-02-12T16:51:40.150Z", "content": "Thanks for sharing @Brandon.
\n\nOur model/feature engineering have a lot in common, and we are so close in private leaderboard. Now I am wondering what the top 3 are doing.
\n\nI do have two questions about your models.\nFirst, I tried logistic regression and random forest in sklearn for ensemble with nn and xgb, but they didn't improve either local validation or public leaderboard score, are GLMNET and RandomForest (using 'ranger' package in R) different from the algorithms in skearn, how much weight did you give to GLMNET and ranger RF?
\n\nSecond, your 8 XGBs are from different data input(like different transformation of your features) or with different depth, random seeds, is the two stage xgb model one of the 8 models? Same about 4NNs, are they using different network structure or different data input?
", "rawMarkdown": "Thanks for sharing @Brandon. \r\n\r\nOur model/feature engineering have a lot in common, and we are so close in private leaderboard. Now I am wondering what the top 3 are doing. \r\n\r\nI do have two questions about your models.\r\nFirst, I tried logistic regression and random forest in sklearn for ensemble with nn and xgb, but they didn't improve either local validation or public leaderboard score, are GLMNET and RandomForest (using 'ranger' package in R) different from the algorithms in skearn, how much weight did you give to GLMNET and ranger RF?\r\n\r\nSecond, your 8 XGBs are from different data input(like different transformation of your features) or with different depth, random seeds, is the two stage xgb model one of the 8 models? Same about 4NNs, are they using different network structure or different data input?\r\n" }, { "id": 107762, "postDate": "2016-02-12T10:48:25.537Z", "content": "[quote=Branden Murray;107744]
\n\nMy desktop is an 8-core Xeon, 128gb RAM, and my GPU is a Nvidia GTX 980 Ti. My laptop has a 4-core i7 and 32gb RAM. Not sure if/when I will be putting my code up yet. It will be on my GitHub if I do.
\n\n[/quote]
\n\nReally impressive specs. How much time did you spend per iteration in average?
", "rawMarkdown": "[quote=Branden Murray;107744]\r\n\r\nMy desktop is an 8-core Xeon, 128gb RAM, and my GPU is a Nvidia GTX 980 Ti. My laptop has a 4-core i7 and 32gb RAM. Not sure if/when I will be putting my code up yet. It will be on my GitHub if I do.\r\n\r\n[/quote]\r\n\r\nReally impressive specs. How much time did you spend per iteration in average?" }, { "id": 107742, "postDate": "2016-02-12T08:01:12.180Z", "content": "Thanks for sharing those findings. What kind of hardware did you use? I was struggling with a 8GB machine. Also, it would be great to take a look over your code, will this be possible?
", "rawMarkdown": "Thanks for sharing those findings. What kind of hardware did you use? I was struggling with a 8GB machine. Also, it would be great to take a look over your code, will this be possible?" }, { "id": 107730, "postDate": "2016-02-12T06:52:15.223Z", "content": "Thanks for the sharing. I just wonder did you ever include those users without sessions into your model?
", "rawMarkdown": "Thanks for the sharing. I just wonder did you ever include those users without sessions into your model?" }, { "id": 107727, "postDate": "2016-02-12T06:31:30.600Z", "content": "Congratulations on your 5th! And thank you for sharing your result.
\n\nThe way to create "helper" columns is creative. Could you share more details for this method? Like what is the columns for each country and how those helper features rank in your feature importance?
\n\nThank you!\nLi
", "rawMarkdown": "Congratulations on your 5th! And thank you for sharing your result.\r\n\r\nThe way to create \"helper\" columns is creative. Could you share more details for this method? Like what is the columns for each country and how those helper features rank in your feature importance?\r\n\r\nThank you!\r\nLi" }, { "id": 627461, "postDate": "2019-09-16T02:39:09.633Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 107837, "postDate": "2016-02-12T23:31:26.110Z", "content": "@Branden Murray, Thanks!
", "rawMarkdown": "@Branden Murray, Thanks!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T13:53:06Z", "competition_id": 4651, "competition_slug": "airbnb-recruiting-new-user-bookings", "team_id": 258543, "solution_url": "/competitions/airbnb-recruiting-new-user-bookings/writeups/branden-murray-5th-place-solution" }