I think one key to this competition was choosing what *not* to change from the baseline. We didn’t change B or E at all. We only changed a small percentage of ACDF – usually only ones with shopping_pt=2. Before teaming up about a week ago, Alessandro had a top 10 score only changing G. We basically used his G and my ACDF. I think our solution only had about 2,500 rows that differed from the baseline (so less than 5% difference).
\nSomething else that helped was finding the “Georgia/Florida tricks.” No customers in Georgia had C=1 or D=1 in their final purchase. But some customers had C=1 or D=1 as part of their last quoted plan in the test set. Changing these to 2 gave improvement. Similarly, no customers in Florida had G=2 in their final purchase. Did anyone find any other situations like these?
\nIn addition to the base features, features I found useful were the A, B, C, D, E, F, and G from the previous shopping_pt. Also cost change from the previous shopping_pt.
\nI used GBM to predict individual ACDF values. Something that made this challenge difficult was that customers who could be safely predicted to change one product also had a high propensity to change multiple products – and getting multiple changes correct for one customer was difficult. So the customers that were easiest to predict for individual products turned out to be difficult to predict overall.
\nThank you to Allstate and Kaggle for a fun competition!
", "comments": [ { "id": 45984, "author_name": "Perroquet", "author_url": "", "post_date": "2014-05-21T12:22:47.220000", "content": "Dear all,
\nfor all the models, we used only gbm in R. We didn’t even seriously try anything else, so there may be a potential for more improvements with other methods.
\nThe base of our idea was the same as for almost everybody here. We divided our data according to shopping_pt up to shopping_pt=6. The biggest value was of course for history 2 and we almost gained nothing for 6 as well as 5, but history 3 and 4 was still useful. As Breakfast Pirate posted the main idea was to find customers who have biggest potential that final purchase will be different from the current quote, i.e. for which it make sense to take a risk to change the last quote benchmark. We have trained few classifiers using adaboost metric (which seems to be a reasonable proxy to AUC), one classifier per each shopping point history.
\nThen we made three different models to predict the actual purchase record, all using multinomial metric in gbm.
First model was based on all available features in the provided data plus we added information from previous versions of ABCDEFG attributes and previous costs. This model consisted of 4 submodels – predicting together AF, BE, CD and G. These combinations were chosen based on the correlation matrix of these products. The second model was based on the same pairs of variables as the first model. It had all features as the first model and some more - a variable with mean cost in each state and each location was added. Moreover, after prediction of the first variable (G), a variable stating whether the model output is different from the LQB was added. The last pair (BE) was then modeled with information whether any other variable has changed. The last model was working with only three submodels (ABEF, CD, G) and had some more features relating to what was changed in the known customer history (e.g. ratios of current and previous costs, time differences, replacing of category variables with 1/0 feature vectors and feature subset vectors, etc.)
These three models were combined with corresponding classifier (models for history 2 with classifier for history 2, etc.) so we got the final predictions for each history. The algorithm used for the combining was based on the classifier prediction
- If the probability of the last quoted benchmark change was low, we kept the last quoted benchmark
- If the probability of the last quoted benchmark change was middle, we changed the last quoted benchmark only when all 3 models predicted the same value
- If the probability of the last quoted benchmark change was high, we changed the last quoted benchmark when at least 2 models predicted the same value
Our final solutions was putting together all predictions for all history (from 2 to 6). The most important parts of this blending was finding out the appropriate setting of the combining algorithm (i.e. which numbers to use for low/middle/high probabilities described above).
By the way, no specific handling for certain cases (such as Florida state) was done, however it looks from the feature importance that these specifics were found and used by the gbm trees.
\nAdditionally, we didn’t use any specific cross validation within each gbm model, but we think the way the last step was done (kind of strict majority voting from the three models – about 2100 records was changed overall) is what kept as from overfitting the public leaderboard).
\nAlternative point of view is that in such tight competition with so many experienced people, the most important factor that moved us on the top place in the private leaderboard is just a bit (or a lot?!) of a luck.. :-)
\nThanks all for a great competition (special thanks to Kaggle and Allstate) and congratulations to all to placing teams.
\nPrazaci team (maternaj, Perroquet, Matfyzak, nimo, Emil Skultety)
", "votes": 30, "replies": [] }, { "id": 47362, "author_name": "Alessandro Mariani", "author_url": "", "post_date": "2014-05-30T19:17:14.387000", "content": "It took me a while to clean up everything, and write some stuff... but here it is! Hopefully somebody is still reading :)
\nThe main idea is that we don't know at what shopping_pt the purchase will be made. We know which plans fit more each profiles, so basically I'm training the model on the whole dataset using the purchased plan as target at each shopping_pt during the transaction history for all the customers. Even though purchases never happen at shopping_pt #1, it is included in the training data. The main reason is because patterns which occur at shopping_pt #1 for some customer can occur at different shopping_pt for others customers, leading to the same plan purchased.
\nI’m using is a Random Forest (scikit-learn implementation) as base model, which by itself only can give quite good results. To produce a robust model, I’ve ensemble 9 Random Forest out of other 50. If five out nine models agree on the same plan then this change is made, otherwise the last quote is used (majority vote).
\nThe final ensemble, which led our team to place 2nd in the private leader board, is the combination of my predicted G and Steve’s ABCEDF.
\n## Extra Features
I’ve used all the features provided, at exception of date & time. To help tree interaction and improve the accuracy I’ve also included the following features, group by category for your convenience.
Category Interactions (2-way)
\nCategory & Interaction mapped at arithmetic mean of the cost
\nAverage of target variable
\nContinuous Interactions
\nNaming Convention
Product: A, B, C, D, E, F and G are all products.
Plan: Combination of A, B, C, D, E, F and G.
Baseline: Is the last plan or product quoted at the latest shopping_pt available.
Metric
The score I’ve used to determine how good a model is defined as follow. The difference between the baseline accuracy and the model accuracy measured at each single shopping_pt, times the number of samples in the test set at that shopping_pt. For example, the difference between the model and the baseline for shopping_pt #2 is 0.4160-0.4116=0.0044 times the count of the test samples where the latest shopping_pt available is #2 is 0.0044x18,943=58.5. I’ll be addressing to score at the sum product of these differences and the test set distribution.
## Modelling Techniques & Training
A Random Forest is by itself an ensemble of decision trees. Each decision tree in a Random Forest is trained on different subset of data, leading to many different trees. The Random Forest predictive power comes with the ensemble of all these trees, stacking the class probabilities.
The higher is the number of tree we’ll build, the more accurate and more stable the prediction will be. This is what happens usually, but not in this problem since the gain over the baseline is very low. Making this more sensible to randomness and harder to fix only increasing the number of tree!
\nInstead of keep stacking the class probabilities and increasing the number of tree, we can keep the number of tree in the Random Forest lower and look at the output at a number of Random Forests. If the majority of these agree on the same outcome, then is quite likely that change is actually occurring. If the majority have the same outcome, then chose this as final prediction. Otherwise use the safest option: the baseline. Making this strategy is less prone to randomness.
\nI’ve trained several Random Forest using the same data but using different seed, which approximately lead to ~300 different predictions out of 55,716. Is quite a low number, but in this particular competition one more accurate prediction is the difference between the 2nd and 3rd place! Hence here comes the need to have not just a good model, but a very stable model which will generalize as much constantly as it could on unseen data.
\nUsing the majority vote ideas gave a quite stable prediction (and more accurate). What helped a little bit further was selecting a subset of all the Random Forests which are expected to have a better accuracy. How? While looking for a way to identify more accurate Random Forest I’ve noticed that for higher train set scores, usually there is an higher cross validation score. Following this intuition I could discard model whom their train set score was worse than the others as they are more likely to be not good as the others. Do a majority vote on the best 9 Random Forests instead of using all the 50 improved the results too!
\nFinally, if you're not bored about reading all this - here is the link to the GitHub repository: https://github.com/alzmcr/allstate
", "votes": 25, "replies": [] }, { "id": 44872, "author_name": "Euclides Filho", "author_url": "", "post_date": "2014-05-20T00:59:13.527000", "content": "I've just entered in this competition last Monday and I'm really surprise with my result.
\nHere follows my approach.
\nCongrats to all :)
", "votes": 13, "replies": [] }, { "id": 44879, "author_name": "Shize Su", "author_url": "", "post_date": "2014-05-20T01:44:51.847000", "content": "Our team was ranked 1st on public board but crashed on private board. I think it is really easy to be over fitting in this competition, and it is difficult to select which submission might be over fitting and which not.
\nActually, before team merging, one submission made by our team member Nitai Dean who just develop a Random Forest classifier for G option will already give top 10 score in final private board, and an ensemble model from Jagannath's model and my model will also give top 10 score in final private board. Besides, a model which just adds FL/Georgia tricks(similar to team of Alessandro&BreakfastPirate, we also found these tricks for FL/Georgia) to Nitai Dean's G_only model, will give us top 5 score on final private board.
\nHowever, it seems to be very difficult to decide which submission is over fitting and which not, and we seem to underestimate the over fitting issue in this competition and the two submissions we choose for final evaluation turn out to perform poorly on the private board.
\nFinally, I do agree that, the key in this competition is to decide when to correct the last-quote baseline prediction and when not. And the over fitting issue as well as some randomness about the public board and private board data split contribute to the big shake of ranking on the private board.
\nP.S.: About the FL/GA tricks, actually we also only find these three tricks same as BreakfastPirate, namely, for example, there are no purchased G=2 for FL, but test set last-quote has some entries such that G=2. And we do find some other tricks that is not like such 0:100 split but something like 10:90 split. For example, for state NY, if last-quote G=1 and C_previous=1, then the purchased G would be highly possible(more than 85% correct) to be G=2 based on training set. And such corrections (only 99 such corrections in the whole test set) does also contribute to significant improvement in public board score (improved by 13*0.00006). Since such corrections contribute to significant improvement to both training set 10-fold cross-validation and public board score, we thought that this probably will also contribute to significant improvement in private board, but a post-deadline submission shows that such corrections about G option of NY just doesn't change the score on private board at all. Therefore, I guess that the existing randomness about data split of public board and private board might be one major reason for the big shake on ranking of public board and private board.
\nAnyway, this competition is still a great competition, and thanks Kaggle and Allstate! And congrats to the winners!
\nBest wishes,
\nShize
", "votes": 11, "replies": [] }, { "id": 44922, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2014-05-20T08:51:10.223000", "content": "Congrats to the winners!
\nI joined fairly late and I did not have much time to experiment a lot, however I immediately saw that there was something going on with the cities as I could barely beat benchmark without them and got top 50 in private with them only with G ! In short :
\n[quote=Joshua Weiner;44874]
\n@Euclides - would you consider sharing your code? Since you are the only one who has expressed using many different ensembles and had success I would be very interested in seeing how you did it.
\n[/quote]
\nYes sure, after a clean up I'll put it on github and post the link here.
", "votes": 10, "replies": [] }, { "id": 46118, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2014-05-23T09:27:46.427000", "content": "Marat,
\n\nIt takes a little bit of time, but you need to automate what you do (e.g. everything in code). Also save everything you've done in the past. I am spending less than half the time on a competition in comparison to when I started kaggling and I am scoring much better, because everything is automated and to some extend more optimized . I also don't repeat stuff that did not work many times in the past so , which also saves lots of time. After while the important question will be how much and how fast your computer can handle the data rather than how much time you have :) . Don't get me wrong, you always need to spend time to understand the data, but even that gets faster the more you play.
", "votes": 5, "replies": [] }, { "id": 44903, "author_name": "Gert", "author_url": "", "post_date": "2014-05-20T05:03:42.980000", "content": "Nice reading this forum, thanks to everyone! My approach was a combination of random forests and logistic regression.
\nFor some models I threw away records where more than one single option changed between last observed quote and purchase. Those models worked best for A-F (which makes sense because A-F often change together, making it harder for the model to attribute ‘imbalance in the quote’ to one option). I included dummy encoded features, counts of values and likelihood features, but they did not add much to the raw features.
\nI trained using the last two shopping points and applied the model equeally to all shoppingpoints. For cross validation I randomly truncated shoppingpoints until I had 0.537. I never managed to improve that beyond 0.539, but every time I had a consistent improvement (over five folds) the leaderboard surprised me with a better improvement. So I think there is something not random about the truncating.
\nIt also helped to use a more conservative threshold (for changing last observed quotes) for some changes, like G from 1 to 2 and G from 4 to 3.
\nI used the public leaderboard just for checking improvement (tried to avoid comparing alternatives) and in the end to tune the number of last observed quotes to change. In my best selected model, I changed just over 1400, but for the private set I should have changed more.
", "votes": 5, "replies": [] }, { "id": 44900, "author_name": "mlandry", "author_url": "", "post_date": "2014-05-20T04:50:57.850000", "content": "I looked at this as a two-step process, as BreakfastPirate discussed. But I never got anything intelligent working for the second step
\nThe situations I targeted, with rough probabilities of the last seen being purchased and the number of customers:
\nExcept the last, these all had very low probabilities based on the train set, so doing just about anything would help your score.
\nSomebody asked how to find these, and I found the 0011004 during the data exploration phase as well--probably the first hour or two with the data. I initially truncated the data at "final shopping point - 4" for everybody. Then I simply ran a SQL query counting the number of times each plan appeared as the last seen, and then purchased, and sorted by the difference. 0011002 was simply more likely for those that were last shown 0011004. Sending that in was worth about 6 correct. And I just happened to observe that FL kept coming up when I was going through the rest of that list. Once I saw that, I scanned every 3-field combination to see if I could find such a pattern, and the GA pair came up.
\nI tried GBM many times on many different subsets--full plans and just the components; all the data, just those rules listed above and other less probable pieces. I just couldn't get anything out of it better than the most common full plans, given the full last-seen plan in that situation.
\nSo for the GA and FL rules, I swapped in the most likely single value. The others, I usually used the full plan. I tried to work with overall net migration into and out of particular plans, but no improvement.
\nCongratulations to the winners! I look forward to learning from you all and seeing what I was doing wrong on the ML side of things.
", "votes": 5, "replies": [] }, { "id": 44881, "author_name": "Alessandro Mariani", "author_url": "", "post_date": "2014-05-20T01:54:37.917000", "content": "WOW!!! I can't describe how afraid was to check Kaggle and the LB today - so wrong!!!! xD
\nI'll be super happy to share my approach & code in details as long this is fine with Allstate, after a beer and pizza!
", "votes": 5, "replies": [] }, { "id": 46055, "author_name": "maternaj", "author_url": "", "post_date": "2014-05-22T11:50:22.720000", "content": "[quote=K-czar;46001]
\n@Perroquet Thanks so much for sharing! That was really awesome to read b/c it validates a lot of the ideas we had intended to try with more time! I also wasn't aware GBM (in R) even had a multinomial metric, or the capability to predict in multiclass, multioutput problems. I don't believe it does in python. That was the reason we stuck to Random Forrests! But when we predicted variables in pairs - similarly based on the correlation matrix - we found that predicting the variables independently almost always drew higher accuracy.
\nSo far, I think you're also the only other group I've seen that incorporated a "consensus" approach like we did.
\nDid you have any memory issues using R's GBM with all of the locations as variables? Unless R's GBM implementation has been updated to serve sparse matrices, I would have guessed it would cause considerable memory problems.
\nThanks so much for sharing, and congratulations! Great work!
\n[/quote]
\nThis was the first time we really used distribution="multinomial" parameter for gbm in R and it served its purpose very well.
\nAs far as I know you are right that gbm doesn't support sparse matrices. We didn't actually used all locations as variables, only added average cost per location as a feature and average cost per state as a feature (the later one being more helpful).
\nGenerally, regarding memory issues, there is a memory leakage in the latest version from CRAN (2.1 I think) for multinomial distribution which causes the whole RAM to be eaten up quite quickly.
\nIn https://github.com/harrysouthworth/gbm (thanks to Harry and all contributors for supporting and further developing this excellent package!) there are newer minor versions that fix the memory leakage (namely 2.1-0.3; beware of 2.1-0.5 as the changes for multinomial seem to broke it somehow).
\nLastly, for larger number of classes (which was the case for ABEF model that has overall 48 classes) there is another bug which causes the training model at some point starts producing NaN values - we hopefully were able to fix this on our own and will try to push this fix to the next releases of gbm.
", "votes": 3, "replies": [] }, { "id": 44911, "author_name": "Rafal Cycon (blaine)", "author_url": "", "post_date": "2014-05-20T07:49:00.507000", "content": "Thank you for sharing your approaches! You got some really nice insights :-)
\nAs for me, at the very start of the competition I loaded the data. Then I trained 7 random forests, one for each option, on selected informative features (most importantly time, location and customerID) but they weren't working nicely. So I did many cross validations to select appropriate seed and when I finally achieved quite high performance I used the obtained seed on the test set and it got me to the 10th place.
\nSeriously.
\nI entered just yesterday but I was working on this competition earlier for about a month after it launched. I created some time-varying features like mean cost, cumulative gradient of cost, number of changes of options, previous A etc. After I noticed the importance and "changeness" of G I created several new features that had sensible correlation with coverage options and could improve the performance. Then I used the whole data set for training, treating each observation independently, as I thought that truncating the data set would simply delete too much information.
\nI also employed a two-step approach - first a classificator (will the X option change?) then a preditor (if it changes, what it will be?), both GBMs. Each option had its 3 classifiers and 3 predictors, trained on 3 customer-wise folds. I trained predictors on lots of data (classificator_output > 0.2), but used a more conservative threshold in the test set (0.45 for G, 0.7 for the rest). Now I see I should have used an even lower threshold on G, but well :-)
\nThank you all for taking part, it was a very interesting competition, good luck in the next ones!
", "votes": 3, "replies": [] }, { "id": 44905, "author_name": "Hiroyuki", "author_url": "", "post_date": "2014-05-20T05:57:08.417000", "content": "[quote=TayShin;44892]
\n@Hiroyuki Can you please elaborate more? because I used exactly same procedure as you did but my score didnt beat the bench mark. variables that I used is given variables + previous quote(A through G). Thank you in advance!
\n[/quote]
\nHere is myRFcode.
\nI'm afraid it's not exactly same as my best submission.
\nMy feature, child, couple and single didn't work well maybe.
", "votes": 3, "replies": [] }, { "id": 44885, "author_name": "Hiroyuki", "author_url": "", "post_date": "2014-05-20T02:22:17.633000", "content": "My strategy was quite simple. I used randomforest model with almost all given variables and 2nd lastquote and then I got (0.53556) on private leaderboad.
\nUsing firstquote, separating plans into the pair of "CD", "AEF" and "BG" and building a model for each shopping_pt (ie. 2,3,4...) didn't work well.
\nRandomForest worked better than gbm, nnet and svm on my prediction.
\nIt was difficult to do CV. I had no idea which data should I use for CV, the truncated train data set or no truncated data. On my prediction, no truncated data gave me better result but I couldn't understand why it went successful.
\nThank you for exciting competition, and see you next!
", "votes": 3, "replies": [] }, { "id": 44880, "author_name": "Taffey Lewis", "author_url": "", "post_date": "2014-05-20T01:50:02.047000", "content": "My best private LB score was just the last quote + change of all Florida customers who still had G2 as option and arbitrarily changed it to G3. That's all.
\nG quirks were:
\nFlorida customers apparently could choose G2 in their quotes, but never purchased it. G1 was apparently not an option.
\nSD and ND customers only had G2 as option. Any model that gave ND or SD customers anything other than G2 was an easy fix.
\nOhio customers did not have G1 as an option.
\nI didn't notice the Georgia quirk, but wonder how much of a difference it would have made.
\nI noticed high polychoric correlations in purchased plans between AF, BE, and CD pairs. I also noticed that (generally) classifiers using these pairs for targets (with G modeled separately) did better than classifiers for each of the options individually. However, none of them ever beat my score with last-quote and Florida fix only.
", "votes": 3, "replies": [] }, { "id": 44860, "author_name": "BreakfastPirate", "author_url": "", "post_date": "2014-05-20T00:14:49.557000", "content": "And congrats Prazaci team for your surprise jump from 11th to 1st!
", "votes": 3, "replies": [] }, { "id": 44883, "author_name": "K-czar", "author_url": "", "post_date": "2014-05-20T02:15:32.153000", "content": "My teammate and I got didn't make a whole lot of submissions on this (8 - but only a couple real ones), but managed to put up a top 5% score, so I'll share what got us there and our insights.
\nAlmost all of our gain came from customers in the test set with only two shopping points given, as that was where we focused most of our time. The results we submitted primarily resulted from two models - logistic and Random Forest. Each model produced fairly good results predicting each purchase option (A-G) independently - significantly better than the benchmark for each individual variable, though neither was actually able to beat the benchmark when predicting all seven variables together.
\nWe found that a pretty good way of ensembling the two models was to simply require a consensus between both models to override the benchmark. If there was no consensus (or the consensus was the benchmark) then we used the benchmark. That was good enough for 72 place.
\nWe actually mostly used this project as a learning exercise to experiment with Pylearn2's Deep Neural network learning packages. As it turns out (we checked by submitting just after the deadline, since we ran out of submissions today), using a NN on shopping point two alone would have vaulted is up another 20-30 spots on the leaderboard.
\nI'm curious if anyone made use of a model trained on a binary output of whether or not a person's purchase choices will be the same as a benchmark. That was something I had wanted to work on if we had more time, so I'm curious if it worked for anyone.
", "votes": 4, "replies": [] }, { "id": 46157, "author_name": "Joshua Weiner", "author_url": "", "post_date": "2014-05-24T03:14:20.233000", "content": "I try running your code to a T and in the doSubmission when I try to train the GBM I get an error...Please see the traceback below:
\n3 stop("gbm does not currently handle categorical variables with more than 1024 levels. Variable ",
i, ": ", var.names[i], " has ", length(levels(x[, i])), " levels.")
2 gbm.fit(x, y, offset = offset, distribution = distribution, w = w,
var.monotone = var.monotone, n.trees = n.trees, interaction.depth = interaction.depth,
n.minobsinnode = n.minobsinnode, shrinkage = shrinkage, bag.fraction = bag.fraction,
nTrain = nTrain, keep.data = keep.data, verbose = lVerbose,
var.names = var.names, response.name = response.name, group = group)
1 gbm(willChange ~ ., data[train, -omitClass], distribution = "bernoulli",
keep.data = F, verbose = F, n.cores = 1, n.trees = classTrees,
shrinkage = class_shrinkage, interaction.depth = class_depth,
n.minobsinnode = class_minobs)
[quote=Stephen McInerney;46132]
\nWhat does 'hasF' mean,I don't see it computed anywhere? Does it mean 1) 'was not NA on all previous quotes' or 2) are you assuming F==0 means no coverage (which is not the case, per Deanonymizing coverage levels and options, also by state)
\n[/quote]
\nThis was one of the very minor changes :-) I fixed it on git, thanks :-)
\nYour guess nr 2 is correct - I assumed that F==0 means that there is no coverage. It was motivated by the description of C_previous, i.e. "C_previous - What the customer formerly had or currently has for product option C (0=nothing, 1, 2, 3,4)". Given that C "normally" takes values 1..4 such inference for other options seemed to be reasonable for me.
\nOn the other hand your analysis looks interesting, although I did not discover such cost <-> options relationships when I did the exploration part.
", "votes": 1, "replies": [] }, { "id": 46132, "author_name": "Stephen McInerney", "author_url": "", "post_date": "2014-05-23T14:52:01.573000", "content": "[quote=blaine;46126]
\nT/F did the client had the coverage option A/B/E/F in any of her previous quotes?
\nhttps://github.com/B1aine/kaggle-allstate/blob/master/prepareData.R
\n mat[customer,"hadF"][obs] = mat[customer,"hadF"][obs-1] | mat[customer,"hasF"][obs]
[/quote]
\nWhat does 'hasF' mean,I don't see it computed anywhere? Does it mean 1) 'was not NA on all previous quotes' or 2) are you assuming F==0 means no coverage (which is not the case, per Deanonymizing coverage levels and options, also by state)
", "votes": 1, "replies": [] }, { "id": 46111, "author_name": "maternaj", "author_url": "", "post_date": "2014-05-23T04:48:46.397000", "content": "@blaine
\nSure, even though we were updating 2.1-0.3 (2.1-0.5 seems to be "more" broken as NaNs for multinomial start to appear immediately if I remember correctly). For 2.1-0.3, the fix we made was in multinomial.cpp and relates to the division by zero error - in some places for very low values the denominator was rounded to 0.
\nIf interested in our "hacked" version of multinomial.cpp, just drop me an email.
\nEdit: wrong formatting
", "votes": 1, "replies": [] }, { "id": 46060, "author_name": "Rafal Cycon (blaine)", "author_url": "", "post_date": "2014-05-22T13:48:25.217000", "content": "[quote=maternaj;46055]
\nIn https://github.com/harrysouthworth/gbm (thanks to Harry and all contributors for supporting and further developing this excellent package!) there are newer minor versions that fix the memory leakage (namely 2.1-0.3; beware of 2.1-0.5 as the changes for multinomial seem to broke it somehow).
\nLastly, for larger number of classes (which was the case for ABEF model that has overall 48 classes) there is another bug which causes the training model at some point starts producing NaN values - we hopefully were able to fix this on our own and will try to push this fix to the next releases of gbm.
\n[/quote]
\nI also used multinomial GBM in R and very quickly encountered the NaN problem, with 2.1-0.5 from Harry Southworth's git. Moreover, even when the seed was fixed it tended to produce very different models on each run. What is interesting, these problems disappear when the very same code and GBM + R are run on Windows. Well done with fixing it! And I really hope your fix will be included in the next release :-) I am very curious what was actually the problem, do you mind sharing?
\n[quote=pythonomic;44969]
\n@blaine : Thanks for sharing your approach. would appreciate if you could share your code as well. Thanks.
\n[/quote]
\nI will upload my code within a few days. However after I started cleaning it up I discovered a huge thinking error I made when training the models :-) I will share the code and describe the mistake so people might learn something from it.
", "votes": 1, "replies": [] }, { "id": 46006, "author_name": "K-czar", "author_url": "", "post_date": "2014-05-21T17:39:09.173000", "content": "@mlandry I notice your y is "yG" so I assume you're doing a multiclass classification, as opposed to multiclass + multioutput simultaneously.
\nPerroquet said they predicted AF, BE, CD and G together in pairs - I assume they simply constructed special y features to accomplish that where each combination of AF was considered a separate class. For Random Forest in sklearn you don't have to do that though - which I guess just makes it simpler, though perhaps not substantively different.
\nDefinitely useful to know though. Thanks!
", "votes": 1, "replies": [] }, { "id": 46005, "author_name": "mlandry", "author_url": "", "post_date": "2014-05-21T17:26:26.400000", "content": "Not sure if this is the best way to do it, but you get a matrix of probabilities from R's GBM implementation, and here's how I was using that (to get the highest, but you can analyze the full range, if desired)
\ng<-gbm.fit(x=x[1:trainEnd,],y=yG[1:trainEnd],distribution="multinomial"
\n,n.trees=trees,shrinkage=shrink,interaction.depth=depth,n.minobsinnode=minobs)
\npH<-predict.gbm(object=g,newdata=x[holdStart:nrow(x),], type="response",300)
mat<-matrix(pH,dim(pH)[1],dim(pH)[2],byrow=FALSE)
predictHoldout<-as.data.frame(mat)
colnames(predictHoldout)<-colnames(pH)
holdFinalPred<-as.factor(colnames(predictHoldout)[max.col(predictHoldout)])
@K-czar, sklearn does have a multiclass, multi-label capabilities. http://scikit-learn.org/stable/modules/multiclass.html Is this what you are looking for?
", "votes": 1, "replies": [] }, { "id": 44908, "author_name": "Yvan", "author_url": "", "post_date": "2014-05-20T07:17:58.077000", "content": "Hello everyone and congrats to the winners.
\nI made rank 170 using only logistic regression, for those interested here is the github link to my winning submission.
\nMy mistake I think was trying to predict *all* A, B, C, D, E, F and G and not letting some at rest.
\nTrying to be too smart at predicting everything is not obviously the best solution sometimes.
\nThanks again to breakfast pirate and all for sharing their ideas and to Kaggle and AllState for this very interesting competition
", "votes": 1, "replies": [] }, { "id": 44887, "author_name": "pythonomic", "author_url": "", "post_date": "2014-05-20T02:46:33.160000", "content": "Congratulations to all the winners. Would appreciate if you could share your approaches and code as well. Thanks .
", "votes": 1, "replies": [] }, { "id": 44884, "author_name": "Joshua Weiner", "author_url": "", "post_date": "2014-05-20T02:21:16.347000", "content": "Hi K-czar, my team did just that. We built a RF to predict if they would buy the policy they viewed the most, in addition to a RF to predict if they would buy the policy they viewed last.
\nIf the probability was high, we went forward and used the most frequently viewed or last observation, and if not we flew into a modeling mechanism where we tried to predict all 7.
", "votes": 1, "replies": [] }, { "id": 44876, "author_name": "Joshua Weiner", "author_url": "", "post_date": "2014-05-20T01:28:23.497000", "content": "@Euclides - thank you so much. I am very interested to see how you combined the ensembles. Given that it was categorical predicted values, did you use a voting method.... or did you average probabilities? Some algorithms don't easily spit out probabilities.
\nVery interested to see how you do it. Thank you very much.
", "votes": 1, "replies": [] }, { "id": 44871, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:54:49.053000", "content": "", "votes": 1, "replies": [] }, { "id": 44868, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:32:53.247000", "content": "", "votes": 1, "replies": [] }, { "id": 44866, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:30:26.350000", "content": "", "votes": 1, "replies": [] }, { "id": 44864, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:20:34.860000", "content": "", "votes": 1, "replies": [] }, { "id": 44858, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:11:27.160000", "content": "", "votes": 1, "replies": [] }, { "id": 46126, "author_name": "", "author_url": "", "post_date": "2014-05-23T14:05:48.813000", "content": "", "votes": 2, "replies": [] }, { "id": 46008, "author_name": "", "author_url": "", "post_date": "2014-05-21T17:57:53.037000", "content": "", "votes": 2, "replies": [] }, { "id": 44901, "author_name": "", "author_url": "", "post_date": "2014-05-20T04:54:24.937000", "content": "", "votes": 2, "replies": [] }, { "id": 44878, "author_name": "", "author_url": "", "post_date": "2014-05-20T01:31:54.693000", "content": "", "votes": 2, "replies": [] }, { "id": 44869, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:33:12.630000", "content": "", "votes": 2, "replies": [] }, { "id": 44863, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:18:42.417000", "content": "", "votes": 2, "replies": [] }, { "id": 141476, "author_name": "", "author_url": "", "post_date": "2016-10-27T02:44:36.350000", "content": "", "votes": -1, "replies": [] }, { "id": 135201, "author_name": "", "author_url": "", "post_date": "2016-09-12T04:29:04.510000", "content": "", "votes": 0, "replies": [] }, { "id": 135200, "author_name": "", "author_url": "", "post_date": "2016-09-12T04:25:40.907000", "content": "", "votes": 0, "replies": [] }, { "id": 135185, "author_name": "", "author_url": "", "post_date": "2016-09-11T22:47:15.103000", "content": "", "votes": 0, "replies": [] }, { "id": 135013, "author_name": "", "author_url": "", "post_date": "2016-09-10T04:33:32.007000", "content": "", "votes": 0, "replies": [] }, { "id": 48959, "author_name": "", "author_url": "", "post_date": "2014-06-10T17:34:47.687000", "content": "", "votes": 0, "replies": [] }, { "id": 47364, "author_name": "", "author_url": "", "post_date": "2014-05-30T19:19:46.907000", "content": "", "votes": 0, "replies": [] }, { "id": 46186, "author_name": "", "author_url": "", "post_date": "2014-05-24T20:00:35.740000", "content": "", "votes": 0, "replies": [] }, { "id": 46139, "author_name": "", "author_url": "", "post_date": "2014-05-23T17:20:49.347000", "content": "", "votes": 0, "replies": [] }, { "id": 46117, "author_name": "", "author_url": "", "post_date": "2014-05-23T09:17:19.417000", "content": "", "votes": 0, "replies": [] }, { "id": 46115, "author_name": "", "author_url": "", "post_date": "2014-05-23T08:26:55.920000", "content": "", "votes": 0, "replies": [] }, { "id": 46114, "author_name": "", "author_url": "", "post_date": "2014-05-23T08:17:52.913000", "content": "", "votes": 0, "replies": [] }, { "id": 46071, "author_name": "", "author_url": "", "post_date": "2014-05-22T16:53:12.367000", "content": "", "votes": 0, "replies": [] }, { "id": 46054, "author_name": "", "author_url": "", "post_date": "2014-05-22T11:19:19.883000", "content": "", "votes": 0, "replies": [] }, { "id": 46025, "author_name": "", "author_url": "", "post_date": "2014-05-21T23:52:47.123000", "content": "", "votes": 0, "replies": [] }, { "id": 46016, "author_name": "", "author_url": "", "post_date": "2014-05-21T20:17:08.090000", "content": "", "votes": 0, "replies": [] }, { "id": 46007, "author_name": "", "author_url": "", "post_date": "2014-05-21T17:46:38.930000", "content": "", "votes": 0, "replies": [] }, { "id": 46004, "author_name": "", "author_url": "", "post_date": "2014-05-21T17:20:11.270000", "content": "", "votes": 0, "replies": [] }, { "id": 46001, "author_name": "", "author_url": "", "post_date": "2014-05-21T17:04:13.297000", "content": "", "votes": 0, "replies": [] }, { "id": 45992, "author_name": "", "author_url": "", "post_date": "2014-05-21T15:14:37.337000", "content": "", "votes": 0, "replies": [] }, { "id": 45985, "author_name": "", "author_url": "", "post_date": "2014-05-21T12:39:12.977000", "content": "", "votes": 0, "replies": [] }, { "id": 44994, "author_name": "", "author_url": "", "post_date": "2014-05-21T02:39:50.573000", "content": "", "votes": 0, "replies": [] }, { "id": 44993, "author_name": "", "author_url": "", "post_date": "2014-05-21T02:03:02.133000", "content": "", "votes": 0, "replies": [] }, { "id": 44981, "author_name": "", "author_url": "", "post_date": "2014-05-20T21:31:57.123000", "content": "", "votes": 0, "replies": [] }, { "id": 44980, "author_name": "", "author_url": "", "post_date": "2014-05-20T21:29:39.967000", "content": "", "votes": 0, "replies": [] }, { "id": 44969, "author_name": "", "author_url": "", "post_date": "2014-05-20T17:41:26.770000", "content": "", "votes": 0, "replies": [] }, { "id": 44938, "author_name": "", "author_url": "", "post_date": "2014-05-20T13:24:36.783000", "content": "", "votes": 0, "replies": [] }, { "id": 44928, "author_name": "", "author_url": "", "post_date": "2014-05-20T09:44:43.867000", "content": "", "votes": 0, "replies": [] }, { "id": 44894, "author_name": "", "author_url": "", "post_date": "2014-05-20T04:08:12.970000", "content": "", "votes": 0, "replies": [] }, { "id": 44892, "author_name": "", "author_url": "", "post_date": "2014-05-20T04:03:15.247000", "content": "", "votes": 0, "replies": [] }, { "id": 44889, "author_name": "", "author_url": "", "post_date": "2014-05-20T03:20:00.277000", "content": "", "votes": 0, "replies": [] }, { "id": 44874, "author_name": "", "author_url": "", "post_date": "2014-05-20T01:09:47.703000", "content": "", "votes": 0, "replies": [] }, { "id": 44867, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:31:59.460000", "content": "", "votes": 0, "replies": [] }, { "id": 44865, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:25:39.037000", "content": "", "votes": 0, "replies": [] }, { "id": 44861, "author_name": "", "author_url": "", "post_date": "2014-05-20T00:15:30.827000", "content": "", "votes": 0, "replies": [] }, { "id": 134984, "author_name": "", "author_url": "", "post_date": "2016-09-09T20:22:23.793000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 76, "messages": [ { "id": 44856, "postDate": "2014-05-20T00:04:22.210Z", "content": "I think one key to this competition was choosing what *not* to change from the baseline. We didn’t change B or E at all. We only changed a small percentage of ACDF – usually only ones with shopping_pt=2. Before teaming up about a week ago, Alessandro had a top 10 score only changing G. We basically used his G and my ACDF. I think our solution only had about 2,500 rows that differed from the baseline (so less than 5% difference).
\nSomething else that helped was finding the “Georgia/Florida tricks.” No customers in Georgia had C=1 or D=1 in their final purchase. But some customers had C=1 or D=1 as part of their last quoted plan in the test set. Changing these to 2 gave improvement. Similarly, no customers in Florida had G=2 in their final purchase. Did anyone find any other situations like these?
\nIn addition to the base features, features I found useful were the A, B, C, D, E, F, and G from the previous shopping_pt. Also cost change from the previous shopping_pt.
\nI used GBM to predict individual ACDF values. Something that made this challenge difficult was that customers who could be safely predicted to change one product also had a high propensity to change multiple products – and getting multiple changes correct for one customer was difficult. So the customers that were easiest to predict for individual products turned out to be difficult to predict overall.
\nThank you to Allstate and Kaggle for a fun competition!
", "votes": 52 }, { "id": 45984, "postDate": "2014-05-21T12:22:47.220Z", "content": "Dear all,
\nfor all the models, we used only gbm in R. We didn’t even seriously try anything else, so there may be a potential for more improvements with other methods.
\nThe base of our idea was the same as for almost everybody here. We divided our data according to shopping_pt up to shopping_pt=6. The biggest value was of course for history 2 and we almost gained nothing for 6 as well as 5, but history 3 and 4 was still useful. As Breakfast Pirate posted the main idea was to find customers who have biggest potential that final purchase will be different from the current quote, i.e. for which it make sense to take a risk to change the last quote benchmark. We have trained few classifiers using adaboost metric (which seems to be a reasonable proxy to AUC), one classifier per each shopping point history.
\nThen we made three different models to predict the actual purchase record, all using multinomial metric in gbm.
First model was based on all available features in the provided data plus we added information from previous versions of ABCDEFG attributes and previous costs. This model consisted of 4 submodels – predicting together AF, BE, CD and G. These combinations were chosen based on the correlation matrix of these products. The second model was based on the same pairs of variables as the first model. It had all features as the first model and some more - a variable with mean cost in each state and each location was added. Moreover, after prediction of the first variable (G), a variable stating whether the model output is different from the LQB was added. The last pair (BE) was then modeled with information whether any other variable has changed. The last model was working with only three submodels (ABEF, CD, G) and had some more features relating to what was changed in the known customer history (e.g. ratios of current and previous costs, time differences, replacing of category variables with 1/0 feature vectors and feature subset vectors, etc.)
These three models were combined with corresponding classifier (models for history 2 with classifier for history 2, etc.) so we got the final predictions for each history. The algorithm used for the combining was based on the classifier prediction
- If the probability of the last quoted benchmark change was low, we kept the last quoted benchmark
- If the probability of the last quoted benchmark change was middle, we changed the last quoted benchmark only when all 3 models predicted the same value
- If the probability of the last quoted benchmark change was high, we changed the last quoted benchmark when at least 2 models predicted the same value
Our final solutions was putting together all predictions for all history (from 2 to 6). The most important parts of this blending was finding out the appropriate setting of the combining algorithm (i.e. which numbers to use for low/middle/high probabilities described above).
By the way, no specific handling for certain cases (such as Florida state) was done, however it looks from the feature importance that these specifics were found and used by the gbm trees.
\nAdditionally, we didn’t use any specific cross validation within each gbm model, but we think the way the last step was done (kind of strict majority voting from the three models – about 2100 records was changed overall) is what kept as from overfitting the public leaderboard).
\nAlternative point of view is that in such tight competition with so many experienced people, the most important factor that moved us on the top place in the private leaderboard is just a bit (or a lot?!) of a luck.. :-)
\nThanks all for a great competition (special thanks to Kaggle and Allstate) and congratulations to all to placing teams.
\nPrazaci team (maternaj, Perroquet, Matfyzak, nimo, Emil Skultety)
", "votes": 30 }, { "id": 47362, "postDate": "2014-05-30T19:17:14.387Z", "content": "It took me a while to clean up everything, and write some stuff... but here it is! Hopefully somebody is still reading :)
\nThe main idea is that we don't know at what shopping_pt the purchase will be made. We know which plans fit more each profiles, so basically I'm training the model on the whole dataset using the purchased plan as target at each shopping_pt during the transaction history for all the customers. Even though purchases never happen at shopping_pt #1, it is included in the training data. The main reason is because patterns which occur at shopping_pt #1 for some customer can occur at different shopping_pt for others customers, leading to the same plan purchased.
\nI’m using is a Random Forest (scikit-learn implementation) as base model, which by itself only can give quite good results. To produce a robust model, I’ve ensemble 9 Random Forest out of other 50. If five out nine models agree on the same plan then this change is made, otherwise the last quote is used (majority vote).
\nThe final ensemble, which led our team to place 2nd in the private leader board, is the combination of my predicted G and Steve’s ABCEDF.
\n## Extra Features
I’ve used all the features provided, at exception of date & time. To help tree interaction and improve the accuracy I’ve also included the following features, group by category for your convenience.
Category Interactions (2-way)
\nCategory & Interaction mapped at arithmetic mean of the cost
\nAverage of target variable
\nContinuous Interactions
\nNaming Convention
Product: A, B, C, D, E, F and G are all products.
Plan: Combination of A, B, C, D, E, F and G.
Baseline: Is the last plan or product quoted at the latest shopping_pt available.
Metric
The score I’ve used to determine how good a model is defined as follow. The difference between the baseline accuracy and the model accuracy measured at each single shopping_pt, times the number of samples in the test set at that shopping_pt. For example, the difference between the model and the baseline for shopping_pt #2 is 0.4160-0.4116=0.0044 times the count of the test samples where the latest shopping_pt available is #2 is 0.0044x18,943=58.5. I’ll be addressing to score at the sum product of these differences and the test set distribution.
## Modelling Techniques & Training
A Random Forest is by itself an ensemble of decision trees. Each decision tree in a Random Forest is trained on different subset of data, leading to many different trees. The Random Forest predictive power comes with the ensemble of all these trees, stacking the class probabilities.
The higher is the number of tree we’ll build, the more accurate and more stable the prediction will be. This is what happens usually, but not in this problem since the gain over the baseline is very low. Making this more sensible to randomness and harder to fix only increasing the number of tree!
\nInstead of keep stacking the class probabilities and increasing the number of tree, we can keep the number of tree in the Random Forest lower and look at the output at a number of Random Forests. If the majority of these agree on the same outcome, then is quite likely that change is actually occurring. If the majority have the same outcome, then chose this as final prediction. Otherwise use the safest option: the baseline. Making this strategy is less prone to randomness.
\nI’ve trained several Random Forest using the same data but using different seed, which approximately lead to ~300 different predictions out of 55,716. Is quite a low number, but in this particular competition one more accurate prediction is the difference between the 2nd and 3rd place! Hence here comes the need to have not just a good model, but a very stable model which will generalize as much constantly as it could on unseen data.
\nUsing the majority vote ideas gave a quite stable prediction (and more accurate). What helped a little bit further was selecting a subset of all the Random Forests which are expected to have a better accuracy. How? While looking for a way to identify more accurate Random Forest I’ve noticed that for higher train set scores, usually there is an higher cross validation score. Following this intuition I could discard model whom their train set score was worse than the others as they are more likely to be not good as the others. Do a majority vote on the best 9 Random Forests instead of using all the 50 improved the results too!
\nFinally, if you're not bored about reading all this - here is the link to the GitHub repository: https://github.com/alzmcr/allstate
", "votes": 25 }, { "id": 44872, "postDate": "2014-05-20T00:59:13.527Z", "content": "I've just entered in this competition last Monday and I'm really surprise with my result.
\nHere follows my approach.
\nCongrats to all :)
", "votes": 13 }, { "id": 44879, "postDate": "2014-05-20T01:44:51.847Z", "content": "Our team was ranked 1st on public board but crashed on private board. I think it is really easy to be over fitting in this competition, and it is difficult to select which submission might be over fitting and which not.
\nActually, before team merging, one submission made by our team member Nitai Dean who just develop a Random Forest classifier for G option will already give top 10 score in final private board, and an ensemble model from Jagannath's model and my model will also give top 10 score in final private board. Besides, a model which just adds FL/Georgia tricks(similar to team of Alessandro&BreakfastPirate, we also found these tricks for FL/Georgia) to Nitai Dean's G_only model, will give us top 5 score on final private board.
\nHowever, it seems to be very difficult to decide which submission is over fitting and which not, and we seem to underestimate the over fitting issue in this competition and the two submissions we choose for final evaluation turn out to perform poorly on the private board.
\nFinally, I do agree that, the key in this competition is to decide when to correct the last-quote baseline prediction and when not. And the over fitting issue as well as some randomness about the public board and private board data split contribute to the big shake of ranking on the private board.
\nP.S.: About the FL/GA tricks, actually we also only find these three tricks same as BreakfastPirate, namely, for example, there are no purchased G=2 for FL, but test set last-quote has some entries such that G=2. And we do find some other tricks that is not like such 0:100 split but something like 10:90 split. For example, for state NY, if last-quote G=1 and C_previous=1, then the purchased G would be highly possible(more than 85% correct) to be G=2 based on training set. And such corrections (only 99 such corrections in the whole test set) does also contribute to significant improvement in public board score (improved by 13*0.00006). Since such corrections contribute to significant improvement to both training set 10-fold cross-validation and public board score, we thought that this probably will also contribute to significant improvement in private board, but a post-deadline submission shows that such corrections about G option of NY just doesn't change the score on private board at all. Therefore, I guess that the existing randomness about data split of public board and private board might be one major reason for the big shake on ranking of public board and private board.
\nAnyway, this competition is still a great competition, and thanks Kaggle and Allstate! And congrats to the winners!
\nBest wishes,
\nShize
", "votes": 11 }, { "id": 44922, "postDate": "2014-05-20T08:51:10.223Z", "content": "Congrats to the winners!
\nI joined fairly late and I did not have much time to experiment a lot, however I immediately saw that there was something going on with the cities as I could barely beat benchmark without them and got top 50 in private with them only with G ! In short :
\n[quote=Joshua Weiner;44874]
\n@Euclides - would you consider sharing your code? Since you are the only one who has expressed using many different ensembles and had success I would be very interested in seeing how you did it.
\n[/quote]
\nYes sure, after a clean up I'll put it on github and post the link here.
", "votes": 10 }, { "id": 46118, "postDate": "2014-05-23T09:27:46.427Z", "content": "Marat,
\n\nIt takes a little bit of time, but you need to automate what you do (e.g. everything in code). Also save everything you've done in the past. I am spending less than half the time on a competition in comparison to when I started kaggling and I am scoring much better, because everything is automated and to some extend more optimized . I also don't repeat stuff that did not work many times in the past so , which also saves lots of time. After while the important question will be how much and how fast your computer can handle the data rather than how much time you have :) . Don't get me wrong, you always need to spend time to understand the data, but even that gets faster the more you play.
", "votes": 5 }, { "id": 44903, "postDate": "2014-05-20T05:03:42.980Z", "content": "Nice reading this forum, thanks to everyone! My approach was a combination of random forests and logistic regression.
\nFor some models I threw away records where more than one single option changed between last observed quote and purchase. Those models worked best for A-F (which makes sense because A-F often change together, making it harder for the model to attribute ‘imbalance in the quote’ to one option). I included dummy encoded features, counts of values and likelihood features, but they did not add much to the raw features.
\nI trained using the last two shopping points and applied the model equeally to all shoppingpoints. For cross validation I randomly truncated shoppingpoints until I had 0.537. I never managed to improve that beyond 0.539, but every time I had a consistent improvement (over five folds) the leaderboard surprised me with a better improvement. So I think there is something not random about the truncating.
\nIt also helped to use a more conservative threshold (for changing last observed quotes) for some changes, like G from 1 to 2 and G from 4 to 3.
\nI used the public leaderboard just for checking improvement (tried to avoid comparing alternatives) and in the end to tune the number of last observed quotes to change. In my best selected model, I changed just over 1400, but for the private set I should have changed more.
", "votes": 5 }, { "id": 44900, "postDate": "2014-05-20T04:50:57.850Z", "content": "I looked at this as a two-step process, as BreakfastPirate discussed. But I never got anything intelligent working for the second step
\nThe situations I targeted, with rough probabilities of the last seen being purchased and the number of customers:
\nExcept the last, these all had very low probabilities based on the train set, so doing just about anything would help your score.
\nSomebody asked how to find these, and I found the 0011004 during the data exploration phase as well--probably the first hour or two with the data. I initially truncated the data at "final shopping point - 4" for everybody. Then I simply ran a SQL query counting the number of times each plan appeared as the last seen, and then purchased, and sorted by the difference. 0011002 was simply more likely for those that were last shown 0011004. Sending that in was worth about 6 correct. And I just happened to observe that FL kept coming up when I was going through the rest of that list. Once I saw that, I scanned every 3-field combination to see if I could find such a pattern, and the GA pair came up.
\nI tried GBM many times on many different subsets--full plans and just the components; all the data, just those rules listed above and other less probable pieces. I just couldn't get anything out of it better than the most common full plans, given the full last-seen plan in that situation.
\nSo for the GA and FL rules, I swapped in the most likely single value. The others, I usually used the full plan. I tried to work with overall net migration into and out of particular plans, but no improvement.
\nCongratulations to the winners! I look forward to learning from you all and seeing what I was doing wrong on the ML side of things.
", "votes": 5 }, { "id": 44881, "postDate": "2014-05-20T01:54:37.917Z", "content": "WOW!!! I can't describe how afraid was to check Kaggle and the LB today - so wrong!!!! xD
\nI'll be super happy to share my approach & code in details as long this is fine with Allstate, after a beer and pizza!
", "votes": 5 }, { "id": 46055, "postDate": "2014-05-22T11:50:22.720Z", "content": "[quote=K-czar;46001]
\n@Perroquet Thanks so much for sharing! That was really awesome to read b/c it validates a lot of the ideas we had intended to try with more time! I also wasn't aware GBM (in R) even had a multinomial metric, or the capability to predict in multiclass, multioutput problems. I don't believe it does in python. That was the reason we stuck to Random Forrests! But when we predicted variables in pairs - similarly based on the correlation matrix - we found that predicting the variables independently almost always drew higher accuracy.
\nSo far, I think you're also the only other group I've seen that incorporated a "consensus" approach like we did.
\nDid you have any memory issues using R's GBM with all of the locations as variables? Unless R's GBM implementation has been updated to serve sparse matrices, I would have guessed it would cause considerable memory problems.
\nThanks so much for sharing, and congratulations! Great work!
\n[/quote]
\nThis was the first time we really used distribution="multinomial" parameter for gbm in R and it served its purpose very well.
\nAs far as I know you are right that gbm doesn't support sparse matrices. We didn't actually used all locations as variables, only added average cost per location as a feature and average cost per state as a feature (the later one being more helpful).
\nGenerally, regarding memory issues, there is a memory leakage in the latest version from CRAN (2.1 I think) for multinomial distribution which causes the whole RAM to be eaten up quite quickly.
\nIn https://github.com/harrysouthworth/gbm (thanks to Harry and all contributors for supporting and further developing this excellent package!) there are newer minor versions that fix the memory leakage (namely 2.1-0.3; beware of 2.1-0.5 as the changes for multinomial seem to broke it somehow).
\nLastly, for larger number of classes (which was the case for ABEF model that has overall 48 classes) there is another bug which causes the training model at some point starts producing NaN values - we hopefully were able to fix this on our own and will try to push this fix to the next releases of gbm.
", "votes": 3 }, { "id": 44911, "postDate": "2014-05-20T07:49:00.507Z", "content": "Thank you for sharing your approaches! You got some really nice insights :-)
\nAs for me, at the very start of the competition I loaded the data. Then I trained 7 random forests, one for each option, on selected informative features (most importantly time, location and customerID) but they weren't working nicely. So I did many cross validations to select appropriate seed and when I finally achieved quite high performance I used the obtained seed on the test set and it got me to the 10th place.
\nSeriously.
\nI entered just yesterday but I was working on this competition earlier for about a month after it launched. I created some time-varying features like mean cost, cumulative gradient of cost, number of changes of options, previous A etc. After I noticed the importance and "changeness" of G I created several new features that had sensible correlation with coverage options and could improve the performance. Then I used the whole data set for training, treating each observation independently, as I thought that truncating the data set would simply delete too much information.
\nI also employed a two-step approach - first a classificator (will the X option change?) then a preditor (if it changes, what it will be?), both GBMs. Each option had its 3 classifiers and 3 predictors, trained on 3 customer-wise folds. I trained predictors on lots of data (classificator_output > 0.2), but used a more conservative threshold in the test set (0.45 for G, 0.7 for the rest). Now I see I should have used an even lower threshold on G, but well :-)
\nThank you all for taking part, it was a very interesting competition, good luck in the next ones!
", "votes": 3 }, { "id": 44905, "postDate": "2014-05-20T05:57:08.417Z", "content": "[quote=TayShin;44892]
\n@Hiroyuki Can you please elaborate more? because I used exactly same procedure as you did but my score didnt beat the bench mark. variables that I used is given variables + previous quote(A through G). Thank you in advance!
\n[/quote]
\nHere is myRFcode.
\nI'm afraid it's not exactly same as my best submission.
\nMy feature, child, couple and single didn't work well maybe.
", "votes": 3 }, { "id": 44885, "postDate": "2014-05-20T02:22:17.633Z", "content": "My strategy was quite simple. I used randomforest model with almost all given variables and 2nd lastquote and then I got (0.53556) on private leaderboad.
\nUsing firstquote, separating plans into the pair of "CD", "AEF" and "BG" and building a model for each shopping_pt (ie. 2,3,4...) didn't work well.
\nRandomForest worked better than gbm, nnet and svm on my prediction.
\nIt was difficult to do CV. I had no idea which data should I use for CV, the truncated train data set or no truncated data. On my prediction, no truncated data gave me better result but I couldn't understand why it went successful.
\nThank you for exciting competition, and see you next!
", "votes": 3 }, { "id": 44880, "postDate": "2014-05-20T01:50:02.047Z", "content": "My best private LB score was just the last quote + change of all Florida customers who still had G2 as option and arbitrarily changed it to G3. That's all.
\nG quirks were:
\nFlorida customers apparently could choose G2 in their quotes, but never purchased it. G1 was apparently not an option.
\nSD and ND customers only had G2 as option. Any model that gave ND or SD customers anything other than G2 was an easy fix.
\nOhio customers did not have G1 as an option.
\nI didn't notice the Georgia quirk, but wonder how much of a difference it would have made.
\nI noticed high polychoric correlations in purchased plans between AF, BE, and CD pairs. I also noticed that (generally) classifiers using these pairs for targets (with G modeled separately) did better than classifiers for each of the options individually. However, none of them ever beat my score with last-quote and Florida fix only.
", "votes": 3 }, { "id": 44860, "postDate": "2014-05-20T00:14:49.557Z", "content": "And congrats Prazaci team for your surprise jump from 11th to 1st!
", "votes": 3 }, { "id": 44883, "postDate": "2014-05-20T02:15:32.153Z", "content": "My teammate and I got didn't make a whole lot of submissions on this (8 - but only a couple real ones), but managed to put up a top 5% score, so I'll share what got us there and our insights.
\nAlmost all of our gain came from customers in the test set with only two shopping points given, as that was where we focused most of our time. The results we submitted primarily resulted from two models - logistic and Random Forest. Each model produced fairly good results predicting each purchase option (A-G) independently - significantly better than the benchmark for each individual variable, though neither was actually able to beat the benchmark when predicting all seven variables together.
\nWe found that a pretty good way of ensembling the two models was to simply require a consensus between both models to override the benchmark. If there was no consensus (or the consensus was the benchmark) then we used the benchmark. That was good enough for 72 place.
\nWe actually mostly used this project as a learning exercise to experiment with Pylearn2's Deep Neural network learning packages. As it turns out (we checked by submitting just after the deadline, since we ran out of submissions today), using a NN on shopping point two alone would have vaulted is up another 20-30 spots on the leaderboard.
\nI'm curious if anyone made use of a model trained on a binary output of whether or not a person's purchase choices will be the same as a benchmark. That was something I had wanted to work on if we had more time, so I'm curious if it worked for anyone.
", "votes": 4 }, { "id": 46157, "postDate": "2014-05-24T03:14:20.233Z", "content": "I try running your code to a T and in the doSubmission when I try to train the GBM I get an error...Please see the traceback below:
\n3 stop("gbm does not currently handle categorical variables with more than 1024 levels. Variable ",
i, ": ", var.names[i], " has ", length(levels(x[, i])), " levels.")
2 gbm.fit(x, y, offset = offset, distribution = distribution, w = w,
var.monotone = var.monotone, n.trees = n.trees, interaction.depth = interaction.depth,
n.minobsinnode = n.minobsinnode, shrinkage = shrinkage, bag.fraction = bag.fraction,
nTrain = nTrain, keep.data = keep.data, verbose = lVerbose,
var.names = var.names, response.name = response.name, group = group)
1 gbm(willChange ~ ., data[train, -omitClass], distribution = "bernoulli",
keep.data = F, verbose = F, n.cores = 1, n.trees = classTrees,
shrinkage = class_shrinkage, interaction.depth = class_depth,
n.minobsinnode = class_minobs)
[quote=Stephen McInerney;46132]
\nWhat does 'hasF' mean,I don't see it computed anywhere? Does it mean 1) 'was not NA on all previous quotes' or 2) are you assuming F==0 means no coverage (which is not the case, per Deanonymizing coverage levels and options, also by state)
\n[/quote]
\nThis was one of the very minor changes :-) I fixed it on git, thanks :-)
\nYour guess nr 2 is correct - I assumed that F==0 means that there is no coverage. It was motivated by the description of C_previous, i.e. "C_previous - What the customer formerly had or currently has for product option C (0=nothing, 1, 2, 3,4)". Given that C "normally" takes values 1..4 such inference for other options seemed to be reasonable for me.
\nOn the other hand your analysis looks interesting, although I did not discover such cost <-> options relationships when I did the exploration part.
", "votes": 1 }, { "id": 46132, "postDate": "2014-05-23T14:52:01.573Z", "content": "[quote=blaine;46126]
\nT/F did the client had the coverage option A/B/E/F in any of her previous quotes?
\nhttps://github.com/B1aine/kaggle-allstate/blob/master/prepareData.R
\n mat[customer,"hadF"][obs] = mat[customer,"hadF"][obs-1] | mat[customer,"hasF"][obs]
[/quote]
\nWhat does 'hasF' mean,I don't see it computed anywhere? Does it mean 1) 'was not NA on all previous quotes' or 2) are you assuming F==0 means no coverage (which is not the case, per Deanonymizing coverage levels and options, also by state)
", "votes": 1 }, { "id": 46111, "postDate": "2014-05-23T04:48:46.397Z", "content": "@blaine
\nSure, even though we were updating 2.1-0.3 (2.1-0.5 seems to be "more" broken as NaNs for multinomial start to appear immediately if I remember correctly). For 2.1-0.3, the fix we made was in multinomial.cpp and relates to the division by zero error - in some places for very low values the denominator was rounded to 0.
\nIf interested in our "hacked" version of multinomial.cpp, just drop me an email.
\nEdit: wrong formatting
", "votes": 1 }, { "id": 46060, "postDate": "2014-05-22T13:48:25.217Z", "content": "[quote=maternaj;46055]
\nIn https://github.com/harrysouthworth/gbm (thanks to Harry and all contributors for supporting and further developing this excellent package!) there are newer minor versions that fix the memory leakage (namely 2.1-0.3; beware of 2.1-0.5 as the changes for multinomial seem to broke it somehow).
\nLastly, for larger number of classes (which was the case for ABEF model that has overall 48 classes) there is another bug which causes the training model at some point starts producing NaN values - we hopefully were able to fix this on our own and will try to push this fix to the next releases of gbm.
\n[/quote]
\nI also used multinomial GBM in R and very quickly encountered the NaN problem, with 2.1-0.5 from Harry Southworth's git. Moreover, even when the seed was fixed it tended to produce very different models on each run. What is interesting, these problems disappear when the very same code and GBM + R are run on Windows. Well done with fixing it! And I really hope your fix will be included in the next release :-) I am very curious what was actually the problem, do you mind sharing?
\n[quote=pythonomic;44969]
\n@blaine : Thanks for sharing your approach. would appreciate if you could share your code as well. Thanks.
\n[/quote]
\nI will upload my code within a few days. However after I started cleaning it up I discovered a huge thinking error I made when training the models :-) I will share the code and describe the mistake so people might learn something from it.
", "votes": 1 }, { "id": 46006, "postDate": "2014-05-21T17:39:09.173Z", "content": "@mlandry I notice your y is "yG" so I assume you're doing a multiclass classification, as opposed to multiclass + multioutput simultaneously.
\nPerroquet said they predicted AF, BE, CD and G together in pairs - I assume they simply constructed special y features to accomplish that where each combination of AF was considered a separate class. For Random Forest in sklearn you don't have to do that though - which I guess just makes it simpler, though perhaps not substantively different.
\nDefinitely useful to know though. Thanks!
", "votes": 1 }, { "id": 46005, "postDate": "2014-05-21T17:26:26.400Z", "content": "Not sure if this is the best way to do it, but you get a matrix of probabilities from R's GBM implementation, and here's how I was using that (to get the highest, but you can analyze the full range, if desired)
\ng<-gbm.fit(x=x[1:trainEnd,],y=yG[1:trainEnd],distribution="multinomial"
\n,n.trees=trees,shrinkage=shrink,interaction.depth=depth,n.minobsinnode=minobs)
\npH<-predict.gbm(object=g,newdata=x[holdStart:nrow(x),], type="response",300)
mat<-matrix(pH,dim(pH)[1],dim(pH)[2],byrow=FALSE)
predictHoldout<-as.data.frame(mat)
colnames(predictHoldout)<-colnames(pH)
holdFinalPred<-as.factor(colnames(predictHoldout)[max.col(predictHoldout)])
@K-czar, sklearn does have a multiclass, multi-label capabilities. http://scikit-learn.org/stable/modules/multiclass.html Is this what you are looking for?
", "votes": 1 }, { "id": 44908, "postDate": "2014-05-20T07:17:58.077Z", "content": "Hello everyone and congrats to the winners.
\nI made rank 170 using only logistic regression, for those interested here is the github link to my winning submission.
\nMy mistake I think was trying to predict *all* A, B, C, D, E, F and G and not letting some at rest.
\nTrying to be too smart at predicting everything is not obviously the best solution sometimes.
\nThanks again to breakfast pirate and all for sharing their ideas and to Kaggle and AllState for this very interesting competition
", "votes": 1 }, { "id": 44887, "postDate": "2014-05-20T02:46:33.160Z", "content": "Congratulations to all the winners. Would appreciate if you could share your approaches and code as well. Thanks .
", "votes": 1 }, { "id": 44884, "postDate": "2014-05-20T02:21:16.347Z", "content": "Hi K-czar, my team did just that. We built a RF to predict if they would buy the policy they viewed the most, in addition to a RF to predict if they would buy the policy they viewed last.
\nIf the probability was high, we went forward and used the most frequently viewed or last observation, and if not we flew into a modeling mechanism where we tried to predict all 7.
", "votes": 1 }, { "id": 44876, "postDate": "2014-05-20T01:28:23.497Z", "content": "@Euclides - thank you so much. I am very interested to see how you combined the ensembles. Given that it was categorical predicted values, did you use a voting method.... or did you average probabilities? Some algorithms don't easily spit out probabilities.
\nVery interested to see how you do it. Thank you very much.
", "votes": 1 }, { "id": 44871, "postDate": "2014-05-20T00:54:49.053Z", "content": "I barely beat the benchmark using handpicked/bruteforced rules for plan options vs. cost.
\nif cost > 810 and option_g == "2" then flip to "1"
\nif cost > 750 and option_g == "4" then flip to "2"
\netc.
\nchanging only 59 plans from last selected benchmark.
\nI did this by checking at what costs the option would flip > 50% of the times in the train set. Then I took most popular replacement option for the flip.
\nSomehow with my last two submissions (RF's) I either equaled the benchmark or I forgot to properly select my final submission. Anyway, I don't rank for it on private leaderboard (0.53274).
", "votes": 1 }, { "id": 44868, "postDate": "2014-05-20T00:32:53.247Z", "content": "[quote=Marat;44861]
\nBreakfastPirate, how did you manage to find Georgia/Florida tricks?
\n[/quote]
\n\nI found it too. It's easy.
\nfor : each state
\nmean(coverage A == median(coverage A in each state))
\nmean(coverage B == median(coverage B in each state))................(goes on)
", "votes": 1 }, { "id": 44866, "postDate": "2014-05-20T00:30:26.350Z", "content": "I only build a model for shopping_pt==2 because of time constraints
\nI think I had a good idea.
\nMy idea is simple.
\nFirst, I separated the customers in to two groups. One with the customers who viewed the coverage (A though G) at least one time before they buy and the customers who didnt viewed the coverage.
\nThe last quote strategy gives 96% coverage accuracy (considering A through G) for group 1 and 0% for group2 (ofc)
\nso I thought I could improve this by this easily: binary prediction on whether customer will buy a coverage that they have seen or not. I used random forest with AUC technique to solve the problem for customer who has shopping pt==2. Finally, I used random forest to build a model to predict coverage for the customer who has not seen the product before they purchased. So my strategy is just 1. if customer is going to buy a coverage they have seen -> last quote strategy 2. if customer is going to buy a converage they havent seen-> random forest.
\nI only build a model for shopping pt==2 and it worked great. I wish I had a little more time to build a model for rest of shopping pt
\n\nanyway goob job guys!
", "votes": 1 }, { "id": 44864, "postDate": "2014-05-20T00:20:34.860Z", "content": "I found it in the data exploration phase. I did a cross-product of state vs each A, B, C, D, E, F, G's final value to get counts. I saw some states had zeros for certain product values. So I looked to see if anyone in those states had those values for their last quote plan. In most cases no one did, but for GA/FL there were a few.
", "votes": 1 }, { "id": 44858, "postDate": "2014-05-20T00:11:27.160Z", "content": "Interesting. I found G harder to predict than other columns (around 86% accuracy). My classifiers seem to work well (92-93%) on the other columns, so I went with the predictions for A-F plus 'last value G'. Big mistake, My solution go trounced in the final leaderboard.
\nCongratulations to the winners.
", "votes": 1 }, { "id": 46126, "postDate": "2014-05-23T14:05:48.813Z", "content": "Hey miners,
\nHere is my code along with a small description of the approach I took: https://github.com/B1aine/kaggle-allstate
\nAs I said I made a huge thinking mistake - its described in the 'warning' section of README. A funny thing about this code is that even though it was created quite early its results were not submitted until the very last day of the competition. I suppose this saved me from having time to do overfits to the public leaderboard.
\nI hope it will be useful to someone. If you have any questions/suggestions please mail me.
\nI also want to note that when you run this code you will not get my exact submission due to a few very minor changes I made - but it should be very similar.
", "votes": 2 }, { "id": 46008, "postDate": "2014-05-21T17:57:53.037Z", "content": "My method is similar to Euclides Filho, Perroquet and others. I build my model using random forest based on the correlation matrix of the seven options. One for each shopping_pt. I then use bootstrap like method to train the model on 13 different subsets of the train data. So the total would be 4X13=52 classifiers. To combine the results, I use cross-validation to find out the threshold. The idea is that, for example, when we only have 2 history quotes, the probability the purchased policy is the same as the last available one is around 0.4. So if 6 of 13 submodels agree on one policy, then we should choose the one regardless whether it is the same as the last quote, because 6/13>0.4. However, due to the correlations between the 13 committee members, the actual threshold is 8 based on my cross-validation. This model gives positive results for shopping_pt from 2 to 7.
\nAs for the features, I used two last quote history and all other features given. The only two additional features are the mean cost for each location, and the cost difference between the last two quotes. I did try to experiment on various sets of features. But I found they did not bring improvement. One interesting observation is that, the smaller the location number is, the more samples it have in the training set. So using the location as a numerical feature instead of a categorical feature can actually improves my result a little bit.
\nThe other lesson I learned from this competition is to never try to experiment on new ideas in the last days of a competition. In the last hour of the competition, I was busy in combining the results. And the clock ran out before I could choose the two final submissions. Kaggle automatically chose the two overfitted results and my ranked crashed from No.5 to No. 22. :(
", "votes": 2 }, { "id": 44901, "postDate": "2014-05-20T04:54:24.937Z", "content": "Hi everyone, great thread. I tried lots of different approaches, and ultimately came up with one simple approach that beat the private leaderboard benchmark (though unfortunately it was not one of the two submissions I selected!)
\nBased on Jos Theelen's tip in the forums, I tried to find complete plans (combinations of all 7 options) that were viewed but rarely purchased in the training set, and replace them with the "most likely replacement plan."
\nFirst, I calculated the percentage of times each plan was purchased out of all of the times it was viewed (quoted). Then, I calculated the total number of times each plan was viewed. If a plan's purchase percentage was less than 5% and it was viewed in at least 500 quotes, I considered it "unlikely to be purchased".
\nFor each of those plans, I calculated the best "replacement" by tallying up what was the most common plan purchased by people who viewed those plans. And, I required that the replacement had to be purchased by at least 5% of those customers, otherwise I wouldn't consider it to be a "common enough" replacement.
\nThere were only two pairs of plans that met this criteria. 0011004 was unlikely and should be replaced by 0011002. 0022004 was unlikely and should be replaced by 0022002.
\nSo in summary, if you simply use the last quoted plan as your baseline prediction, and convert 0011004 to 0011002, and convert 0022004 to 0022002, you beat the benchmark on the private leaderboard by 3 picks (0.53277). This affects a total of 305 rows.
\nNote that I did play around with the three threshold values (5%, 500, and 5%) considerably, and depending on the different values it replaced a different number of plans. If I had done some cross-validation on this approach I probably would have chose to submit these predictions, but I instead selected the submissions that did the best on the public leaderboard (which did worse than the benchmark on the private leaderboard!)
\nHere's a link to my R code, as well as my paper (in which I discuss this approach and many others in more detail). If you have any feedback on the paper, I created another thread and would love to hear from you!
\nThanks,
\nKevin
", "votes": 2 }, { "id": 44878, "postDate": "2014-05-20T01:31:54.693Z", "content": "[quote=Joshua Weiner;44876]
\n@Euclides - thank you so much. I am very interested to see how you combined the ensembles. Given that it was categorical predicted values, did you use a voting method.... or did you average probabilities? Some algorithms don't easily spit out probabilities.
\nVery interested to see how you do it. Thank you very much.
\n[/quote]
\nBasically I used the individual classifier's probabilities of each class as features on a Logistic Regression classifier to predict the final classes.
", "votes": 2 }, { "id": 44869, "postDate": "2014-05-20T00:33:12.630Z", "content": "I tried a similar strategy as the state tricks but using location code. I was not successful. When you filter by location code there is less data. I created a dictionary for location codes that had mostly a specific plan in the train set (more than 10 observations, more than 65 percent a particular plan). If the last quoted plan in test wasn't found in the filtered train and the prob of the last quoted plan was smaller, I would switch the plan to what I had in my dictionary. I changed 11 entries. I got the benchmark exactly. I am not sure why.
", "votes": 2 }, { "id": 44863, "postDate": "2014-05-20T00:18:42.417Z", "content": "We had a similar strategy as well, only predict a certain number of shopping_pt=2 but predicted all of the values. For shopping_pt > 3, only predict G.
\n\nI noticed the Florida usually being G3, and New York usually being G2 I think, but didn't explicitly change them in the final output and let the classification figure it out. State, last seen, and cost were the most important features to us.
\n\nIt was an interesting challenge trying to predict the outcome of something that was likely to happen > 90% of the time, so seeing that G stayed as the last seen in the ~80-85% range it was the one plan that could be predicted to find any gains on.
", "votes": 2 }, { "id": 141476, "postDate": "2016-10-27T02:44:36.350Z", "content": "Hi All,\nI have gone through a few comments/solutions on this one, however it appears that in reality no model or solution was practical enough to deploy in the real life scenario ?
", "rawMarkdown": "Hi All,\r\nI have gone through a few comments/solutions on this one, however it appears that in reality no model or solution was practical enough to deploy in the real life scenario ?", "votes": -1 }, { "id": 135201, "postDate": "2016-09-12T04:29:04.510Z", "content": "Anyway thanks for your comment weiyu wan. Are you a Data analyst or Data Scientist.
", "rawMarkdown": "Anyway thanks for your comment weiyu wan. Are you a Data analyst or Data Scientist. " }, { "id": 135200, "postDate": "2016-09-12T04:25:40.907Z", "content": "I have predicted whether the customer will purchase or not with a balanced accuracy of 0.532116.
", "rawMarkdown": "I have predicted whether the customer will purchase or not with a balanced accuracy of 0.532116. " }, { "id": 135185, "postDate": "2016-09-11T22:47:15.103Z", "content": "[quote=Chirantan;135013]
\n\nInstead of predicting which options to buy among A-G. I have predicted whether a person will purchase or not. what you guys think about that.
\n\n[/quote]\nI also think we should divide people into two types. One type is the person that will not change from the last quote into a new one. Another is the person will change from the last quote. Then, we will predict the value of G for this kind person. However, I am not sure whether we could classify types of people accurately.
", "rawMarkdown": "[quote=Chirantan;135013]\r\n\r\nInstead of predicting which options to buy among A-G. I have predicted whether a person will purchase or not. what you guys think about that. \r\n\r\n[/quote]\r\nI also think we should divide people into two types. One type is the person that will not change from the last quote into a new one. Another is the person will change from the last quote. Then, we will predict the value of G for this kind person. However, I am not sure whether we could classify types of people accurately." }, { "id": 135013, "postDate": "2016-09-10T04:33:32.007Z", "content": "Instead of predicting which options to buy among A-G. I have predicted whether a person will purchase or not. what you guys think about that.
", "rawMarkdown": "Instead of predicting which options to buy among A-G. I have predicted whether a person will purchase or not. what you guys think about that. " }, { "id": 48959, "postDate": "2014-06-10T17:34:47.687Z", "content": "Thanks! Alessandro for sharing your solution.
" }, { "id": 47364, "postDate": "2014-05-30T19:19:46.907Z", "content": "@Alessandro - not bored at all reading this, and very thankful that you took the time to do the write-up.
" }, { "id": 46186, "postDate": "2014-05-24T20:00:35.740Z", "content": "[quote=Joshua Weiner;46139]
\nHi Blaine, where in your code do you collapse your dataset to have one observation per person? Or do you create a prediction for each shopping point for each person?
\n[/quote]
\nI trained models on each shopping point and each customer, and I also made predictions that way (i.e. models are applied to whole test data). I was going to do an analysis of prediction results, but due to time constrains I went the easy road and picked the last quote as the final prediction anyway.
\nAs for the error, it is fixed. I apologize for these mistakes, I have lots of duties at this moment and I cannot spend so much time cleaning it up and testing as I would want to. Everything seems to be working fine now though.
" }, { "id": 46139, "postDate": "2014-05-23T17:20:49.347Z", "content": "Hi Blaine, where in your code do you collapse your dataset to have one observation per person? Or do you create a prediction for each shopping point for each person?
Marat,
\nHaving a full time job and participating can be difficult if one plans on doing some serious stuff and just getting started with machine learning.
\nI return from work, take a hour break for cooking and other errands and then spend some time on competitions. This includes reading up stuff as well. Days, when there are ridiculously long meetings or project deadlines coming up, sleep is the one that loses out in all the power struggle :-)
\nWe might just as well start a new thread on time management!!
" }, { "id": 46115, "postDate": "2014-05-23T08:26:55.920Z", "content": "Hi, everyone!
\nIf you work full-time at your job, how is it possible to find time to participate in the contest?
\nFor example I came to kaggling about an hour before my job every day, rather than kaggling in weekends or after my job -- unfortunately I found it only 10 days before the deadline. But It worth it.
\nPlease, can you describe how you shared your kaggling time with your main job time.
" }, { "id": 46114, "postDate": "2014-05-23T08:17:52.913Z", "content": "thank you, I'm really looking forward to your code. thanks.
" }, { "id": 46071, "postDate": "2014-05-22T16:53:12.367Z", "content": "Hi, my approach was very much like mlandry's, and you folks can refer to his post for more details. Here I just briefly write down my thoughts why I picked this approach.
\nThis is a quite unique competition in the sense that there is already a benchmarking model (i.e. use the current plan as the prediction of your final plan) that produces a very high plan hit rate (more than 53%). I realized in the middle of my way that no matter how hard I try, it is virtually impossible to develop a alternative model that can produce a higher hit rate than the benchmarking model.
\nI then tried to find out which segments (or you can call it sub-portfolios) have the lowest hit rate -- for instance, if the benchmarking model hit rate for the segment of "risk_factor=4 and state=NY and g=1" is 15%, I could then build alternative model of which the hit rate might surpass the benchmarking 15% for that segment. I tried various models but mostly I only changed G, on a few occasions changing C and D or change C and D together.
\nI am a new Kaggler and joined this competition 3 weeks before. I ranked 52th on public board and 14th on private. On hindsight I guess it was the way that I chose a model prudently to avoid overfitting -- that I mostly would discard an alternative model and stick to the benchmarking model for a segment if that alternative model gives me less than 5 cases improvements in that segment on training dataset. By the way I used all the data for training and no validation set.
" }, { "id": 46054, "postDate": "2014-05-22T11:19:19.883Z", "content": "@K-czar: We made special features AF, BE, CD from previous history for learning model, which have as many classes as it is the number of combinations. Same for y fetaures, i.e. our y featrues were AF, BE, CD combinations of final purchase.
" }, { "id": 46025, "postDate": "2014-05-21T23:52:47.123Z", "content": "Wow, thank you all for sharing the wonderful ideas. I already see how many mistakes we made! We don't notice the difference of each option and try to predict them all as a whole. We did find state has an impact on purchase point but we can only improve very few points from the last shopping point. Anyway, we are complete novices and there are some approaches you mentioned we even never heard before. Wonderful experience. Thank kaggle and all state : )
" }, { "id": 46016, "postDate": "2014-05-21T20:17:08.090Z", "content": "@JWANG I am having a little trouble grasping your explanation, specifically how you accounted for the correlations. Would you consider sharing your code?
" }, { "id": 46007, "postDate": "2014-05-21T17:46:38.930Z", "content": "You're right, I was using independent models for each of the estimators and made the wrong assumption on what multiclass-multioutput was. So I am not sure whether it can handle that or not (and now know to try out scikit in some cases).
\nProper explanation (from scikit-learn):
\nMultioutput-multiclass classification and multi-task classification means that an estimators have to handle jointly several classification tasks. This is a generalization of the multi-label classification task, where the set of classification problem is restricted to binary classification, and of the multi-class classification task.
" }, { "id": 46004, "postDate": "2014-05-21T17:20:11.270Z", "content": "@JWANG multiclass yes, but multiclass + multioutput for Gradient boosting no, I'm pretty sure. We did try the multiclass + multiouput functionality for sklearn's RandomForrestClassifier though.
" }, { "id": 46001, "postDate": "2014-05-21T17:04:13.297Z", "content": "@Perroquet Thanks so much for sharing! That was really awesome to read b/c it validates a lot of the ideas we had intended to try with more time! I also wasn't aware GBM (in R) even had a multinomial metric, or the capability to predict in multiclass, multioutput problems. I don't believe it does in python. That was the reason we stuck to Random Forrests! But when we predicted variables in pairs - similarly based on the correlation matrix - we found that predicting the variables independently almost always drew higher accuracy.
\nSo far, I think you're also the only other group I've seen that incorporated a "consensus" approach like we did.
\nDid you have any memory issues using R's GBM with all of the locations as variables? Unless R's GBM implementation has been updated to serve sparse matrices, I would have guessed it would cause considerable memory problems.
\nThanks so much for sharing, and congratulations! Great work!
" }, { "id": 45992, "postDate": "2014-05-21T15:14:37.337Z", "content": "I barely made it to the top 10% but I learned a lot from my first entry.
\nThanks for sharing your solutions and insights. Here is my story...
\nAt first I was trying to train classifiers to predict the coverage options using the data from previous shopping points. I just barely got into the leaderboard with this method.
\nFor myself the breakthrough was when I trained a naive bayes classifier to learn the probability of each code using the last row for each customer (the purchase record). Using just the last row seemed to perform better than using all the rows. From there I was able to improve slightly by tweaking parameters. I tried SVM/etc on smaller data sets but it performed slightly worse.
\nLearned a lot as the data science field is new to me, next time I will do more research into patterns and probabilities / correlations as if I did that I might have noticed some of the patterns mentioned in other solutions.
" }, { "id": 45985, "postDate": "2014-05-21T12:39:12.977Z", "content": "@Perroquet
\nWow, that is some world class problem modelling. I need to step my game up!
\nThanks.
" }, { "id": 44994, "postDate": "2014-05-21T02:39:50.573Z", "content": "Tried both. Will clean up code and share.
\n\nEnded up going with the threshold model to get to around 200 on leaderboard.
\nWe overfit our model I think
" }, { "id": 44993, "postDate": "2014-05-21T02:03:02.133Z", "content": "@Perroquet Congratulations !
\nI am trying to learn from the various solutions.
\nCan you please share your soultion ?
\nThanks,
\nAmbarish
" }, { "id": 44981, "postDate": "2014-05-20T21:31:57.123Z", "content": "@Perroquet pretty girls in a nerdfest math competition? This 1st place winner needs to be investigated for witchcraft. Shenanigans!
\nCongrats :-)
" }, { "id": 44980, "postDate": "2014-05-20T21:29:39.967Z", "content": "@Joshua Weiner - Thanks for the input. Did you try a bayesian approach with the "same as benchmark or not" model (i.e. train a model to predict benchmark or not, then train models to predict "given not benchmark, predict choices"?) or did you only try the "threshold" approach?
" }, { "id": 44969, "postDate": "2014-05-20T17:41:26.770Z", "content": "@blaine : Thanks for sharing your approach. would appreciate if you could share your code as well. Thanks.
" }, { "id": 44938, "postDate": "2014-05-20T13:24:36.783Z", "content": "It was also a big surprise for us:-)
" }, { "id": 44928, "postDate": "2014-05-20T09:44:43.867Z", "content": "I wrote a program that handled 'formulas' like 0xx10x4-0011014-0011034 (every combination with A = 0, D = 1, E = 0, G = 4, without 0011014 and without 0011034). I tried to find combinations with a very high probability of changing. And in the testfile I changed those combinations with the 'next best' combination, the different combination that occured most.
\nI got some points about the benchmark, but not much. I decided too quick that I should not look at individual values.
" }, { "id": 44894, "postDate": "2014-05-20T04:08:12.970Z", "content": "Agree here too. Exactly what features did you use in your RF? My RF did not beat benchmark on its own.
" }, { "id": 44892, "postDate": "2014-05-20T04:03:15.247Z", "content": "[quote=Hiroyuki;44885]
\nMy strategy was quite simple. I used randomforest model with almost all given variables and 2nd lastquote and then I got (0.53556) on private leaderboad.
\nUsing firstquote, separating plans into the pair of "CD", "AEF" and "BG" and building a model for each shopping_pt (ie. 2,3,4...) didn't work well.
\nRandomForest worked better than gbm, nnet and svm on my prediction.
\nIt was difficult to do CV. I had no idea which data should I use for CV, the truncated train data set or no truncated data. On my prediction, no truncated data gave me better result but I couldn't understand why it went successful.
\nThank you for exciting competition, and see you next!
\n[/quote]
\n\n@Hiroyuki Can you please elaborate more? because I used exactly same procedure as you did but my score didnt beat the bench mark. variables that I used is given variables + previous quote(A through G). Thank you in advance!
" }, { "id": 44889, "postDate": "2014-05-20T03:20:00.277Z", "content": "@Hiroyuki: congratulations on beating the benchmark.
\ndo you mind sharing your code please. thanks in advance.
" }, { "id": 44874, "postDate": "2014-05-20T01:09:47.703Z", "content": "@Euclides - would you consider sharing your code? Since you are the only one who has expressed using many different ensembles and had success I would be very interested in seeing how you did it.
" }, { "id": 44867, "postDate": "2014-05-20T00:31:59.460Z", "content": "I just used a tree classifier for G with last choice, state, C_previous, Risk_factor and cost.
\nThe tree classifier was able to produce same score (CV) as if I used the Florida/NY trick.
\n\nI also noticed that changing the C_previous values from NA to the mode=3, gives some little improvements in my CV score.
" }, { "id": 44865, "postDate": "2014-05-20T00:25:39.037Z", "content": "I ended up predicting only G value, since it was by far the one that changed the most from the last quote.
\nUnfortunately, only today I realized that the real stuff was about understanding the patterns in each location ! Has anybody explored this extensively ?
" }, { "id": 44861, "postDate": "2014-05-20T00:15:30.827Z", "content": "BreakfastPirate, how did you manage to find Georgia/Florida tricks?
" }, { "id": 134984, "postDate": "2016-09-09T20:22:23.793Z", "content": "Thank you! Masters!
", "rawMarkdown": "Thank you! Masters!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:35:35Z", "competition_id": 3788, "competition_slug": "allstate-purchase-prediction-challenge", "team_id": 77617, "solution_url": "/competitions/allstate-purchase-prediction-challenge/writeups/alessandro-breakfastpirate-solution-sharing" }