All of our predictions are based on Triskelion's features, http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7688/feature-engineering-and-beat-the-benchmark-0-59347, and we stuck at 0.605 for quite a long time. I'm just wondering what's the limit if only Triskelion's features are used. What's the best score in that case? Maybe it's time for us to move on to look at the much larger data set. In general, I'm curious about how to find out whether it is the model or data set that is the bottleneck. Thank you very much and enjoy the world cup! We witnessed 10 goals today! Crazzzy!
", "messages": [ { "id": "49389", "postDate": "06/23/2014 01:51:17", "content": "All of our predictions are based on Triskelion's features, http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7688/feature-engineering-and-beat-the-benchmark-0-59347, and we stuck at 0.605 for quite a long time. I'm just wondering what's the limit if only Triskelion's features are used. What's the best score in that case? Maybe it's time for us to move on to look at the much larger data set. In general, I'm curious about how to find out whether it is the model or data set that is the bottleneck. Thank you very much and enjoy the world cup! We witnessed 10 goals today! Crazzzy!
", "rawMarkdown": "", "votes": null }, { "id": "49410", "postDate": "06/23/2014 13:21:10", "content": "I would suggest moving on.
Usually in my case, big leaderboard jumps occur just after implementing a new idea and any further optimizations to the new approach lead to only small additional improvements that saturate pretty quickly.
Achieving 0.605 using only Triskelion's features is amazing. what techniques have you used to achieve this. I am not able to move up from 0.583
\n\nThanks
", "rawMarkdown": "", "votes": null }, { "id": "49445", "postDate": "06/24/2014 01:27:42", "content": "[quote=Maverick;49430]
\nAchieving 0.605 using only Triskelion's features is amazing. what techniques have you used to achieve this. I am not able to move up from 0.583
\nThanks
\n[/quote]
\nI trained a dozen of logistic regression models and average their predictions. Each model tweaks Triskelion's features differently. I'll share details if this method also gets good scores in private leaderboard after contest closes. : )
", "rawMarkdown": "", "votes": null }, { "id": "49535", "postDate": "06/25/2014 06:07:33", "content": "I did roughly the same using the idea of Random Forest with Logistic Regression as the base learner. It ends up ~0.607 with a few features from my side. What is really interesting is that, with these same features, the original Random Forest or Gradient Boosting Machine can yield higher local validation score than LR based Random Forest, but really worse LB score (also, quite a large diff between the LB and local validation of those sophisticated model). I suspect that it is due to the diff between the training and testing data, and thus RF/GBM suffer from overfitting.
\n\nThat being said, I quite wonder are there guys have success with Random Forest or Gradient Boosting Machine? Or these models are not good matches for this comp due to the diff between the training and testing data?
", "rawMarkdown": "", "votes": null }, { "id": "49543", "postDate": "06/25/2014 08:24:46", "content": "Logit is giving better performance on LB because you might be avoiding overfitting using regularization. That is not the case with RF or GBM.
\n\nThere are many categories/brands in TestHistory which are not present in TrainHistory.
\n\nSo as @BreakfastPirate noted in a separate thread, the challenge is also to “Figure out how I will respond to a discount on ice cream – given your response to a discount on deodorant.”
\n\nIf you split your training and validation set on brands/categories, the CV scores of RF/GBM are far worse.
", "rawMarkdown": "", "votes": null }, { "id": "49581", "postDate": "06/25/2014 17:48:50", "content": "I have 0.60181 from a small subset of the Triskellion features. Even a very coarsely fitted gbm overfits. I had some success with glmboost, although it couldn't quite match my logistic regression approach. I think I need a new feature. I'm surprised that I can't get anything out of market or chain on the test set, although they do well on train.
", "rawMarkdown": "", "votes": null }, { "id": "49585", "postDate": "06/25/2014 20:08:50", "content": "[quote=yr;49535]
\nThat being said, I quite wonder are there guys have success with Random Forest or Gradient Boosting Machine? Or these models are not good matches for this comp due to the diff between the training and testing data?
\n[/quote]
\n\nI used Gradient Boosted Trees with very shallow trees as the base estimators (around 5 deep).
\nbut I like the random logistic regression idea, I think I'll try that too.
", "rawMarkdown": "", "votes": null }, { "id": "49615", "postDate": "06/26/2014 11:13:55", "content": "to Selfish Gene
\nCan you tell more details about how you applied the Gradient Boosted Trees. I have tested this method using sklearn with the default parameters and the Trischlikons's feature but it is overfitted and the score is around 0.56. Many Thanks.
", "rawMarkdown": "", "votes": null }, { "id": "49619", "postDate": "06/26/2014 12:57:39", "content": "well, first, try changing the default parameters to smaller trees.
\nyou can control that by 'max_depth' or 'min_samples_leaf'. also, I would suggest changing 'subsample' to around 0.7. another parameter you should try and keep small relative to the amount of features used is 'max_features'.
secondly, what I used for final prediction is probably far from optimal since I don't invest much of my time in that direction. so in that regard, the 'random logistic regression' approach described above sounds a little more promising.
mind you, I didn't try GBT with Triskelion's features, but rather I've tried a large amount of other feature sets I've concocted. but for all feature sets I did try, I was forced to use very simple learners to achieve reasonable results.
Also, on a more general note, in my humble and inexperienced opinion, the final prediction phase is not very important at almost any learning problem. almost all 'standard' learners (SVM, shallow NNs, Decision Trees) have practically identical hypothesis spaces, and while the training process is somewhat different for each one, I think that until recently (until the emergence of deep learning), the training algorithms for them were relatively poor at their tasks of choosing the 'best hypothesis' from that space ("best" in statistical generalization terms, not in optimization loss terms).
Also, from my limited practical experience, when hyper parameters are optimized for each of the standard learners separately, the performances they achieve are approximately the same. this is why optimizing the final learner is usually one of the last things I do, and most of my time is dedicated to trying to view the data in a different way, and extract useful features accordingly.
I would suggest moving on.
Usually in my case, big leaderboard jumps occur just after implementing a new idea and any further optimizations to the new approach lead to only small additional improvements that saturate pretty quickly.
Achieving 0.605 using only Triskelion's features is amazing. what techniques have you used to achieve this. I am not able to move up from 0.583
\n\nThanks
", "votes": null, "replies": [] }, { "id": 49445, "author_name": "jiweiliu", "author_url": "", "post_date": "06/24/2014 01:27:42", "content": "[quote=Maverick;49430]
\nAchieving 0.605 using only Triskelion's features is amazing. what techniques have you used to achieve this. I am not able to move up from 0.583
\nThanks
\n[/quote]
\nI trained a dozen of logistic regression models and average their predictions. Each model tweaks Triskelion's features differently. I'll share details if this method also gets good scores in private leaderboard after contest closes. : )
", "votes": null, "replies": [] }, { "id": 49535, "author_name": "chenglongchen", "author_url": "", "post_date": "06/25/2014 06:07:33", "content": "I did roughly the same using the idea of Random Forest with Logistic Regression as the base learner. It ends up ~0.607 with a few features from my side. What is really interesting is that, with these same features, the original Random Forest or Gradient Boosting Machine can yield higher local validation score than LR based Random Forest, but really worse LB score (also, quite a large diff between the LB and local validation of those sophisticated model). I suspect that it is due to the diff between the training and testing data, and thus RF/GBM suffer from overfitting.
\n\nThat being said, I quite wonder are there guys have success with Random Forest or Gradient Boosting Machine? Or these models are not good matches for this comp due to the diff between the training and testing data?
", "votes": null, "replies": [] }, { "id": 49543, "author_name": "hrishi", "author_url": "", "post_date": "06/25/2014 08:24:46", "content": "Logit is giving better performance on LB because you might be avoiding overfitting using regularization. That is not the case with RF or GBM.
\n\nThere are many categories/brands in TestHistory which are not present in TrainHistory.
\n\nSo as @BreakfastPirate noted in a separate thread, the challenge is also to “Figure out how I will respond to a discount on ice cream – given your response to a discount on deodorant.”
\n\nIf you split your training and validation set on brands/categories, the CV scores of RF/GBM are far worse.
", "votes": null, "replies": [] }, { "id": 49581, "author_name": "jfkingiii", "author_url": "", "post_date": "06/25/2014 17:48:50", "content": "I have 0.60181 from a small subset of the Triskellion features. Even a very coarsely fitted gbm overfits. I had some success with glmboost, although it couldn't quite match my logistic regression approach. I think I need a new feature. I'm surprised that I can't get anything out of market or chain on the test set, although they do well on train.
", "votes": null, "replies": [] }, { "id": 49585, "author_name": "selfishgene", "author_url": "", "post_date": "06/25/2014 20:08:50", "content": "[quote=yr;49535]
\nThat being said, I quite wonder are there guys have success with Random Forest or Gradient Boosting Machine? Or these models are not good matches for this comp due to the diff between the training and testing data?
\n[/quote]
\n\nI used Gradient Boosted Trees with very shallow trees as the base estimators (around 5 deep).
\nbut I like the random logistic regression idea, I think I'll try that too.
", "votes": null, "replies": [] }, { "id": 49615, "author_name": "qiujiqiong", "author_url": "", "post_date": "06/26/2014 11:13:55", "content": "to Selfish Gene
\nCan you tell more details about how you applied the Gradient Boosted Trees. I have tested this method using sklearn with the default parameters and the Trischlikons's feature but it is overfitted and the score is around 0.56. Many Thanks.
", "votes": null, "replies": [] }, { "id": 49619, "author_name": "selfishgene", "author_url": "", "post_date": "06/26/2014 12:57:39", "content": "well, first, try changing the default parameters to smaller trees.
\nyou can control that by 'max_depth' or 'min_samples_leaf'. also, I would suggest changing 'subsample' to around 0.7. another parameter you should try and keep small relative to the amount of features used is 'max_features'.
secondly, what I used for final prediction is probably far from optimal since I don't invest much of my time in that direction. so in that regard, the 'random logistic regression' approach described above sounds a little more promising.
mind you, I didn't try GBT with Triskelion's features, but rather I've tried a large amount of other feature sets I've concocted. but for all feature sets I did try, I was forced to use very simple learners to achieve reasonable results.
Also, on a more general note, in my humble and inexperienced opinion, the final prediction phase is not very important at almost any learning problem. almost all 'standard' learners (SVM, shallow NNs, Decision Trees) have practically identical hypothesis spaces, and while the training process is somewhat different for each one, I think that until recently (until the emergence of deep learning), the training algorithms for them were relatively poor at their tasks of choosing the 'best hypothesis' from that space ("best" in statistical generalization terms, not in optimization loss terms).
Also, from my limited practical experience, when hyper parameters are optimized for each of the standard learners separately, the performances they achieve are approximately the same. this is why optimizing the final learner is usually one of the last things I do, and most of my time is dedicated to trying to view the data in a different way, and extract useful features accordingly.