\r\n
I am using L1 Logistic Regression. The best single model is 0.38
\r\nBest KNN model is 0.40
\r\nBest SVM model is 0.41
\r\nWhat is your method?
", "comments": [ { "id": 14044, "author_name": "Marat", "author_url": "", "post_date": "2012-09-07T07:12:32.333000", "content": "@MaBu it was multiclass classification, but as answer I used "decision_functions" function. It gives real values for each class. Usually the class with\r\n the biggest value is selected, but here I needed to get propabilities.
\r\n@MaBu Linear SVC took about 4 minutes on "Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz" and 10 minutes on core2duo.
\r\n@MaBu can describe how you handled logistic regression?
\r\nIf anyone is interested I can share my code.
\r\n", "votes": 1, "replies": [] }, { "id": 13792, "author_name": "MaBu", "author_url": "", "post_date": "2012-09-02T09:03:25.223000", "content": "
I used Naive bayes with TF-IDF got 0.564.
\r\nThen used Logistic regression also with TF-IDF and got 0.44483.
\r\nAfter that I removed 130 empty documents from the training set and got 0.38181.
\r\nLast thing I tried was log-entropy model with Logistic regression. I got 0.293 and this was my final model. I tried to remove some features but calculating the model was too long and I couldn't submit it. Maybe I would get 0.001 improvement :)
\r\nI always used 75% training set and 25% test set. To choose parameters I used cross validation on the training set.
\r\nIt was great that results were very similar on my test set and public/private sets.
\r\nI also tried LDA and LSI with Logistic regression adn with cosine distances but it brought no improvement.
\r\nI used python scikit, also tried R to learn it, but dataset was too big for data.frame and I didn't know what to do.
\r\nI also tried removing features with CHI square, tried PCA (didn't finished), SVD (didn't improve the score), Non negative matrix factorization (Out of memory). :)
\r\nIn the end I used all features. Model runs for 12 minutes.
\r\nWhich SVM implementation did you use?
\r\nI used libsvm and it took ages, I didn't waited for it to finish. It is a lot of classes and features and samples.
\r\nStochastic Gradient Descent was wast and also good but it didn't support prediction probabilities.
", "votes": 1, "replies": [] }, { "id": 13829, "author_name": "Marat", "author_url": "", "post_date": "2012-09-03T10:41:46.397000", "content": ">Can you tell me more about the process scaled Linear SVC's answer to [0,1] and took to the power of 11 and renormalized?
\r\n>Is it:
\r\n>init [0.5, 0.3, 0.2] -> [0.5^11, 0.3^11, 0.2^11] -> make it sum to 1?
\r\n>Why 11 works?
\r\nYes, actually, I missed one thing. It was
\r\ninit [0.5, 0.3, 0.2] -> [0.5^11 + 0.000026, 0.3^11 + 0.000026, 0.2^11 + 0.000026] -> make it sum to 1?
\r\nBoth "11" and "0.000026" were empirically choosen.
\r\nI had to make such a hack because Linear SVC is linear, so I had to make margin between classes much bigger. Rising to a big power made 1s stay 1s while other probabilities approach 0.
\r\n\r\n
Thank you all for sharing your ideas! It is the most valuable part of kaggle.com =)
", "votes": 2, "replies": [] }, { "id": 14704, "author_name": "Marat", "author_url": "", "post_date": "2012-09-22T13:02:17.793000", "content": "so here is my final evaluation code
\r\nhttps://dl.dropbox.com/u/9359214/FinalEvaluation.py
", "votes": 0, "replies": [] }, { "id": 14123, "author_name": "MaBu", "author_url": "", "post_date": "2012-09-09T09:23:17.583000", "content": "@Marat Great thanks. I understand it now. I'll have to look why in my case SVC took so long. Because I have i5-3570K. I assume you used ovr for multiclass classification.
\r\nFor logistic regression I just used LogisticRegression(penalty='l1', C=6) from scikit C was choosen with cross validation. And used predict_proba as answers instead of predictions. Penalty was left at l1 because l2 gave worse results and I read somewhere\r\n that when you have a lot of features l1 is usually better.
", "votes": 0, "replies": [] }, { "id": 14097, "author_name": "Bing Xu", "author_url": "", "post_date": "2012-09-08T06:44:04.597000", "content": "@LI, Wei , For KNN is implemented by my teammate, I can't tell more without his permission. I am sorry. I will send him an email and ask for his opinion\r\n :)
\r\n", "votes": 0, "replies": [] }, { "id": 14095, "author_name": "LI Wei", "author_url": "", "post_date": "2012-09-08T06:13:54.037000", "content": "
@Marat It would be quite nice of you to share your code!
", "votes": 0, "replies": [] }, { "id": 13952, "author_name": "MaBu", "author_url": "", "post_date": "2012-09-05T15:56:10.633000", "content": "\r\nI tried to reduce dimensionality with many methods PCA, SVD, LSI, LDA, Random projections but they didn't work or results weren't imporved.
\r\nSo at the end all models were with all features.
\r\nI used python and scikit's \r\nLogistic Regression which uses one vs all and uses liblinear internally.
\r\nFor random projections you can use provided R sample or \r\nGensim if you use Python. Normally dimensionality reduction is applied to speed up learning and to have more compact representation of features.\r\n\r\nHere are some videos describing it.
\r\n@Marat Those numbers between 0 and 1 in Linear SVC are scaled classes from 0-96 to 0-1 or were you doing regressions\r\n with SVC and then scaled from 0-96 to 0-1.
\r\nAnybody who worked with SVC: How long the learning took and on what machine? Was any dimensionality reduction tehnique used?
\r\nI would like to know If I would get some results If I would be more patient. I just read everywhere that SVC were great for document classifications, but were too slow for me.
", "votes": 0, "replies": [] }, { "id": 13865, "author_name": "LI Wei", "author_url": "", "post_date": "2012-09-04T09:07:54.060000", "content": "@MaBu I have two questions,
\r\nHave you done a dimensionality reduction before applying those models?
\r\nFor logistic regression , you use one-vs-all methods or the multinomial-loss logistic regression to do the model?
\r\n\r\n
As the dataset is too big, I have tried many methods to do the dimensionality reduction, but seems to fail all the time. Can you give me rough description of the dimensionality reduction procedure? I have tried randomized pca without whitening and centering,\r\n but with very ugly answer.
\r\n\r\n
@binghsu, how do you do the KNN and get a 0.4? do you do a dimensionality reduction before applying KNN or other supervised dim reduction?
", "votes": 0, "replies": [] }, { "id": 13825, "author_name": "yuenking", "author_url": "", "post_date": "2012-09-03T07:00:32.840000", "content": "I've used TF-IDF to remove half the number of features; then logistic regression on the rest using only TF. The logistic regression with binomial distribution was ran for each class separately. It took VERY long though; definitely more than 12 minutes on\r\n R... (loading the data itself took more than 12mins!)
\r\nDoes multinomial work any better? It didn't show any progress for half a day and I closed it.
\r\nTried SVD, random projection; but SVD of 200 ran for a day and it was still incomplete; same for random projection of 300.
\r\nI calibrated my results by repeatedly normalizing and pasting the most common classes over. Then I raised those class probabilities of >0.4 by 0.1. Due to the slow speed of my model, I didn't bother building a cross-validation set; I just calibrated my results\r\n to the public leaderboard.
", "votes": 0, "replies": [] }, { "id": 13824, "author_name": "Bing Xu", "author_url": "", "post_date": "2012-09-03T06:42:25.047000", "content": "linear binding is enough. For most cases, half/half works very well.
\r\nAlso I tried Kalman Filter, it worked well when I combined a random forest result with a bad SVM result. Unfortunatily, it didn't work on later models.
\r\nCan you tell me more about the process scaled Linear SVC's answer to [0,1] and took to the power of 11 and renormalized?
\r\n\r\n
Is it:
\r\ninit [0.5, 0.3, 0.2] -> [11^0.5, 11^0.3, 11^0.2] -> make it sum to 1?
\r\nWhy 11 works?
\r\n\r\n
Thanks!
", "votes": 0, "replies": [] }, { "id": 13811, "author_name": "Marat", "author_url": "", "post_date": "2012-09-02T20:17:58.343000", "content": "My single model (Linear SVC C=0.0045 at scikit learn) gave me 0.8 with TF transformation
\r\nThen I scaled Linear SVC's answer to [0,1] and took to the power of 11 and renormalized. It gave me 0.38.\r\n
\r\nThen I counted error for every class and set weights for every class. So my final solution is 0.37.
\r\nPlease, tell us more about blending technics. I also tried sevel different models, but I couldn't blend them into one solution.\r\n
", "votes": 0, "replies": [] }, { "id": 13805, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-09-02T18:01:41.817000", "content": "interesting
\r\nI got pretty bad scores with TF-IDF. Good scores with TF.
\r\nI did termFreq * log (number of docs/doc freq)
\r\nWonder how others did it. Given that matrix was sparse, I did TF-IDF only for non-sparse elements.
\r\n\r\n
TF gave better score than TF-IDF
", "votes": 0, "replies": [] }, { "id": 13793, "author_name": "Bing Xu", "author_url": "", "post_date": "2012-09-02T09:53:46.517000", "content": "Cool.
\r\nI have done PCA and SVD.
\r\nFor the model TF, SVD imporved SVM to 0.79/ PCA 1.56/ Init 1.08
\r\nFor SVM, I use libSVM.
", "votes": 0, "replies": [] }, { "id": 13789, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-09-02T06:17:56.473000", "content": "L1 Logistic regression gave me 0.44 best score
\r\nSVM gave 0.58
\r\n\r\n
I tried to blend - but the blend did not help in taking closer than 0.42. blending gave little to no improvement in this case.
\r\nWhat other strategies were employed?
", "votes": 0, "replies": [] } ], "comment_count": 15, "messages": [ { "id": 14044, "postDate": "2012-09-07T07:12:32.333Z", "content": "@MaBu it was multiclass classification, but as answer I used "decision_functions" function. It gives real values for each class. Usually the class with\r\n the biggest value is selected, but here I needed to get propabilities.
\r\n@MaBu Linear SVC took about 4 minutes on "Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz" and 10 minutes on core2duo.
\r\n@MaBu can describe how you handled logistic regression?
\r\nIf anyone is interested I can share my code.
\r\n", "votes": 1 }, { "id": 13792, "postDate": "2012-09-02T09:03:25.223Z", "content": "
I used Naive bayes with TF-IDF got 0.564.
\r\nThen used Logistic regression also with TF-IDF and got 0.44483.
\r\nAfter that I removed 130 empty documents from the training set and got 0.38181.
\r\nLast thing I tried was log-entropy model with Logistic regression. I got 0.293 and this was my final model. I tried to remove some features but calculating the model was too long and I couldn't submit it. Maybe I would get 0.001 improvement :)
\r\nI always used 75% training set and 25% test set. To choose parameters I used cross validation on the training set.
\r\nIt was great that results were very similar on my test set and public/private sets.
\r\nI also tried LDA and LSI with Logistic regression adn with cosine distances but it brought no improvement.
\r\nI used python scikit, also tried R to learn it, but dataset was too big for data.frame and I didn't know what to do.
\r\nI also tried removing features with CHI square, tried PCA (didn't finished), SVD (didn't improve the score), Non negative matrix factorization (Out of memory). :)
\r\nIn the end I used all features. Model runs for 12 minutes.
\r\nWhich SVM implementation did you use?
\r\nI used libsvm and it took ages, I didn't waited for it to finish. It is a lot of classes and features and samples.
\r\nStochastic Gradient Descent was wast and also good but it didn't support prediction probabilities.
", "votes": 1 }, { "id": 13829, "postDate": "2012-09-03T10:41:46.397Z", "content": ">Can you tell me more about the process scaled Linear SVC's answer to [0,1] and took to the power of 11 and renormalized?
\r\n>Is it:
\r\n>init [0.5, 0.3, 0.2] -> [0.5^11, 0.3^11, 0.2^11] -> make it sum to 1?
\r\n>Why 11 works?
\r\nYes, actually, I missed one thing. It was
\r\ninit [0.5, 0.3, 0.2] -> [0.5^11 + 0.000026, 0.3^11 + 0.000026, 0.2^11 + 0.000026] -> make it sum to 1?
\r\nBoth "11" and "0.000026" were empirically choosen.
\r\nI had to make such a hack because Linear SVC is linear, so I had to make margin between classes much bigger. Rising to a big power made 1s stay 1s while other probabilities approach 0.
\r\n\r\n
Thank you all for sharing your ideas! It is the most valuable part of kaggle.com =)
", "votes": 2 }, { "id": 14704, "postDate": "2012-09-22T13:02:17.793Z", "content": "so here is my final evaluation code
\r\nhttps://dl.dropbox.com/u/9359214/FinalEvaluation.py
" }, { "id": 14123, "postDate": "2012-09-09T09:23:17.583Z", "content": "@Marat Great thanks. I understand it now. I'll have to look why in my case SVC took so long. Because I have i5-3570K. I assume you used ovr for multiclass classification.
\r\nFor logistic regression I just used LogisticRegression(penalty='l1', C=6) from scikit C was choosen with cross validation. And used predict_proba as answers instead of predictions. Penalty was left at l1 because l2 gave worse results and I read somewhere\r\n that when you have a lot of features l1 is usually better.
" }, { "id": 14097, "postDate": "2012-09-08T06:44:04.597Z", "content": "@LI, Wei , For KNN is implemented by my teammate, I can't tell more without his permission. I am sorry. I will send him an email and ask for his opinion\r\n :)
\r\n" }, { "id": 14095, "postDate": "2012-09-08T06:13:54.037Z", "content": "
@Marat It would be quite nice of you to share your code!
" }, { "id": 13952, "postDate": "2012-09-05T15:56:10.633Z", "content": "\r\nI tried to reduce dimensionality with many methods PCA, SVD, LSI, LDA, Random projections but they didn't work or results weren't imporved.
\r\nSo at the end all models were with all features.
\r\nI used python and scikit's \r\nLogistic Regression which uses one vs all and uses liblinear internally.
\r\nFor random projections you can use provided R sample or \r\nGensim if you use Python. Normally dimensionality reduction is applied to speed up learning and to have more compact representation of features.\r\n\r\nHere are some videos describing it.
\r\n@Marat Those numbers between 0 and 1 in Linear SVC are scaled classes from 0-96 to 0-1 or were you doing regressions\r\n with SVC and then scaled from 0-96 to 0-1.
\r\nAnybody who worked with SVC: How long the learning took and on what machine? Was any dimensionality reduction tehnique used?
\r\nI would like to know If I would get some results If I would be more patient. I just read everywhere that SVC were great for document classifications, but were too slow for me.
" }, { "id": 13865, "postDate": "2012-09-04T09:07:54.060Z", "content": "@MaBu I have two questions,
\r\nHave you done a dimensionality reduction before applying those models?
\r\nFor logistic regression , you use one-vs-all methods or the multinomial-loss logistic regression to do the model?
\r\n\r\n
As the dataset is too big, I have tried many methods to do the dimensionality reduction, but seems to fail all the time. Can you give me rough description of the dimensionality reduction procedure? I have tried randomized pca without whitening and centering,\r\n but with very ugly answer.
\r\n\r\n
@binghsu, how do you do the KNN and get a 0.4? do you do a dimensionality reduction before applying KNN or other supervised dim reduction?
" }, { "id": 13825, "postDate": "2012-09-03T07:00:32.840Z", "content": "I've used TF-IDF to remove half the number of features; then logistic regression on the rest using only TF. The logistic regression with binomial distribution was ran for each class separately. It took VERY long though; definitely more than 12 minutes on\r\n R... (loading the data itself took more than 12mins!)
\r\nDoes multinomial work any better? It didn't show any progress for half a day and I closed it.
\r\nTried SVD, random projection; but SVD of 200 ran for a day and it was still incomplete; same for random projection of 300.
\r\nI calibrated my results by repeatedly normalizing and pasting the most common classes over. Then I raised those class probabilities of >0.4 by 0.1. Due to the slow speed of my model, I didn't bother building a cross-validation set; I just calibrated my results\r\n to the public leaderboard.
" }, { "id": 13824, "postDate": "2012-09-03T06:42:25.047Z", "content": "linear binding is enough. For most cases, half/half works very well.
\r\nAlso I tried Kalman Filter, it worked well when I combined a random forest result with a bad SVM result. Unfortunatily, it didn't work on later models.
\r\nCan you tell me more about the process scaled Linear SVC's answer to [0,1] and took to the power of 11 and renormalized?
\r\n\r\n
Is it:
\r\ninit [0.5, 0.3, 0.2] -> [11^0.5, 11^0.3, 11^0.2] -> make it sum to 1?
\r\nWhy 11 works?
\r\n\r\n
Thanks!
" }, { "id": 13811, "postDate": "2012-09-02T20:17:58.343Z", "content": "My single model (Linear SVC C=0.0045 at scikit learn) gave me 0.8 with TF transformation
\r\nThen I scaled Linear SVC's answer to [0,1] and took to the power of 11 and renormalized. It gave me 0.38.\r\n
\r\nThen I counted error for every class and set weights for every class. So my final solution is 0.37.
\r\nPlease, tell us more about blending technics. I also tried sevel different models, but I couldn't blend them into one solution.\r\n
" }, { "id": 13805, "postDate": "2012-09-02T18:01:41.817Z", "content": "interesting
\r\nI got pretty bad scores with TF-IDF. Good scores with TF.
\r\nI did termFreq * log (number of docs/doc freq)
\r\nWonder how others did it. Given that matrix was sparse, I did TF-IDF only for non-sparse elements.
\r\n\r\n
TF gave better score than TF-IDF
" }, { "id": 13793, "postDate": "2012-09-02T09:53:46.517Z", "content": "Cool.
\r\nI have done PCA and SVD.
\r\nFor the model TF, SVD imporved SVM to 0.79/ PCA 1.56/ Init 1.08
\r\nFor SVM, I use libSVM.
" }, { "id": 13789, "postDate": "2012-09-02T06:17:56.473Z", "content": "L1 Logistic regression gave me 0.44 best score
\r\nSVM gave 0.58
\r\n\r\n
I tried to blend - but the blend did not help in taking closer than 0.42. blending gave little to no improvement in this case.
\r\nWhat other strategies were employed?
" }, { "id": 13787, "postDate": "2012-09-02T04:03:29.397Z", "content": "\r\n
I am using L1 Logistic Regression. The best single model is 0.38
\r\nBest KNN model is 0.40
\r\nBest SVM model is 0.41
\r\nWhat is your method?
" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T13:05:25Z", "competition_id": 2958, "competition_slug": "emc-data-science", "team_id": 19269, "solution_url": "/competitions/emc-data-science/writeups/bing-xu-sashi-drago-what-is-your-methods" }