I would be very interested to see what other people used (languages, packages).
\r\nI published my code and wrote about it in my blog.
\r\nBasically I used mainly character n-grams and logistic regression from scikit-learn.
Chris Brew (came in fourth) also commented there. He used a similar approach and also published his code.
\r\n
\r\n
Vivek Sharma, the winner, also used scikit-learn as he said on the scikit-learn mailing list.
\r\n(I don't want to say anything about his approach as I am not sure what he wants to disclose).
I would be really interested if the other leaders also used scikit-learn or maybe R or weka or anything else?
\r\n\r\n
Cheers,
\r\nAndy
", "comments": [ { "id": 15951, "author_name": "Vivek Sharma", "author_url": "", "post_date": "2012-10-19T13:14:00.443000", "content": "Its somewhat late I realize, but I thought I'd post the code for my winning submission. Its a short python script... and is attached with this post.
\r\nThe badwords file was taken from here: (http://urbanoalvarez.es/blog/2008/04/04/bad-words-list/) and modified manually.
", "votes": 17, "replies": [ { "id": 168901, "author_name": "salvu65", "author_url": "", "post_date": "2017-03-18T15:34:41.060000", "content": "Dear Vivek,
\n\nEdit:\nI managed to find the bug that was forcing the predictions to be almost the same. The script was using the Date column as test data. My fault for not checking which columns where being used.
\n\nOriginal post:\nI have tried to run your winning script using Python 2.7 in Anaconda 4.3 (64-bit) as I would like to see how it works and also use it with some data that I have collected but I am running into some problems as the preds.csv file with the results shows the same value on every line.
I have used the same files that you used to train (train.csv and test_with_solutions.csv), the badwords file and as a test file I have tried both impermium_verification_labels.csv and impermium_verification_set.csv but had the same results, i.e. 2236 values all exactly 0.0700027496. I have tried different values for binary, stopwords and analyzer_char in the ngrams function but the results are mostly the same. So I am definitely doing something wrong.
I have had to change the max_n=mx and the min_n=mn in the ngrams function specifically in the CounterVectorizer and TfidfVectorizer to ngram_range=(mx, mn) as the error TypeError: __init__() got an unexpected keyword argument 'max_n' was being reported so I do not know whether this is influencing all the skewed results.
When running the script in debug mode (I'm using PyCharm community edition) and closely watch the runClassifiers function, I notice that the variable pred is an array of tuples where all tuples have the same values and when these values are appended in the preds list they are all the same with the result that the final_pred ndarray is an array of 2235 same values.
Any help would be greatly appreciated. Thank you.
", "votes": 1, "replies": [] } ] }, { "id": 14755, "author_name": "Vivek Sharma", "author_url": "", "post_date": "2012-09-24T18:26:47.183000", "content": "My feature set was almost the same as the char and word features that Andreas used. SVC gave me better performance than regularized LR. And, some normalizations (like tuzzeg mentioned), along with using a bad words list (http://urbanoalvarez.es/blog/2008/04/04/bad-words-list/)\r\n helped quite a bit. Those were probably the only differences between Andreas' score and mine. The single SVC model would have won by itself, although the winning submission combined SVC with RF which improved the score marginally over just SVC. Regularized\r\n LR and GBRT were also tried, but they did not change the score much. I did not use the datetime field.\r\n
\r\nTuzzeg, I experimented a little bit with phrase features, and I'm pretty sure they would be needed in any implementation of such a system. A lot of the insults were of the form: "you are/you're a/an xxxx", "xxxx like you", "you xxxx". I tried to look for\r\n a large +ve/-ve word list to determine sentiment of such phrases with unseen words, but I couldn't find a good word list that was freely available for commercial use. Does anyone know of one? Ultimately, I didn't use any such features except for a very simplified\r\n one based on "you are/you're xxx" which did help the score, although, only to a small extent.\r\n
", "votes": 12, "replies": [] }, { "id": 14751, "author_name": "tuzzeg", "author_url": "", "post_date": "2012-09-24T16:11:49.857000", "content": "I used scikit-learn as well, with Stanford POS tagger and Stanford parser. My approach in general was ensemble of LogisitcRegression classifiers over words, stemmed words, POS tags, char ngrams, words/stems 2,3-grams, word/stem subsequences, language models\r\n over words/stems/tags and a bunch of features over dependency parsing results (110 basic classifiers in final solution). All of them were stacked using ExtraTreesRegressor.
\r\nI didn't use word correction - which could help to detect such phrases like 'r u'=='are you' or 'f#%k'.
\r\nI am curious if somebody used more linguistic features, like in \r\nSmokey or like described here.
", "votes": 10, "replies": [] }, { "id": 15857, "author_name": "Andrei Olariu", "author_url": "", "post_date": "2012-10-17T22:39:57.117000", "content": "Congrats to the winners!
\r\nI also added a description of my approach (in short: SVMs, neural networks and some good tokenizing). I wanted to share this earlier, but I just couldn't find the time.
\r\nhttp://webmining.olariu.org/my-first-kaggle-competition-and-how-i-ranked
", "votes": 5, "replies": [] }, { "id": 15068, "author_name": "tuzzeg", "author_url": "", "post_date": "2012-10-02T17:32:13.177000", "content": "Just published \r\nmy code and \r\nbrief explanation of my approach. Questions, suggestions and any thoughts are welcome!
", "votes": 5, "replies": [] }, { "id": 14834, "author_name": "joshnk", "author_url": "", "post_date": "2012-09-26T13:46:32.023000", "content": "I used character n-grams, tfidf with sublinear_tf and SGDRegressor with early stopping. I am somewhat proud of the early stopping code.
\r\nMy reason for using a regression estimator was that the evaluation was going to be AUC, which is sensitive only to the order of the scores, not the finer details. Had I used a classifier, I would have needed to do something with predict proba to arrange\r\n the items in a good order anyway. SGD is also nice because it works well with sparse inputs lets you explore things like the use of the elastic net penalty while sticking with the same classifier.
\r\nAs I said in my comment on Andreas Mueller's blog, the final order has an element of luck to it, because the final test set was so small and the labeling was rather noisy
\r\nCode is at:
\r\nhttps://github.com/cbrew/Insults/blob/master/Insults/insults.py
", "votes": 3, "replies": [] }, { "id": 14801, "author_name": "Michael Gerasimenko", "author_url": "", "post_date": "2012-09-25T18:44:27.313000", "content": "I have used R and tm package for word stemming, normalisation and creating word-features for documents. I am tryed binary, tf and tfIdf weightings, but binary was best. Also i used as features 1000 letter 3-grams with best information gain and small number\r\n of features like proportion of uppercase letters, punctuations, etc. About 6k features total.
\r\nFor classification I have used gradient boosted trees from R gbm package.
\r\nI realy impressed that people widely use scikit-learn for this competion, i will try to use it next time!
Can somebody explain me, why stochastic gradient descent is better than gradient boosting for this kind of tasks, with a lot of features.
", "votes": 1, "replies": [] }, { "id": 14747, "author_name": "Hrishikesh Huilgolkar", "author_url": "", "post_date": "2012-09-24T15:40:24.463000", "content": "I used stanford classifier
\r\nhttp://nlp.stanford.edu/software/classifier.shtml
It is basically same approach you used. Logistic classifier without regularization using n-gram technique. I made only a few changes to the defaults.
\r\nI finished 31st.
Michael: I don't think that the statement is "SGD is better than gradient boosting" but "linear is better than trees". Not sure in how far that holds. But most people only use linear classifiers with text and it works very well.
\r\nI'm totally new to text, so I can't really judge.
\r\nI think you could argue that there are many important words (I think a lot more than the 1000 features you used - I think my word list was longer!) and that in such a very high-dimensional space with very strong univariate correlations, a linear model should\r\n to the trick.
", "votes": 2, "replies": [] }, { "id": 14829, "author_name": "Steve Poulson", "author_url": "", "post_date": "2012-09-26T10:37:58.967000", "content": "I've added a blog on my approach:
\r\nhttp://steve-p0ulson.blogspot.co.uk/2012/09/recently-i-entered-kaggle-detecting.html
\r\nThe interesting thing from the leading entries was how similar the approaches ended up.
", "votes": 2, "replies": [] }, { "id": 167830, "author_name": "salvu65", "author_url": "", "post_date": "2017-03-15T15:53:18.757000", "content": "I know it has been a long time and most probably this thread is as dead as the dodo but I would like to use some of the code provided by the entrants for a part of my research. To whom shall I attribute the citation please? Thank you.
", "votes": 0, "replies": [] }, { "id": 137052, "author_name": "g7", "author_url": "", "post_date": "2016-09-27T19:34:52.940000", "content": "@tuzzeg how fast does it process information? For example: if its a forum posting, how many posts can it process within 1 minute if trained the way you did it?
", "votes": 0, "replies": [] }, { "id": 15858, "author_name": "Glider", "author_url": "", "post_date": "2012-10-17T22:48:44.777000", "content": "HI Andrei - link appears to be broken
", "votes": 0, "replies": [] }, { "id": 15060, "author_name": "Yasser Tabandeh", "author_url": "", "post_date": "2012-10-02T15:01:57.157000", "content": "Congratulations to winners. I used R for this competition.
\r\n- RWeka package was used for word tokenizing. Iterated Lovins was used as stemmer.
\r\n- Other features which I used:
\r\n-Number of words in the comment
\r\n-Number of bad words (I used a bad word list containing 50 words)
\r\n-Number of phrases such as: "you are a", "you sound like"
\r\n-Type of date (this feature was useless in the verification set)
\r\n- Fisher Score was used as feature selector. 750 features were used for modeling.
\r\n- Both decision trees and functional algorithms were used for modeling. Final model was a combination of SVM, Neural Net, random Forest, GBM, SGD, and GLMNet.
", "votes": 0, "replies": [] }, { "id": 15049, "author_name": "Tony_R", "author_url": "", "post_date": "2012-10-02T10:40:50.267000", "content": "I used SQL Server Express to prep the dataset and build variables (time of day, length of comment, comment groupings: foul, meanspirited, pointed, family reference, etc.), then used R for the models. My best performing model prior to the final validation\r\n phase was using the earth package for Multivariate Adaptive Regression Splines (0.872 public, 0.875 private. Ultimately, the best model (0.757) turned out to be an ensemble of MARS, Neural Nets, Random Forest and GBM.
\r\nI actually like the simplicity of logistic regression and found it quite competitive with some of the other models.
\r\nAnyway, fun competition!
", "votes": 0, "replies": [] }, { "id": 14920, "author_name": "Willie Liao", "author_url": "", "post_date": "2012-09-28T13:13:11.137000", "content": "I used Python and the NLTK package for preprocessing and feature selection. Features included tfidf, bad words list, smileys list, regex for insult phrases, and time categories. I did not try character n-grams like most people here did. It would have improved\r\n my score by 0.00053 if I had added a simple character 2-gram prediction to my blend.
\r\nI was short on time so did not have the chance to play with scikit-learn like I had originally intended. Instead, I went back to R for modelling. It's comfort food for statisticians. I concentrated on algorithms that did their own feature selection: regularized\r\n logistic regression, gradient boosted regression trees, and random forests.
\r\nDid anyone try incorporating similarity scores or other clustering techniques? I found cosine similarity on tfidf to be quite useful in the blend.
", "votes": 0, "replies": [] }, { "id": 14917, "author_name": "DataGeek", "author_url": "", "post_date": "2012-09-28T11:28:07.787000", "content": "@Andreas
\r\nYou seems to be saying right, I used LiblineaR library to make models, but in the final version I removed stemming from my approach. 1 model I submitted using textir package in R, in which I used stemming also that only gave my best result. I didn't do\r\n stemming for LiblineaR otherwise Liblinear was giving lot better results then textir.
", "votes": 0, "replies": [] }, { "id": 14869, "author_name": "Andreas Mueller", "author_url": "", "post_date": "2012-09-27T07:49:55.660000", "content": "B Yang: I tried count features, tfidf features and binarized features (only presence / absence). tfidf worked best. I think it only really made a difference with the char n-grams.
\r\ntuzzeg: I used a semantic word database and counted the number of "strongly positive" and "strongly negative" words. I also used a regexp based on "you are (a)/(the) ...".
\r\nI tried using POS-tag histograms but it took very long and didn't seem to help much, so I dropped it.
\r\nI also used nltk collocations that included "you".
\r\n\r\n
I had one classifier that was only based on these features and it worked quite badly :-/ I didn't have time to figure out how to combine it with the character n-grams.
", "votes": 0, "replies": [] }, { "id": 14803, "author_name": "tuzzeg", "author_url": "", "post_date": "2012-09-25T18:50:09.053000", "content": "[quote=Olivier Grisel;14780]
\r\n@tuzzeg interesting approach. Why did you use ExtraTreesRegressor (as opposed to ExtraTreesClassifier)? What target signal did you use for fitting it?
\r\nI assume the inputs are the predicted probas of the logistic regression models for each feature set, it that correct?
\r\n[/quote]
\r\nYes, final ExtraTreeRegressor takes results of crossvalidation of a bunch basic LogReg classifiers - I think this is called stacking.
", "votes": 0, "replies": [] }, { "id": 14802, "author_name": "tuzzeg", "author_url": "", "post_date": "2012-09-25T18:44:57.603000", "content": "For words/stems I tried 0/1 measure and some kind of p(w_c|w) measure (actual formulae is (pc(w)+1)/(pc(w)+nc(w)+1) where pc(w) is positive count - how many times this word found in positive examples - and nc(w) is negative count). 0/1 shown worse results.\r\n Didn't try TF*IDF measure (but should!).
\r\nFor char ngrams I tried the same p(ngram_c|ngram) and TF measure (count(ngram)/sum_ngram(count(ngram))). According to feature_importance in final ensemble TF is better.
", "votes": 0, "replies": [] }, { "id": 14796, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-09-25T17:26:35.237000", "content": "I used only R.
\r\nTm package for a term-document matrix with binary weighting. A blend of multiple classification models
--deleted--
", "votes": 0, "replies": [] }, { "id": 14780, "author_name": "", "author_url": "", "post_date": "2012-09-25T12:12:27.163000", "content": "@tuzzeg interesting approach. Why did you use ExtraTreesRegressor (as opposed to ExtraTreesClassifier)? What target signal did you use for fitting it?\r\n
\r\nI assume the inputs are the predicted probas of the logistic regression models for each feature set, it that correct?\r\n
", "votes": 0, "replies": [] }, { "id": 14774, "author_name": "razgon", "author_url": "", "post_date": "2012-09-25T08:41:41.653000", "content": "I used RTextTools package, ensembled SVM and GLMNET models. I used weightSMART (ltu) weighting, it performs better than TFIDF.
", "votes": 0, "replies": [] }, { "id": 14773, "author_name": "B Yang", "author_url": "", "post_date": "2012-09-25T05:41:30.480000", "content": "Andreas, welcome to Kaggle. I saw in your code that you used tfidf weighting. Did you find it useful ?
\r\nSame question to everyone else who used tfidf. Was it useful ?
", "votes": 0, "replies": [] }, { "id": 359359, "author_name": "yuwei", "author_url": "", "post_date": "2018-07-20T00:47:20.470000", "content": "thanks
", "votes": 0, "replies": [] }, { "id": 134210, "author_name": "g7", "author_url": "", "post_date": "2016-09-05T19:41:27.227000", "content": "Thanks Vivek!
", "votes": 0, "replies": [] } ], "comment_count": 28, "messages": [ { "id": 14742, "postDate": "2012-09-24T10:42:59.393Z", "content": "I would be very interested to see what other people used (languages, packages).
\r\nI published my code and wrote about it in my blog.
\r\nBasically I used mainly character n-grams and logistic regression from scikit-learn.
Chris Brew (came in fourth) also commented there. He used a similar approach and also published his code.
\r\n
\r\n
Vivek Sharma, the winner, also used scikit-learn as he said on the scikit-learn mailing list.
\r\n(I don't want to say anything about his approach as I am not sure what he wants to disclose).
I would be really interested if the other leaders also used scikit-learn or maybe R or weka or anything else?
\r\n\r\n
Cheers,
\r\nAndy
", "votes": 19 }, { "id": 15951, "postDate": "2012-10-19T13:14:00.443Z", "content": "Its somewhat late I realize, but I thought I'd post the code for my winning submission. Its a short python script... and is attached with this post.
\r\nThe badwords file was taken from here: (http://urbanoalvarez.es/blog/2008/04/04/bad-words-list/) and modified manually.
", "votes": 17, "replies": [ { "id": 168901, "postDate": "2017-03-18T15:34:41.060Z", "content": "Dear Vivek,
\n\nEdit:\nI managed to find the bug that was forcing the predictions to be almost the same. The script was using the Date column as test data. My fault for not checking which columns where being used.
\n\nOriginal post:\nI have tried to run your winning script using Python 2.7 in Anaconda 4.3 (64-bit) as I would like to see how it works and also use it with some data that I have collected but I am running into some problems as the preds.csv file with the results shows the same value on every line.
I have used the same files that you used to train (train.csv and test_with_solutions.csv), the badwords file and as a test file I have tried both impermium_verification_labels.csv and impermium_verification_set.csv but had the same results, i.e. 2236 values all exactly 0.0700027496. I have tried different values for binary, stopwords and analyzer_char in the ngrams function but the results are mostly the same. So I am definitely doing something wrong.
I have had to change the max_n=mx and the min_n=mn in the ngrams function specifically in the CounterVectorizer and TfidfVectorizer to ngram_range=(mx, mn) as the error TypeError: __init__() got an unexpected keyword argument 'max_n' was being reported so I do not know whether this is influencing all the skewed results.
When running the script in debug mode (I'm using PyCharm community edition) and closely watch the runClassifiers function, I notice that the variable pred is an array of tuples where all tuples have the same values and when these values are appended in the preds list they are all the same with the result that the final_pred ndarray is an array of 2235 same values.
Any help would be greatly appreciated. Thank you.
", "rawMarkdown": "Dear Vivek,\n\nEdit:\nI managed to find the bug that was forcing the predictions to be almost the same. The script was using the Date column as test data. My fault for not checking which columns where being used.\n\nOriginal post:\nI have tried to run your winning script using Python 2.7 in Anaconda 4.3 (64-bit) as I would like to see how it works and also use it with some data that I have collected but I am running into some problems as the `preds.csv` file with the results shows the same value on every line.\n\nI have used the same files that you used to train (`train.csv` and `test_with_solutions.csv`), the badwords file and as a test file I have tried both `impermium_verification_labels.csv` and `impermium_verification_set.csv` but had the same results, i.e. 2236 values all exactly 0.0700027496. I have tried different values for binary, stopwords and analyzer_char in the ngrams function but the results are mostly the same. So I am definitely doing something wrong. \n\nI have had to change the `max_n=mx` and the `min_n=mn` in the ngrams function specifically in the CounterVectorizer and TfidfVectorizer to `ngram_range=(mx, mn)` as the error `TypeError: __init__() got an unexpected keyword argument 'max_n'` was being reported so I do not know whether this is influencing all the skewed results.\n\nWhen running the script in debug mode (I'm using PyCharm community edition) and closely watch the runClassifiers function, I notice that the variable `pred` is an array of tuples where all tuples have the same values and when these values are appended in the `preds` list they are all the same with the result that the `final_pred` ndarray is an array of 2235 same values.\n\nAny help would be greatly appreciated. Thank you.", "votes": 1 } ] }, { "id": 14755, "postDate": "2012-09-24T18:26:47.183Z", "content": "My feature set was almost the same as the char and word features that Andreas used. SVC gave me better performance than regularized LR. And, some normalizations (like tuzzeg mentioned), along with using a bad words list (http://urbanoalvarez.es/blog/2008/04/04/bad-words-list/)\r\n helped quite a bit. Those were probably the only differences between Andreas' score and mine. The single SVC model would have won by itself, although the winning submission combined SVC with RF which improved the score marginally over just SVC. Regularized\r\n LR and GBRT were also tried, but they did not change the score much. I did not use the datetime field.\r\n
\r\nTuzzeg, I experimented a little bit with phrase features, and I'm pretty sure they would be needed in any implementation of such a system. A lot of the insults were of the form: "you are/you're a/an xxxx", "xxxx like you", "you xxxx". I tried to look for\r\n a large +ve/-ve word list to determine sentiment of such phrases with unseen words, but I couldn't find a good word list that was freely available for commercial use. Does anyone know of one? Ultimately, I didn't use any such features except for a very simplified\r\n one based on "you are/you're xxx" which did help the score, although, only to a small extent.\r\n
", "votes": 12 }, { "id": 14751, "postDate": "2012-09-24T16:11:49.857Z", "content": "I used scikit-learn as well, with Stanford POS tagger and Stanford parser. My approach in general was ensemble of LogisitcRegression classifiers over words, stemmed words, POS tags, char ngrams, words/stems 2,3-grams, word/stem subsequences, language models\r\n over words/stems/tags and a bunch of features over dependency parsing results (110 basic classifiers in final solution). All of them were stacked using ExtraTreesRegressor.
\r\nI didn't use word correction - which could help to detect such phrases like 'r u'=='are you' or 'f#%k'.
\r\nI am curious if somebody used more linguistic features, like in \r\nSmokey or like described here.
", "votes": 10 }, { "id": 15857, "postDate": "2012-10-17T22:39:57.117Z", "content": "Congrats to the winners!
\r\nI also added a description of my approach (in short: SVMs, neural networks and some good tokenizing). I wanted to share this earlier, but I just couldn't find the time.
\r\nhttp://webmining.olariu.org/my-first-kaggle-competition-and-how-i-ranked
", "votes": 5 }, { "id": 15068, "postDate": "2012-10-02T17:32:13.177Z", "content": "Just published \r\nmy code and \r\nbrief explanation of my approach. Questions, suggestions and any thoughts are welcome!
", "votes": 5 }, { "id": 14834, "postDate": "2012-09-26T13:46:32.023Z", "content": "I used character n-grams, tfidf with sublinear_tf and SGDRegressor with early stopping. I am somewhat proud of the early stopping code.
\r\nMy reason for using a regression estimator was that the evaluation was going to be AUC, which is sensitive only to the order of the scores, not the finer details. Had I used a classifier, I would have needed to do something with predict proba to arrange\r\n the items in a good order anyway. SGD is also nice because it works well with sparse inputs lets you explore things like the use of the elastic net penalty while sticking with the same classifier.
\r\nAs I said in my comment on Andreas Mueller's blog, the final order has an element of luck to it, because the final test set was so small and the labeling was rather noisy
\r\nCode is at:
\r\nhttps://github.com/cbrew/Insults/blob/master/Insults/insults.py
", "votes": 3 }, { "id": 14801, "postDate": "2012-09-25T18:44:27.313Z", "content": "I have used R and tm package for word stemming, normalisation and creating word-features for documents. I am tryed binary, tf and tfIdf weightings, but binary was best. Also i used as features 1000 letter 3-grams with best information gain and small number\r\n of features like proportion of uppercase letters, punctuations, etc. About 6k features total.
\r\nFor classification I have used gradient boosted trees from R gbm package.
\r\nI realy impressed that people widely use scikit-learn for this competion, i will try to use it next time!
Can somebody explain me, why stochastic gradient descent is better than gradient boosting for this kind of tasks, with a lot of features.
", "votes": 1 }, { "id": 14747, "postDate": "2012-09-24T15:40:24.463Z", "content": "I used stanford classifier
\r\nhttp://nlp.stanford.edu/software/classifier.shtml
It is basically same approach you used. Logistic classifier without regularization using n-gram technique. I made only a few changes to the defaults.
\r\nI finished 31st.
Michael: I don't think that the statement is "SGD is better than gradient boosting" but "linear is better than trees". Not sure in how far that holds. But most people only use linear classifiers with text and it works very well.
\r\nI'm totally new to text, so I can't really judge.
\r\nI think you could argue that there are many important words (I think a lot more than the 1000 features you used - I think my word list was longer!) and that in such a very high-dimensional space with very strong univariate correlations, a linear model should\r\n to the trick.
", "votes": 2 }, { "id": 14829, "postDate": "2012-09-26T10:37:58.967Z", "content": "I've added a blog on my approach:
\r\nhttp://steve-p0ulson.blogspot.co.uk/2012/09/recently-i-entered-kaggle-detecting.html
\r\nThe interesting thing from the leading entries was how similar the approaches ended up.
", "votes": 2 }, { "id": 167830, "postDate": "2017-03-15T15:53:18.757Z", "content": "I know it has been a long time and most probably this thread is as dead as the dodo but I would like to use some of the code provided by the entrants for a part of my research. To whom shall I attribute the citation please? Thank you.
", "rawMarkdown": "I know it has been a long time and most probably this thread is as dead as the dodo but I would like to use some of the code provided by the entrants for a part of my research. To whom shall I attribute the citation please? Thank you." }, { "id": 137052, "postDate": "2016-09-27T19:34:52.940Z", "content": "@tuzzeg how fast does it process information? For example: if its a forum posting, how many posts can it process within 1 minute if trained the way you did it?
", "rawMarkdown": "@tuzzeg how fast does it process information? For example: if its a forum posting, how many posts can it process within 1 minute if trained the way you did it?" }, { "id": 15858, "postDate": "2012-10-17T22:48:44.777Z", "content": "HI Andrei - link appears to be broken
" }, { "id": 15060, "postDate": "2012-10-02T15:01:57.157Z", "content": "Congratulations to winners. I used R for this competition.
\r\n- RWeka package was used for word tokenizing. Iterated Lovins was used as stemmer.
\r\n- Other features which I used:
\r\n-Number of words in the comment
\r\n-Number of bad words (I used a bad word list containing 50 words)
\r\n-Number of phrases such as: "you are a", "you sound like"
\r\n-Type of date (this feature was useless in the verification set)
\r\n- Fisher Score was used as feature selector. 750 features were used for modeling.
\r\n- Both decision trees and functional algorithms were used for modeling. Final model was a combination of SVM, Neural Net, random Forest, GBM, SGD, and GLMNet.
" }, { "id": 15049, "postDate": "2012-10-02T10:40:50.267Z", "content": "I used SQL Server Express to prep the dataset and build variables (time of day, length of comment, comment groupings: foul, meanspirited, pointed, family reference, etc.), then used R for the models. My best performing model prior to the final validation\r\n phase was using the earth package for Multivariate Adaptive Regression Splines (0.872 public, 0.875 private. Ultimately, the best model (0.757) turned out to be an ensemble of MARS, Neural Nets, Random Forest and GBM.
\r\nI actually like the simplicity of logistic regression and found it quite competitive with some of the other models.
\r\nAnyway, fun competition!
" }, { "id": 14920, "postDate": "2012-09-28T13:13:11.137Z", "content": "I used Python and the NLTK package for preprocessing and feature selection. Features included tfidf, bad words list, smileys list, regex for insult phrases, and time categories. I did not try character n-grams like most people here did. It would have improved\r\n my score by 0.00053 if I had added a simple character 2-gram prediction to my blend.
\r\nI was short on time so did not have the chance to play with scikit-learn like I had originally intended. Instead, I went back to R for modelling. It's comfort food for statisticians. I concentrated on algorithms that did their own feature selection: regularized\r\n logistic regression, gradient boosted regression trees, and random forests.
\r\nDid anyone try incorporating similarity scores or other clustering techniques? I found cosine similarity on tfidf to be quite useful in the blend.
" }, { "id": 14917, "postDate": "2012-09-28T11:28:07.787Z", "content": "@Andreas
\r\nYou seems to be saying right, I used LiblineaR library to make models, but in the final version I removed stemming from my approach. 1 model I submitted using textir package in R, in which I used stemming also that only gave my best result. I didn't do\r\n stemming for LiblineaR otherwise Liblinear was giving lot better results then textir.
" }, { "id": 14869, "postDate": "2012-09-27T07:49:55.660Z", "content": "B Yang: I tried count features, tfidf features and binarized features (only presence / absence). tfidf worked best. I think it only really made a difference with the char n-grams.
\r\ntuzzeg: I used a semantic word database and counted the number of "strongly positive" and "strongly negative" words. I also used a regexp based on "you are (a)/(the) ...".
\r\nI tried using POS-tag histograms but it took very long and didn't seem to help much, so I dropped it.
\r\nI also used nltk collocations that included "you".
\r\n\r\n
I had one classifier that was only based on these features and it worked quite badly :-/ I didn't have time to figure out how to combine it with the character n-grams.
" }, { "id": 14803, "postDate": "2012-09-25T18:50:09.053Z", "content": "[quote=Olivier Grisel;14780]
\r\n@tuzzeg interesting approach. Why did you use ExtraTreesRegressor (as opposed to ExtraTreesClassifier)? What target signal did you use for fitting it?
\r\nI assume the inputs are the predicted probas of the logistic regression models for each feature set, it that correct?
\r\n[/quote]
\r\nYes, final ExtraTreeRegressor takes results of crossvalidation of a bunch basic LogReg classifiers - I think this is called stacking.
" }, { "id": 14802, "postDate": "2012-09-25T18:44:57.603Z", "content": "For words/stems I tried 0/1 measure and some kind of p(w_c|w) measure (actual formulae is (pc(w)+1)/(pc(w)+nc(w)+1) where pc(w) is positive count - how many times this word found in positive examples - and nc(w) is negative count). 0/1 shown worse results.\r\n Didn't try TF*IDF measure (but should!).
\r\nFor char ngrams I tried the same p(ngram_c|ngram) and TF measure (count(ngram)/sum_ngram(count(ngram))). According to feature_importance in final ensemble TF is better.
" }, { "id": 14796, "postDate": "2012-09-25T17:26:35.237Z", "content": "I used only R.
\r\nTm package for a term-document matrix with binary weighting. A blend of multiple classification models
--deleted--
" }, { "id": 14780, "postDate": "2012-09-25T12:12:27.163Z", "content": "@tuzzeg interesting approach. Why did you use ExtraTreesRegressor (as opposed to ExtraTreesClassifier)? What target signal did you use for fitting it?\r\n
\r\nI assume the inputs are the predicted probas of the logistic regression models for each feature set, it that correct?\r\n
" }, { "id": 14774, "postDate": "2012-09-25T08:41:41.653Z", "content": "I used RTextTools package, ensembled SVM and GLMNET models. I used weightSMART (ltu) weighting, it performs better than TFIDF.
" }, { "id": 14773, "postDate": "2012-09-25T05:41:30.480Z", "content": "Andreas, welcome to Kaggle. I saw in your code that you used tfidf weighting. Did you find it useful ?
\r\nSame question to everyone else who used tfidf. Was it useful ?
" }, { "id": 359359, "postDate": "2018-07-20T00:47:20.470Z", "content": "thanks
", "rawMarkdown": "thanks" }, { "id": 134210, "postDate": "2016-09-05T19:41:27.227Z", "content": "Thanks Vivek!
", "rawMarkdown": "Thanks Vivek!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:33:45Z", "competition_id": 3023, "competition_slug": "detecting-insults-in-social-commentary", "team_id": 21442, "solution_url": "/competitions/detecting-insults-in-social-commentary/writeups/andreas-mueller-what-did-you-use" }