Firstly, congrats to the winners! It's been a mostly fun competition, but also annoying at times.
=)
Seems the top ranks between private/public leaderboard haven't changed much, except of course for the top spot.
\nAnyways, here's a write-up of what I did. It's mostly standard stuff for text processing, but maybe it'll be helpful to some.
\nI used python, sklearn, and numpy/scipy. My computer is a dual core with 4GB ram. My process is broken down into about a dozen scripts, due to hardware restrictons. If it were to be run start to end, from raw data to submission file, I estimate it would take somewhere around 10-15 hours.
\n\n
TL;DR
I cleaned the post texts, applied tf-idf, and combined it with a few meta features. With one-vs-rest approach, I trained a SGD classifier for each of the most common 10K tags. Based on the confidence scores of the classifiers on the test set, I picked the most likely tags for each post among these 10K tags.
Cleaning the data
My assumption was that the code fragments in the texts were too varied to be of any use, so I got rid of them all. I also stripped off the html tags, links, and urls. Also the line breaks and most punctuation, and converted all to lowercase. This left me with cleaner, conversational bits of the text. It also cut down the sizes of both train and test sets by more than half.
Meta features
Seeing I would now be working on less than half of the original input, I decided to incorporate some meta features from the raw text, especially about the bits I dropped off. For each post, I used:
I also used a couple features from the cleaned version of the text:
I scaled all these features to the 0-1 range, using simple min-max.
In the end, these meta features did improve the score (vs using only tf-idf on cleaned text), but not significantly.
Duplicates
As discussed on the forums, a lot of the posts (over a million) in the test set were already in the train set. The tags were already known, so I seperated these out as solved. In some cases (little over 100K) there were different tags given for the same question. I chose to take the union of these tags, instead of the intersection, since it scored slightly better on the leaderboard.
I also chose to prune out the duplicates from the train set. I kept the ones where same question had different results, but dropped all exact matches. My train set was now at little over 4.2 million cases.
To identify duplicates, I took the hash of clean text posts, and compared the resulting integers.
Tf-idf
Up until now, I was basically streaming the data through pipes, instead of loading it into memory (no pandas, I used custom generators to handle the data). But when I tried to apply tf-idf on the train set, I ran out of memory. I was still streaming the text, but my computer apparently couldn't handle all the features (there were around 5 million unique words, iirc). After various trials, I split the train set into chunks of 500K posts, and limited the number of features to 20K. My computer could handle larger numbers for the tf-idf step alone, but they got me into memory problems again in the future steps, so I limited myself at 500K x 20K.
I used the default english stopwords in sklearn. I only used single words, since including 2 or more ngrams seemed to hurt my results during the scoring stage.
And finally, I simply stacked the meta features from earlier on top of the tf-idf matrix. So my final processed input was eight sparse matrices of 500,000 x 20,007, plus the last chunk which had 206K rows. I designated the first chunk as cross validation set, and used it for testing out parameters.
Training and predicting
I took the one-vs-rest approach, and transformed the problem into binary classification. So I would build a model for the tag "c#", and mark the questions positive if their tags included "c#", negative otherwise. I would train seperate models this way for every tag on the input chunk, and then use the confidence scores (predict_proba or decision_function in sklearn) of the models to come up with tags.
Of course, I ran into memory problems again. So I split the process into steps, writing the results of each step to disk. I would train 1000 models, save them to disk. Then load the cv/test set, calculate confidences, save to disk. Repeat for more batches of 1000 tags. Then load the confidences for all batches of 1000 models each and combine them, and spit out tag predictions.
As the model itself, I tried Logistic Regression, Ridge Classifier, and SGD Classifier. SGD with modified huber loss gave best results, and was also the fastest, so I sticked with that.
When guessing the tags, I tried various approaches, like picking top n tags with highest confidence, or with confidence over a threshold. In the end, a mixed method seemed to work best for me. This is my predicting process, in two steps:
- from every batch of 1000 models, pick the ones with confidence over 0.20
- if there are more than 5, pick only the top 5
- if there are none, pick the best tag anyway
- after collecting from all batches in this way, pick the tags with confidence over 0.10
- if there are more than 5, pick only the top 5
- if there are none, pick the best tag anyway
It looks a bit silly, I know, but the 0.20 / 0.10 dual thresholds worked best for me in cross validation, and the scores were in line with the leaderboard. This way, I reason, rarer tags with low confidence get a chance to be picked, but only if there aren't enough frequent tags with high confidence.
Final submission
I ended up using the third 500K chunk for training, since it performed slightly better than others. I used only the first 10K tags (by frequency in train), thus 10K models. Beyond 10K, I started getting tags that were so rare that they weren't present in my 500K cases.
I tried setting up an ensemble of various 500K chunks, averaging the confidence scores, and predicting tags from that. But it showed very little improvement when I tried on small scale, and I was running out of time. So my final submission trains on only 500K cases of the input.
Things I tried that didn't seem to work
Dimensionality reduction didn't seem to work for me. I tried LSA to reduce the 20,007 features to 100-500, but scores went down. I also tried picking only the models with high confidence, but that, too, hurt the final f1 score.
I briefly played with a two tier stacking process, using the confidence scores of the models as input for a second tier of models. Again, I couldn't improve the score.
There were some tags that often come up together, like python & django, or ios & objective-c. But I wasn't able to exploit this to score better than seperate models for each tag.
Things I didn't try
I didn't try feature selection, I thought it would be costly to do so on each of the 10K models.
I thought of employing bagging, boosting, or just simply duplicating cases with rare tags, in order to train models better. Didn't have time to try these out.
I didn't try stemming or part of speech methods. Also, I didn't look into creating tags directly from text. So any tags in test that are not in the most frequent 10K ones in train were on my blind side.
Anyways, the worst part was of course continuously juggling data between ram and disk. I'm left with gigabytes of intermediate dumps, laying around my disk in hundreds of files.
Looking forward to reading about what all others did.
Cheers!
\n", "messages": [ { "id": "36434", "postDate": "12/21/2013 00:18:24", "content": "
Firstly, congrats to the winners! It's been a mostly fun competition, but also annoying at times.
=)
Seems the top ranks between private/public leaderboard haven't changed much, except of course for the top spot.
\nAnyways, here's a write-up of what I did. It's mostly standard stuff for text processing, but maybe it'll be helpful to some.
\nI used python, sklearn, and numpy/scipy. My computer is a dual core with 4GB ram. My process is broken down into about a dozen scripts, due to hardware restrictons. If it were to be run start to end, from raw data to submission file, I estimate it would take somewhere around 10-15 hours.
\n\n
TL;DR
I cleaned the post texts, applied tf-idf, and combined it with a few meta features. With one-vs-rest approach, I trained a SGD classifier for each of the most common 10K tags. Based on the confidence scores of the classifiers on the test set, I picked the most likely tags for each post among these 10K tags.
Cleaning the data
My assumption was that the code fragments in the texts were too varied to be of any use, so I got rid of them all. I also stripped off the html tags, links, and urls. Also the line breaks and most punctuation, and converted all to lowercase. This left me with cleaner, conversational bits of the text. It also cut down the sizes of both train and test sets by more than half.
Meta features
Seeing I would now be working on less than half of the original input, I decided to incorporate some meta features from the raw text, especially about the bits I dropped off. For each post, I used:
I also used a couple features from the cleaned version of the text:
I scaled all these features to the 0-1 range, using simple min-max.
In the end, these meta features did improve the score (vs using only tf-idf on cleaned text), but not significantly.
Duplicates
As discussed on the forums, a lot of the posts (over a million) in the test set were already in the train set. The tags were already known, so I seperated these out as solved. In some cases (little over 100K) there were different tags given for the same question. I chose to take the union of these tags, instead of the intersection, since it scored slightly better on the leaderboard.
I also chose to prune out the duplicates from the train set. I kept the ones where same question had different results, but dropped all exact matches. My train set was now at little over 4.2 million cases.
To identify duplicates, I took the hash of clean text posts, and compared the resulting integers.
Tf-idf
Up until now, I was basically streaming the data through pipes, instead of loading it into memory (no pandas, I used custom generators to handle the data). But when I tried to apply tf-idf on the train set, I ran out of memory. I was still streaming the text, but my computer apparently couldn't handle all the features (there were around 5 million unique words, iirc). After various trials, I split the train set into chunks of 500K posts, and limited the number of features to 20K. My computer could handle larger numbers for the tf-idf step alone, but they got me into memory problems again in the future steps, so I limited myself at 500K x 20K.
I used the default english stopwords in sklearn. I only used single words, since including 2 or more ngrams seemed to hurt my results during the scoring stage.
And finally, I simply stacked the meta features from earlier on top of the tf-idf matrix. So my final processed input was eight sparse matrices of 500,000 x 20,007, plus the last chunk which had 206K rows. I designated the first chunk as cross validation set, and used it for testing out parameters.
Training and predicting
I took the one-vs-rest approach, and transformed the problem into binary classification. So I would build a model for the tag "c#", and mark the questions positive if their tags included "c#", negative otherwise. I would train seperate models this way for every tag on the input chunk, and then use the confidence scores (predict_proba or decision_function in sklearn) of the models to come up with tags.
Of course, I ran into memory problems again. So I split the process into steps, writing the results of each step to disk. I would train 1000 models, save them to disk. Then load the cv/test set, calculate confidences, save to disk. Repeat for more batches of 1000 tags. Then load the confidences for all batches of 1000 models each and combine them, and spit out tag predictions.
As the model itself, I tried Logistic Regression, Ridge Classifier, and SGD Classifier. SGD with modified huber loss gave best results, and was also the fastest, so I sticked with that.
When guessing the tags, I tried various approaches, like picking top n tags with highest confidence, or with confidence over a threshold. In the end, a mixed method seemed to work best for me. This is my predicting process, in two steps:
- from every batch of 1000 models, pick the ones with confidence over 0.20
- if there are more than 5, pick only the top 5
- if there are none, pick the best tag anyway
- after collecting from all batches in this way, pick the tags with confidence over 0.10
- if there are more than 5, pick only the top 5
- if there are none, pick the best tag anyway
It looks a bit silly, I know, but the 0.20 / 0.10 dual thresholds worked best for me in cross validation, and the scores were in line with the leaderboard. This way, I reason, rarer tags with low confidence get a chance to be picked, but only if there aren't enough frequent tags with high confidence.
Final submission
I ended up using the third 500K chunk for training, since it performed slightly better than others. I used only the first 10K tags (by frequency in train), thus 10K models. Beyond 10K, I started getting tags that were so rare that they weren't present in my 500K cases.
I tried setting up an ensemble of various 500K chunks, averaging the confidence scores, and predicting tags from that. But it showed very little improvement when I tried on small scale, and I was running out of time. So my final submission trains on only 500K cases of the input.
Things I tried that didn't seem to work
Dimensionality reduction didn't seem to work for me. I tried LSA to reduce the 20,007 features to 100-500, but scores went down. I also tried picking only the models with high confidence, but that, too, hurt the final f1 score.
I briefly played with a two tier stacking process, using the confidence scores of the models as input for a second tier of models. Again, I couldn't improve the score.
There were some tags that often come up together, like python & django, or ios & objective-c. But I wasn't able to exploit this to score better than seperate models for each tag.
Things I didn't try
I didn't try feature selection, I thought it would be costly to do so on each of the 10K models.
I thought of employing bagging, boosting, or just simply duplicating cases with rare tags, in order to train models better. Didn't have time to try these out.
I didn't try stemming or part of speech methods. Also, I didn't look into creating tags directly from text. So any tags in test that are not in the most frequent 10K ones in train were on my blind side.
Anyways, the worst part was of course continuously juggling data between ram and disk. I'm left with gigabytes of intermediate dumps, laying around my disk in hundreds of files.
Looking forward to reading about what all others did.
Cheers!
\n", "rawMarkdown": "", "votes": null }, { "id": "36448", "postDate": "12/21/2013 04:29:03", "content": "
I went in as anonymous, finished 42nd with 0.77091 which was fairly robust compared to my public 0.77100 last submission score.
\nApproach:
\nsklearn, 16GB RAM on OSX, goal of making a pipeline capable of submitting in a business day.
\nThe approach then did the following after having each classifier predict against the test set:
\nFirst, if the test example was in the training set, predict the training tags.
\nFor anything left, take the union of the output of classifiers 1 and 2, and toss out the lowest probability tag if the number of predictions was over 5.
\nFor anything left unpredicted after that, take the prediction from the appropriate cluster that the test example belonged to.
\nIf there still isn't a prediction after that, use the top 5 tags.
\nIt turns out that Classifier 2 (the vocabulary approach) made Classifier 3 (the lexical approach) unnecessary.
\n\n
Dead Ends
\nBefore I knew what I was doing, I tried clustering the training set and then training OneVsRest classifiers for each cluster. This was intended to make it easier to train more tags in less memory but letting me train say, 450 OvR classifiers on a small number of examples, but it got comparable to slightly inferior results with much more training time. Ultimately, just training one or two OvR classifiers that used somewhat skewed vectorization approaches for their features produced better results in less time.
\nIf I had to do it again, I'd go "all in" on using a vocabulary approach with TfidfVectorizer and then spend my time working on preprocessing. Oddly, many substitutions and rollups of synonyms degraded the results. I haven't figured out why yet.
\nUsing SVD to reduce dimensions on raw Tfidf wasted a ton of time and didn't make a difference. You're much better off defining your vocabulary, or using the min_df to reduce dimensions.
\nI had a fixation on exposing myself to as many training examples as possible--turns out that with my skewed training set, 80,000 examples (42000 culled, plus 38000 random) was often good enough, so it was much better to spend computation on more features rather than more training examples.
\nAs others pointed out, when you train a generic model, you need a 0.024 improvement in your test set to get a 0.01 improvement in your submission score once you account for identical posts. That makes 42nd look less impressive. I did one test where I calculated an f1 score if I could pick the best from the predictions of the 10 classifiers (sans identical matches) and I think it was something like 0.57. I didn't have time to figure out a good ensemble scheme or weighting approach to get closer to that theoretical max the models could produce. Part of the problem was that sklearn and SGDClassifier require settings to get raw probabilities that might help--and these settings got poorer results than the "hinge" approach I landed on.
\nKey Insights
\nIf you're working with raw tokenized text, ngrams are your friend. As is min_df
\nUsing a vocabulary with TfidfVectorizer is a great idea, particular with preprocessing.
\nNot much gain going beyond training on the top 5000 tags. It'll move you up a few spots, but nothing earth shaking.
\n\n
", "rawMarkdown": "", "votes": null }, { "id": "36450", "postDate": "12/21/2013 04:51:33", "content": "
Congrats to all, its been fun, educational, entertaining, and...controversial :P This will be long, i will try to keep it as short as possible.
\n\n
PART I.
\n----------------------------------------
\nSoftware: I don't like to enter into programming language debates, but this will come up. In my experience, Python, and especially R, have been too slow for industrial/large scale/optimized custom-designed work. C is not exactly a joy to code in. I'm very good with SAS, but its specialized in what it does, and even if i wanted to use it, I don't have access to a licence via which using it for this competition wouldn't be a fire-able offense.
\nSubsequently, I coded the entire thing in Lisp. Common Lisp, SBCL, using emacs and SLIME on Linux Mint to be precise. I won't bore you with the history of Lisp. Read up on it if you're curious. Its one of the few languages that is relatively dynamic/high-level/low-level/compiled/REPL at the same time, but with atypical polish-prefix syntax.
\nIf my memory serves me correctly, the libraries I used were read-csv, and cl-utilities...which probably saved a bit of time here and there. Read-csv was used to parse the original raw data into lisp forms that could be written out in printed representation to disk. I split the body of the post into a code section and a non-code section, based roughly upon being between the <code> </code> tags. I lower-cased everything, and for the body text that wasn't deemed to be code, I roughly removed the HTML tags via an algorithm that searched for text between the '>' and '<' characters, since i considered parsing it more accurately to be outside help :P In the end I merged to the two back together, but there's good reason to believe that this is another area for potential improvement.
\nThe first few important functions I wrote were:
\ndata accessing macro: For exploratory work. I coded up a form that read in each record one at a time from the main file and looped through the input text file, kind of like the SAS data step. Each field in each record was referenced with ID, BODY, TITLE, CODE, and TAGS respectively. In a tribute to SAS, I made a variable _N_ which was an automatic variable that kept track of the observation number irrespective of the ID. I made some additional tools that I never really had to use. One was an array which recorded the position of each record within the file via the array index. A macro/function would use this to enable arbitrary random access to records within the original data file based upon their IDs, but I never really needed to use it.
\nn-gram generators: I wrote a character-ngram, a word-ngram, and a unique-ngram-function. Each is capable of returning an array of arbitrary length n-grams within a text. I only used 1-gram words in my final model, but its perfectly reasonable to assume that using more evidence would be a further improvement. Phrases like common lisp, for instance, would presumably be picked up under word-2-grams, but lisp turns up as significant evidence for both the common-lisp tag, and the lisp tag. Character n-grams would allow for evidence to be gathered irrespective of arbitrary separators, and would in some way enable negation of some of my sloppy parsing/cleaning/user entry.
\n-----------------
\nNow before I go on there were a number of irritating controversies which probably deserve some commentary. When I discovered the competition I started because I believed that it was literally a competition on "tag prediction" with a test set and a training set. Two things about that...
\nFirstly, I observed that the scores in the leader-board just weren't theoretically possible for such a task. An observation I stand by. Subsequently to be sure, I programmed up simulator, which let me enter some information on number of tags, accuracy of predictions for tags and number of tags in a post, and tell me what score would be returned for such performance. I did, and still do consider, duplicates to be against the spirit/goal of such a task, so was put in the awkward position of either calling out the scores and potentially pissing off a lot of people and possibly hurt my chances of working at facebook to find out what was going on, or keeping quiet and use it to my advantage to game the system for a high score. I chose the former. Mr Chester Grant soon started the next thread exposing the duplicate problem. My point about incentives to dishonesty for such a competition, i feel, still stands.
\nThere was one more problem, which has to do with the quality metric. For much of the time I was working under the belief that the metric being used was the F1 measure as calculated for true positives/falsepositives/falsenegatives across all posts (which is what I interpreted as the "average F1 measure"), rather than an F1 measure taken within each post and then each F1 score averaged. This is a subtle, but quite important difference. One results in a totally different strategy for score maximisation compared to the other. For one, the first gives you an incentive to not make spurious and unnecessary guesses, while the later gives you at least 1 free guess that it is optimal to take in all possible strategies. I'm sure I'm not the only person who misunderstood this, and that such an explanation on scoring could be made clearer and more prominent.
\n---------------
\nDuplicate Removal: This to me was a relatively unimportant part of the competition, so I spent about a day or two coding it up, deduping the training file, and then making the links between the test and train files for fun :P
\nA deterministic join on parts of the title/body gets you most of them. You can do a LITTLE bit better by going fuzzy/data linking. As I've said earlier, this is too close to my actual work, so I didn't spend much time on this.
\nTo fuzzily link the duplicates, I held the body of the test set in memory, and created a hash table key'd on individual words. I already had a word count frequency loaded into another hash table, so I used the 5 least common words in each post, and pushed the ID of that post onto the hash table key'd on those words.
\nI then coded up a quick implementation of the Dice-sorenson coefficient, which would act as a simple distance metric between any two posts. I read through the train set one record at a time, found the bottom 5 words in each post with the smallest frequency, and used the dice function to compare them to any of the original posts indexed by those words in the test set. I had one more hash table, keyed on IDs in the test set, in which I stored the highest scoring dice metric and subsequent ID in the train set for each individual ID. I arbitrarily chose a cutoff, and those ID's in the test set with a dice metric score to a record in the train set above this cutoff, i deemed duplicates. There's lots of theory here I will neither bore you with, nor explain, since I didn't bother being too careful, but I did notice during some reviews of these links the problems with certain short common phrased questions "What's the difference between "X" and "Y"?" Only two words different, but obviously different questions that look very similar :P
\nMoving on... to be cont'...
", "rawMarkdown": "", "votes": null }, { "id": "36451", "postDate": "12/21/2013 05:06:44", "content": "Thanks for posting the details! With OneVsRestClassifier did you use multi-label format(given as is) with LabelBinarizer or you changed input format of your input rows somehow?
", "rawMarkdown": "", "votes": null }, { "id": "36452", "postDate": "12/21/2013 05:22:35", "content": "My basic idea is to find predictive words. A predictive word for a tag is a word with high ratio of posts in which the word is contained and are labeled with that tag vs posts in which this word is contained.
\nI have the following pipeline.
\nStep 1: Read input in python, tokenize text and create features. Title words are prefixed by "T_", words in code segments are prefixed by "C_". That's because a "main" in the code might have higher predictive power than a "main" in the text. I got a big bump by doing the same for the title with the "T_" prefix. I guess something like "java" in the title is more predictive than "java" in the text.
\nStep 2: C++ code to remove words with very high and very low occurrence. I remove the top ~1000 words and the bottom 80% of the words. The bottom 80% have about 7 occurrences in the text, so I can't even use them reliably to predict.
\nStep 3: Train a classifier. C++ again. This is just computing the ratio which I pointed at earlier. I keep a label to words matrix in memory. To save memory, I hash label and words. The output is a map from words to labels which the words predict with high probability. So "C_iostream" might predict C++ with probability of 0.79. I also produce bigrams from words in the title. Bigrams from words in the post were blowing up my memory usage.
\nStep 4: Prediction. I go through every word and see what labels does it predict and keep the maximum probability for each label that comes up. I select the top 5 predictions and output them. Most of the time there are less than 5 predictions.
\nDupe detection is done by using a djb hash on the label text.
\nI tried coding a naive bayes with x2 feature selection step in both C++ and using scikit, but didn't get very far with F1.
", "rawMarkdown": "", "votes": null }, { "id": "36453", "postDate": "12/21/2013 05:36:01", "content": "I've worked on this challenge as part of class project (cs229 stanford), attached final report for reference.
\nReally enjoyed working on this, although didn't get as much time to work on tuning up the submission in the end.
\nLet me know your feedback.
", "rawMarkdown": "", "votes": null }, { "id": "36454", "postDate": "12/21/2013 06:18:23", "content": "PART II
\nThis is where it probably gets interesting, and I'm probably going to struggle to explain this without diagrams...
\nLets get this out of the way. I didn't use stop words, and I don't generally like matrices. In many real world problems they're too inefficient.
\nMy initial plan was to use a simple bayes formula on the individual words within title/body/code of a post. This involved calculating the requisite probabilities of observing a given word for each tag/not-tag combination...which is not exactly a small task. So...
\n
I am guessing everyone here is able to generate some basic summary stats. I calculated the relevant ones and stored them in memory: number of posts, probability of each tag, frequency for each word, etc, etc. One efficiency i noted here was that you don't need to store frequencies for words that only appear once. If you observe a word in the train set later, and its not contained in such a frequency hash, you DEDUCE that it must be a word that is only observed once in the train set, and it is observed once only in a post with the qualities of the post in which you have currently observed it.
Two-deep-hashes:
\nI created some functions to work with double-keyed hashes. That is to say, a hash-table whose key is a tag, and which maps each key to another hash-table that is keyed on words.
\nSo for example, if we are training on the tags "c#" and "java" and we observe the word "c#" in a post with tags "c#" and "java", the first hash table keys are "c#" and "java", which directs us to two more hash-tables, both of these are then accessed with keys of "c#". The values in these hashes then have their requisite counts incremented. Upon one iteration for the tags you are training through the training set, you can then see the frequency of words for each tag.
\nTo find out how many instances of that tag did not contain that word, and how many posts that didn't carry those tags contained that word, you iterate through the two-deep-hash-tables values and compare the requisite key/values to the original word frequencies/tag frequencies.
\nTraining and Evidence
\nThis is all well and good, but you've now got a two-deep-hash key'd on tags->words->frequencies. What you need for your classifier to make decisions are data structures keyed on words->tags->evidence, so each time you observe a word, you know how much evidence such an observation contributes to each tag. For this I chose a data structure that is a hash, keyed on words, mapping to an array of #(tag evidence) arrays, which can be iterated across incredibly fast. Each time a word is observed in a post, you hash it, which takes you to an array of evidence for requisite tags. The function remember-inverse-two-deep-hash is responsible for converting information from the two-deep-hash in the training phase into that needed for the classification phase.
\nThe functions save-valuable-words and load-valuable-words are responsible for writing the current valuable word->tag-evidence structures out to disk, and reading them in from disk respectively.
\nOne optimization I included relatively late was the decision to change the way load-valuable-words worked to ensure that it would resolve any clashes between evidence it reads in from multiple files. This means that you can theoretically create valuable-word structures from multiple files, allowing some form of combinations of different amounts of evidence, or allowing a later loaded file to overwrite evidence loaded from an earlier file.
\nOf course, this is all still relatively romantic. The resulting structures created from these are still far too big to store all possible evidence at once, even if it doesn't have to worry about sparsity quite like a matrix.
\nThat's where the cull-valuable-words/cull-two-deep-hash functions come in. They are relatively blunt, but cut down on the amount of work a busy person like myself has to do :P
\nThey iterate through the valuable-words/two-deep-hash structures, eliminating mappings that don't meet basic requirements of minimum probability/evidence/frequencies. Hence I don't worry about stop words, because such words will be culled by such functions during their sweep, because by definition, they won't be contributing very much evidence to particular tags, and if they aren't eliminated, then they probably aren't stop words for that particular tag.
\nTo classify a post, there's functions of produce-evidence-array and make-simple-prediction. The first uses the valuable words to produce a ranked prediction of all tags for which relevant words have been observed. The second uses the produced evidence array with a small number of parameters to make a prediction of tags for each post: say the maximum amount of evidence required to make a prediction for example.
\nQuality/Administrative Functions
\nThere are two more aspects which might need direct commentary. These are the very basic functions random-sampler and judgement. The first is a simple way of iterating through the training set and holding a certain proportion of posts in memory. The second can take predictions from the like of make-simple-prediction (or indeed any list of tags) and compare them to a truth list of tags to produce a representive F1 score. It actually operates as several functions sharing a closure, so you can just keep feeding it records, and it increments stats as it goes, and you can ask it at any time to report how the F1 score is doing so far, check how another prediction went, or reset it.
\nI think that's basically it...i hope i'm not forgetting anything...
\nImprovements/lessons learnt
\nThere's actually a fair number of improvements you can make to my model not already mentioned. Two obvious ones i thought of include better use of the evidence-culling functions. They are currently quite blunt, but you can change them into higher-order-functions such that their behaviors are driven more directly by the probability of observing a particular tag. This leads me to my second improvement. I know that my current method doesn't produce an optimal distribution of predicted tags, and optimizing this would likely result in some decent score improvements.
\nAlthough I haven't tested it, this model provides a possible claimant to the "good model with very little memory" aspect, you would just have to be much more violent with using the culling functions to make the evidence fit into memory. Classification is a relatively memory stable operation once the valuable words have been loaded, since all that happens is that a record is read in, and its classification is written out to disk. Of course you will do worse if you use less memory, but I'm not sure to what degree.
\nThe classification is, relative to most of the other entries i've heard of so far, fast. Indeed it is what I would call practically real-time. It runs in about as fast as it takes to read the data in from disk and write out the results, although processing each uncleaned-unformatted record to establish the unique words to fit into the classification function would slow it down, i don't think we're talking hours here.
\nEdit: Oh, just remembered, my submitted versions effectively culled records that didn't contribute a certain amount of evidence for a tag. I'm sure you could also add some evidence which would make certain tags less likely if a certain word is observed, but those wouldn't have been included in my model.
\nEdit2: Actually, there's a fair few improvements I can think of, but didn't get round to implementation. Another one is integrating post length stats into the prediction. My current entry basically has one classification rule across the entire set, which is obviously suboptimal. Post length does indeed correlate with the number of tags, and the amount of evidence that could ever be collected for a post, so this is another fix if one really wanted to get that top spot :P
\nEdit 3: In case it wasn't clear, there were different amounts of evidence for seeing a word in the title vs seeing a word in the body of a post.
", "rawMarkdown": "", "votes": null }, { "id": "36457", "postDate": "12/21/2013 10:31:48", "content": "Tried a different challenge. From .csv's to submission in under 1 hour, with less than 1GB of memory, in 30 days.
\nThis inspired further constraints: No use of external algorithm libraries, single-pass, online learning, parallel learning.
\nStarted with observation that top 200 tags are good for ~50% of all tag occurrences.
\nFirst algorithm idea: 75% of documents with the word "stylesheet" are tagged with "css". So Bro Bayes.
\nUse of a FTS4 Sqlite database played it safe with memory, but gave me too much to count and update (couldn't resist), so I predicted this would not meet time constraint. If you give yourself 30 minutes for the predictions then you are predicting at 1k+ a second.
\nI figured out the dupes in the train and test sets, and the dupes between them. To dedupe you could match the post titles. Using that with only a tag dictionary generated from the train set and substring matching those against the titles in the test set gave you a f1-score of around #20 at the leaderboard. This was 3 weeks ago. It took about 700Mb of memory and under 3 minutes.
\nStemming text was too slow. Preprocessing thus was more cleaning of text and removing stopwords. For stop words I first tried a top 10k stopword file from Google with the tag dictionary words removed. I found that counter to the competition rules, so I generated a stopword file from the train set with words that are found with many different tags attached, and hand-picked a top 200.
\nTrained Vowpal Wabbit on top 40 tags. Got a good result, but was approaching 1 hour. Trained it again on top 200 tags and got a better result, but took 3 hours. I believe I saw that top 1024 tags are good for over 70% of all tag occurrences, but realized this route was not going to meet constraints.
\nI ended up using 3 count dictionaries. One tag counter, one word counter and one word-to-tag counter. This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory. Not using an external big stopword dictionary meant I had to prune the word-to-tag dictionary with every 100k chunk. Once the number of tags per keyword reaches a certain threshold, I take a top n and only increase the counts of those top n tags in the future.
\nThis gave me my result that met all my constraints except for one, memory. It took 1.7Gb max. for model creation. Predictions with 1.1Gb on only the title words.
\nTried to do keyword extraction without a dictionary or statistical historical knowledge. First term frequency on a top 10k stopword removed file. Then co-occurrence: this paper http://ymatsuo.com/papers/flairs03.pdf found me reasonable results. All took either too long to run or did not add substantially to a dictionary-based approach.
\nFun things. Ran online LDA over the test set to obtain topics. Very clear topics appeared like all string manipulation commands. "adolf-hitler" is a tag. The hardest question to predict only by title was something like: "why?! why! why! why! why?!?!", because after stopwords and cleaning nothing remained :).
\nWhat I learned: Simply count ALL the things. Don't count 'em twice.
", "rawMarkdown": "", "votes": null }, { "id": "36463", "postDate": "12/21/2013 12:47:22", "content": "My approach seems too simple compared to a lot of these. Anyway, this got me 0.78 at 26th on the leaderboard. Everything coded from scratch in the D programming language.
\nVectorization
\nBag of words with binary TF (i.e. 1 if word there, 0 otherwise). Title words counted as separate features. Features with less than 20 total occurrences removed. No stop words removed or stemming (stop words are often very important in programming languages!)
\nTraining
\nUsed only the first million samples due to me only starting this method a few days ago. Just trained a one-vs-rest decision tree for each of the most frequent 10,000 tags using the ID3 algorithm. Tuned a few hyper-parameters for when to prune a tree based on frequency of the tag and feature to split on to avoid overfitting. The tree leaves give a probability of the tag rather than a binary decision.
\nClassification
\nFirst, use duplicates from training (hash lookup), then test all the decision trees, choosing those that gave the top 5 probabilities, if the probability was above 0.22 (tuned hyper-parameter).
\n\n
Notes
\nRig: Old 2GHz MacBook from 2008 with 2GB RAM
\nTraining took about a day using 300MB RAM, and classification took 30 minutes using 50MB RAM.
\nAt first I tried to use support vector machines but I found these to overfit too much and were much too slow to train on large datasets anyway, so that's when I moved to decision trees. Used a lot of time implementing the SVM trainer (SMO style, similar to libsvm), but it was a fun learning exercise (pun unintended).
\nIf I had more time I'd use the whole training set and use random forests instead. I'm certain that would give >0.8. Maybe I'll try that over the holidays.
", "rawMarkdown": "", "votes": null }, { "id": "36471", "postDate": "12/21/2013 17:29:42", "content": "First I started out with Naive Bayes Classifier. It was too slow and it only give a 0.16 f1 score on the non-duplicates. Then, I decided to tag a post with a tag if the tag appears in the title or body. This took my f1 score up to 0.26 on non-duplicates. Then, I tried word association. This required me to prune the number of words. I selected only words that had more than 10 appearance in the dataset. Word association took me up to about 0.40
\nAfter this is just weighing the associations and doing parameter optimisation of the weights using genetic algorithm. This took me to 0.506 f1 score.
\nI did try neural nets and random forest, but they only gave me the same results as word associations and since you can't can only do classification on a subset of tags with neural nets and random forest, I decided to go with word association.
\nThere were some tags that were unpredictable eg. html,file,database,form,exception,image. In the end, I was making customs classification for these unpredictable tags but that only took me from 0506 to 0.509. 0.506 translated to 0.77674 with the duplicate. I am happy with my performance.
", "rawMarkdown": "", "votes": null }, { "id": "36475", "postDate": "12/22/2013 00:20:49", "content": "Congrats to the winners and thanks to barisumog for starting this thread with a great write-up!
\nI also used one-vs-rest approach and here are my few comments on it.
\nCleaning the data and feature extraction
\nI tried TfIdf, Count, and Hashing vectorizers from sklearn with various parameters and hashing one ended up giving the best cv score and it was fast and it had a low memory usage since it doesn't need to store its vocabulary. The things which helped here:
\nAnd things which didn't help:
\nI also wrote a custom vectorizer based on TfIdf vectorizer to implement several supervised term weighting schemes, but none of them helped too.
\nTraining
\nFor each tag among top tags I trained a logistic regression model and pickled it for later usage. I started with training on the first several tens of thousands samples, but it had enough number of positive samples only for the most frequent tags. So I decided to generate a set of training samples for each tag separately: I kept reading Train.csv until I find some fixed number of positive and negative samples. Since my cv score changed significantly when I was asking for different numbers of positive and negative samples, I had a grid search to find their optimal values.
\nClassification
\nFind all duplicates using hash lookup and store non-duplicates in a separate file. Un-pickle all trained models, predict probabilities of each trained tag, and choose the ones with probability higher than some threshold, if more than five tags are chosen leave top five. My leaderboard score somewhat improved when I tried several ways of adjusting predicted probability with precision score I stored during training, for example, I would leave a tag with somewhat low predicted probability of 0.8 and precision score of 0.9 (like 'android') rather than leave a tag with high predicted probability of 0.9, but with low precision score of 0.15 (like 'database').
\nI also tried to use predicted probabilities to train another model to take into account possible correlations between tags, for example, some tags are more likely to be used together, but this additional step didn't seem to help much too.
\nOne final note, I only trained on first 2K tags which already correspond to 80% of all tag occurrences and I thought it's not worth it to train on another several thousand tags to gain just a few more percents, but it looks like people from top 10% trained on about ~10K tags.
", "rawMarkdown": "", "votes": null }, { "id": "36479", "postDate": "12/22/2013 01:32:23", "content": "First congrats to the winners and thanks to the other participants who shared their approaches. At the beginning I also spent lots of time extracting meta-features like has_code, is_html_tag, is_email, etc. My idea was to use this "improved" representation of the documents plus all the usual bag of words features to make similarity queries to each one of the documents in the test set (bow->tf-idf->lsa). From the top most similar documents in the training set I extracted the tags and combined them to make my prediction. This made more sense to me than just blindly clustering as it would be difficult to determine the right number of clusters. Nevertheless I tried both and didn't get the results I was expecting (under 0.70 f1-score). I still think this approach has potential and needs to be further explored.
\nSo in the last day I decided to go for a simpler approach. It was a little dramatic as I finished the code 19 hours before the submission deadline and according to my estimations the model would also take 19 hours to complete the prediction. Fortunately, the algorithm was relatively easy to parallelize and thanks to Amazon EC2 it took around 3 hours to finish the prediction in a cc2.8xlarge (wonderful machines by the way). I adapted the winning solution of the ECML PKDD Discovery Challenge 2009 by Evangelio Milos et. al. plus some small improvements from other solutions given there [1]. The algorithm goes like this, there are three basic tag recommenders which get combined to form the final submission:
\nAt the end I combined all tag recommendations probabilistically plus some small last minute optimizations. I'd say the main takeaway is how much statistical information is contained in the title. I intend to post the code as soon as I get to clean it a bit (it was made in a rush).
\nTools used: scikits-learn (hash vectorizer, quick tests), gensim (LSA, document similarity queries), networkx(for the graphs), pypy(I didn't got any visible speed up from this, most of the time was spent accessing hash-tables within networkx).
\n\n
UPDATE: Blogged about it with some code here.
\n\n
\n
Credits:
\n[1] http://www.kde.cs.uni-kassel.de/ws/dc09/papers/proceedings.pdf
", "rawMarkdown": "", "votes": null }, { "id": "36496", "postDate": "12/22/2013 04:19:06", "content": "Congrats to Owen, and others who did well. Also thanks to those who are sharing their approaches and experiences from this competition.
\nAfter some time experimenting with a range of different methods (OvA classifiers, and simple baseline models mostly), my most successful approach (0.79479) used association rules and was quite straightforward.
\nFeature Extraction
\nI stripped html tags, stop words (NLTK stop words), and special characters (except '+' and '#'). I kept the code blocks, as they seemed like dead giveaways for language tags. Then I extracted word unigrams, bigrams, and trigrams separately from both the title and body texts. Adding trigrams didn't give much improvement (about 0.01), and increased memory usage by quite a bit, but an improvement is an improvement.
\nTraining
\nI learned association rules separately from the title and body texts. This was essentially counting n-gram-tag cooccurrences in a Python dict, and keeping a count of n-gram occurrences in another Python dict. As the dictionaries grew I pruned out n-gram-tag cooccurrences which were very low in number and would be insignificant during prediction. This kept the total size of the dictionaries to about 25GB, so I could stay in RAM. I dumped the dictionaries to disk as a pickle. I trained over the whole training set (except for a small 10k sample for cross-validation).
\nPrediction
\nI found that the title text was far more useful than the body text, which was surprising.
\nI loaded the dictionaries back into RAM, then made predictions based on the title association rules. I tuned the confidences and support thresholds for 1st, 2nd, 3rd, etc predicted tags. Thresholds increased for each consecutive tag predicted.
\nIf there were less than 5 tags predicted from the title association rules I made further predictions based on the body association rules. Again, I tuned the confidence and support thresholds.
\nIf no predictions were made, I forced a single prediction by removing the thresholds, and failing that, I predicted "c#".
\nSince some duplicates had more than one tag set associated with them in the training data, I tried to predict the most frequently occurring tag set for a given duplicate. Surprisingly, that performed worse than simply predicting the last occurring tag set, for a given duplicate, in the training set.
\nI quite enjoyed the challenges which came with the size of this data set. More please!
", "rawMarkdown": "", "votes": null }, { "id": "36500", "postDate": "12/22/2013 09:22:28", "content": "Congratulations to all. This was a fun and challenging problem. My approach was as follows:
\n1. Clean the data by removing html tags and some punctuation, converting everything to lowercase and removing duplicate observations.
\n2. Generate tfidf vectors on title and body separately (1-grams and 2-grams), and limiting the terms to 70k due to memory. Also, the tfidf vectors were based on a sample of about 50% of the non-dup observations.
\n3. Reduced the sample to 500k observations and ran standard one-vs-rest linear regression models on the top tags (top 7000 for titles and top 3000 for bodies). Because of memory limitations, I processed the tags in groups of a couple hundred and pickled each resulting model. This produced two sets of models, one for titles and one for bodies.
\n4. Created two additional models based on whether a tag appeared in the title or in the first 50 characters of the body. For these exact matches, it was necessary first to process the titles and bodies by replacing synonyms with their tag representation (e.g., 'win7' in a title was changed to 'windows-7'). This step produced two additional models, one for titles and bodies. In both cases, I calculated the probability that the appearance of a tag term in the title or body meant that the tag was also included in the tags field.
\n5. In the end, I had four models, each of which gave the probability of a specific tag appearing for a specific observation. To merge these models, I removed duplicates (e.g. same tag produced by different models), ordered them in descending order of probability and selected the first 1 to 5 tags based on probability thresholds. The thresholds were determined by fitting to a validation set. The final thresholds were [0, .16, .28, .41, .46]. For example, the highest-probability tag was first selected; then the second if it's probability was over .16, then the third if its probability was over .28, etc..
\nMy final f1 score on the non-duplicated validation set was .526.
\nFor me, the most challenging aspect was dealing with time and memory limitations. There were a lot of factors (size of sample, number of terms in tfidf, number of tags in one-vs-rest, etc.) that affected the amount of memory and time to execute. I spent a lot of time trying to find the optimal balance of all these variables through trial-and-error. In the end, my program needed about 5 hours to generate the models and another 3 hours to generate the predictions.
\n\n
\n
", "rawMarkdown": "", "votes": null }, { "id": "36501", "postDate": "12/22/2013 09:32:55", "content": "
I have 0.76814 on private leaderboard (44th rank) (public: 0.76770). I have used information about duplicates.
\nMy minimalistic approach:
\nCreate bag of words WITHOUT any preprocessing for each post combining title and body (probably should I have done this separately). I have counted conditional probability of tag|word.
\nClassification was done by generating ranking P(tag|word)-P(tag).
\nI had problem with RAM (on laptop have only 8GB), but rewrite from Python to C helped. Also I count only these words which was in test set.
", "rawMarkdown": "", "votes": null }, { "id": "36502", "postDate": "12/22/2013 11:15:38", "content": "Hi djstrong - your idea to only use words in the test set is really nice - so obvious in hindsight :) You say you did your classification based in P(tag|word) - P(tag). Did you calculate that for all words in a question and pick the tag with the highest value, or combine the values in some other way?
\n\n
", "rawMarkdown": "", "votes": null }, { "id": "36505", "postDate": "12/22/2013 11:24:36", "content": "
Hi, thanks :)
\nYes, I calculate this for every word in question and combine the values (AFAIR by simply adding values).
\nFor example (capitalized is tag):
\nP(JAVA|java) - P(JAVA) is probably positive, but P(JAVA|bicycle)-P(JAVA) is negative. So every word in question have impact on value of (almost) every tag (also negative).
\n", "rawMarkdown": "", "votes": null }, { "id": "36506", "postDate": "12/22/2013 12:39:45", "content": "
I'm more of a computer scientist more than a mathematician, and I'm new to data science, so i'm quite happy with my results (also didn't have a massive amount of time!). Pleased to see a lot of citations to papers and models which I'll keep in mind for any further contests.
\n\n
This is a summary of what I did:
\n1. Matched up all duplicates between the Test and Training set
\n2. Saved a count of each tag per word and per bigram from each title in the Training set. (I also did this for the body, but didn't end up using it in the end). Utilised a stoplist.
\n3. During the results processing, I split the title into words and bigrams, calculated the bayesian probability of a tag given the word or bigram, and then summed up the probabilities given the tags.
\n3a. I then put some logic in to say that the tag likelihoods had to be within x% of the top tag (this was to increase the precision).
\n3b. There was also some logic in there to prevent known unlikely tags (e.g. c# java python) with were produced from the FScore calculations on the Training set. If one of these unknowns was encountered, the system would downweight these tags.
\n4. I ran some simulations of the model on the Training set using a self implemented version of the FScore calculation in order to test which variables were best (e.g. the x% for 3a).
\nCoded the whole thing up in Python (which I'm new to, I usually develop in c#) using pandas and utilised a postgres db for saving data. Also heavy usage of csv files to split, process and combine data (ala mapreduce) seeing as I only have 4GB of RAM. I think having an SSD harddrive also drastically helped with loading speeds from disk to RAM.
\n\n
", "rawMarkdown": "", "votes": null }, { "id": "36508", "postDate": "12/22/2013 14:58:20", "content": "
I used spreading activation. At the beginning I used only tags with no. of occurrences more than 100. In my last & best model I used tags which appeared more than 4 times in the dataset. It improved my f1 score by a very small amount and didn't slow my training and prediction. So I decided to go with it. All the f1-scores reported here are for predicting non-duplicate last 5000 questions in the dataset( after pruning all the duplicates using a murmur hasher applied on the title of the questions). I coded everything from scratch in python by using numpy, scipy and sklearn DictVectorizer.
\n -> predict top 6 tags and break if
->the difference in activation of two consecutive tags is more than 0.7 of the max and the activation is below 0.12.
->after predicting top 2 tags, the activation drops below 0.13.
I used my i5 dual core machine with 8gb ram. The feature selection process(from 1 million questions) took around 5 hours.The training on the entire dataset( 4.1 million) took 8 hours and the prediction for non-duplicates( 0.9 million or so) took 1.5-2 hours. RAM usage was never more than 3gb and during prediction it was only 80-100mb.
\nI approached this problem with associative rules algorithm (not sure if it’s the proper name but I never seen this approach in the books and basically it is a simplification of naive bayes approach).
\n1. Match duplicates in test set and remove multiple duplicated entries from train set.
\n2. Clean the data: I removed HTML tags and separated words with regular expression. Extracted all blocks with code from body. That seems important, as the code appears to be a strong indicator of programming language used and hence is good predictor for tag. Later on I added StanfordNLP parser http://nlp.stanford.edu/software/lex-parser.shtml to extract lemmas. Title seems to have a bigger impact then the body so title words were extracted as separate features. Code blocks were case sensitive and processed differently, plus, I extracted some syntactic features like usage of different arrows ==> or different type of comments // or #
\n3. Build associative rules for first 30 000 tags. I built rules on top of TF vectors, where TF(Word_j) = 1 + log(Count(Word_j)). The rule can be either positive or negative. For positive rule p(Tag_i|Word_j) > 0.3 and Sum (Word_j)>40. For negative rule p(Tag_i|Word_j) = 0 and Count(Word_j) > 50 000, where I had different weights for different type of rules and had to run multiple CV tests to select proper value. I had to run all the rules on both train and test set. Train set run was used to weight them properly. This alone seems to give ~ 0.78 and the next major improvement was introducing bigrams. I used a variation of Bloom Filter to extract candidate bigrams and had to run my algorithm in multiple stages, but that essentially pushed my score up to 0.8.
\n4. Naïve Bayes on the first 2000 tags. I mixed results of this model into my core algorithm whenever I had 0 tags predicted.
\n5. VowpalWabbit linear model on the first 20 tags. This marginally improved results but I decided to keep it anyway.
\n6. The last step was to predict some additional tags based on already predicted tags. I only used predictions with high confidence. Basically I found all cases there p(Tag_i|Tag_j) is high and used them to improve predictions.
", "rawMarkdown": "", "votes": null }, { "id": "36520", "postDate": "12/22/2013 19:16:53", "content": "My approach:
\n1. Take duplicates out like many others did
\n2. Calculate the association between all 1-gram and 2-grams in title and body. I kept the first 400 chars and last 100 of body to keep the size under control. I keep only the top 30 tags for each 1-gram and 2-gram
\n3. Predict the tags by combining all the tags associated with all the 1-gram and 2-grams in Title+body. Each 1/2 gram's impact is weighted by its support and entropy. The entropy part only improve the score very marginally.
\n4. Take the top tags based on their score. The cut off threshold is determined by the ratio between the score of tag_k and the average score of all tags with higher scores than k.
\n5. I scored the above on 200k posts in training, and computed the bias for each predicted tag. Some tags have more FP than FN, some have more FN than FP. The goal is to have FP and FN about equal.
\n6. Score test data and adjust for the tag bias from step 5, i.e., if a tag has #FP>#FN, decrease its score, and vice versa.
\nNote:
\nThere are quite a number of tuning parameters that I "hand optimized" by evaluating the impact on 2% of training data I kept as a validation dataset.
\nTitle is much more important than body -- I gave a 3:1 weight to title (but this is somewhat cancelled out by the fact that there are more words in body usually)
\nI thought about doing some <code></code> extraction but didn't get time to do so.
\nI also extracted quoted posts in the body and tried to match that with training, just like the duplicates. But this barely improves the result. So it seems the association of tags between posts and quoted posts are not very strong.
\nI also tried some TF-IDF based methods but didn't get good results.
\nI didn't spend too much time on "NLP" like tasks such as stemming/tagging, etc. Part of the reason is time required, and part of reason is that the posts are not very "natural", compared to texts from news archives and books. I figure I probably would lose as much information on the code part, v.s. what I can gain in the "natural language" part.
\n\n
The "RIG":
\nAs noted by everyone, the bottleneck of this competition is RAM. I am fortunate enough to have access to a machine with 256GB ram to try out different parameters.
\nThe final solution would run on a desktop with 64GB of ram in about 4 hours. The training only takes about 1.5 hour and scoring will take 2.5 -- partially because it runs out of ram. Training of the model will eat up almost all of the 64GB, and the rest will swap to disk. There has to be at least 90GB swap space for this to run with out error.
\nI used Python, which is not very "memory efficient". If we were to compete on "efficiency" I think c/c++ would be necessary.
", "rawMarkdown": "", "votes": null }, { "id": "36557", "postDate": "12/23/2013 09:37:13", "content": "[quote=djstrong;36501]I have 0.76814 on private leaderboard (44th rank) (public: 0.76770). I have used information about duplicates.My minimalistic approach:Create
bag of words WITHOUT any preprocessing for each post combining title
and body (probably should I have done this separately). I have counted
conditional probability of tag|word.Classification was done by generating ranking P(tag|word)-P(tag).I
had problem with RAM (on laptop have only 8GB), but rewrite from Python
to C helped. Also I count only these words which was in test set.[/quote]
As an additional curiosity, how to decide the number of tags for your prediction ?
", "rawMarkdown": "", "votes": null }, { "id": "36558", "postDate": "12/23/2013 09:38:59", "content": "[quote=Triskelion;36457]
\nTried a different challenge. From .csv's to submission in under 1 hour, with less than 1GB of memory, in 30 days.
\nThis inspired further constraints: No use of external algorithm libraries, single-pass, online learning, parallel learning.
\nStarted with observation that top 200 tags are good for ~50% of all tag occurrences.
\nFirst algorithm idea: 75% of documents with the word "stylesheet" are tagged with "css". So Bro Bayes.
\nUse of a FTS4 Sqlite database played it safe with memory, but gave me too much to count and update (couldn't resist), so I predicted this would not meet time constraint. If you give yourself 30 minutes for the predictions then you are predicting at 1k+ a second.
\nI figured out the dupes in the train and test sets, and the dupes between them. To dedupe you could match the post titles. Using that with only a tag dictionary generated from the train set and substring matching those against the titles in the test set gave you a f1-score of around #20 at the leaderboard. This was 3 weeks ago. It took about 700Mb of memory and under 3 minutes.
\nStemming text was too slow. Preprocessing thus was more cleaning of text and removing stopwords. For stop words I first tried a top 10k stopword file from Google with the tag dictionary words removed. I found that counter to the competition rules, so I generated a stopword file from the train set with words that are found with many different tags attached, and hand-picked a top 200.
\nTrained Vowpal Wabbit on top 40 tags. Got a good result, but was approaching 1 hour. Trained it again on top 200 tags and got a better result, but took 3 hours. I believe I saw that top 1024 tags are good for over 70% of all tag occurrences, but realized this route was not going to meet constraints.
\nI ended up using 3 count dictionaries. One tag counter, one word counter and one word-to-tag counter. This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory. Not using an external big stopword dictionary meant I had to prune the word-to-tag dictionary with every 100k chunk. Once the number of tags per keyword reaches a certain threshold, I take a top n and only increase the counts of those top n tags in the future.
\nThis gave me my result that met all my constraints except for one, memory. It took 1.7Gb max. for model creation. Predictions with 1.1Gb on only the title words.
\nTried to do keyword extraction without a dictionary or statistical historical knowledge. First term frequency on a top 10k stopword removed file. Then co-occurrence: this paper http://ymatsuo.com/papers/flairs03.pdf found me reasonable results. All took either too long to run or did not add substantially to a dictionary-based approach.
\nFun things. Ran online LDA over the test set to obtain topics. Very clear topics appeared like all string manipulation commands. "adolf-hitler" is a tag. The hardest question to predict only by title was something like: "why?! why! why! why! why?!?!", because after stopwords and cleaning nothing remained :).
\nWhat I learned: Simply count ALL the things. Don't count 'em twice.
\n[/quote]
\n\n
Thanks for your conscience share. However one question ?
What you mean by "This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory"
Can You give a better explanation ?
[quote=Owen;36520]
\n4. Take the top tags based on their score. The cut off threshold is determined by the ratio between the score of tag_k and the average score of all tags with higher scores than k.
\n5. I scored the above on 200k posts in training, and computed the bias for each predicted tag. Some tags have more FP than FN, some have more FN than FP. The goal is to have FP and FN about equal.
\n6. Score test data and adjust for the tag bias from step 5, i.e., if a tag has #FP>#FN, decrease its score, and vice versa.
\n[/quote]
\nFirst of all Congratulations for the Rank :)
\nOne question;
\nIs the cut of threshold defined by tag_k is used for the prediction of tag_k or k is an free parameter that you set it via x-validation ??
\n\n
By the way, your method is very much alike to http://iccm-conference.org/2013-proceedings/papers/0077/paper0077.pdf
\n[quote=Eren Gölge;36558]
\n\n
What you mean by "This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory"
Can You give a better explanation ?
[/quote]
\nI'll try. The "tf" part in tf*idf stands for term frequency. This can prevent a bias to longer documents.
\nFor example: two documents both mention the term "Kaggle" one time. Document A has 10 words, document B has 1000 words. A very simple "tf" can be calculated like 1/10 for Document A and 1/1000 for Document B. So Document A is probably more relevant to the term "Kaggle" than Document B. Because 10% of Document A is about the term "Kaggle" and only 0.1% of Document B is about the term "Kaggle".
\n"idf" stands for inverse document frequency. This can prevent a bias to common words. Let's say we have 20 documents total. The IDF for a very rare word that appears only in 1 of those documents would simply be 20/1 = 20. The IDF for a fairly common word that appears in half of those documents would be 20/10 = 2. The IDF for a word that appears in all of those documents would be 20/20 = 1.
\nSince tf*idf is the product of tf and idf, a very common word would get a lower tf*idf score (idf=1, so simply the term frequency).
\nTo calculate idf one needs to check if a word is present in all of the documents that you have observed so far. This is the costly part of tf*idf. It requires lots of memory or multiple passes to calculate idf.
\nSince idf is used to prevent bias to common words, we could substitute with inverse word frequency. If word A was counted 990 times vs. word B that was counted 10 times, the iwf would be 1000 / 990 vs. 1000 / 10. Word B gets a higher iwf score because it is less common.
\nTo prevent bias to common tags, I also did inverse tag frequency. Some words appear a lot tagged with common tags, not because those words are descriptive to those tags, but because the tag is added to so many documents.
\nI also gave a little boost to words that are exact matches of tags (so I didn't need to substring match anymore), and took into account the distance of the word to the beginning of a paragraph or sentence (reasoning, more important keywords are found early on in sentences, paragraphs and texts).
\nFinally I did some logs to smooth it out. TF*IDF can be as complex as you want to. Binary, relative, logarithms etc. The Yahoo Learning to Rank competition has solid practice-tested TF*IDF formulas if you want to know more.
\nWith this tf*iwf*itf score for every word, you can see that my model gives tags like "php" and "ruby-on-rails" a higher score for the word "bug" than tags like "c#" or ".net", even though the word "bug" may be counted more in documents tagged "c#" or ".net".
", "rawMarkdown": "", "votes": null }, { "id": "36573", "postDate": "12/23/2013 14:35:55", "content": "How did you fit your input to tfidf vectorizer at once while you have memory problem? Can anyone share sklearn code about that?
\n\n
Is there any way to fit partially for tfidf?
", "rawMarkdown": "", "votes": null }, { "id": "36578", "postDate": "12/23/2013 15:05:18", "content": "[quote=Güngör Yılmaz;36573]
\nHow did you fit your input to tfidf vectorizer at once while you have memory problem? Can anyone share sklearn code about that?
\nIs there any way to fit partially for tfidf?
\n[/quote]
\nThis may be relevant reading: http://scikit-learn.org/stable/modules/feature_extraction.html#vectorizing-a-large-text-corpus-with-the-hashing-trick
\nAs for partial fit, I think some people in this very thread have fit tfidf on chunks, instead of the entire data set (for example tfidf on a 500k chunk, then use that to transform the rest of the data).
", "rawMarkdown": "", "votes": null }, { "id": "36586", "postDate": "12/23/2013 19:15:25", "content": "@Eren: yes I did read that paper, which is one of the better (more practical) ones I managed to find.
\nOn the topic of cut-off threshold of k, it is a standalone parameter I set through cross-validation. First I sort all the Tags that was positively correlated with 1/2 grams by the strength of correlation. The threshold is applied to the ratio of the correlation of Tag_k to the average correlation of Tag_1 to Tag_k-1. For each position (0-4) there is a different k.
\nThis will determine the number of Tags to predict.
\nI did some very "back of the napkin" theoretical analysis on the incremental impact of adding a Tag with Prob(TP)=pk to the existing prediction with given number of TP, FP, and FN. The thresholding approach is derived from this analysis to maximize expected F1.
", "rawMarkdown": "", "votes": null }, { "id": "36607", "postDate": "12/23/2013 23:39:23", "content": "Hello everyone, just wanted to share my approach as well. Like most of you, the biggest constraint for me was memory. I was equipped with a 16GB linux machine (after exhausting my 8GB Macbook), but more memory would have allowed me a bit more flexibility with my model.
\n1. I initially performed tokenization by combining, lowercasing and removing any non-alphanumeric characters (except '+' and '#') from the title/body after removing html tags. Extra whitespace was removed and I was left with a space-delimited file containing the tokens. I used a bag-of-words model, so the first goal was to extract the most relevant tokens from this list - this was done by maximizing the frequency of words within each sample while minimizing their frequency across all the documents (hence, I was trying to find tokens that would maximize tf-idf values). This produced a matrix with 40K features that represented the frequency for each unigram.
\n2. Term-frequency (tf) was done by taking the log and dividing by the maximum log-transformed frequency for each sample (log-transformation was done after adding 1).
\n3. The response matrix was essentially a giant indicator matrix for every one of the 42050 tags in the training set. A separate response matrix contained the tag count data represented by a 5 column indicator matrix.
\n4. Learning was done using least squares using a one vs all approach only with the non-duplicated data set. Now at this point, I created 2 models: one to estimate the most probable tags given the feature matrix, the other to estimate the number of tags given the feature matrix. After a bit of experimentation, the first model was giving the best f1 scores after performing dimensionality reduction on the feature matrix - accomplished by using the eigendecomposition of the covariance matrix, trimming off eigenvectors with lower eigenvalues (~5K dimensions omitted) and projecting the mean-centered feature matrix into this new space.
\nThis reduced model was used to find the most likely tags, while the other model (which didn't use any dimensionality reduction) predicted the number of tags.
\n5. A significant amount of optimization was necessary to do least squares on these huge matrices. Both the feature and response matrices were represented using a sparse matrix data structure. Everything was implemented in C and I ended up using single-precision floating points (rather than double) for all the calculations - this way a 40K x 40K matrix of eigenvectors would take about 6.4GB and could be stored in RAM. I also used OpenBLAS, which was a lot more efficient than the default BLAS on linux.
\n", "rawMarkdown": "", "votes": null }, { "id": "36639", "postDate": "12/25/2013 10:20:16", "content": "
[quote=Eren Gölge;36557]As an additional curiosity, how to decide the number of tags for your prediction ?[/quote]
\nUnfortunately I did not any smart solution for this. I cut off whan next tag hav value 2 times less than previous. If is not then I take 5 tags.
", "rawMarkdown": "", "votes": null }, { "id": "36831", "postDate": "12/30/2013 21:52:08", "content": "It seems the word-tag association, although needing a lot of memory works very well for this data set! Very creative approach, I didn't think of this.
\nI created my solution on a laptop, and it runs pretty fast. In a few hours it delivers the result. The solution was created in Python. My solution is quite straightforward, although I am sure with a bit more tweaking the results can be much better. I just used Logistic Ridge Regression without cross validating, and without averaging with other estimators or bagging.
\nHere a quick overview of what I did:
\nMy Solution:
\nLater I got much improved results for individual tags by averaging Logistic Regression and Random Forests (using the 25 best features). I however did not have the time to run this on all examples due to work obligations the last days of the competition.
", "rawMarkdown": "", "votes": null }, { "id": "37398", "postDate": "01/15/2014 11:57:40", "content": "[quote=Petko Minkov;36452]
\nMy basic idea is to find predictive words. A predictive word for a tag is a word with high ratio of posts in which the word is contained and are labeled with that tag vs posts in which this word is contained.
\nI have the following pipeline.
\nStep 1: Read input in python, tokenize text and create features. Title words are prefixed by "T_", words in code segments are prefixed by "C_". That's because a "main" in the code might have higher predictive power than a "main" in the text. I got a big bump by doing the same for the title with the "T_" prefix. I guess something like "java" in the title is more predictive than "java" in the text.
\nStep 2: C++ code to remove words with very high and very low occurrence. I remove the top ~1000 words and the bottom 80% of the words. The bottom 80% have about 7 occurrences in the text, so I can't even use them reliably to predict.
\nStep 3: Train a classifier. C++ again. This is just computing the ratio which I pointed at earlier. I keep a label to words matrix in memory. To save memory, I hash label and words. The output is a map from words to labels which the words predict with high probability. So "C_iostream" might predict C++ with probability of 0.79. I also produce bigrams from words in the title. Bigrams from words in the post were blowing up my memory usage.
\nStep 4: Prediction. I go through every word and see what labels does it predict and keep the maximum probability for each label that comes up. I select the top 5 predictions and output them. Most of the time there are less than 5 predictions.
\nDupe detection is done by using a djb hash on the label text.
\nI tried coding a naive bayes with x2 feature selection step in both C++ and using scikit, but didn't get very far with F1.
\n[/quote]
\nHow do you tokenize the text in Python?
", "rawMarkdown": "", "votes": null }, { "id": "37401", "postDate": "01/15/2014 13:22:36", "content": "[quote=JorgeBlasco;37398]
\nHow do you tokenize the text in Python?
\n[/quote]
\nIn its most basic form (called unigram tokenization):
\nimport re
s = 'This text will get tokenized'tokens = re.findall('\\w+',s)
#['This', 'text', 'will', 'get', 'tokenized']
Or did I misunderstand the question?
", "rawMarkdown": "", "votes": null }, { "id": "37448", "postDate": "01/16/2014 08:32:40", "content": "@Owen: Is there any chance that you could make your code public?
", "rawMarkdown": "", "votes": null }, { "id": "37457", "postDate": "01/16/2014 13:28:06", "content": "I feel the need to say that the challenge was very interesting. It is a shame that i haven't finished my solution until the end of the challenge but maybe it would be interesting to someone so here it is:
\nThe idea is to compare each question title and description to the collection of tag titles and descriptions like its done in categorization tasks. I've founded an article which compares categorization methods and decided that Weighted Inverse Document Frequency (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.49.7015&rep=rep1&type=pdf ) may perform best.
\nWIDF vectors
\n1. Create a collection of words vs its frequencies for each tag in a training set. There should be two collections for each tag - one for titles and one for question bodies.
2. Create a key value dictionary of words vs its frequencies for all training set.
3. Create a key value dictionary of words vs its occurrences in the current document for each title and body.
3. Create WIDF vectors for each tag collection (for both titles and bodies as it is in 1st list item).
Training by the distance
\n1. Compare the similarity between question title WIDF vector and each tag collection using cosine similarity function. The result should be that cosine angle between related tags is closer to 1 and to unrelated tags is closer to 0.
2. Also compare the similarity between question body WIDF vector the same as title was compared. Now we should have X as similarity between titles and Y as similarity between bodies.
3. Plot these similarity points. Red pints for unrelated tags and green points for related. This should help to define weather Euclidean or Mahalanobis distance is needed.
4. Find the best Euclidean/Mahalanobis distance by counting precision/recall for each distance step.
Speed
\nAs you could see this method requires a lot of resources to process the text. I've tried using multithreaded C++ and MySql query pools but it was too slow so the best way i think should be to write Hadoop mapreduce chain for it.
\nI've missed the tokenizer part because i think this algorithm could perform without anything more than splitting the text by \\w, but it would be interesting to test some more options.
\n", "rawMarkdown": "", "votes": null }, { "id": "128701", "postDate": "07/22/2016 20:14:35", "content": "
sklearn now supports partial fit on OneVsRestClassifier! (development branch Jan 2015).
", "rawMarkdown": "sklearn now supports partial fit on OneVsRestClassifier! (development branch Jan 2015).", "votes": null } ], "comments": [ { "id": 36448, "author_name": "jerboa", "author_url": "", "post_date": "12/21/2013 04:29:03", "content": "I went in as anonymous, finished 42nd with 0.77091 which was fairly robust compared to my public 0.77100 last submission score.
\nApproach:
\nsklearn, 16GB RAM on OSX, goal of making a pipeline capable of submitting in a business day.
\nThe approach then did the following after having each classifier predict against the test set:
\nFirst, if the test example was in the training set, predict the training tags.
\nFor anything left, take the union of the output of classifiers 1 and 2, and toss out the lowest probability tag if the number of predictions was over 5.
\nFor anything left unpredicted after that, take the prediction from the appropriate cluster that the test example belonged to.
\nIf there still isn't a prediction after that, use the top 5 tags.
\nIt turns out that Classifier 2 (the vocabulary approach) made Classifier 3 (the lexical approach) unnecessary.
\n\n
Dead Ends
\nBefore I knew what I was doing, I tried clustering the training set and then training OneVsRest classifiers for each cluster. This was intended to make it easier to train more tags in less memory but letting me train say, 450 OvR classifiers on a small number of examples, but it got comparable to slightly inferior results with much more training time. Ultimately, just training one or two OvR classifiers that used somewhat skewed vectorization approaches for their features produced better results in less time.
\nIf I had to do it again, I'd go "all in" on using a vocabulary approach with TfidfVectorizer and then spend my time working on preprocessing. Oddly, many substitutions and rollups of synonyms degraded the results. I haven't figured out why yet.
\nUsing SVD to reduce dimensions on raw Tfidf wasted a ton of time and didn't make a difference. You're much better off defining your vocabulary, or using the min_df to reduce dimensions.
\nI had a fixation on exposing myself to as many training examples as possible--turns out that with my skewed training set, 80,000 examples (42000 culled, plus 38000 random) was often good enough, so it was much better to spend computation on more features rather than more training examples.
\nAs others pointed out, when you train a generic model, you need a 0.024 improvement in your test set to get a 0.01 improvement in your submission score once you account for identical posts. That makes 42nd look less impressive. I did one test where I calculated an f1 score if I could pick the best from the predictions of the 10 classifiers (sans identical matches) and I think it was something like 0.57. I didn't have time to figure out a good ensemble scheme or weighting approach to get closer to that theoretical max the models could produce. Part of the problem was that sklearn and SGDClassifier require settings to get raw probabilities that might help--and these settings got poorer results than the "hinge" approach I landed on.
\nKey Insights
\nIf you're working with raw tokenized text, ngrams are your friend. As is min_df
\nUsing a vocabulary with TfidfVectorizer is a great idea, particular with preprocessing.
\nNot much gain going beyond training on the top 5000 tags. It'll move you up a few spots, but nothing earth shaking.
\n\n
", "votes": null, "replies": [] }, { "id": 36450, "author_name": "damienmelksham", "author_url": "", "post_date": "12/21/2013 04:51:33", "content": "
Congrats to all, its been fun, educational, entertaining, and...controversial :P This will be long, i will try to keep it as short as possible.
\n\n
PART I.
\n----------------------------------------
\nSoftware: I don't like to enter into programming language debates, but this will come up. In my experience, Python, and especially R, have been too slow for industrial/large scale/optimized custom-designed work. C is not exactly a joy to code in. I'm very good with SAS, but its specialized in what it does, and even if i wanted to use it, I don't have access to a licence via which using it for this competition wouldn't be a fire-able offense.
\nSubsequently, I coded the entire thing in Lisp. Common Lisp, SBCL, using emacs and SLIME on Linux Mint to be precise. I won't bore you with the history of Lisp. Read up on it if you're curious. Its one of the few languages that is relatively dynamic/high-level/low-level/compiled/REPL at the same time, but with atypical polish-prefix syntax.
\nIf my memory serves me correctly, the libraries I used were read-csv, and cl-utilities...which probably saved a bit of time here and there. Read-csv was used to parse the original raw data into lisp forms that could be written out in printed representation to disk. I split the body of the post into a code section and a non-code section, based roughly upon being between the <code> </code> tags. I lower-cased everything, and for the body text that wasn't deemed to be code, I roughly removed the HTML tags via an algorithm that searched for text between the '>' and '<' characters, since i considered parsing it more accurately to be outside help :P In the end I merged to the two back together, but there's good reason to believe that this is another area for potential improvement.
\nThe first few important functions I wrote were:
\ndata accessing macro: For exploratory work. I coded up a form that read in each record one at a time from the main file and looped through the input text file, kind of like the SAS data step. Each field in each record was referenced with ID, BODY, TITLE, CODE, and TAGS respectively. In a tribute to SAS, I made a variable _N_ which was an automatic variable that kept track of the observation number irrespective of the ID. I made some additional tools that I never really had to use. One was an array which recorded the position of each record within the file via the array index. A macro/function would use this to enable arbitrary random access to records within the original data file based upon their IDs, but I never really needed to use it.
\nn-gram generators: I wrote a character-ngram, a word-ngram, and a unique-ngram-function. Each is capable of returning an array of arbitrary length n-grams within a text. I only used 1-gram words in my final model, but its perfectly reasonable to assume that using more evidence would be a further improvement. Phrases like common lisp, for instance, would presumably be picked up under word-2-grams, but lisp turns up as significant evidence for both the common-lisp tag, and the lisp tag. Character n-grams would allow for evidence to be gathered irrespective of arbitrary separators, and would in some way enable negation of some of my sloppy parsing/cleaning/user entry.
\n-----------------
\nNow before I go on there were a number of irritating controversies which probably deserve some commentary. When I discovered the competition I started because I believed that it was literally a competition on "tag prediction" with a test set and a training set. Two things about that...
\nFirstly, I observed that the scores in the leader-board just weren't theoretically possible for such a task. An observation I stand by. Subsequently to be sure, I programmed up simulator, which let me enter some information on number of tags, accuracy of predictions for tags and number of tags in a post, and tell me what score would be returned for such performance. I did, and still do consider, duplicates to be against the spirit/goal of such a task, so was put in the awkward position of either calling out the scores and potentially pissing off a lot of people and possibly hurt my chances of working at facebook to find out what was going on, or keeping quiet and use it to my advantage to game the system for a high score. I chose the former. Mr Chester Grant soon started the next thread exposing the duplicate problem. My point about incentives to dishonesty for such a competition, i feel, still stands.
\nThere was one more problem, which has to do with the quality metric. For much of the time I was working under the belief that the metric being used was the F1 measure as calculated for true positives/falsepositives/falsenegatives across all posts (which is what I interpreted as the "average F1 measure"), rather than an F1 measure taken within each post and then each F1 score averaged. This is a subtle, but quite important difference. One results in a totally different strategy for score maximisation compared to the other. For one, the first gives you an incentive to not make spurious and unnecessary guesses, while the later gives you at least 1 free guess that it is optimal to take in all possible strategies. I'm sure I'm not the only person who misunderstood this, and that such an explanation on scoring could be made clearer and more prominent.
\n---------------
\nDuplicate Removal: This to me was a relatively unimportant part of the competition, so I spent about a day or two coding it up, deduping the training file, and then making the links between the test and train files for fun :P
\nA deterministic join on parts of the title/body gets you most of them. You can do a LITTLE bit better by going fuzzy/data linking. As I've said earlier, this is too close to my actual work, so I didn't spend much time on this.
\nTo fuzzily link the duplicates, I held the body of the test set in memory, and created a hash table key'd on individual words. I already had a word count frequency loaded into another hash table, so I used the 5 least common words in each post, and pushed the ID of that post onto the hash table key'd on those words.
\nI then coded up a quick implementation of the Dice-sorenson coefficient, which would act as a simple distance metric between any two posts. I read through the train set one record at a time, found the bottom 5 words in each post with the smallest frequency, and used the dice function to compare them to any of the original posts indexed by those words in the test set. I had one more hash table, keyed on IDs in the test set, in which I stored the highest scoring dice metric and subsequent ID in the train set for each individual ID. I arbitrarily chose a cutoff, and those ID's in the test set with a dice metric score to a record in the train set above this cutoff, i deemed duplicates. There's lots of theory here I will neither bore you with, nor explain, since I didn't bother being too careful, but I did notice during some reviews of these links the problems with certain short common phrased questions "What's the difference between "X" and "Y"?" Only two words different, but obviously different questions that look very similar :P
\nMoving on... to be cont'...
", "votes": null, "replies": [] }, { "id": 36451, "author_name": "monami", "author_url": "", "post_date": "12/21/2013 05:06:44", "content": "Thanks for posting the details! With OneVsRestClassifier did you use multi-label format(given as is) with LabelBinarizer or you changed input format of your input rows somehow?
", "votes": null, "replies": [] }, { "id": 36452, "author_name": "pminkov", "author_url": "", "post_date": "12/21/2013 05:22:35", "content": "My basic idea is to find predictive words. A predictive word for a tag is a word with high ratio of posts in which the word is contained and are labeled with that tag vs posts in which this word is contained.
\nI have the following pipeline.
\nStep 1: Read input in python, tokenize text and create features. Title words are prefixed by "T_", words in code segments are prefixed by "C_". That's because a "main" in the code might have higher predictive power than a "main" in the text. I got a big bump by doing the same for the title with the "T_" prefix. I guess something like "java" in the title is more predictive than "java" in the text.
\nStep 2: C++ code to remove words with very high and very low occurrence. I remove the top ~1000 words and the bottom 80% of the words. The bottom 80% have about 7 occurrences in the text, so I can't even use them reliably to predict.
\nStep 3: Train a classifier. C++ again. This is just computing the ratio which I pointed at earlier. I keep a label to words matrix in memory. To save memory, I hash label and words. The output is a map from words to labels which the words predict with high probability. So "C_iostream" might predict C++ with probability of 0.79. I also produce bigrams from words in the title. Bigrams from words in the post were blowing up my memory usage.
\nStep 4: Prediction. I go through every word and see what labels does it predict and keep the maximum probability for each label that comes up. I select the top 5 predictions and output them. Most of the time there are less than 5 predictions.
\nDupe detection is done by using a djb hash on the label text.
\nI tried coding a naive bayes with x2 feature selection step in both C++ and using scikit, but didn't get very far with F1.
", "votes": null, "replies": [] }, { "id": 36453, "author_name": "chintansparikh", "author_url": "", "post_date": "12/21/2013 05:36:01", "content": "I've worked on this challenge as part of class project (cs229 stanford), attached final report for reference.
\nReally enjoyed working on this, although didn't get as much time to work on tuning up the submission in the end.
\nLet me know your feedback.
", "votes": null, "replies": [] }, { "id": 36454, "author_name": "damienmelksham", "author_url": "", "post_date": "12/21/2013 06:18:23", "content": "PART II
\nThis is where it probably gets interesting, and I'm probably going to struggle to explain this without diagrams...
\nLets get this out of the way. I didn't use stop words, and I don't generally like matrices. In many real world problems they're too inefficient.
\nMy initial plan was to use a simple bayes formula on the individual words within title/body/code of a post. This involved calculating the requisite probabilities of observing a given word for each tag/not-tag combination...which is not exactly a small task. So...
\n
I am guessing everyone here is able to generate some basic summary stats. I calculated the relevant ones and stored them in memory: number of posts, probability of each tag, frequency for each word, etc, etc. One efficiency i noted here was that you don't need to store frequencies for words that only appear once. If you observe a word in the train set later, and its not contained in such a frequency hash, you DEDUCE that it must be a word that is only observed once in the train set, and it is observed once only in a post with the qualities of the post in which you have currently observed it.
Two-deep-hashes:
\nI created some functions to work with double-keyed hashes. That is to say, a hash-table whose key is a tag, and which maps each key to another hash-table that is keyed on words.
\nSo for example, if we are training on the tags "c#" and "java" and we observe the word "c#" in a post with tags "c#" and "java", the first hash table keys are "c#" and "java", which directs us to two more hash-tables, both of these are then accessed with keys of "c#". The values in these hashes then have their requisite counts incremented. Upon one iteration for the tags you are training through the training set, you can then see the frequency of words for each tag.
\nTo find out how many instances of that tag did not contain that word, and how many posts that didn't carry those tags contained that word, you iterate through the two-deep-hash-tables values and compare the requisite key/values to the original word frequencies/tag frequencies.
\nTraining and Evidence
\nThis is all well and good, but you've now got a two-deep-hash key'd on tags->words->frequencies. What you need for your classifier to make decisions are data structures keyed on words->tags->evidence, so each time you observe a word, you know how much evidence such an observation contributes to each tag. For this I chose a data structure that is a hash, keyed on words, mapping to an array of #(tag evidence) arrays, which can be iterated across incredibly fast. Each time a word is observed in a post, you hash it, which takes you to an array of evidence for requisite tags. The function remember-inverse-two-deep-hash is responsible for converting information from the two-deep-hash in the training phase into that needed for the classification phase.
\nThe functions save-valuable-words and load-valuable-words are responsible for writing the current valuable word->tag-evidence structures out to disk, and reading them in from disk respectively.
\nOne optimization I included relatively late was the decision to change the way load-valuable-words worked to ensure that it would resolve any clashes between evidence it reads in from multiple files. This means that you can theoretically create valuable-word structures from multiple files, allowing some form of combinations of different amounts of evidence, or allowing a later loaded file to overwrite evidence loaded from an earlier file.
\nOf course, this is all still relatively romantic. The resulting structures created from these are still far too big to store all possible evidence at once, even if it doesn't have to worry about sparsity quite like a matrix.
\nThat's where the cull-valuable-words/cull-two-deep-hash functions come in. They are relatively blunt, but cut down on the amount of work a busy person like myself has to do :P
\nThey iterate through the valuable-words/two-deep-hash structures, eliminating mappings that don't meet basic requirements of minimum probability/evidence/frequencies. Hence I don't worry about stop words, because such words will be culled by such functions during their sweep, because by definition, they won't be contributing very much evidence to particular tags, and if they aren't eliminated, then they probably aren't stop words for that particular tag.
\nTo classify a post, there's functions of produce-evidence-array and make-simple-prediction. The first uses the valuable words to produce a ranked prediction of all tags for which relevant words have been observed. The second uses the produced evidence array with a small number of parameters to make a prediction of tags for each post: say the maximum amount of evidence required to make a prediction for example.
\nQuality/Administrative Functions
\nThere are two more aspects which might need direct commentary. These are the very basic functions random-sampler and judgement. The first is a simple way of iterating through the training set and holding a certain proportion of posts in memory. The second can take predictions from the like of make-simple-prediction (or indeed any list of tags) and compare them to a truth list of tags to produce a representive F1 score. It actually operates as several functions sharing a closure, so you can just keep feeding it records, and it increments stats as it goes, and you can ask it at any time to report how the F1 score is doing so far, check how another prediction went, or reset it.
\nI think that's basically it...i hope i'm not forgetting anything...
\nImprovements/lessons learnt
\nThere's actually a fair number of improvements you can make to my model not already mentioned. Two obvious ones i thought of include better use of the evidence-culling functions. They are currently quite blunt, but you can change them into higher-order-functions such that their behaviors are driven more directly by the probability of observing a particular tag. This leads me to my second improvement. I know that my current method doesn't produce an optimal distribution of predicted tags, and optimizing this would likely result in some decent score improvements.
\nAlthough I haven't tested it, this model provides a possible claimant to the "good model with very little memory" aspect, you would just have to be much more violent with using the culling functions to make the evidence fit into memory. Classification is a relatively memory stable operation once the valuable words have been loaded, since all that happens is that a record is read in, and its classification is written out to disk. Of course you will do worse if you use less memory, but I'm not sure to what degree.
\nThe classification is, relative to most of the other entries i've heard of so far, fast. Indeed it is what I would call practically real-time. It runs in about as fast as it takes to read the data in from disk and write out the results, although processing each uncleaned-unformatted record to establish the unique words to fit into the classification function would slow it down, i don't think we're talking hours here.
\nEdit: Oh, just remembered, my submitted versions effectively culled records that didn't contribute a certain amount of evidence for a tag. I'm sure you could also add some evidence which would make certain tags less likely if a certain word is observed, but those wouldn't have been included in my model.
\nEdit2: Actually, there's a fair few improvements I can think of, but didn't get round to implementation. Another one is integrating post length stats into the prediction. My current entry basically has one classification rule across the entire set, which is obviously suboptimal. Post length does indeed correlate with the number of tags, and the amount of evidence that could ever be collected for a post, so this is another fix if one really wanted to get that top spot :P
\nEdit 3: In case it wasn't clear, there were different amounts of evidence for seeing a word in the title vs seeing a word in the body of a post.
", "votes": null, "replies": [] }, { "id": 36457, "author_name": "triskelion", "author_url": "", "post_date": "12/21/2013 10:31:48", "content": "Tried a different challenge. From .csv's to submission in under 1 hour, with less than 1GB of memory, in 30 days.
\nThis inspired further constraints: No use of external algorithm libraries, single-pass, online learning, parallel learning.
\nStarted with observation that top 200 tags are good for ~50% of all tag occurrences.
\nFirst algorithm idea: 75% of documents with the word "stylesheet" are tagged with "css". So Bro Bayes.
\nUse of a FTS4 Sqlite database played it safe with memory, but gave me too much to count and update (couldn't resist), so I predicted this would not meet time constraint. If you give yourself 30 minutes for the predictions then you are predicting at 1k+ a second.
\nI figured out the dupes in the train and test sets, and the dupes between them. To dedupe you could match the post titles. Using that with only a tag dictionary generated from the train set and substring matching those against the titles in the test set gave you a f1-score of around #20 at the leaderboard. This was 3 weeks ago. It took about 700Mb of memory and under 3 minutes.
\nStemming text was too slow. Preprocessing thus was more cleaning of text and removing stopwords. For stop words I first tried a top 10k stopword file from Google with the tag dictionary words removed. I found that counter to the competition rules, so I generated a stopword file from the train set with words that are found with many different tags attached, and hand-picked a top 200.
\nTrained Vowpal Wabbit on top 40 tags. Got a good result, but was approaching 1 hour. Trained it again on top 200 tags and got a better result, but took 3 hours. I believe I saw that top 1024 tags are good for over 70% of all tag occurrences, but realized this route was not going to meet constraints.
\nI ended up using 3 count dictionaries. One tag counter, one word counter and one word-to-tag counter. This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory. Not using an external big stopword dictionary meant I had to prune the word-to-tag dictionary with every 100k chunk. Once the number of tags per keyword reaches a certain threshold, I take a top n and only increase the counts of those top n tags in the future.
\nThis gave me my result that met all my constraints except for one, memory. It took 1.7Gb max. for model creation. Predictions with 1.1Gb on only the title words.
\nTried to do keyword extraction without a dictionary or statistical historical knowledge. First term frequency on a top 10k stopword removed file. Then co-occurrence: this paper http://ymatsuo.com/papers/flairs03.pdf found me reasonable results. All took either too long to run or did not add substantially to a dictionary-based approach.
\nFun things. Ran online LDA over the test set to obtain topics. Very clear topics appeared like all string manipulation commands. "adolf-hitler" is a tag. The hardest question to predict only by title was something like: "why?! why! why! why! why?!?!", because after stopwords and cleaning nothing remained :).
\nWhat I learned: Simply count ALL the things. Don't count 'em twice.
", "votes": null, "replies": [] }, { "id": 36463, "author_name": "peteralexander0", "author_url": "", "post_date": "12/21/2013 12:47:22", "content": "My approach seems too simple compared to a lot of these. Anyway, this got me 0.78 at 26th on the leaderboard. Everything coded from scratch in the D programming language.
\nVectorization
\nBag of words with binary TF (i.e. 1 if word there, 0 otherwise). Title words counted as separate features. Features with less than 20 total occurrences removed. No stop words removed or stemming (stop words are often very important in programming languages!)
\nTraining
\nUsed only the first million samples due to me only starting this method a few days ago. Just trained a one-vs-rest decision tree for each of the most frequent 10,000 tags using the ID3 algorithm. Tuned a few hyper-parameters for when to prune a tree based on frequency of the tag and feature to split on to avoid overfitting. The tree leaves give a probability of the tag rather than a binary decision.
\nClassification
\nFirst, use duplicates from training (hash lookup), then test all the decision trees, choosing those that gave the top 5 probabilities, if the probability was above 0.22 (tuned hyper-parameter).
\n\n
Notes
\nRig: Old 2GHz MacBook from 2008 with 2GB RAM
\nTraining took about a day using 300MB RAM, and classification took 30 minutes using 50MB RAM.
\nAt first I tried to use support vector machines but I found these to overfit too much and were much too slow to train on large datasets anyway, so that's when I moved to decision trees. Used a lot of time implementing the SVM trainer (SMO style, similar to libsvm), but it was a fun learning exercise (pun unintended).
\nIf I had more time I'd use the whole training set and use random forests instead. I'm certain that would give >0.8. Maybe I'll try that over the holidays.
", "votes": null, "replies": [] }, { "id": 36471, "author_name": "chestergrant", "author_url": "", "post_date": "12/21/2013 17:29:42", "content": "First I started out with Naive Bayes Classifier. It was too slow and it only give a 0.16 f1 score on the non-duplicates. Then, I decided to tag a post with a tag if the tag appears in the title or body. This took my f1 score up to 0.26 on non-duplicates. Then, I tried word association. This required me to prune the number of words. I selected only words that had more than 10 appearance in the dataset. Word association took me up to about 0.40
\nAfter this is just weighing the associations and doing parameter optimisation of the weights using genetic algorithm. This took me to 0.506 f1 score.
\nI did try neural nets and random forest, but they only gave me the same results as word associations and since you can't can only do classification on a subset of tags with neural nets and random forest, I decided to go with word association.
\nThere were some tags that were unpredictable eg. html,file,database,form,exception,image. In the end, I was making customs classification for these unpredictable tags but that only took me from 0506 to 0.509. 0.506 translated to 0.77674 with the duplicate. I am happy with my performance.
", "votes": null, "replies": [] }, { "id": 36475, "author_name": "aysent", "author_url": "", "post_date": "12/22/2013 00:20:49", "content": "Congrats to the winners and thanks to barisumog for starting this thread with a great write-up!
\nI also used one-vs-rest approach and here are my few comments on it.
\nCleaning the data and feature extraction
\nI tried TfIdf, Count, and Hashing vectorizers from sklearn with various parameters and hashing one ended up giving the best cv score and it was fast and it had a low memory usage since it doesn't need to store its vocabulary. The things which helped here:
\nAnd things which didn't help:
\nI also wrote a custom vectorizer based on TfIdf vectorizer to implement several supervised term weighting schemes, but none of them helped too.
\nTraining
\nFor each tag among top tags I trained a logistic regression model and pickled it for later usage. I started with training on the first several tens of thousands samples, but it had enough number of positive samples only for the most frequent tags. So I decided to generate a set of training samples for each tag separately: I kept reading Train.csv until I find some fixed number of positive and negative samples. Since my cv score changed significantly when I was asking for different numbers of positive and negative samples, I had a grid search to find their optimal values.
\nClassification
\nFind all duplicates using hash lookup and store non-duplicates in a separate file. Un-pickle all trained models, predict probabilities of each trained tag, and choose the ones with probability higher than some threshold, if more than five tags are chosen leave top five. My leaderboard score somewhat improved when I tried several ways of adjusting predicted probability with precision score I stored during training, for example, I would leave a tag with somewhat low predicted probability of 0.8 and precision score of 0.9 (like 'android') rather than leave a tag with high predicted probability of 0.9, but with low precision score of 0.15 (like 'database').
\nI also tried to use predicted probabilities to train another model to take into account possible correlations between tags, for example, some tags are more likely to be used together, but this additional step didn't seem to help much too.
\nOne final note, I only trained on first 2K tags which already correspond to 80% of all tag occurrences and I thought it's not worth it to train on another several thousand tags to gain just a few more percents, but it looks like people from top 10% trained on about ~10K tags.
", "votes": null, "replies": [] }, { "id": 36479, "author_name": "elyase", "author_url": "", "post_date": "12/22/2013 01:32:23", "content": "First congrats to the winners and thanks to the other participants who shared their approaches. At the beginning I also spent lots of time extracting meta-features like has_code, is_html_tag, is_email, etc. My idea was to use this "improved" representation of the documents plus all the usual bag of words features to make similarity queries to each one of the documents in the test set (bow->tf-idf->lsa). From the top most similar documents in the training set I extracted the tags and combined them to make my prediction. This made more sense to me than just blindly clustering as it would be difficult to determine the right number of clusters. Nevertheless I tried both and didn't get the results I was expecting (under 0.70 f1-score). I still think this approach has potential and needs to be further explored.
\nSo in the last day I decided to go for a simpler approach. It was a little dramatic as I finished the code 19 hours before the submission deadline and according to my estimations the model would also take 19 hours to complete the prediction. Fortunately, the algorithm was relatively easy to parallelize and thanks to Amazon EC2 it took around 3 hours to finish the prediction in a cc2.8xlarge (wonderful machines by the way). I adapted the winning solution of the ECML PKDD Discovery Challenge 2009 by Evangelio Milos et. al. plus some small improvements from other solutions given there [1]. The algorithm goes like this, there are three basic tag recommenders which get combined to form the final submission:
\nAt the end I combined all tag recommendations probabilistically plus some small last minute optimizations. I'd say the main takeaway is how much statistical information is contained in the title. I intend to post the code as soon as I get to clean it a bit (it was made in a rush).
\nTools used: scikits-learn (hash vectorizer, quick tests), gensim (LSA, document similarity queries), networkx(for the graphs), pypy(I didn't got any visible speed up from this, most of the time was spent accessing hash-tables within networkx).
\n\n
UPDATE: Blogged about it with some code here.
\n\n
\n
Credits:
\n[1] http://www.kde.cs.uni-kassel.de/ws/dc09/papers/proceedings.pdf
", "votes": null, "replies": [] }, { "id": 36496, "author_name": "philipwahrlich", "author_url": "", "post_date": "12/22/2013 04:19:06", "content": "Congrats to Owen, and others who did well. Also thanks to those who are sharing their approaches and experiences from this competition.
\nAfter some time experimenting with a range of different methods (OvA classifiers, and simple baseline models mostly), my most successful approach (0.79479) used association rules and was quite straightforward.
\nFeature Extraction
\nI stripped html tags, stop words (NLTK stop words), and special characters (except '+' and '#'). I kept the code blocks, as they seemed like dead giveaways for language tags. Then I extracted word unigrams, bigrams, and trigrams separately from both the title and body texts. Adding trigrams didn't give much improvement (about 0.01), and increased memory usage by quite a bit, but an improvement is an improvement.
\nTraining
\nI learned association rules separately from the title and body texts. This was essentially counting n-gram-tag cooccurrences in a Python dict, and keeping a count of n-gram occurrences in another Python dict. As the dictionaries grew I pruned out n-gram-tag cooccurrences which were very low in number and would be insignificant during prediction. This kept the total size of the dictionaries to about 25GB, so I could stay in RAM. I dumped the dictionaries to disk as a pickle. I trained over the whole training set (except for a small 10k sample for cross-validation).
\nPrediction
\nI found that the title text was far more useful than the body text, which was surprising.
\nI loaded the dictionaries back into RAM, then made predictions based on the title association rules. I tuned the confidences and support thresholds for 1st, 2nd, 3rd, etc predicted tags. Thresholds increased for each consecutive tag predicted.
\nIf there were less than 5 tags predicted from the title association rules I made further predictions based on the body association rules. Again, I tuned the confidence and support thresholds.
\nIf no predictions were made, I forced a single prediction by removing the thresholds, and failing that, I predicted "c#".
\nSince some duplicates had more than one tag set associated with them in the training data, I tried to predict the most frequently occurring tag set for a given duplicate. Surprisingly, that performed worse than simply predicting the last occurring tag set, for a given duplicate, in the training set.
\nI quite enjoyed the challenges which came with the size of this data set. More please!
", "votes": null, "replies": [] }, { "id": 36500, "author_name": "psilogram", "author_url": "", "post_date": "12/22/2013 09:22:28", "content": "Congratulations to all. This was a fun and challenging problem. My approach was as follows:
\n1. Clean the data by removing html tags and some punctuation, converting everything to lowercase and removing duplicate observations.
\n2. Generate tfidf vectors on title and body separately (1-grams and 2-grams), and limiting the terms to 70k due to memory. Also, the tfidf vectors were based on a sample of about 50% of the non-dup observations.
\n3. Reduced the sample to 500k observations and ran standard one-vs-rest linear regression models on the top tags (top 7000 for titles and top 3000 for bodies). Because of memory limitations, I processed the tags in groups of a couple hundred and pickled each resulting model. This produced two sets of models, one for titles and one for bodies.
\n4. Created two additional models based on whether a tag appeared in the title or in the first 50 characters of the body. For these exact matches, it was necessary first to process the titles and bodies by replacing synonyms with their tag representation (e.g., 'win7' in a title was changed to 'windows-7'). This step produced two additional models, one for titles and bodies. In both cases, I calculated the probability that the appearance of a tag term in the title or body meant that the tag was also included in the tags field.
\n5. In the end, I had four models, each of which gave the probability of a specific tag appearing for a specific observation. To merge these models, I removed duplicates (e.g. same tag produced by different models), ordered them in descending order of probability and selected the first 1 to 5 tags based on probability thresholds. The thresholds were determined by fitting to a validation set. The final thresholds were [0, .16, .28, .41, .46]. For example, the highest-probability tag was first selected; then the second if it's probability was over .16, then the third if its probability was over .28, etc..
\nMy final f1 score on the non-duplicated validation set was .526.
\nFor me, the most challenging aspect was dealing with time and memory limitations. There were a lot of factors (size of sample, number of terms in tfidf, number of tags in one-vs-rest, etc.) that affected the amount of memory and time to execute. I spent a lot of time trying to find the optimal balance of all these variables through trial-and-error. In the end, my program needed about 5 hours to generate the models and another 3 hours to generate the predictions.
\n\n
\n
", "votes": null, "replies": [] }, { "id": 36501, "author_name": "djstrong", "author_url": "", "post_date": "12/22/2013 09:32:55", "content": "
I have 0.76814 on private leaderboard (44th rank) (public: 0.76770). I have used information about duplicates.
\nMy minimalistic approach:
\nCreate bag of words WITHOUT any preprocessing for each post combining title and body (probably should I have done this separately). I have counted conditional probability of tag|word.
\nClassification was done by generating ranking P(tag|word)-P(tag).
\nI had problem with RAM (on laptop have only 8GB), but rewrite from Python to C helped. Also I count only these words which was in test set.
", "votes": null, "replies": [] }, { "id": 36502, "author_name": "eoinlawless", "author_url": "", "post_date": "12/22/2013 11:15:38", "content": "Hi djstrong - your idea to only use words in the test set is really nice - so obvious in hindsight :) You say you did your classification based in P(tag|word) - P(tag). Did you calculate that for all words in a question and pick the tag with the highest value, or combine the values in some other way?
\n\n
", "votes": null, "replies": [] }, { "id": 36505, "author_name": "djstrong", "author_url": "", "post_date": "12/22/2013 11:24:36", "content": "
Hi, thanks :)
\nYes, I calculate this for every word in question and combine the values (AFAIR by simply adding values).
\nFor example (capitalized is tag):
\nP(JAVA|java) - P(JAVA) is probably positive, but P(JAVA|bicycle)-P(JAVA) is negative. So every word in question have impact on value of (almost) every tag (also negative).
\n", "votes": null, "replies": [] }, { "id": 36506, "author_name": "ahallan", "author_url": "", "post_date": "12/22/2013 12:39:45", "content": "
I'm more of a computer scientist more than a mathematician, and I'm new to data science, so i'm quite happy with my results (also didn't have a massive amount of time!). Pleased to see a lot of citations to papers and models which I'll keep in mind for any further contests.
\n\n
This is a summary of what I did:
\n1. Matched up all duplicates between the Test and Training set
\n2. Saved a count of each tag per word and per bigram from each title in the Training set. (I also did this for the body, but didn't end up using it in the end). Utilised a stoplist.
\n3. During the results processing, I split the title into words and bigrams, calculated the bayesian probability of a tag given the word or bigram, and then summed up the probabilities given the tags.
\n3a. I then put some logic in to say that the tag likelihoods had to be within x% of the top tag (this was to increase the precision).
\n3b. There was also some logic in there to prevent known unlikely tags (e.g. c# java python) with were produced from the FScore calculations on the Training set. If one of these unknowns was encountered, the system would downweight these tags.
\n4. I ran some simulations of the model on the Training set using a self implemented version of the FScore calculation in order to test which variables were best (e.g. the x% for 3a).
\nCoded the whole thing up in Python (which I'm new to, I usually develop in c#) using pandas and utilised a postgres db for saving data. Also heavy usage of csv files to split, process and combine data (ala mapreduce) seeing as I only have 4GB of RAM. I think having an SSD harddrive also drastically helped with loading speeds from disk to RAM.
\n\n
", "votes": null, "replies": [] }, { "id": 36508, "author_name": "shendu", "author_url": "", "post_date": "12/22/2013 14:58:20", "content": "
I used spreading activation. At the beginning I used only tags with no. of occurrences more than 100. In my last & best model I used tags which appeared more than 4 times in the dataset. It improved my f1 score by a very small amount and didn't slow my training and prediction. So I decided to go with it. All the f1-scores reported here are for predicting non-duplicate last 5000 questions in the dataset( after pruning all the duplicates using a murmur hasher applied on the title of the questions). I coded everything from scratch in python by using numpy, scipy and sklearn DictVectorizer.
\n -> predict top 6 tags and break if
->the difference in activation of two consecutive tags is more than 0.7 of the max and the activation is below 0.12.
->after predicting top 2 tags, the activation drops below 0.13.
I used my i5 dual core machine with 8gb ram. The feature selection process(from 1 million questions) took around 5 hours.The training on the entire dataset( 4.1 million) took 8 hours and the prediction for non-duplicates( 0.9 million or so) took 1.5-2 hours. RAM usage was never more than 3gb and during prediction it was only 80-100mb.
\nI approached this problem with associative rules algorithm (not sure if it’s the proper name but I never seen this approach in the books and basically it is a simplification of naive bayes approach).
\n1. Match duplicates in test set and remove multiple duplicated entries from train set.
\n2. Clean the data: I removed HTML tags and separated words with regular expression. Extracted all blocks with code from body. That seems important, as the code appears to be a strong indicator of programming language used and hence is good predictor for tag. Later on I added StanfordNLP parser http://nlp.stanford.edu/software/lex-parser.shtml to extract lemmas. Title seems to have a bigger impact then the body so title words were extracted as separate features. Code blocks were case sensitive and processed differently, plus, I extracted some syntactic features like usage of different arrows ==> or different type of comments // or #
\n3. Build associative rules for first 30 000 tags. I built rules on top of TF vectors, where TF(Word_j) = 1 + log(Count(Word_j)). The rule can be either positive or negative. For positive rule p(Tag_i|Word_j) > 0.3 and Sum (Word_j)>40. For negative rule p(Tag_i|Word_j) = 0 and Count(Word_j) > 50 000, where I had different weights for different type of rules and had to run multiple CV tests to select proper value. I had to run all the rules on both train and test set. Train set run was used to weight them properly. This alone seems to give ~ 0.78 and the next major improvement was introducing bigrams. I used a variation of Bloom Filter to extract candidate bigrams and had to run my algorithm in multiple stages, but that essentially pushed my score up to 0.8.
\n4. Naïve Bayes on the first 2000 tags. I mixed results of this model into my core algorithm whenever I had 0 tags predicted.
\n5. VowpalWabbit linear model on the first 20 tags. This marginally improved results but I decided to keep it anyway.
\n6. The last step was to predict some additional tags based on already predicted tags. I only used predictions with high confidence. Basically I found all cases there p(Tag_i|Tag_j) is high and used them to improve predictions.
", "votes": null, "replies": [] }, { "id": 36520, "author_name": "owenzhang1", "author_url": "", "post_date": "12/22/2013 19:16:53", "content": "My approach:
\n1. Take duplicates out like many others did
\n2. Calculate the association between all 1-gram and 2-grams in title and body. I kept the first 400 chars and last 100 of body to keep the size under control. I keep only the top 30 tags for each 1-gram and 2-gram
\n3. Predict the tags by combining all the tags associated with all the 1-gram and 2-grams in Title+body. Each 1/2 gram's impact is weighted by its support and entropy. The entropy part only improve the score very marginally.
\n4. Take the top tags based on their score. The cut off threshold is determined by the ratio between the score of tag_k and the average score of all tags with higher scores than k.
\n5. I scored the above on 200k posts in training, and computed the bias for each predicted tag. Some tags have more FP than FN, some have more FN than FP. The goal is to have FP and FN about equal.
\n6. Score test data and adjust for the tag bias from step 5, i.e., if a tag has #FP>#FN, decrease its score, and vice versa.
\nNote:
\nThere are quite a number of tuning parameters that I "hand optimized" by evaluating the impact on 2% of training data I kept as a validation dataset.
\nTitle is much more important than body -- I gave a 3:1 weight to title (but this is somewhat cancelled out by the fact that there are more words in body usually)
\nI thought about doing some <code></code> extraction but didn't get time to do so.
\nI also extracted quoted posts in the body and tried to match that with training, just like the duplicates. But this barely improves the result. So it seems the association of tags between posts and quoted posts are not very strong.
\nI also tried some TF-IDF based methods but didn't get good results.
\nI didn't spend too much time on "NLP" like tasks such as stemming/tagging, etc. Part of the reason is time required, and part of reason is that the posts are not very "natural", compared to texts from news archives and books. I figure I probably would lose as much information on the code part, v.s. what I can gain in the "natural language" part.
\n\n
The "RIG":
\nAs noted by everyone, the bottleneck of this competition is RAM. I am fortunate enough to have access to a machine with 256GB ram to try out different parameters.
\nThe final solution would run on a desktop with 64GB of ram in about 4 hours. The training only takes about 1.5 hour and scoring will take 2.5 -- partially because it runs out of ram. Training of the model will eat up almost all of the 64GB, and the rest will swap to disk. There has to be at least 90GB swap space for this to run with out error.
\nI used Python, which is not very "memory efficient". If we were to compete on "efficiency" I think c/c++ would be necessary.
", "votes": null, "replies": [] }, { "id": 36557, "author_name": "erengolge", "author_url": "", "post_date": "12/23/2013 09:37:13", "content": "[quote=djstrong;36501]I have 0.76814 on private leaderboard (44th rank) (public: 0.76770). I have used information about duplicates.My minimalistic approach:Create
bag of words WITHOUT any preprocessing for each post combining title
and body (probably should I have done this separately). I have counted
conditional probability of tag|word.Classification was done by generating ranking P(tag|word)-P(tag).I
had problem with RAM (on laptop have only 8GB), but rewrite from Python
to C helped. Also I count only these words which was in test set.[/quote]
As an additional curiosity, how to decide the number of tags for your prediction ?
", "votes": null, "replies": [] }, { "id": 36558, "author_name": "erengolge", "author_url": "", "post_date": "12/23/2013 09:38:59", "content": "[quote=Triskelion;36457]
\nTried a different challenge. From .csv's to submission in under 1 hour, with less than 1GB of memory, in 30 days.
\nThis inspired further constraints: No use of external algorithm libraries, single-pass, online learning, parallel learning.
\nStarted with observation that top 200 tags are good for ~50% of all tag occurrences.
\nFirst algorithm idea: 75% of documents with the word "stylesheet" are tagged with "css". So Bro Bayes.
\nUse of a FTS4 Sqlite database played it safe with memory, but gave me too much to count and update (couldn't resist), so I predicted this would not meet time constraint. If you give yourself 30 minutes for the predictions then you are predicting at 1k+ a second.
\nI figured out the dupes in the train and test sets, and the dupes between them. To dedupe you could match the post titles. Using that with only a tag dictionary generated from the train set and substring matching those against the titles in the test set gave you a f1-score of around #20 at the leaderboard. This was 3 weeks ago. It took about 700Mb of memory and under 3 minutes.
\nStemming text was too slow. Preprocessing thus was more cleaning of text and removing stopwords. For stop words I first tried a top 10k stopword file from Google with the tag dictionary words removed. I found that counter to the competition rules, so I generated a stopword file from the train set with words that are found with many different tags attached, and hand-picked a top 200.
\nTrained Vowpal Wabbit on top 40 tags. Got a good result, but was approaching 1 hour. Trained it again on top 200 tags and got a better result, but took 3 hours. I believe I saw that top 1024 tags are good for over 70% of all tag occurrences, but realized this route was not going to meet constraints.
\nI ended up using 3 count dictionaries. One tag counter, one word counter and one word-to-tag counter. This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory. Not using an external big stopword dictionary meant I had to prune the word-to-tag dictionary with every 100k chunk. Once the number of tags per keyword reaches a certain threshold, I take a top n and only increase the counts of those top n tags in the future.
\nThis gave me my result that met all my constraints except for one, memory. It took 1.7Gb max. for model creation. Predictions with 1.1Gb on only the title words.
\nTried to do keyword extraction without a dictionary or statistical historical knowledge. First term frequency on a top 10k stopword removed file. Then co-occurrence: this paper http://ymatsuo.com/papers/flairs03.pdf found me reasonable results. All took either too long to run or did not add substantially to a dictionary-based approach.
\nFun things. Ran online LDA over the test set to obtain topics. Very clear topics appeared like all string manipulation commands. "adolf-hitler" is a tag. The hardest question to predict only by title was something like: "why?! why! why! why! why?!?!", because after stopwords and cleaning nothing remained :).
\nWhat I learned: Simply count ALL the things. Don't count 'em twice.
\n[/quote]
\n\n
Thanks for your conscience share. However one question ?
What you mean by "This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory"
Can You give a better explanation ?
[quote=Owen;36520]
\n4. Take the top tags based on their score. The cut off threshold is determined by the ratio between the score of tag_k and the average score of all tags with higher scores than k.
\n5. I scored the above on 200k posts in training, and computed the bias for each predicted tag. Some tags have more FP than FN, some have more FN than FP. The goal is to have FP and FN about equal.
\n6. Score test data and adjust for the tag bias from step 5, i.e., if a tag has #FP>#FN, decrease its score, and vice versa.
\n[/quote]
\nFirst of all Congratulations for the Rank :)
\nOne question;
\nIs the cut of threshold defined by tag_k is used for the prediction of tag_k or k is an free parameter that you set it via x-validation ??
\n\n
By the way, your method is very much alike to http://iccm-conference.org/2013-proceedings/papers/0077/paper0077.pdf
\n[quote=Eren Gölge;36558]
\n\n
What you mean by "This word-to-tag counter gets updated with a term frequency * inverse word frequency * inverse tag frequency, so you get a score that approximates tf*idf, without having to do inverse document frequency calculations on a second pass or in memory"
Can You give a better explanation ?
[/quote]
\nI'll try. The "tf" part in tf*idf stands for term frequency. This can prevent a bias to longer documents.
\nFor example: two documents both mention the term "Kaggle" one time. Document A has 10 words, document B has 1000 words. A very simple "tf" can be calculated like 1/10 for Document A and 1/1000 for Document B. So Document A is probably more relevant to the term "Kaggle" than Document B. Because 10% of Document A is about the term "Kaggle" and only 0.1% of Document B is about the term "Kaggle".
\n"idf" stands for inverse document frequency. This can prevent a bias to common words. Let's say we have 20 documents total. The IDF for a very rare word that appears only in 1 of those documents would simply be 20/1 = 20. The IDF for a fairly common word that appears in half of those documents would be 20/10 = 2. The IDF for a word that appears in all of those documents would be 20/20 = 1.
\nSince tf*idf is the product of tf and idf, a very common word would get a lower tf*idf score (idf=1, so simply the term frequency).
\nTo calculate idf one needs to check if a word is present in all of the documents that you have observed so far. This is the costly part of tf*idf. It requires lots of memory or multiple passes to calculate idf.
\nSince idf is used to prevent bias to common words, we could substitute with inverse word frequency. If word A was counted 990 times vs. word B that was counted 10 times, the iwf would be 1000 / 990 vs. 1000 / 10. Word B gets a higher iwf score because it is less common.
\nTo prevent bias to common tags, I also did inverse tag frequency. Some words appear a lot tagged with common tags, not because those words are descriptive to those tags, but because the tag is added to so many documents.
\nI also gave a little boost to words that are exact matches of tags (so I didn't need to substring match anymore), and took into account the distance of the word to the beginning of a paragraph or sentence (reasoning, more important keywords are found early on in sentences, paragraphs and texts).
\nFinally I did some logs to smooth it out. TF*IDF can be as complex as you want to. Binary, relative, logarithms etc. The Yahoo Learning to Rank competition has solid practice-tested TF*IDF formulas if you want to know more.
\nWith this tf*iwf*itf score for every word, you can see that my model gives tags like "php" and "ruby-on-rails" a higher score for the word "bug" than tags like "c#" or ".net", even though the word "bug" may be counted more in documents tagged "c#" or ".net".
", "votes": null, "replies": [] }, { "id": 36573, "author_name": "gungoryilmaz", "author_url": "", "post_date": "12/23/2013 14:35:55", "content": "How did you fit your input to tfidf vectorizer at once while you have memory problem? Can anyone share sklearn code about that?
\n\n
Is there any way to fit partially for tfidf?
", "votes": null, "replies": [] }, { "id": 36578, "author_name": "triskelion", "author_url": "", "post_date": "12/23/2013 15:05:18", "content": "[quote=Güngör Yılmaz;36573]
\nHow did you fit your input to tfidf vectorizer at once while you have memory problem? Can anyone share sklearn code about that?
\nIs there any way to fit partially for tfidf?
\n[/quote]
\nThis may be relevant reading: http://scikit-learn.org/stable/modules/feature_extraction.html#vectorizing-a-large-text-corpus-with-the-hashing-trick
\nAs for partial fit, I think some people in this very thread have fit tfidf on chunks, instead of the entire data set (for example tfidf on a 500k chunk, then use that to transform the rest of the data).
", "votes": null, "replies": [] }, { "id": 36586, "author_name": "owenzhang1", "author_url": "", "post_date": "12/23/2013 19:15:25", "content": "@Eren: yes I did read that paper, which is one of the better (more practical) ones I managed to find.
\nOn the topic of cut-off threshold of k, it is a standalone parameter I set through cross-validation. First I sort all the Tags that was positively correlated with 1/2 grams by the strength of correlation. The threshold is applied to the ratio of the correlation of Tag_k to the average correlation of Tag_1 to Tag_k-1. For each position (0-4) there is a different k.
\nThis will determine the number of Tags to predict.
\nI did some very "back of the napkin" theoretical analysis on the incremental impact of adding a Tag with Prob(TP)=pk to the existing prediction with given number of TP, FP, and FN. The thresholding approach is derived from this analysis to maximize expected F1.
", "votes": null, "replies": [] }, { "id": 36607, "author_name": "omaro126129", "author_url": "", "post_date": "12/23/2013 23:39:23", "content": "Hello everyone, just wanted to share my approach as well. Like most of you, the biggest constraint for me was memory. I was equipped with a 16GB linux machine (after exhausting my 8GB Macbook), but more memory would have allowed me a bit more flexibility with my model.
\n1. I initially performed tokenization by combining, lowercasing and removing any non-alphanumeric characters (except '+' and '#') from the title/body after removing html tags. Extra whitespace was removed and I was left with a space-delimited file containing the tokens. I used a bag-of-words model, so the first goal was to extract the most relevant tokens from this list - this was done by maximizing the frequency of words within each sample while minimizing their frequency across all the documents (hence, I was trying to find tokens that would maximize tf-idf values). This produced a matrix with 40K features that represented the frequency for each unigram.
\n2. Term-frequency (tf) was done by taking the log and dividing by the maximum log-transformed frequency for each sample (log-transformation was done after adding 1).
\n3. The response matrix was essentially a giant indicator matrix for every one of the 42050 tags in the training set. A separate response matrix contained the tag count data represented by a 5 column indicator matrix.
\n4. Learning was done using least squares using a one vs all approach only with the non-duplicated data set. Now at this point, I created 2 models: one to estimate the most probable tags given the feature matrix, the other to estimate the number of tags given the feature matrix. After a bit of experimentation, the first model was giving the best f1 scores after performing dimensionality reduction on the feature matrix - accomplished by using the eigendecomposition of the covariance matrix, trimming off eigenvectors with lower eigenvalues (~5K dimensions omitted) and projecting the mean-centered feature matrix into this new space.
\nThis reduced model was used to find the most likely tags, while the other model (which didn't use any dimensionality reduction) predicted the number of tags.
\n5. A significant amount of optimization was necessary to do least squares on these huge matrices. Both the feature and response matrices were represented using a sparse matrix data structure. Everything was implemented in C and I ended up using single-precision floating points (rather than double) for all the calculations - this way a 40K x 40K matrix of eigenvectors would take about 6.4GB and could be stored in RAM. I also used OpenBLAS, which was a lot more efficient than the default BLAS on linux.
\n", "votes": null, "replies": [] }, { "id": 36639, "author_name": "djstrong", "author_url": "", "post_date": "12/25/2013 10:20:16", "content": "
[quote=Eren Gölge;36557]As an additional curiosity, how to decide the number of tags for your prediction ?[/quote]
\nUnfortunately I did not any smart solution for this. I cut off whan next tag hav value 2 times less than previous. If is not then I take 5 tags.
", "votes": null, "replies": [] }, { "id": 36831, "author_name": "jaap39657", "author_url": "", "post_date": "12/30/2013 21:52:08", "content": "It seems the word-tag association, although needing a lot of memory works very well for this data set! Very creative approach, I didn't think of this.
\nI created my solution on a laptop, and it runs pretty fast. In a few hours it delivers the result. The solution was created in Python. My solution is quite straightforward, although I am sure with a bit more tweaking the results can be much better. I just used Logistic Ridge Regression without cross validating, and without averaging with other estimators or bagging.
\nHere a quick overview of what I did:
\nMy Solution:
\nLater I got much improved results for individual tags by averaging Logistic Regression and Random Forests (using the 25 best features). I however did not have the time to run this on all examples due to work obligations the last days of the competition.
", "votes": null, "replies": [] }, { "id": 37398, "author_name": "jorgeblasco", "author_url": "", "post_date": "01/15/2014 11:57:40", "content": "[quote=Petko Minkov;36452]
\nMy basic idea is to find predictive words. A predictive word for a tag is a word with high ratio of posts in which the word is contained and are labeled with that tag vs posts in which this word is contained.
\nI have the following pipeline.
\nStep 1: Read input in python, tokenize text and create features. Title words are prefixed by "T_", words in code segments are prefixed by "C_". That's because a "main" in the code might have higher predictive power than a "main" in the text. I got a big bump by doing the same for the title with the "T_" prefix. I guess something like "java" in the title is more predictive than "java" in the text.
\nStep 2: C++ code to remove words with very high and very low occurrence. I remove the top ~1000 words and the bottom 80% of the words. The bottom 80% have about 7 occurrences in the text, so I can't even use them reliably to predict.
\nStep 3: Train a classifier. C++ again. This is just computing the ratio which I pointed at earlier. I keep a label to words matrix in memory. To save memory, I hash label and words. The output is a map from words to labels which the words predict with high probability. So "C_iostream" might predict C++ with probability of 0.79. I also produce bigrams from words in the title. Bigrams from words in the post were blowing up my memory usage.
\nStep 4: Prediction. I go through every word and see what labels does it predict and keep the maximum probability for each label that comes up. I select the top 5 predictions and output them. Most of the time there are less than 5 predictions.
\nDupe detection is done by using a djb hash on the label text.
\nI tried coding a naive bayes with x2 feature selection step in both C++ and using scikit, but didn't get very far with F1.
\n[/quote]
\nHow do you tokenize the text in Python?
", "votes": null, "replies": [] }, { "id": 37401, "author_name": "mattvonrohr", "author_url": "", "post_date": "01/15/2014 13:22:36", "content": "[quote=JorgeBlasco;37398]
\nHow do you tokenize the text in Python?
\n[/quote]
\nIn its most basic form (called unigram tokenization):
\nimport re
s = 'This text will get tokenized'tokens = re.findall('\\w+',s)
#['This', 'text', 'will', 'get', 'tokenized']
Or did I misunderstand the question?
", "votes": null, "replies": [] }, { "id": 37448, "author_name": "paithan", "author_url": "", "post_date": "01/16/2014 08:32:40", "content": "@Owen: Is there any chance that you could make your code public?
", "votes": null, "replies": [] }, { "id": 37457, "author_name": "vykintasmak", "author_url": "", "post_date": "01/16/2014 13:28:06", "content": "I feel the need to say that the challenge was very interesting. It is a shame that i haven't finished my solution until the end of the challenge but maybe it would be interesting to someone so here it is:
\nThe idea is to compare each question title and description to the collection of tag titles and descriptions like its done in categorization tasks. I've founded an article which compares categorization methods and decided that Weighted Inverse Document Frequency (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.49.7015&rep=rep1&type=pdf ) may perform best.
\nWIDF vectors
\n1. Create a collection of words vs its frequencies for each tag in a training set. There should be two collections for each tag - one for titles and one for question bodies.
2. Create a key value dictionary of words vs its frequencies for all training set.
3. Create a key value dictionary of words vs its occurrences in the current document for each title and body.
3. Create WIDF vectors for each tag collection (for both titles and bodies as it is in 1st list item).
Training by the distance
\n1. Compare the similarity between question title WIDF vector and each tag collection using cosine similarity function. The result should be that cosine angle between related tags is closer to 1 and to unrelated tags is closer to 0.
2. Also compare the similarity between question body WIDF vector the same as title was compared. Now we should have X as similarity between titles and Y as similarity between bodies.
3. Plot these similarity points. Red pints for unrelated tags and green points for related. This should help to define weather Euclidean or Mahalanobis distance is needed.
4. Find the best Euclidean/Mahalanobis distance by counting precision/recall for each distance step.
Speed
\nAs you could see this method requires a lot of resources to process the text. I've tried using multithreaded C++ and MySql query pools but it was too slow so the best way i think should be to write Hadoop mapreduce chain for it.
\nI've missed the tokenizer part because i think this algorithm could perform without anything more than splitting the text by \\w, but it would be interesting to test some more options.
\n", "votes": null, "replies": [] }, { "id": 128701, "author_name": "jonclement", "author_url": "", "post_date": "07/22/2016 20:14:35", "content": "
sklearn now supports partial fit on OneVsRestClassifier! (development branch Jan 2015).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "36434": "", "36448": "", "36450": "", "36451": "", "36452": "", "36453": "", "36454": "", "36457": "", "36463": "", "36471": "", "36475": "", "36479": "", "36496": "", "36500": "", "36501": "", "36502": "", "36505": "", "36506": "", "36508": "", "36510": "", "36520": "", "36557": "", "36558": "", "36559": "", "36567": "", "36573": "", "36578": "", "36586": "", "36607": "", "36639": "", "36831": "", "37398": "", "37401": "", "37448": "", "37457": "", "128701": "sklearn now supports partial fit on OneVsRestClassifier! (development branch Jan 2015)." }, "source": "meta" }