Here's a brief description of what I did. I'm simplifying quite a bit and omitting a number of details in order to focus on the big picture.
\nFirst, an overview of the sequence of steps in processing a single line:
\n1) Guess which position the missing word is at.
\n2) Generate candidate word insertions for the hypothesized position.
\n3) Guess what word (from the candidates) should be inserted at the missing position.
\n4) Compute a confidence score, C1, for the insertion of the hypothesized word at the hypothesized position.
\n5) If C1 is high enough, output a line with the hypothesized word inserted at the hypothesized position.
\n6) Compute a confidence score, C2, for the hypothesized position.
\n7) If C2 is high enough, output a line with a space inserted at the hypothesized position.
\n8) Otherwise just output the input line with no changes.
\nFor the two major tasks (guessing the position and guessing the word), I used relatively unsophisticated n-gram models. In particular, I used a rather naive additive smoothing approach for estimating n-gram probabilities. For example, for bigrams:
\nP(w2|w1) = (C(w1,w2) + d) / ( d * |V| + C(w1))
\n(|V| is the size of the bigram vocabulary, C(w1, w2) and C(w1) are the raw bigram/unigram counts in the training corpus.) In other words, I add a count of d to every n-gram where d is a parameter tuned empirically on a held-out data set.
\nI used n-grams of length up to 6. When I calculate the probability of a given n-gram, I used only values of n such that the n-gram count was > 0. To combine the n-gram estimates for different values of n, I employed a logistic regression model.
\nI also incorporated a part-of-speech model into the missing position calculation using the Stanford POS tagger. This was just an n-gram model of the sort described above with part-of-speech tags rather than words as the constituents.
\nGuessing the missing word is done with the same sort of n-gram model as just described.
\nFor the imputation task, choosing whether to guess a word has some subtleties. Notice that if you insert the wrong word at the right position, you may not suffer any increased edit distance. For example:
\nInput: the bit the man
\nTrue sentence: the dog bit the man
\nGuessed sentence: the cat bit the man
\nNote that although we have guessed the wrong we do not suffer any edit distance penalty (as compared to not guessing a word). This leads us to some conclusions that impact the design of confidence scores:
\n1) Guessing a word should be done more conservatively if you are not confident you are inserting at the right position.
\n2) Guessing long words should be done more conservatively than guessing short words.
\nI used a linear regression model to predict the edit distance gain (or loss) from an insertion based on certain features. These features include things like the gap between the n-gram language model score for the top candidate and the next candidate. Likewise the gap between the position model scores for the top position and the next best position.
\nI trained a separate regression model for different word lengths of inserted words to account for the second remark above; i.e., that we want to guess more or less aggressively depending on the length of the inserted word.
\nThere's a similar regression model that gives us a confidence score for positions. This one is a logistic regression model because we are simply trying to predict a binary value (have we guessed the right position or not?).
\nAll n-gram models were built with lower-cased text. When outputting an inserted word, we used the most common capitalization for that word (e.g., usa->USA, john->John, and->and).
", "messages": [ { "id": "78335", "postDate": "05/14/2015 00:21:00", "content": "Here's a brief description of what I did. I'm simplifying quite a bit and omitting a number of details in order to focus on the big picture.
\nFirst, an overview of the sequence of steps in processing a single line:
\n1) Guess which position the missing word is at.
\n2) Generate candidate word insertions for the hypothesized position.
\n3) Guess what word (from the candidates) should be inserted at the missing position.
\n4) Compute a confidence score, C1, for the insertion of the hypothesized word at the hypothesized position.
\n5) If C1 is high enough, output a line with the hypothesized word inserted at the hypothesized position.
\n6) Compute a confidence score, C2, for the hypothesized position.
\n7) If C2 is high enough, output a line with a space inserted at the hypothesized position.
\n8) Otherwise just output the input line with no changes.
\nFor the two major tasks (guessing the position and guessing the word), I used relatively unsophisticated n-gram models. In particular, I used a rather naive additive smoothing approach for estimating n-gram probabilities. For example, for bigrams:
\nP(w2|w1) = (C(w1,w2) + d) / ( d * |V| + C(w1))
\n(|V| is the size of the bigram vocabulary, C(w1, w2) and C(w1) are the raw bigram/unigram counts in the training corpus.) In other words, I add a count of d to every n-gram where d is a parameter tuned empirically on a held-out data set.
\nI used n-grams of length up to 6. When I calculate the probability of a given n-gram, I used only values of n such that the n-gram count was > 0. To combine the n-gram estimates for different values of n, I employed a logistic regression model.
\nI also incorporated a part-of-speech model into the missing position calculation using the Stanford POS tagger. This was just an n-gram model of the sort described above with part-of-speech tags rather than words as the constituents.
\nGuessing the missing word is done with the same sort of n-gram model as just described.
\nFor the imputation task, choosing whether to guess a word has some subtleties. Notice that if you insert the wrong word at the right position, you may not suffer any increased edit distance. For example:
\nInput: the bit the man
\nTrue sentence: the dog bit the man
\nGuessed sentence: the cat bit the man
\nNote that although we have guessed the wrong we do not suffer any edit distance penalty (as compared to not guessing a word). This leads us to some conclusions that impact the design of confidence scores:
\n1) Guessing a word should be done more conservatively if you are not confident you are inserting at the right position.
\n2) Guessing long words should be done more conservatively than guessing short words.
\nI used a linear regression model to predict the edit distance gain (or loss) from an insertion based on certain features. These features include things like the gap between the n-gram language model score for the top candidate and the next candidate. Likewise the gap between the position model scores for the top position and the next best position.
\nI trained a separate regression model for different word lengths of inserted words to account for the second remark above; i.e., that we want to guess more or less aggressively depending on the length of the inserted word.
\nThere's a similar regression model that gives us a confidence score for positions. This one is a logistic regression model because we are simply trying to predict a binary value (have we guessed the right position or not?).
\nAll n-gram models were built with lower-cased text. When outputting an inserted word, we used the most common capitalization for that word (e.g., usa->USA, john->John, and->and).
", "rawMarkdown": "", "votes": null }, { "id": "78397", "postDate": "05/14/2015 13:38:22", "content": "Congratulations on the win!
\nMy main take-away is that your winning methodology didn't focus on creating the most comprehensive language model (and then applying it to the task) - but rather concentrated more effort on understanding how to win against the (rather linguistically obtuse) scoring system.
\nSo, while I guess there are some eyebrows being raised at your use of the "Stanford POS tagger" (unless you trained it on the training data alone), that probably didn't have as big an impact as the sophisticated insertion-choice heuristics you developed.
\nThanks for the write-up.
", "rawMarkdown": "", "votes": null }, { "id": "78401", "postDate": "05/14/2015 15:00:09", "content": ""My main take-away is that your winning methodology didn't focus on creating the most comprehensive language model (and then applying it to the task) - but rather concentrated more effort on understanding how to win against the (rather linguistically obtuse) scoring system."
\nYes, there's something to that. In particular, the confidence scoring mechanism is optimized specifically for this task, taking into account the way this competition is scored (edit distance).
\nI did gloss over some of the details in my language model - so it's probably not quite as simple or naive as I made it sound - but nevertheless I tend to agree with you that the language model was probably not the key differentiating factor of my system.
\nIf this competition were to be done again, I'm not sure that I'd recommend using the edit distance metric. It leads to work being done to optimize for that metric (as I did) which arguably distracts from what competitors "should" be working on. I think just giving a score of 0 or 1 depending on whether you reproduce the full sentence exactly correctly would be better.
\n"So, while I guess there are some eyebrows being raised at your use of the "Stanford POS tagger" (unless you trained it on the training data alone), that probably didn't have as big an impact as the sophisticated insertion-choice heuristics you developed."
\nNot sure why this would raise eyebrows. Also, to clarify, this is not something that I trained. The Stanford POS tagger works "out of the box"; i.e., it does not need to be trained by the user.
\nIn any case, I think I agree that it probably didn't add that much, although I don't have my notes in front of me at the moment.
", "rawMarkdown": "", "votes": null }, { "id": "78409", "postDate": "05/14/2015 16:38:15", "content": "Some statistics that may be of interest.
\nPosition detection:
\nApplying the model to every line, we guess the correct position 69.9% of the time. (This is assessed using a cross-validation set.) Compare to team mern who report 71% on this task; so my system is actually worse, although in the same ballpark.
\nUsing a confidence score with one particular threshold, we make a prediction 70.9% of the time and achieve 83.7% accuracy when making a prediction.
\nOverall task:
\nWe guess a word 57.9% of the time. When we guess, we guess exactly correctly (i.e., edit distance is zero) 56.6% of the time. Putting those things together, over the whole cross-validation set we produce the exactly correct output sentence 32.8% of the time.
", "rawMarkdown": "", "votes": null }, { "id": "78443", "postDate": "05/14/2015 21:56:41", "content": "Ahhgg - we've finished?! So many interesting things to do, so little time...
\nWell done Eric seems like a good solid engineering approach.
\nSorry Merl - the combination of better LMs and all your other bits should have pulled it off - my fault completly for not having time.
\nThe main page for the Stanford POS tagger at http://nlp.stanford.edu/software/tagger.shtml says that "The basic download contains two trained tagger models for English". I don't think these can be allowed, the rules are clear that the training data is the only data to be used.
\nEric - so you didn't try any nonsense words?? The basic approach is clear, find the right slot and fill it. But there was nothing in the rules to say that you had to fill it with a word from English and indeed someone (sorry, don't remember who) made the suggestion that if you knew the slot but not the filler then use two blanks but not one. There's an obvious generalisation, search over all possible strings for the minimum edit distance between the words that can fill the slot and weight the edit distance by the posterior of the word filling the slot. For example, as many words contain the letter "i" then if that's true of the weighted list then it's better to replace the single " " representing the missing word with " i " so gaining one edit distance assuming the slot was correct and another assuming the letter "i" was present.
\nKaggle - can we continue to post results and get them evaluated? If not, will you release the code that does the evaluation? There were several obvious flaws in the setup, even after including the test data in the training was fixed. There's quite some scope for using this problem as part of taught course.
\nIf anyone wants our n-best lists to continue this work then just email me.
\n\nTony
\n--
Dr A J Robinson, Founder
We are hiring: www.speechmatics.com/careers
Speechmatics is a trading name of Cantab Research Limited
Phone direct: 01223 778240 office: 01223 794497
Company reg no GB 05697423, VAT reg no 925606030
51 Canterbury Street, Cambridge, CB4 3QG, UK
"The main page for the Stanford POS tagger at http://nlp.stanford.edu/software/tagger.shtml says that "The basic download contains two trained tagger models for English". I don't think these can be allowed, the rules are clear that the training data is the only data to be used."
\nOh, I see the concern now. This is presumably what mdda was getting at too. Hmm, the competition is over now, but for my own interest I reran my system without the POS tagger. On my cross-validation set I get 3.944 which I estimate would correspond to 3.908 on the real test set.
\n"Eric - so you didn't try any nonsense words??"
\nNo, that wasn't part of my final submission. The only thing I did, as already described, was insert a space for cases where I had confidence in the position but no good guess as to the word. Very early on I tried inserting "e" in such cases and didn't get good results, but that might have been before I had good position detection.
\nArguably it's kind of silly to play games like inserting nonsense words, and doesn't further the research goals of the competition, so this may be another argument for not using edit distance as the scoring metric for this task.
", "rawMarkdown": "", "votes": null }, { "id": "78591", "postDate": "05/15/2015 23:19:59", "content": "[quote=Tony Robinson;78443]...there was nothing in the rules to say that you had to fill it with a word from English ...an obvious generalization, search over all possible strings for the minimum edit distance between the words that can fill the slot and weight the edit distance by the posterior of the word filling the slot. [/quote]
\nI tried that exact approach. Finding a median string is an NP complete problem, but I found a library that calculated median strings in a reasonable amount of time. I took a list of probable words (based on completing 4-grams or 3-grams), then calculated the weighted median string of those words.
\nUsing the median string instead of the most probable word gave an improvement of 0.25 - 0.3 on the leaderboard. So there's a non-trivial gain to be had by optimizing for the edit-distance scoring metric.
\nOn the other hand, if you start with a weak language model (as I did) these gains don't help much :)
\n[quote=Tony Robinson;78443] Well done Eric seems like a good solid engineering approach. [/quote]
\nDitto -- congratulations!.
", "rawMarkdown": "", "votes": null }, { "id": "78620", "postDate": "05/16/2015 08:20:30", "content": "Congratulations!
\nInteresting to see a lot of similarities in our approaches...
\nI just posted a more detailed report describing my system in the "my approach" thread, but picking a couple of points from there:
\n- I used a somewhat similar strategy regarding word insertion, but having actually 4 possible actions: insert word / leave as is / insert space / insert filler word ("ae"), with a rule based decision (using tunable thresholds) depending on the length of the inserted word.
\n- A link from the word2vec site points to http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz, which includes a version of the test set reference, which matches 99.8% with the competition's test set. I re-calculated the distance score (using a plain edit distance function) for my system at several points of the process using this matched set, and got a result within 0.01 of my "official" submissions.
\n- I actually thought the edit distance was a nice twist. Another possible variation would be to allow competitors several entries for each sentence, with different weights.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 78397, "author_name": "mdda123", "author_url": "", "post_date": "05/14/2015 13:38:22", "content": "Congratulations on the win!
\nMy main take-away is that your winning methodology didn't focus on creating the most comprehensive language model (and then applying it to the task) - but rather concentrated more effort on understanding how to win against the (rather linguistically obtuse) scoring system.
\nSo, while I guess there are some eyebrows being raised at your use of the "Stanford POS tagger" (unless you trained it on the training data alone), that probably didn't have as big an impact as the sophisticated insertion-choice heuristics you developed.
\nThanks for the write-up.
", "votes": null, "replies": [] }, { "id": 78401, "author_name": "onemillionmonkeys", "author_url": "", "post_date": "05/14/2015 15:00:09", "content": ""My main take-away is that your winning methodology didn't focus on creating the most comprehensive language model (and then applying it to the task) - but rather concentrated more effort on understanding how to win against the (rather linguistically obtuse) scoring system."
\nYes, there's something to that. In particular, the confidence scoring mechanism is optimized specifically for this task, taking into account the way this competition is scored (edit distance).
\nI did gloss over some of the details in my language model - so it's probably not quite as simple or naive as I made it sound - but nevertheless I tend to agree with you that the language model was probably not the key differentiating factor of my system.
\nIf this competition were to be done again, I'm not sure that I'd recommend using the edit distance metric. It leads to work being done to optimize for that metric (as I did) which arguably distracts from what competitors "should" be working on. I think just giving a score of 0 or 1 depending on whether you reproduce the full sentence exactly correctly would be better.
\n"So, while I guess there are some eyebrows being raised at your use of the "Stanford POS tagger" (unless you trained it on the training data alone), that probably didn't have as big an impact as the sophisticated insertion-choice heuristics you developed."
\nNot sure why this would raise eyebrows. Also, to clarify, this is not something that I trained. The Stanford POS tagger works "out of the box"; i.e., it does not need to be trained by the user.
\nIn any case, I think I agree that it probably didn't add that much, although I don't have my notes in front of me at the moment.
", "votes": null, "replies": [] }, { "id": 78409, "author_name": "onemillionmonkeys", "author_url": "", "post_date": "05/14/2015 16:38:15", "content": "Some statistics that may be of interest.
\nPosition detection:
\nApplying the model to every line, we guess the correct position 69.9% of the time. (This is assessed using a cross-validation set.) Compare to team mern who report 71% on this task; so my system is actually worse, although in the same ballpark.
\nUsing a confidence score with one particular threshold, we make a prediction 70.9% of the time and achieve 83.7% accuracy when making a prediction.
\nOverall task:
\nWe guess a word 57.9% of the time. When we guess, we guess exactly correctly (i.e., edit distance is zero) 56.6% of the time. Putting those things together, over the whole cross-validation set we produce the exactly correct output sentence 32.8% of the time.
", "votes": null, "replies": [] }, { "id": 78443, "author_name": "tonyrobinson", "author_url": "", "post_date": "05/14/2015 21:56:41", "content": "Ahhgg - we've finished?! So many interesting things to do, so little time...
\nWell done Eric seems like a good solid engineering approach.
\nSorry Merl - the combination of better LMs and all your other bits should have pulled it off - my fault completly for not having time.
\nThe main page for the Stanford POS tagger at http://nlp.stanford.edu/software/tagger.shtml says that "The basic download contains two trained tagger models for English". I don't think these can be allowed, the rules are clear that the training data is the only data to be used.
\nEric - so you didn't try any nonsense words?? The basic approach is clear, find the right slot and fill it. But there was nothing in the rules to say that you had to fill it with a word from English and indeed someone (sorry, don't remember who) made the suggestion that if you knew the slot but not the filler then use two blanks but not one. There's an obvious generalisation, search over all possible strings for the minimum edit distance between the words that can fill the slot and weight the edit distance by the posterior of the word filling the slot. For example, as many words contain the letter "i" then if that's true of the weighted list then it's better to replace the single " " representing the missing word with " i " so gaining one edit distance assuming the slot was correct and another assuming the letter "i" was present.
\nKaggle - can we continue to post results and get them evaluated? If not, will you release the code that does the evaluation? There were several obvious flaws in the setup, even after including the test data in the training was fixed. There's quite some scope for using this problem as part of taught course.
\nIf anyone wants our n-best lists to continue this work then just email me.
\n\nTony
\n--
Dr A J Robinson, Founder
We are hiring: www.speechmatics.com/careers
Speechmatics is a trading name of Cantab Research Limited
Phone direct: 01223 778240 office: 01223 794497
Company reg no GB 05697423, VAT reg no 925606030
51 Canterbury Street, Cambridge, CB4 3QG, UK
"The main page for the Stanford POS tagger at http://nlp.stanford.edu/software/tagger.shtml says that "The basic download contains two trained tagger models for English". I don't think these can be allowed, the rules are clear that the training data is the only data to be used."
\nOh, I see the concern now. This is presumably what mdda was getting at too. Hmm, the competition is over now, but for my own interest I reran my system without the POS tagger. On my cross-validation set I get 3.944 which I estimate would correspond to 3.908 on the real test set.
\n"Eric - so you didn't try any nonsense words??"
\nNo, that wasn't part of my final submission. The only thing I did, as already described, was insert a space for cases where I had confidence in the position but no good guess as to the word. Very early on I tried inserting "e" in such cases and didn't get good results, but that might have been before I had good position detection.
\nArguably it's kind of silly to play games like inserting nonsense words, and doesn't further the research goals of the competition, so this may be another argument for not using edit distance as the scoring metric for this task.
", "votes": null, "replies": [] }, { "id": 78591, "author_name": "chefele", "author_url": "", "post_date": "05/15/2015 23:19:59", "content": "[quote=Tony Robinson;78443]...there was nothing in the rules to say that you had to fill it with a word from English ...an obvious generalization, search over all possible strings for the minimum edit distance between the words that can fill the slot and weight the edit distance by the posterior of the word filling the slot. [/quote]
\nI tried that exact approach. Finding a median string is an NP complete problem, but I found a library that calculated median strings in a reasonable amount of time. I took a list of probable words (based on completing 4-grams or 3-grams), then calculated the weighted median string of those words.
\nUsing the median string instead of the most probable word gave an improvement of 0.25 - 0.3 on the leaderboard. So there's a non-trivial gain to be had by optimizing for the edit-distance scoring metric.
\nOn the other hand, if you start with a weak language model (as I did) these gains don't help much :)
\n[quote=Tony Robinson;78443] Well done Eric seems like a good solid engineering approach. [/quote]
\nDitto -- congratulations!.
", "votes": null, "replies": [] }, { "id": 78620, "author_name": "meron1", "author_url": "", "post_date": "05/16/2015 08:20:30", "content": "Congratulations!
\nInteresting to see a lot of similarities in our approaches...
\nI just posted a more detailed report describing my system in the "my approach" thread, but picking a couple of points from there:
\n- I used a somewhat similar strategy regarding word insertion, but having actually 4 possible actions: insert word / leave as is / insert space / insert filler word ("ae"), with a rule based decision (using tunable thresholds) depending on the length of the inserted word.
\n- A link from the word2vec site points to http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz, which includes a version of the test set reference, which matches 99.8% with the competition's test set. I re-calculated the distance score (using a plain edit distance function) for my system at several points of the process using this matched set, and got a result within 0.01 of my "official" submissions.
\n- I actually thought the edit distance was a nice twist. Another possible variation would be to allow competitors several entries for each sentence, with different weights.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "78335": "", "78397": "", "78401": "", "78409": "", "78443": "", "78550": "", "78591": "", "78620": "" }, "source": "existing_discussion", "complete": true, "competition_id": 3927, "competition_slug": "billion-word-imputation", "team_id": 112039, "solution_url": "/competitions/billion-word-imputation/writeups/one-million-monkeys-one-million-monkeys-approach" }