I've worked on this problem for a couple months now, and it's been a blast. Thanks for organizing it! My primary motivation is to get hands on experience with recent research in language processing, while have a fair benchmark to evaluate progress.
\nAt the moment, I'm using a feedforward neural network to identify the missing word location, with about 55% success rate and a recurrent neural network language model to predict the missing word with about 50% success rate. Frankly, having a language model that, given a missing word location, gets half the sentences right exceeds my wildest initial expectations.
\nThe combined success rate is only about 25%, because the models are largely independent. Given the way scoring works, I only submit about 16% of the phrases, and only get 11% of the words rate, based on a crude grid search. All this for a score of "5.02". While I'm having a lot of fun on the coding / experimentation side, I find it very difficult to relate my progress to either published results or other competitors:
\n* The score is an aggregation of the hole model, the language model and the aggregation scheme. I can't tell which model works well and which one needs improvement.
\n* The problem is designed such that comparison with published research is hard. Most published research uses just a language model, and uses either perplexity or, more rarely, word error rate. I could not find a single paper that deals with finding the location where to insert a missing word.
\n* Perplexity comparison is a dead end, because the contest encourages considering the full pre/post context for each word, while published research I found is concerned only with the pre context. For the record, I get perplexity of 11.2, which is so low that I suspect a bug somewhere.
\n* The evaluation code is not open source. It would be super nice to have the source of the evaluation code public, so I could run it on my cross-validation data set, instead of using crufty approximations and/or spending precious time trying to replicate it.
\nI hope that the numbers I made public above will make easier for other competitors to evaluate their approach and/or progress on different aspects of the problem.
\nRegards.
", "messages": [ { "id": "56914", "postDate": "10/28/2014 17:42:04", "content": "I've worked on this problem for a couple months now, and it's been a blast. Thanks for organizing it! My primary motivation is to get hands on experience with recent research in language processing, while have a fair benchmark to evaluate progress.
\nAt the moment, I'm using a feedforward neural network to identify the missing word location, with about 55% success rate and a recurrent neural network language model to predict the missing word with about 50% success rate. Frankly, having a language model that, given a missing word location, gets half the sentences right exceeds my wildest initial expectations.
\nThe combined success rate is only about 25%, because the models are largely independent. Given the way scoring works, I only submit about 16% of the phrases, and only get 11% of the words rate, based on a crude grid search. All this for a score of "5.02". While I'm having a lot of fun on the coding / experimentation side, I find it very difficult to relate my progress to either published results or other competitors:
\n* The score is an aggregation of the hole model, the language model and the aggregation scheme. I can't tell which model works well and which one needs improvement.
\n* The problem is designed such that comparison with published research is hard. Most published research uses just a language model, and uses either perplexity or, more rarely, word error rate. I could not find a single paper that deals with finding the location where to insert a missing word.
\n* Perplexity comparison is a dead end, because the contest encourages considering the full pre/post context for each word, while published research I found is concerned only with the pre context. For the record, I get perplexity of 11.2, which is so low that I suspect a bug somewhere.
\n* The evaluation code is not open source. It would be super nice to have the source of the evaluation code public, so I could run it on my cross-validation data set, instead of using crufty approximations and/or spending precious time trying to replicate it.
\nI hope that the numbers I made public above will make easier for other competitors to evaluate their approach and/or progress on different aspects of the problem.
\nRegards.
", "rawMarkdown": "", "votes": null }, { "id": "56998", "postDate": "10/29/2014 16:29:15", "content": "Podragu, thanks for sharing. I'm also having a lot of fun with this problem & have a few thoughts to share.
\nFirst, I like the longer-than-average duration of this contest. It certainly gives plenty of time to mull over the problem & try multiple approaches.
\nSecond, I know some people are trying neural-nets for this, though FYI, I've been been using a simpler approach using simple statistics over n-grams. I identify a good insertion point about 58% of the time (compared to podragu's 55% above using NN). There's probably only so much you can extract from the dataset, regardless of the method, due to the ambiguities of language, etc.
\nAlso, one of the biggest challenges has been writing (and rewriting!) code to make it efficient. I'm limited to a single machine & this task seems hungry for both time & memory.
\n[quote=podragu;56914]...I could not find a single paper that deals with finding the location where to insert a missing word.[/quote]
\nTrue, this task doesn't match up well with the existing literature, but maybe this means that this is an area ready for some new research! Given the distinct clusters of scores on the leaderboard, I suspect there are distinct groups of approaches people are using, so it'll be interesting to compare approaches when it's all over.
\n[quote=podragu;56914]* The evaluation code is not open source. It would be super nice to have the source of the evaluation code public, so I could run it on my cross-validation data set, instead of using crufty approximations and/or spending precious time trying to replicate it.[/quote]
\nI'm not sure what you mean -- do you mean the Levenshtein distance metric? (If so, Google "edit distance in {your-programming-language-of-choice}" to find a library, or look at something like this.)
\nThanks & good luck everyone.
", "rawMarkdown": "", "votes": null }, { "id": "57007", "postDate": "10/29/2014 18:32:43", "content": "I'm also taking an n-gram based approach.
\nHere's one little trick that may be helpful to some. If you can identify the position of the missing word (with some confidence), but not the word itself, generate an output sentence with a space inserted at the missing position. Your edit distance will improve assuming you guessed the position correctly.
", "rawMarkdown": "", "votes": null }, { "id": "57921", "postDate": "11/12/2014 16:12:44", "content": "Interesting...
\nTwo questions about this:
\n1) What's the syntax to add a space?
\n2) If the Levenshtein distance uses the same penalty for deletion and substitution, why would the score improve
", "rawMarkdown": "", "votes": null }, { "id": "57922", "postDate": "11/12/2014 16:31:39", "content": "Oops... my mistake...
\nI guess I misunderstood the metric...
\nA "regular" edit distance would just give a distance of 1.0 when you remove one word - so where is this 5.5 coming from?
", "rawMarkdown": "", "votes": null }, { "id": "57935", "postDate": "11/12/2014 17:52:34", "content": "The Levenshtein distance is the minimum number of single-character edits (i.e. insertions, deletions or substitutions) required to change one sentence into the other. So if you take a sentence and randomly delete a 4 character word & the space after it, the "distance" to the original sentence 4+1=5 (not 1).
\nNext, the evaluation metric for this contest is average distance across multiple sentences. Since word lengths differ, you can get a non-integer average distance (e.g. 5.5).
", "rawMarkdown": "", "votes": null }, { "id": "57939", "postDate": "11/12/2014 19:01:39", "content": "Thanks!
", "rawMarkdown": "", "votes": null }, { "id": "63318", "postDate": "01/29/2015 18:29:20", "content": "Hello All
This is Sahithya from India
I'm pursuing engineering in Information Science
I'm planning to work on this project since it seems very interesting
I'm very new to language modelling , and I very badly want to learn and make a very good project, definitely not for this competition.
would anyone guide me do so?
do notify
\nsahitya.mj@gmail.com
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 56998, "author_name": "chefele", "author_url": "", "post_date": "10/29/2014 16:29:15", "content": "Podragu, thanks for sharing. I'm also having a lot of fun with this problem & have a few thoughts to share.
\nFirst, I like the longer-than-average duration of this contest. It certainly gives plenty of time to mull over the problem & try multiple approaches.
\nSecond, I know some people are trying neural-nets for this, though FYI, I've been been using a simpler approach using simple statistics over n-grams. I identify a good insertion point about 58% of the time (compared to podragu's 55% above using NN). There's probably only so much you can extract from the dataset, regardless of the method, due to the ambiguities of language, etc.
\nAlso, one of the biggest challenges has been writing (and rewriting!) code to make it efficient. I'm limited to a single machine & this task seems hungry for both time & memory.
\n[quote=podragu;56914]...I could not find a single paper that deals with finding the location where to insert a missing word.[/quote]
\nTrue, this task doesn't match up well with the existing literature, but maybe this means that this is an area ready for some new research! Given the distinct clusters of scores on the leaderboard, I suspect there are distinct groups of approaches people are using, so it'll be interesting to compare approaches when it's all over.
\n[quote=podragu;56914]* The evaluation code is not open source. It would be super nice to have the source of the evaluation code public, so I could run it on my cross-validation data set, instead of using crufty approximations and/or spending precious time trying to replicate it.[/quote]
\nI'm not sure what you mean -- do you mean the Levenshtein distance metric? (If so, Google "edit distance in {your-programming-language-of-choice}" to find a library, or look at something like this.)
\nThanks & good luck everyone.
", "votes": null, "replies": [] }, { "id": 57007, "author_name": "onemillionmonkeys", "author_url": "", "post_date": "10/29/2014 18:32:43", "content": "I'm also taking an n-gram based approach.
\nHere's one little trick that may be helpful to some. If you can identify the position of the missing word (with some confidence), but not the word itself, generate an output sentence with a space inserted at the missing position. Your edit distance will improve assuming you guessed the position correctly.
", "votes": null, "replies": [] }, { "id": 57921, "author_name": "meron1", "author_url": "", "post_date": "11/12/2014 16:12:44", "content": "Interesting...
\nTwo questions about this:
\n1) What's the syntax to add a space?
\n2) If the Levenshtein distance uses the same penalty for deletion and substitution, why would the score improve
", "votes": null, "replies": [] }, { "id": 57922, "author_name": "meron1", "author_url": "", "post_date": "11/12/2014 16:31:39", "content": "Oops... my mistake...
\nI guess I misunderstood the metric...
\nA "regular" edit distance would just give a distance of 1.0 when you remove one word - so where is this 5.5 coming from?
", "votes": null, "replies": [] }, { "id": 57935, "author_name": "chefele", "author_url": "", "post_date": "11/12/2014 17:52:34", "content": "The Levenshtein distance is the minimum number of single-character edits (i.e. insertions, deletions or substitutions) required to change one sentence into the other. So if you take a sentence and randomly delete a 4 character word & the space after it, the "distance" to the original sentence 4+1=5 (not 1).
\nNext, the evaluation metric for this contest is average distance across multiple sentences. Since word lengths differ, you can get a non-integer average distance (e.g. 5.5).
", "votes": null, "replies": [] }, { "id": 57939, "author_name": "meron1", "author_url": "", "post_date": "11/12/2014 19:01:39", "content": "Thanks!
", "votes": null, "replies": [] }, { "id": 63318, "author_name": "sahityama", "author_url": "", "post_date": "01/29/2015 18:29:20", "content": "Hello All
This is Sahithya from India
I'm pursuing engineering in Information Science
I'm planning to work on this project since it seems very interesting
I'm very new to language modelling , and I very badly want to learn and make a very good project, definitely not for this competition.
would anyone guide me do so?
do notify
\nsahitya.mj@gmail.com
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "56914": "", "56998": "", "57007": "", "57921": "", "57922": "", "57935": "", "57939": "", "63318": "" }, "source": "meta" }