I've just sent the email below to 1-billion-word-language-modeling-benchmark-discuss@googlegroups.com. There are major problems with this task, I suggest that it is closed and we discuss on the mailing list and then task is reopened when the problems are sorted out.
\n\nTony Robinson
\n\nTo: 1-billion-word-language-modeling-benchmark-discuss@googlegroups.com
\n\nHas anyone worked out what Kaggle have done with our data?
\n\nThere are two issues:
\n\nthey seem to have put all the data in train.txt (that is not just news.en-00001-of-00100 to news.en-00099-of-00100 but the heldout part 00000 as well)
they have also chosen a different partitioning of the heldout data, not just news.en.heldout-00000-of-00050 and news.en.heldout-00001-of-00050 but randomly from all of them.
I'll also post to the Kaggle group and point them over here.
\n\n** Cantab is hiring: www.cantabResearch.com/openings **
\nDr A J Robinson, Founder, Cantab Research Ltd
\nCompany reg no GB 05697423, VAT reg no 925606030
\n51 Canterbury Street, Cambridge, CB4 3QG, UK
I said that I'd report back once we'd processed the entire test set and we've just done that.
\n\nI thought it would take a week or so and my last post on this was 58 days ago - so a month or so would have been a better estimate.
\n\nWe took something like 160 CPU days (assuming single core - so 20 8 core days) for our current submission and our earlier runs had to be aborted as they were going to take too long.
\n\nSo I think this task is all about really good models of language and very efficient search - which is two things we love doing. It'll be a real challenge keeping everything going fast enough - and that's the engineering problem it should be.
\n\nThanks,
\n\n\nTony
", "votes": 4 }, { "id": 50269, "postDate": "2014-07-10T14:09:13.947Z", "content": "Excellent work! I get nervous if an algorithm runs for more than a few hours, and I'm downright antsy after a couple of days... I'm impressed with your patience and confidence, but it looks like it paid off. Looks like the competition is on. :)
", "votes": 1 }, { "id": 50233, "postDate": "2014-07-10T00:06:03.887Z", "content": "How amazing! Grats on the nr. 1 position too. This contest is really pushing the boundaries.
", "votes": 1 }, { "id": 44316, "postDate": "2014-05-09T16:42:55.077Z", "content": "Would it help if you gave the location of the missing word?
\nTask complexity is very high:
\nWith Levenshtein distance I think the location does not matter for the score:
\n"I shot the xxxx, but i did not shoot the deputy"
\n"I shot the, but i did not shoot the xxxx deputy"
\nwould have an equal edit distance to the correct sentence. So you won't know if your word location detection is working, and finding the correct location is not rewarded?
\nI'll have a look if predicting the average word length for all in test, using chars that are most popular in the English language, in a random location gives a workable score. A word like "etaoins" to see if substitutions are rewarded more than insertions (predicting 'sheriffs' when correct is 'sheriff').
\nFrom test:
\n99,"Now add the egg and continue to ."
\nWhat to do here?
\nNow you add the egg and continue to
\nNow add the boiled egg and continue to
\nNow add the egg and happily continue to
\n163,"Welcome to Crazy ."
\n201,"Kevin had 20 points ."
\n293041,"up ."
\n293035,"Spencer ."
\nEdit: I am enthusiastic for this competition either way. A massive dataset. An interesting problem. Ty Kaggle and researchers for letting me fool around with this.
", "votes": 1 }, { "id": 54585, "postDate": "2014-09-24T07:15:49.837Z", "content": "These are not problematic sentences. One of the first things that you have to realise about this tasks is that if you haven't got a clue where to insert a word then don't.
\n\nI don't regard this task as algorithmically hard, we've already made good progress so everyone can see that there is useful work you can do here. It does take a fair amount of compute, and I'd imagine that has limited the popularity.
", "votes": 2 }, { "id": 44318, "postDate": "2014-05-09T17:00:31.257Z", "content": "I hear your concerns. This is a for-fun playground competition, so we can always reboot and dial down the difficulty if it turns out to too masochistic. Half the fun of playground competitions is pushing things (like our scoring servers!) to a breaking point. The reverse game of life competition turned into a lot of fun, despite the computational arms race it created.
\nThe Levenshtein indeed requires the right word in the right place. There wouldn't be much use in putting the right word in the wrong place! The alternative was just to ask for the missing word only, which I find a bit more boring and less NLP-intensive. I didn't want the winning model to be some clever variation of a list of the most common words.
", "votes": 2 }, { "id": 44313, "postDate": "2014-05-09T15:34:43.657Z", "content": "Hi William,
\nThanks for helping sort this out.
\nI recommend that you cat all 99 files in training-monolingual.tokenized.shuffled to create train.txt, leaving out the one that is in the heldout directory.
\nRunning our big language models on just one of the heldout shards takes about three days of compute, so all 50 would be 150 CPU days. This is just to compute the log probabilities of the next word (as we have made publicly available on the first ten shards), I would guess that to search over possibilities of missing words as well may be one or two orders of magnitude more work, so perhaps 4 or 40 CPU years. The lower end of this isn't impossible for us, but wouldn't be possible for the bright coder that wants to play with this task to see what they might learn, so you may find that the winner is the group with the most compute and not the best ideas. Taking the test size down would allow more people to play, but as you say, the results would be noisier.
\nTony
", "votes": 2 }, { "id": 67547, "postDate": "2015-03-21T18:50:45.063Z", "content": "Interesting experiment. I can see that my system frequently makes mistakes that no human would make which might suggest we are far below human performance. On the other hand, it is surely also the case that the system is better than human performance in some situations.
" }, { "id": 67003, "postDate": "2015-03-18T16:17:00.583Z", "content": "A simple experiment:
\nRandomly selected 100 sentences (held out from the training set), and randomly omitted one word in each sentence.
\nWithout seeing the original sentences, tried to impute the missing word as best I could (sometimes not adding any word, if in doubt)
\nFinally - calculated the distance to the original sentences:
\n- omitted sentences (before imputation) : 5.68 (reasonably close to the expected 5.55)
\n- manually imputed sentences: 3.78 - not too far from the current top entry...
\n\nWould be interesting to test this on a larger set (and have native English speakers...)
" }, { "id": 66183, "postDate": "2015-03-14T14:42:40.140Z", "content": "I'll take a shot at the crazy examples.
\n"He added that people should not mess with mother nature , and let the sharks be ."
\n"The 's bloody body was discovered on a bed ." It' almost certainly man or woman which is like 50/50.
\n"Now add the egg and continue to stir." Other cooking verbs would also fit.
\n"Welcome to Crazy town."
\n"Kevin had 20 points left."
\n"Look up ."
\n"Spencer stopped." Although any verb could work there, so just pick the most common.
" }, { "id": 44570, "postDate": "2014-05-15T03:02:23.587Z", "content": "My guess is that the "final" solution will require multiple types of models. One to create inference as to the location of the missing word and another to select the needed word within context. The first problem is hard. The second is much harder.
" }, { "id": 44427, "postDate": "2014-05-12T20:01:18.497Z", "content": "[quote=William Cukierski;44318]
\nThe Levenshtein indeed requires the right word in the right place. There wouldn't be much use in putting the right word in the wrong place! The alternative was just to ask for the missing word only, which I find a bit more boring and less NLP-intensive. I didn't want the winning model to be some clever variation of a list of the most common words.
\n[/quote]
\nMan, that's rough. Looking at some of this data I can hardly guess the correct word and location even when googling for an exact quote, and this is in samples I'm pulling because I can identify that the sentences are obviously incomplete (by hand, mind you.... no algos yet).
\nShould be an interesting competition!
" }, { "id": 44409, "postDate": "2014-05-12T16:07:07.650Z", "content": "Now that we've cleared up the training data issue we'll give this task a go and report back fairly soon as to how long the whole test test (all heldout data) takes. We've already thought of some ways to make our model run faster, no doubt we'll think of more - this is the sort of thing we want to get out of running this task anyway.
" }, { "id": 44304, "postDate": "2014-05-09T14:04:32.707Z", "content": "Forgot to add: I can add a warning to the data page if you're concerned people will take the Kaggle dataset and publish perplexities on it.
" }, { "id": 44298, "postDate": "2014-05-09T13:56:52.740Z", "content": "[quote=Tony Robinson;44291]
\n* they seem to have put all the data in train.txt (that is not just news.en-00001-of-00100 to news.en-00099-of-00100 but the heldout part 00000 as well)
\n* they have also chosen a different partitioning of the heldout data, not just news.en.heldout-00000-of-00050 and news.en.heldout-00001-of-00050 but randomly from all of them.
\n[/quote]
\nHey Tony, thanks for reaching out!
\nTo summarize what we've done, it was just a big cat of all the heldout files into a the test set (order shuffled) and a big cat of all the non-heldout files into the train set.
" }, { "id": 44291, "postDate": "2014-05-09T10:46:51.227Z", "content": "I've just sent the email below to 1-billion-word-language-modeling-benchmark-discuss@googlegroups.com. There are major problems with this task, I suggest that it is closed and we discuss on the mailing list and then task is reopened when the problems are sorted out.
\n\nTony Robinson
\n\nTo: 1-billion-word-language-modeling-benchmark-discuss@googlegroups.com
\n\nHas anyone worked out what Kaggle have done with our data?
\n\nThere are two issues:
\n\nthey seem to have put all the data in train.txt (that is not just news.en-00001-of-00100 to news.en-00099-of-00100 but the heldout part 00000 as well)
they have also chosen a different partitioning of the heldout data, not just news.en.heldout-00000-of-00050 and news.en.heldout-00001-of-00050 but randomly from all of them.
I'll also post to the Kaggle group and point them over here.
\n\n** Cantab is hiring: www.cantabResearch.com/openings **
\nDr A J Robinson, Founder, Cantab Research Ltd
\nCompany reg no GB 05697423, VAT reg no 925606030
\n51 Canterbury Street, Cambridge, CB4 3QG, UK
I said that I'd report back once we'd processed the entire test set and we've just done that.
\n\nI thought it would take a week or so and my last post on this was 58 days ago - so a month or so would have been a better estimate.
\n\nWe took something like 160 CPU days (assuming single core - so 20 8 core days) for our current submission and our earlier runs had to be aborted as they were going to take too long.
\n\nSo I think this task is all about really good models of language and very efficient search - which is two things we love doing. It'll be a real challenge keeping everything going fast enough - and that's the engineering problem it should be.
\n\nThanks,
\n\n\nTony
", "votes": 4, "replies": [] }, { "id": 50269, "author_name": "ChipMonkey", "author_url": "", "post_date": "2014-07-10T14:09:13.947000", "content": "Excellent work! I get nervous if an algorithm runs for more than a few hours, and I'm downright antsy after a couple of days... I'm impressed with your patience and confidence, but it looks like it paid off. Looks like the competition is on. :)
", "votes": 1, "replies": [] }, { "id": 50233, "author_name": "Triskelion", "author_url": "", "post_date": "2014-07-10T00:06:03.887000", "content": "How amazing! Grats on the nr. 1 position too. This contest is really pushing the boundaries.
", "votes": 1, "replies": [] }, { "id": 44316, "author_name": "Triskelion", "author_url": "", "post_date": "2014-05-09T16:42:55.077000", "content": "Would it help if you gave the location of the missing word?
\nTask complexity is very high:
\nWith Levenshtein distance I think the location does not matter for the score:
\n"I shot the xxxx, but i did not shoot the deputy"
\n"I shot the, but i did not shoot the xxxx deputy"
\nwould have an equal edit distance to the correct sentence. So you won't know if your word location detection is working, and finding the correct location is not rewarded?
\nI'll have a look if predicting the average word length for all in test, using chars that are most popular in the English language, in a random location gives a workable score. A word like "etaoins" to see if substitutions are rewarded more than insertions (predicting 'sheriffs' when correct is 'sheriff').
\nFrom test:
\n99,"Now add the egg and continue to ."
\nWhat to do here?
\nNow you add the egg and continue to
\nNow add the boiled egg and continue to
\nNow add the egg and happily continue to
\n163,"Welcome to Crazy ."
\n201,"Kevin had 20 points ."
\n293041,"up ."
\n293035,"Spencer ."
\nEdit: I am enthusiastic for this competition either way. A massive dataset. An interesting problem. Ty Kaggle and researchers for letting me fool around with this.
", "votes": 1, "replies": [] }, { "id": 54585, "author_name": "Tony Robinson", "author_url": "", "post_date": "2014-09-24T07:15:49.837000", "content": "These are not problematic sentences. One of the first things that you have to realise about this tasks is that if you haven't got a clue where to insert a word then don't.
\n\nI don't regard this task as algorithmically hard, we've already made good progress so everyone can see that there is useful work you can do here. It does take a fair amount of compute, and I'd imagine that has limited the popularity.
", "votes": 2, "replies": [] }, { "id": 44318, "author_name": "Will Cukierski", "author_url": "", "post_date": "2014-05-09T17:00:31.257000", "content": "I hear your concerns. This is a for-fun playground competition, so we can always reboot and dial down the difficulty if it turns out to too masochistic. Half the fun of playground competitions is pushing things (like our scoring servers!) to a breaking point. The reverse game of life competition turned into a lot of fun, despite the computational arms race it created.
\nThe Levenshtein indeed requires the right word in the right place. There wouldn't be much use in putting the right word in the wrong place! The alternative was just to ask for the missing word only, which I find a bit more boring and less NLP-intensive. I didn't want the winning model to be some clever variation of a list of the most common words.
", "votes": 2, "replies": [] }, { "id": 44313, "author_name": "Tony Robinson", "author_url": "", "post_date": "2014-05-09T15:34:43.657000", "content": "Hi William,
\nThanks for helping sort this out.
\nI recommend that you cat all 99 files in training-monolingual.tokenized.shuffled to create train.txt, leaving out the one that is in the heldout directory.
\nRunning our big language models on just one of the heldout shards takes about three days of compute, so all 50 would be 150 CPU days. This is just to compute the log probabilities of the next word (as we have made publicly available on the first ten shards), I would guess that to search over possibilities of missing words as well may be one or two orders of magnitude more work, so perhaps 4 or 40 CPU years. The lower end of this isn't impossible for us, but wouldn't be possible for the bright coder that wants to play with this task to see what they might learn, so you may find that the winner is the group with the most compute and not the best ideas. Taking the test size down would allow more people to play, but as you say, the results would be noisier.
\nTony
", "votes": 2, "replies": [] }, { "id": 67547, "author_name": "Eric Jackson", "author_url": "", "post_date": "2015-03-21T18:50:45.063000", "content": "Interesting experiment. I can see that my system frequently makes mistakes that no human would make which might suggest we are far below human performance. On the other hand, it is surely also the case that the system is better than human performance in some situations.
", "votes": 0, "replies": [] }, { "id": 67003, "author_name": "meron", "author_url": "", "post_date": "2015-03-18T16:17:00.583000", "content": "A simple experiment:
\nRandomly selected 100 sentences (held out from the training set), and randomly omitted one word in each sentence.
\nWithout seeing the original sentences, tried to impute the missing word as best I could (sometimes not adding any word, if in doubt)
\nFinally - calculated the distance to the original sentences:
\n- omitted sentences (before imputation) : 5.68 (reasonably close to the expected 5.55)
\n- manually imputed sentences: 3.78 - not too far from the current top entry...
\n\nWould be interesting to test this on a larger set (and have native English speakers...)
", "votes": 0, "replies": [] }, { "id": 66183, "author_name": "Houshalter", "author_url": "", "post_date": "2015-03-14T14:42:40.140000", "content": "I'll take a shot at the crazy examples.
\n"He added that people should not mess with mother nature , and let the sharks be ."
\n"The 's bloody body was discovered on a bed ." It' almost certainly man or woman which is like 50/50.
\n"Now add the egg and continue to stir." Other cooking verbs would also fit.
\n"Welcome to Crazy town."
\n"Kevin had 20 points left."
\n"Look up ."
\n"Spencer stopped." Although any verb could work there, so just pick the most common.
", "votes": 0, "replies": [] }, { "id": 44570, "author_name": "Jared Thompson", "author_url": "", "post_date": "2014-05-15T03:02:23.587000", "content": "My guess is that the "final" solution will require multiple types of models. One to create inference as to the location of the missing word and another to select the needed word within context. The first problem is hard. The second is much harder.
", "votes": 0, "replies": [] }, { "id": 44427, "author_name": "ChipMonkey", "author_url": "", "post_date": "2014-05-12T20:01:18.497000", "content": "[quote=William Cukierski;44318]
\nThe Levenshtein indeed requires the right word in the right place. There wouldn't be much use in putting the right word in the wrong place! The alternative was just to ask for the missing word only, which I find a bit more boring and less NLP-intensive. I didn't want the winning model to be some clever variation of a list of the most common words.
\n[/quote]
\nMan, that's rough. Looking at some of this data I can hardly guess the correct word and location even when googling for an exact quote, and this is in samples I'm pulling because I can identify that the sentences are obviously incomplete (by hand, mind you.... no algos yet).
\nShould be an interesting competition!
", "votes": 0, "replies": [] }, { "id": 44409, "author_name": "Tony Robinson", "author_url": "", "post_date": "2014-05-12T16:07:07.650000", "content": "Now that we've cleared up the training data issue we'll give this task a go and report back fairly soon as to how long the whole test test (all heldout data) takes. We've already thought of some ways to make our model run faster, no doubt we'll think of more - this is the sort of thing we want to get out of running this task anyway.
", "votes": 0, "replies": [] }, { "id": 44304, "author_name": "Will Cukierski", "author_url": "", "post_date": "2014-05-09T14:04:32.707000", "content": "Forgot to add: I can add a warning to the data page if you're concerned people will take the Kaggle dataset and publish perplexities on it.
", "votes": 0, "replies": [] }, { "id": 44298, "author_name": "Will Cukierski", "author_url": "", "post_date": "2014-05-09T13:56:52.740000", "content": "[quote=Tony Robinson;44291]
\n* they seem to have put all the data in train.txt (that is not just news.en-00001-of-00100 to news.en-00099-of-00100 but the heldout part 00000 as well)
\n* they have also chosen a different partitioning of the heldout data, not just news.en.heldout-00000-of-00050 and news.en.heldout-00001-of-00050 but randomly from all of them.
\n[/quote]
\nHey Tony, thanks for reaching out!
\nTo summarize what we've done, it was just a big cat of all the heldout files into a the test set (order shuffled) and a big cat of all the non-heldout files into the train set.
", "votes": 0, "replies": [] }, { "id": 54562, "author_name": "", "author_url": "", "post_date": "2014-09-24T03:38:00.103000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "50221": "", "50269": "", "50233": "", "44316": "", "54585": "", "44318": "", "44313": "", "67547": "", "67003": "", "66183": "", "44570": "", "44427": "", "44409": "", "44304": "", "44298": "", "44291": "I've just sent the email below to 1-billion-word-language-modeling-benchmark-discuss@googlegroups.com. There are major problems with this task, I suggest that it is closed and we discuss on the mailing list and then task is reopened when the problems are sorted out.\n\nTony Robinson\n\nTo: 1-billion-word-language-modeling-benchmark-discuss@googlegroups.com\n\nHas anyone worked out what Kaggle have done with our data?\n\nThere are two issues:\n\n* they seem to have put all the data in train.txt (that is not just news.en-00001-of-00100 to news.en-00099-of-00100 but the heldout part 00000 as well)\n\n* they have also chosen a different partitioning of the heldout data, not just news.en.heldout-00000-of-00050 and news.en.heldout-00001-of-00050 but randomly from all of them.\n\nI'll also post to the Kaggle group and point them over here.\n\nTony \n-- \n** Cantab is hiring: www.cantabResearch.com/openings ** \nDr A J Robinson, Founder, Cantab Research Ltd \nCompany reg no GB 05697423, VAT reg no 925606030 \n51 Canterbury Street, Cambridge, CB4 3QG, UK", "54562": "" } }