The following is a description of the method I used used to impute missing words - I'll try to make it as short as possible.
\nHopefully there will be an opportunity for us to share more detailed information about our systems - perhaps some (joint?) publication?
\nTo impute the missing words, a two stage process is used. In the first stage, the position of the missing word is detected,
and in the second stage, a replacement word is selected.
Computational resources:
All the computation were done on a single home computer (used to be high end, 4 years ago...) with a fairly new nvidia gtx780 GPU.
Due to the time required to train models, only a very limited number of experiments were performed. In particular,
a lot of guesstimating was done in selecting the features to be used, model topology and size, and practically every tunable parameter.
Better selection is certainly possible, given more time and computational resources.
1) detecting position of missing word
The position of the missing word is detected using a DNN.
The Stanford 'Caffe' framework was used in this work. It supports GPU computation, and allows fairly easy custom extensions.
Description of the network:
\nThe simplest way to construct a network would be to predict, for a given context, if a word is missing in the middle of this context.
The topology actually used is a bit more complicated - the output predicts the "probabilities" of word omission for each possible positions in the context.
It would have been possible to build separate nets for predicting the probability for each position separately, but by making it into one net, it could learn some shared features.
Input:
- a context of 14 consecutive words taken out of a sentence from which a word was omitted.
The context is padded with sentence boundary markers if necessary
The context may or may not include a position with a missing word.
- each word is represented as by vector of 256 features.
- For training: a label is also given, corresponding to the position of the missing word within this context (relative to the beginning of the context). A label of 0 is given if no word is missing within the sequence.
Each word in the context vector is represented by a vector of 256 values, comprising of:
- word2vec representation of the word (dimension 223).
- POS features (binary, dimension 27) - a list of around 250 words was manually assembled for closed-class POS (prepositions, pronouns, symbols...). a word outside this list (nouns, verbs...) is assigned 0 for all these binary features. Numbers are specially detected and marked as numbers.
- special tags (binary) for: sentence boundary, unknown word
- indicators for character level word features - currently just two such features: "word includes a digit?" and "word end with s?" (intended to have more of these, but didn't have time)
- word probability (taking into account 4 previous words)- calculated by a small ngram language model (details below)
Intermediate layers:
From the input, The network splits to two main "trunks", which are then combined to produce the output.
The first trunk consists of standard stack of repetitions of : (inner product layer + ReLU).
The second trunk consists of a single convolution layer, followed by repetitions of : (inner product layer + ReLU).
The convolution layer looks at a sliding window of 8 consecutive words at a time.
The outputs of the two trunks are then concatenated and followed by repetitions of : (inner product layer + ReLU).
The reason for adding the convolutional trunk is that it may help catch some of the more local features which are common for all positions.
In total there are 5 hidden layers (for any path from input to output). The convolution trunk has 1024 features at its base, and the inner product trunk has 2048 nodes at its base (both get narrower towards the top).
\nOutput:
The output layer of the network consists of 14 units, indicating the position of the missing word:
e.g. unit 5 corresponds to the probability of a word missing between the 5th and 6th words.
unit 0 corresponds to no missing word.
Detection:
To perform actual detection, the system is fed with consecutive sliding-window contexts of 18 words (14 context words + 4 preceding words needed to calculate LM scores).
each context is run through the net, which outputs its prediction for the position of the missing word.
Consecutive contexts assign different values to each position in the sentence. These values are then averaged, and the position with the highest value is selected as the position to insert a word.
On my development set, the position detection accuracy was about 71% (the DNN training still has not reached its peak...)
In addition to predicting the position of the missing word, the DNN output can also serves as a useful confidence score (i.e. for the sentences whose confidence score is in the top 55%, the detection accuracy is over 90%), which in this task is very important.
2) selecting replacement word
\nOnce the position of the missing word is determined, the word with the highest probability, according to a language model, is selected.
\nTwo ngram LM's were built:
- a small LM - used to calculate input features for the position detection DNN - 5gram KN discounting, 20 million ngram entries
- a large LM - used to find word to insert into detected position - 6gram KN discounting, 220 million ngrams
Both are standard models, trained using the SRILM toolkit.
A lot of effort (and even more time) went into trying different configurations and modifications of rnnlm (rnnlm.org).
Numerous experiments with rnnlm either did not yield significant improvement, or had to be abandoned due to their slowness or memory thirst.
The best results I could get (using the original code, hidden=120, direct=3500) did at best match the ngram results in my setup.
On my development set, the accuracy of correct word selection (for a position correctly detected) was around 48% (case sensitive) for either ngram or rnn, or 53% (case insensitive).
Linear combination of the two gave an improvement of about 1%.
The following is a description of the method I used used to impute missing words - I'll try to make it as short as possible.
\nHopefully there will be an opportunity for us to share more detailed information about our systems - perhaps some (joint?) publication?
\nTo impute the missing words, a two stage process is used. In the first stage, the position of the missing word is detected,
and in the second stage, a replacement word is selected.
Computational resources:
All the computation were done on a single home computer (used to be high end, 4 years ago...) with a fairly new nvidia gtx780 GPU.
Due to the time required to train models, only a very limited number of experiments were performed. In particular,
a lot of guesstimating was done in selecting the features to be used, model topology and size, and practically every tunable parameter.
Better selection is certainly possible, given more time and computational resources.
1) detecting position of missing word
The position of the missing word is detected using a DNN.
The Stanford 'Caffe' framework was used in this work. It supports GPU computation, and allows fairly easy custom extensions.
Description of the network:
\nThe simplest way to construct a network would be to predict, for a given context, if a word is missing in the middle of this context.
The topology actually used is a bit more complicated - the output predicts the "probabilities" of word omission for each possible positions in the context.
It would have been possible to build separate nets for predicting the probability for each position separately, but by making it into one net, it could learn some shared features.
Input:
- a context of 14 consecutive words taken out of a sentence from which a word was omitted.
The context is padded with sentence boundary markers if necessary
The context may or may not include a position with a missing word.
- each word is represented as by vector of 256 features.
- For training: a label is also given, corresponding to the position of the missing word within this context (relative to the beginning of the context). A label of 0 is given if no word is missing within the sequence.
Each word in the context vector is represented by a vector of 256 values, comprising of:
- word2vec representation of the word (dimension 223).
- POS features (binary, dimension 27) - a list of around 250 words was manually assembled for closed-class POS (prepositions, pronouns, symbols...). a word outside this list (nouns, verbs...) is assigned 0 for all these binary features. Numbers are specially detected and marked as numbers.
- special tags (binary) for: sentence boundary, unknown word
- indicators for character level word features - currently just two such features: "word includes a digit?" and "word end with s?" (intended to have more of these, but didn't have time)
- word probability (taking into account 4 previous words)- calculated by a small ngram language model (details below)
Intermediate layers:
From the input, The network splits to two main "trunks", which are then combined to produce the output.
The first trunk consists of standard stack of repetitions of : (inner product layer + ReLU).
The second trunk consists of a single convolution layer, followed by repetitions of : (inner product layer + ReLU).
The convolution layer looks at a sliding window of 8 consecutive words at a time.
The outputs of the two trunks are then concatenated and followed by repetitions of : (inner product layer + ReLU).
The reason for adding the convolutional trunk is that it may help catch some of the more local features which are common for all positions.
In total there are 5 hidden layers (for any path from input to output). The convolution trunk has 1024 features at its base, and the inner product trunk has 2048 nodes at its base (both get narrower towards the top).
\nOutput:
The output layer of the network consists of 14 units, indicating the position of the missing word:
e.g. unit 5 corresponds to the probability of a word missing between the 5th and 6th words.
unit 0 corresponds to no missing word.
Detection:
To perform actual detection, the system is fed with consecutive sliding-window contexts of 18 words (14 context words + 4 preceding words needed to calculate LM scores).
each context is run through the net, which outputs its prediction for the position of the missing word.
Consecutive contexts assign different values to each position in the sentence. These values are then averaged, and the position with the highest value is selected as the position to insert a word.
On my development set, the position detection accuracy was about 71% (the DNN training still has not reached its peak...)
In addition to predicting the position of the missing word, the DNN output can also serves as a useful confidence score (i.e. for the sentences whose confidence score is in the top 55%, the detection accuracy is over 90%), which in this task is very important.
2) selecting replacement word
\nOnce the position of the missing word is determined, the word with the highest probability, according to a language model, is selected.
\nTwo ngram LM's were built:
- a small LM - used to calculate input features for the position detection DNN - 5gram KN discounting, 20 million ngram entries
- a large LM - used to find word to insert into detected position - 6gram KN discounting, 220 million ngrams
Both are standard models, trained using the SRILM toolkit.
A lot of effort (and even more time) went into trying different configurations and modifications of rnnlm (rnnlm.org).
Numerous experiments with rnnlm either did not yield significant improvement, or had to be abandoned due to their slowness or memory thirst.
The best results I could get (using the original code, hidden=120, direct=3500) did at best match the ngram results in my setup.
On my development set, the accuracy of correct word selection (for a position correctly detected) was around 48% (case sensitive) for either ngram or rnn, or 53% (case insensitive).
Linear combination of the two gave an improvement of about 1%.
Thanks for sharing. I will write up something about my approach too, although I may not be able to get to it until mid-May due to travel.
", "rawMarkdown": "", "votes": null }, { "id": "74483", "postDate": "04/24/2015 13:21:13", "content": "Looking forward to that!
", "rawMarkdown": "", "votes": null }, { "id": "77308", "postDate": "05/06/2015 13:07:43", "content": "I had a pretty simple model -- I built a list of the most common word pairs, then performed the word removal (multiple times) and created another list. Anything that moved significantly up the ranking of common pairs implied a grammatical anomaly: simple pairs like ("and", ".") or ("the", "of") tended to dominate the list.
\nI looked for instances in the test set of these new combinations and anywhere I found one I put "eeee" in between (I tried a few word combinations, but this reliably scored the best).
\nThis scored surprisingly well until people started actually thinking about the problem. :-) I'd have done more, but my Kaggle available time has been really low lately, and I was happy at how well this performed, particularly for the low effort. But yeah, not ridiculously useful.
", "rawMarkdown": "", "votes": null }, { "id": "78616", "postDate": "05/16/2015 07:41:26", "content": "I've put together a fuller description of the system, including some scores for intermediate steps in the process.
\n\nCheers
", "rawMarkdown": "", "votes": null }, { "id": "222278", "postDate": "09/18/2017 09:04:15", "content": "Hello Meron! Nice work!\nI am wondering that how did you generate missing word from the training data?\nLike randomly remove a word from original training sentence or something else?
", "rawMarkdown": "Hello Meron! Nice work!\nI am wondering that how did you generate missing word from the training data?\nLike randomly remove a word from original training sentence or something else?", "votes": null }, { "id": "577265", "postDate": "07/16/2019 14:50:12", "content": "Could you please share a link to your code. I am working on a project to find out the location of missing articles(a, an, the) in sentences. Please help me with the approach and some code. \nWaiting for a reply :} \nThanks!
", "rawMarkdown": "Could you please share a link to your code. I am working on a project to find out the location of missing articles(a, an, the) in sentences. Please help me with the approach and some code. \nWaiting for a reply :} \nThanks!", "votes": null } ], "comments": [ { "id": 74414, "author_name": "onemillionmonkeys", "author_url": "", "post_date": "04/24/2015 00:27:20", "content": "Thanks for sharing. I will write up something about my approach too, although I may not be able to get to it until mid-May due to travel.
", "votes": null, "replies": [] }, { "id": 74483, "author_name": "meron1", "author_url": "", "post_date": "04/24/2015 13:21:13", "content": "Looking forward to that!
", "votes": null, "replies": [] }, { "id": 77308, "author_name": "chipmonkey", "author_url": "", "post_date": "05/06/2015 13:07:43", "content": "I had a pretty simple model -- I built a list of the most common word pairs, then performed the word removal (multiple times) and created another list. Anything that moved significantly up the ranking of common pairs implied a grammatical anomaly: simple pairs like ("and", ".") or ("the", "of") tended to dominate the list.
\nI looked for instances in the test set of these new combinations and anywhere I found one I put "eeee" in between (I tried a few word combinations, but this reliably scored the best).
\nThis scored surprisingly well until people started actually thinking about the problem. :-) I'd have done more, but my Kaggle available time has been really low lately, and I was happy at how well this performed, particularly for the low effort. But yeah, not ridiculously useful.
", "votes": null, "replies": [] }, { "id": 78616, "author_name": "meron1", "author_url": "", "post_date": "05/16/2015 07:41:26", "content": "I've put together a fuller description of the system, including some scores for intermediate steps in the process.
\n\nCheers
", "votes": null, "replies": [ { "id": 222278, "author_name": "yuchunlo", "author_url": "", "post_date": "09/18/2017 09:04:15", "content": "Hello Meron! Nice work!\nI am wondering that how did you generate missing word from the training data?\nLike randomly remove a word from original training sentence or something else?
", "votes": null, "replies": [] } ] }, { "id": 577265, "author_name": "firdoussultana", "author_url": "", "post_date": "07/16/2019 14:50:12", "content": "Could you please share a link to your code. I am working on a project to find out the location of missing articles(a, an, the) in sentences. Please help me with the approach and some code. \nWaiting for a reply :} \nThanks!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "74314": "", "74414": "", "74483": "", "77308": "", "78616": "", "222278": "Hello Meron! Nice work!\nI am wondering that how did you generate missing word from the training data?\nLike randomly remove a word from original training sentence or something else?", "577265": "Could you please share a link to your code. I am working on a project to find out the location of missing articles(a, an, the) in sentences. Please help me with the approach and some code. \nWaiting for a reply :} \nThanks!" }, "source": "meta" }