# 21st - A Bag of Words is All You Need

Thank you everyone for an interesting competition.
My solution ignores sentence grammar entirely, embedding entire cells as single tokens with a sparse-dense-matmul between the tfidf-matrix and word/word-ngram embeddings. Using this method, it's possible to have global attention between every cell of each document. An epoch of training takes about 20 minutes.
The rest of my model is very similar to BERT, except that the model has the option to add an eye matrix, or input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.
My highest scoring notebook doesn't use external data as I was running out of time towards the end of the competition, but given the short epoch time requirement I think my score likely could have been improved quite a bit by incorporating a large volume of external data.