{ "id": 343614, "title": "21st - A Bag of Words is All You Need", "url": "/competitions/AI4Code/writeups/jacoby-jaeger-21st-a-bag-of-words-is-all-you-need", "author_name": "", "post_date": "2022-08-12T01:33:12.549848600Z", "votes": 37, "comment_count": 10, "views": 0, "content": "

Thank you everyone for an interesting competition.

\n

My solution ignores sentence grammar entirely, embedding entire cells as single tokens with a sparse-dense-matmul between the tfidf-matrix and word/word-ngram embeddings. Using this method, it's possible to have global attention between every cell of each document. An epoch of training takes about 20 minutes.

\n

The rest of my model is very similar to BERT, except that the model has the option to add an eye matrix, or input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.

\n

My highest scoring notebook doesn't use external data as I was running out of time towards the end of the competition, but given the short epoch time requirement I think my score likely could have been improved quite a bit by incorporating a large volume of external data.

", "messages": [ { "id": "1895136", "postDate": "08/12/2022 01:33:12", "content": "

Thank you everyone for an interesting competition.

\n

My solution ignores sentence grammar entirely, embedding entire cells as single tokens with a sparse-dense-matmul between the tfidf-matrix and word/word-ngram embeddings. Using this method, it's possible to have global attention between every cell of each document. An epoch of training takes about 20 minutes.

\n

The rest of my model is very similar to BERT, except that the model has the option to add an eye matrix, or input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.

\n

My highest scoring notebook doesn't use external data as I was running out of time towards the end of the competition, but given the short epoch time requirement I think my score likely could have been improved quite a bit by incorporating a large volume of external data.

", "rawMarkdown": "Thank you everyone for an interesting competition. \n\nMy solution ignores sentence grammar entirely, embedding entire cells as single tokens with a sparse-dense-matmul between the tfidf-matrix and word/word-ngram embeddings. Using this method, it's possible to have global attention between every cell of each document. An epoch of training takes about 20 minutes.\n\nThe rest of my model is very similar to BERT, except that the model has the option to add an eye matrix, or input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.\n\nMy highest scoring notebook doesn't use external data as I was running out of time towards the end of the competition, but given the short epoch time requirement I think my score likely could have been improved quite a bit by incorporating a large volume of external data.", "votes": null }, { "id": "1895167", "postDate": "08/12/2022 02:08:41", "content": "

Very interesting! I'd love to see the code for that solution if you are willing to share.

", "rawMarkdown": "Very interesting! I'd love to see the code for that solution if you are willing to share.", "votes": null }, { "id": "1895171", "postDate": "08/12/2022 02:13:30", "content": "

Thank you, it's not very readable, but here it is: https://www.kaggle.com/code/jacobyjaeger/a-bag-of-words-is-all-you-need/

", "rawMarkdown": "Thank you, it's not very readable, but here it is: [https://www.kaggle.com/code/jacobyjaeger/a-bag-of-words-is-all-you-need/](https://www.kaggle.com/code/jacobyjaeger/a-bag-of-words-is-all-you-need/)", "votes": null }, { "id": "1895220", "postDate": "08/12/2022 02:59:56", "content": "

Thank you very much for sharing and congrats on the (public) 21st!

\n

Did you use just Kaggle Nbs for this comp? I'm curious what your setup was like

", "rawMarkdown": "Thank you very much for sharing and congrats on the (public) 21st! \n\nDid you use just Kaggle Nbs for this comp? I'm curious what your setup was like", "votes": null }, { "id": "1895245", "postDate": "08/12/2022 03:25:12", "content": "

Thank you, congrats on 13th.

\n

I used colab notebooks for everything. Usually I get a v100 gpu.

", "rawMarkdown": "Thank you, congrats on 13th. \n\nI used colab notebooks for everything. Usually I get a v100 gpu.", "votes": null }, { "id": "1895445", "postDate": "08/12/2022 06:24:48", "content": "

Thank you for sharing such interesting solution and codes.

", "rawMarkdown": "Thank you for sharing such interesting solution and codes.", "votes": null }, { "id": "1895474", "postDate": "08/12/2022 06:56:47", "content": "

Thank you, it's an interesting and unexpected solution.

", "rawMarkdown": "Thank you, it's an interesting and unexpected solution.", "votes": null }, { "id": "1895553", "postDate": "08/12/2022 08:04:44", "content": "

@jacobyjaeger Congrats on a very novel approach. Worth its weight in gold.

\n

I didnt quite follow how you de-bias or bias attention. Could you slightly explain or share any external link or reference?

\n
\n

input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.

\n
", "rawMarkdown": "jacobyjaeger Congrats on a very novel approach. Worth its weight in gold.\n\nI didnt quite follow how you de-bias or bias attention. Could you slightly explain or share any external link or reference?\n\n> input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.", "votes": null }, { "id": "1895570", "postDate": "08/12/2022 08:28:03", "content": "

Thanks, Allohvk.

\n

If normal attention calculates its attention matrix as:

\n

A = matmul(query, key.T)

\n

I calculate mine as:

\n

A = matmul(query, key.T) + sum_i[ matmul( augment_query_i, augment_key_i.T ) * augment_matrix_i ]

\n

where in this case the augment matrices are the identity matrix and two ngram similarity matrices.

\n

This approach allows the transformer to make use of relationships between tokens that are not necessarily easy to represent with token embeddings alone. For example it could be very hard for a token to attend the token that is most similar to itself without attending to itself using normal attention. My approach allows the identity matrix to be subtracted from the attention matrix making this operation simple.

\n

I'm not sure if I'm the first to come up with this however.

", "rawMarkdown": "Thanks, Allohvk.\n\nIf normal attention calculates its attention matrix as:\n\n`A = matmul(query, key.T)`\n\nI calculate mine as:\n\n`A = matmul(query, key.T) + sum_i[ matmul( augment_query_i, augment_key_i.T ) * augment_matrix_i ] `\n\nwhere in this case the augment matrices are the identity matrix and two ngram similarity matrices. \n\nThis approach allows the transformer to make use of relationships between tokens that are not necessarily easy to represent with token embeddings alone. For example it could be very hard for a token to attend the token that is most similar to itself without attending to itself using normal attention. My approach allows the identity matrix to be subtracted from the attention matrix making this operation simple. \n\nI'm not sure if I'm the first to come up with this however.", "votes": null }, { "id": "1895612", "postDate": "08/12/2022 09:15:13", "content": "

Awesome. Thanks for the explanation

", "rawMarkdown": "Awesome. Thanks for the explanation", "votes": null }, { "id": "1896636", "postDate": "08/13/2022 02:14:08", "content": "

wow, glad to see traditional NLP works. Thanks for sharing

", "rawMarkdown": "wow, glad to see traditional NLP works. Thanks for sharing", "votes": null } ], "comments": [ { "id": 1895167, "author_name": "cabbage972", "author_url": "", "post_date": "08/12/2022 02:08:41", "content": "

Very interesting! I'd love to see the code for that solution if you are willing to share.

", "votes": null, "replies": [ { "id": 1895171, "author_name": "jacobyjaeger", "author_url": "", "post_date": "08/12/2022 02:13:30", "content": "

Thank you, it's not very readable, but here it is: https://www.kaggle.com/code/jacobyjaeger/a-bag-of-words-is-all-you-need/

", "votes": null, "replies": [] }, { "id": 1895220, "author_name": "init27", "author_url": "", "post_date": "08/12/2022 02:59:56", "content": "

Thank you very much for sharing and congrats on the (public) 21st!

\n

Did you use just Kaggle Nbs for this comp? I'm curious what your setup was like

", "votes": null, "replies": [] }, { "id": 1895245, "author_name": "jacobyjaeger", "author_url": "", "post_date": "08/12/2022 03:25:12", "content": "

Thank you, congrats on 13th.

\n

I used colab notebooks for everything. Usually I get a v100 gpu.

", "votes": null, "replies": [] } ] }, { "id": 1895445, "author_name": "brandonhu0215", "author_url": "", "post_date": "08/12/2022 06:24:48", "content": "

Thank you for sharing such interesting solution and codes.

", "votes": null, "replies": [] }, { "id": 1895474, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "08/12/2022 06:56:47", "content": "

Thank you, it's an interesting and unexpected solution.

", "votes": null, "replies": [] }, { "id": 1895553, "author_name": "allohvk", "author_url": "", "post_date": "08/12/2022 08:04:44", "content": "

@jacobyjaeger Congrats on a very novel approach. Worth its weight in gold.

\n

I didnt quite follow how you de-bias or bias attention. Could you slightly explain or share any external link or reference?

\n
\n

input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.

\n
", "votes": null, "replies": [ { "id": 1895570, "author_name": "jacobyjaeger", "author_url": "", "post_date": "08/12/2022 08:28:03", "content": "

Thanks, Allohvk.

\n

If normal attention calculates its attention matrix as:

\n

A = matmul(query, key.T)

\n

I calculate mine as:

\n

A = matmul(query, key.T) + sum_i[ matmul( augment_query_i, augment_key_i.T ) * augment_matrix_i ]

\n

where in this case the augment matrices are the identity matrix and two ngram similarity matrices.

\n

This approach allows the transformer to make use of relationships between tokens that are not necessarily easy to represent with token embeddings alone. For example it could be very hard for a token to attend the token that is most similar to itself without attending to itself using normal attention. My approach allows the identity matrix to be subtracted from the attention matrix making this operation simple.

\n

I'm not sure if I'm the first to come up with this however.

", "votes": null, "replies": [] }, { "id": 1895612, "author_name": "allohvk", "author_url": "", "post_date": "08/12/2022 09:15:13", "content": "

Awesome. Thanks for the explanation

", "votes": null, "replies": [] } ] }, { "id": 1896636, "author_name": "zacchaeus", "author_url": "", "post_date": "08/13/2022 02:14:08", "content": "

wow, glad to see traditional NLP works. Thanks for sharing

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1895136": "Thank you everyone for an interesting competition. \n\nMy solution ignores sentence grammar entirely, embedding entire cells as single tokens with a sparse-dense-matmul between the tfidf-matrix and word/word-ngram embeddings. Using this method, it's possible to have global attention between every cell of each document. An epoch of training takes about 20 minutes.\n\nThe rest of my model is very similar to BERT, except that the model has the option to add an eye matrix, or input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.\n\nMy highest scoring notebook doesn't use external data as I was running out of time towards the end of the competition, but given the short epoch time requirement I think my score likely could have been improved quite a bit by incorporating a large volume of external data.", "1895167": "Very interesting! I'd love to see the code for that solution if you are willing to share.", "1895171": "Thank you, it's not very readable, but here it is: [https://www.kaggle.com/code/jacobyjaeger/a-bag-of-words-is-all-you-need/](https://www.kaggle.com/code/jacobyjaeger/a-bag-of-words-is-all-you-need/)", "1895220": "Thank you very much for sharing and congrats on the (public) 21st! \n\nDid you use just Kaggle Nbs for this comp? I'm curious what your setup was like", "1895245": "Thank you, congrats on 13th. \n\nI used colab notebooks for everything. Usually I get a v100 gpu.", "1895445": "Thank you for sharing such interesting solution and codes.", "1895474": "Thank you, it's an interesting and unexpected solution.", "1895553": "jacobyjaeger Congrats on a very novel approach. Worth its weight in gold.\n\nI didnt quite follow how you de-bias or bias attention. Could you slightly explain or share any external link or reference?\n\n> input character n-gram similarity matrix weighted attention to the attention matrix eg. to bias attention away from cells attending to themselves or to bias attention to cells that have similar character-ngram compositions.", "1895570": "Thanks, Allohvk.\n\nIf normal attention calculates its attention matrix as:\n\n`A = matmul(query, key.T)`\n\nI calculate mine as:\n\n`A = matmul(query, key.T) + sum_i[ matmul( augment_query_i, augment_key_i.T ) * augment_matrix_i ] `\n\nwhere in this case the augment matrices are the identity matrix and two ngram similarity matrices. \n\nThis approach allows the transformer to make use of relationships between tokens that are not necessarily easy to represent with token embeddings alone. For example it could be very hard for a token to attend the token that is most similar to itself without attending to itself using normal attention. My approach allows the identity matrix to be subtracted from the attention matrix making this operation simple. \n\nI'm not sure if I'm the first to come up with this however.", "1895612": "Awesome. Thanks for the explanation", "1896636": "wow, glad to see traditional NLP works. Thanks for sharing" }, "source": "existing_discussion", "complete": true, "competition_id": 35887, "competition_slug": "AI4Code", "team_id": 8638105, "solution_url": "/competitions/AI4Code/writeups/jacoby-jaeger-21st-a-bag-of-words-is-all-you-need" }