So based on some of the information here, I am trying to explore different ways so that I can have codebert take in more inputs than a few hundred or less. (https://underline.io/events/122/sessions/4103/lecture/22352-beyond-paragraphs-nlp-for-long-sequences-part-4)
\nHas anyone looked into this, is considering this and thinks it's a dead end? I'm new to this field so I wanted to know if anyone had experiences with them.
So based on some of the information here, I am trying to explore different ways so that I can have codebert take in more inputs than a few hundred or less. (https://underline.io/events/122/sessions/4103/lecture/22352-beyond-paragraphs-nlp-for-long-sequences-part-4)
\nHas anyone looked into this, is considering this and thinks it's a dead end? I'm new to this field so I wanted to know if anyone had experiences with them.
The longformer paper describes how to modify existing transformers to be used for long inputs (without resorting to truncation). It has a modification of the original transformer's attention layer,which is quite simple, so it should be easily adaptable to any existing transformer model.
\ncheck this out.
", "rawMarkdown": "The longformer paper describes how to modify existing transformers to be used for long inputs (without resorting to truncation). It has a modification of the original transformer's attention layer,which is quite simple, so it should be easily adaptable to any existing transformer model.\n\ncheck [this](https://github.com/ccdv-ai/convert_checkpoint_to_lsg) out.", "votes": null }, { "id": "1859409", "postDate": "07/17/2022 15:53:15", "content": "I actually tried copying the weights and positional embeddings from codebert to longformer and for some reason even with more parameters it seems to be performing worse. Any ideas on what could be potential causes?
", "rawMarkdown": "I actually tried copying the weights and positional embeddings from codebert to longformer and for some reason even with more parameters it seems to be performing worse. Any ideas on what could be potential causes?", "votes": null } ], "comments": [ { "id": 1839815, "author_name": "thedevastator", "author_url": "", "post_date": "07/01/2022 17:52:38", "content": "The longformer paper describes how to modify existing transformers to be used for long inputs (without resorting to truncation). It has a modification of the original transformer's attention layer,which is quite simple, so it should be easily adaptable to any existing transformer model.
\ncheck this out.
", "votes": null, "replies": [ { "id": 1859409, "author_name": "brianyoon", "author_url": "", "post_date": "07/17/2022 15:53:15", "content": "I actually tried copying the weights and positional embeddings from codebert to longformer and for some reason even with more parameters it seems to be performing worse. Any ideas on what could be potential causes?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1836512": "So based on some of the information here, I am trying to explore different ways so that I can have codebert take in more inputs than a few hundred or less. (https://underline.io/events/122/sessions/4103/lecture/22352-beyond-paragraphs-nlp-for-long-sequences-part-4)\nHas anyone looked into this, is considering this and thinks it's a dead end? I'm new to this field so I wanted to know if anyone had experiences with them.", "1839815": "The longformer paper describes how to modify existing transformers to be used for long inputs (without resorting to truncation). It has a modification of the original transformer's attention layer,which is quite simple, so it should be easily adaptable to any existing transformer model.\n\ncheck [this](https://github.com/ccdv-ai/convert_checkpoint_to_lsg) out.", "1859409": "I actually tried copying the weights and positional embeddings from codebert to longformer and for some reason even with more parameters it seems to be performing worse. Any ideas on what could be potential causes?" }, "source": "meta" }