![\"\"](\"https://github.com/google-research/bigbird/raw/master/comparison.png\")
One of the challenges of this competition is the text length. Most sections are longer than 512 tokens, which is the limit for classical transformers.
\nThe attention being quadratic, it is already quite long to train a model with a maximum text length of 512 tokens.
Google recently released Big Bird which allows for the use of longer sequences : its attention mechanism is sparse and of linear complexity.
\nAccording to the Long Range Arena benchmark, Big Bird is the only model (so far) that reduces the memory cost without any performance loss (see bellow). But keep in mind that the benchmark is made by Google as well …
\n
\n", "messages": [ { "id": "1256994", "postDate": "03/30/2021 12:42:37", "content": "Paper : https://arxiv.org/abs/2007.14062
\n
\n Code : https://github.com/google-research/bigbird
One of the challenges of this competition is the text length. Most sections are longer than 512 tokens, which is the limit for classical transformers.
\nThe attention being quadratic, it is already quite long to train a model with a maximum text length of 512 tokens.
Google recently released Big Bird which allows for the use of longer sequences : its attention mechanism is sparse and of linear complexity.
\nAccording to the Long Range Arena benchmark, Big Bird is the only model (so far) that reduces the memory cost without any performance loss (see bellow). But keep in mind that the benchmark is made by Google as well …
\n
\n", "rawMarkdown": "One of the challenges of this competition is the text length. Most sections are longer than 512 tokens, which is the limit for classical transformers.\nThe attention being quadratic, it is already quite long to train a model with a maximum text length of 512 tokens.\n\nGoogle recently released [Big Bird](https://github.com/google-research/bigbird) which allows for the use of longer sequences : its attention mechanism is **sparse** and of **linear complexity**.\n\nAccording to the [Long Range Arena](https://arxiv.org/pdf/2011.04006.pdf) benchmark, Big Bird is the only model (so far) that reduces the memory cost without any performance loss (see bellow). But keep in mind that the benchmark is made by Google as well ...\n\n\n\n> Paper : https://arxiv.org/abs/2007.14062\n> Code : https://github.com/google-research/bigbird", "votes": null }, { "id": "1257037", "postDate": "03/30/2021 13:26:29", "content": "Paper : https://arxiv.org/abs/2007.14062
\n
\n Code : https://github.com/google-research/bigbird
Big Bird is worth looking into.
\nHowever, keep in mind that support for longer sequences is only upto 8x length (compared to previous models).
\nAnother limitation of BigBird is that moving to sparse-attention may require polynomially more layers, because of Orthogonal Vector Conjecture (OVC) problem.
It is now available in huggingfaces master….
", "rawMarkdown": "It is now available in huggingfaces master....", "votes": null }, { "id": "1259486", "postDate": "04/01/2021 13:01:33", "content": "HuggingFace: How to use BigBird Transformer
\nBigBird Colab notebook
\nHuggingFace BigBird documentation
\nHuggingFace also has BigBird alternatives such as Longformer and Reformer
@cpmpml Good to know, Huggingface's implementation should be more convenient to use than Google's.
", "rawMarkdown": "cpmpml Good to know, Huggingface's implementation should be more convenient to use than Google's.", "votes": null }, { "id": "1259569", "postDate": "04/01/2021 14:08:16", "content": "Esp if you use pytorch ;)
\nMaybe time to move to real modeling here…
", "rawMarkdown": "Esp if you use pytorch ;)\n\nMaybe time to move to real modeling here...", "votes": null }, { "id": "1259639", "postDate": "04/01/2021 14:57:45", "content": "Cannot use Huggingface's BigBird Transformer in the kernel environment, I guess the transformers library needs to be updated.
", "rawMarkdown": "Cannot use Huggingface's BigBird Transformer in the kernel environment, I guess the transformers library needs to be updated.", "votes": null }, { "id": "1259675", "postDate": "04/01/2021 15:17:29", "content": "I looked for BigBirdForConditionalGeneration. It doesn't exist yet.
\nHow are supposed to use it for finetuning so ?
\nFor instance I use Model_nameForConditionalGeneration to finetune generative models like T5, MT5, Bart etc.
", "rawMarkdown": "I looked for BigBirdForConditionalGeneration. It doesn't exist yet. \n\nHow are supposed to use it for finetuning so ? \n\nFor instance I use Model_nameForConditionalGeneration to finetune generative models like T5, MT5, Bart etc.", "votes": null }, { "id": "1260416", "postDate": "04/02/2021 05:35:14", "content": "According to the memory of the previous comp, the update of Huggingface is still very fast, and sometimes even two releases can be experienced within a three-month competition. So I think maybe it's better to wait for a stable version of it🙈
", "rawMarkdown": "According to the memory of the previous comp, the update of Huggingface is still very fast, and sometimes even two releases can be experienced within a three-month competition. So I think maybe it's better to wait for a stable version of it🙈", "votes": null }, { "id": "1262224", "postDate": "04/04/2021 01:47:44", "content": "In RIIID comp, the winning solution by Keetar also addressed longer sequences by changing Transformer complexity from O(n^2) to O(n). See post here
", "rawMarkdown": "In RIIID comp, the winning solution by Keetar also addressed longer sequences by changing Transformer complexity from O(n^2) to O(n). See post [here][1]\n\n[1]: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/218318", "votes": null }, { "id": "1263528", "postDate": "04/05/2021 13:49:36", "content": "What is the OVC problem?
", "rawMarkdown": "What is the OVC problem?", "votes": null }, { "id": "1273173", "postDate": "04/14/2021 06:26:13", "content": "Good idea!
", "rawMarkdown": "Good idea!", "votes": null }, { "id": "1274204", "postDate": "04/15/2021 05:30:06", "content": "I download the bigbird-base-trivia-itc model but can`t find its vocabulary file.
", "rawMarkdown": "I download the bigbird-base-trivia-itc model but can`t find its vocabulary file.", "votes": null }, { "id": "1284602", "postDate": "04/26/2021 06:07:22", "content": "@chewzy , in the conventional transformer every token 'knows' every other token, wheras in the sparser versions (BigBird, Longformer, most X-mers etc) this is not the case. So while token 'A' has info pertaining to Token 'M' in conventional transformer and just 1 layer is needed to extract this info, in the sparser versions, we may have to build another few layers on top to achieve this effect. If 'A' knows 'H' and 'H' knows 'M' then it will take at least 1 additional layer to connect 'A' to 'M'. This is somewhat like convolutions where the field of vision keeps getting bigger and bigger as one adds layer upon layer (the visibility cone originating from the topmost layer expands to cover more tokens at the bottom-most layer as the number of layers increase). I believe this is what @sirishks was implying and I agree with him. He also brings out an imp point that after all this gimmicks and tricks, the improvement is 8X instead of X (with a slightly lower accuracy due to the sparse tokens) and we need to see how well it applies to the current context (show US the data)…
\ncoming to the OVC problem itself, in simple words it states the above intuition. Let us take a simpler problem of trying to find the most dis-similar token for each token in the given input sequence. This simply boils down to taking the minimum of the inner product of every token with every other token - we do this on the encoder side when creating self-attention (taking the inner-products). So with just 1 layer, we are able to give out the above information as far as a regular transformer is concerned. Now consider the case of sparse attention models (not just BigB but all the X-mers). They simply cannot do this with 1 layer (in sub-quadratic time). This is simply because they don't have all the inner products (that is what makes them sparse). We would have to add layers on top to get what we want (viz the minimum of all the possible inner-products). Adding layers increases time and in the end we are forced to do it in quadratic time. The BB paper for some reason goes into extreme details of this - giving mathematical proofs and interpretations and conjectures and lemmas and what not. Instead it should have focussed more on what makes BigB different from the other transformers which have already introduced the concepts of global tokens and sliding window tokens. For e.g. a little more details on Random tokens, their possible configuration, their tuning - what settings work what dont, the intuitive aspects behind why it gives better results. In fact there are two versions of BigB and the one without the random tokens is the one used in most of the benchmarks (Hugginface has the other one I think) In that case, what is new that BigB offers and why is it doing better. It would have been good if the paper had focussed more on these aspects instead of going into great details into proving limitations that all sparse transformers have.
\nSorry if this came out as a bit of a rant. The folks have done some really good work. It would really help to know why their model is doing so good. As of now there are no good blogs or discussions on BigB other than the original paper itself. I am sure there would be lots of follow-up studies and interesting discussions ahead.
", "rawMarkdown": "chewzy , in the conventional transformer every token 'knows' every other token, wheras in the sparser versions (BigBird, Longformer, most X-mers etc) this is not the case. So while token 'A' has info pertaining to Token 'M' in conventional transformer and just 1 layer is needed to extract this info, in the sparser versions, we may have to build another few layers on top to achieve this effect. If 'A' knows 'H' and 'H' knows 'M' then it will take at least 1 additional layer to connect 'A' to 'M'. This is somewhat like convolutions where the field of vision keeps getting bigger and bigger as one adds layer upon layer (the visibility cone originating from the topmost layer expands to cover more tokens at the bottom-most layer as the number of layers increase). I believe this is what @sirishks was implying and I agree with him. He also brings out an imp point that after all this gimmicks and tricks, the improvement is 8X instead of X (with a slightly lower accuracy due to the sparse tokens) and we need to see how well it applies to the current context (show US the data)...\n\ncoming to the OVC problem itself, in simple words it states the above intuition. Let us take a simpler problem of trying to find the most dis-similar token for each token in the given input sequence. This simply boils down to taking the minimum of the inner product of every token with every other token - we do this on the encoder side when creating self-attention (taking the inner-products). So with just 1 layer, we are able to give out the above information as far as a regular transformer is concerned. Now consider the case of sparse attention models (not just BigB but all the X-mers). They simply cannot do this with 1 layer (in sub-quadratic time). This is simply because they don't have all the inner products (that is what makes them sparse). We would have to add layers on top to get what we want (viz the minimum of all the possible inner-products). Adding layers increases time and in the end we are forced to do it in quadratic time. The BB paper for some reason goes into extreme details of this - giving mathematical proofs and interpretations and conjectures and lemmas and what not. Instead it should have focussed more on what makes BigB different from the other transformers which have already introduced the concepts of global tokens and sliding window tokens. For e.g. a little more details on Random tokens, their possible configuration, their tuning - what settings work what dont, the intuitive aspects behind why it gives better results. In fact there are two versions of BigB and the one without the random tokens is the one used in most of the benchmarks (Hugginface has the other one I think) In that case, what is new that BigB offers and why is it doing better. It would have been good if the paper had focussed more on these aspects instead of going into great details into proving limitations that all sparse transformers have. \n\nSorry if this came out as a bit of a rant. The folks have done some really good work. It would really help to know why their model is doing so good. As of now there are no good blogs or discussions on BigB other than the original paper itself. I am sure there would be lots of follow-up studies and interesting discussions ahead.", "votes": null }, { "id": "1284612", "postDate": "04/26/2021 06:24:05", "content": "Hmm…It is interesting. Going by the discussions, while some of the folks in the comp seems to have intuitively guessed that it was only the last question that mattered and used that info in their versions of code, looks like they didn't actually update the transformer implementation (1 line change) and hence stuck at quadratic time which limited the tokens.
\nI am just trying to imagine why this worked. I guess this is somewhat similar to the case like a longformer where we use global tokens but in Riiid case the number of global tokens is just 1….which is the last token in the series. It attends to every other token and its representation contains relevant information from all the previous tokens. So if we have a 1000 questions in history and we are required to guess the answer of the 1001'st question, then we can base this answer just on the outcome of the 1000th question (AS LONG AS the 1000th question EMBEDDING contains relevant info of all the relevant questions from 1-999). I guess this works because (a) this is not a tradition seq-seq transformation. It is more like a time-series and we are supposed to guess only the answer to the 1001th question (b) there is an LSTM on top of this encoder which anyway brings in captures of sequence and relationships to past questions…so rather than looking at this as the encoder-side of a transformer architecture it may be more appropriate to look at it like a traditional LSTM implementation on the encoder side where the embeddings of the last Q have more contextual information
\nCan this approach be generalized to time-series? That is an interesting thought as well…
\nAnyways thanks @cdeotte for sharing this wonderful link. Do guide in case I have been wrong anywhere
", "rawMarkdown": "Hmm...It is interesting. Going by the discussions, while some of the folks in the comp seems to have intuitively guessed that it was only the last question that mattered and used that info in their versions of code, looks like they didn't actually update the transformer implementation (1 line change) and hence stuck at quadratic time which limited the tokens.\n\nI am just trying to imagine why this worked. I guess this is somewhat similar to the case like a longformer where we use global tokens but in Riiid case the number of global tokens is just 1....which is the last token in the series. It attends to every other token and its representation contains relevant information from all the previous tokens. So if we have a 1000 questions in history and we are required to guess the answer of the 1001'st question, then we can base this answer just on the outcome of the 1000th question (AS LONG AS the 1000th question EMBEDDING contains relevant info of all the relevant questions from 1-999). I guess this works because (a) this is not a tradition seq-seq transformation. It is more like a time-series and we are supposed to guess only the answer to the 1001th question (b) there is an LSTM on top of this encoder which anyway brings in captures of sequence and relationships to past questions...so rather than looking at this as the encoder-side of a transformer architecture it may be more appropriate to look at it like a traditional LSTM implementation on the encoder side where the embeddings of the last Q have more contextual information\n\nCan this approach be generalized to time-series? That is an interesting thought as well...\n\nAnyways thanks @cdeotte for sharing this wonderful link. Do guide in case I have been wrong anywhere", "votes": null }, { "id": "1290321", "postDate": "05/01/2021 19:57:12", "content": "@prettyprettyboy you could load the tokenizer in a notebook with tokenizer = BigBird.from_pretrained('google/bigbird-base-trivia-itc') and then save it with tokenizer.save_pretrained(path_to_directory) and commit the notebook.
\nJust not sure what is the best way to access the classes, for instance BigBirdModel in a notebook without internet. @ibtesama did you find a solution?
", "rawMarkdown": "prettyprettyboy you could load the tokenizer in a notebook with tokenizer = BigBird.from_pretrained('google/bigbird-base-trivia-itc') and then save it with tokenizer.save_pretrained(*path_to_directory*) and commit the notebook.\n\nJust not sure what is the best way to access the classes, for instance BigBirdModel in a notebook without internet. @ibtesama did you find a solution?", "votes": null } ], "comments": [ { "id": 1257037, "author_name": "sirishks", "author_url": "", "post_date": "03/30/2021 13:26:29", "content": "Big Bird is worth looking into.
\nHowever, keep in mind that support for longer sequences is only upto 8x length (compared to previous models).
\nAnother limitation of BigBird is that moving to sparse-attention may require polynomially more layers, because of Orthogonal Vector Conjecture (OVC) problem.
What is the OVC problem?
", "votes": null, "replies": [] }, { "id": 1284602, "author_name": "allohvk", "author_url": "", "post_date": "04/26/2021 06:07:22", "content": "@chewzy , in the conventional transformer every token 'knows' every other token, wheras in the sparser versions (BigBird, Longformer, most X-mers etc) this is not the case. So while token 'A' has info pertaining to Token 'M' in conventional transformer and just 1 layer is needed to extract this info, in the sparser versions, we may have to build another few layers on top to achieve this effect. If 'A' knows 'H' and 'H' knows 'M' then it will take at least 1 additional layer to connect 'A' to 'M'. This is somewhat like convolutions where the field of vision keeps getting bigger and bigger as one adds layer upon layer (the visibility cone originating from the topmost layer expands to cover more tokens at the bottom-most layer as the number of layers increase). I believe this is what @sirishks was implying and I agree with him. He also brings out an imp point that after all this gimmicks and tricks, the improvement is 8X instead of X (with a slightly lower accuracy due to the sparse tokens) and we need to see how well it applies to the current context (show US the data)…
\ncoming to the OVC problem itself, in simple words it states the above intuition. Let us take a simpler problem of trying to find the most dis-similar token for each token in the given input sequence. This simply boils down to taking the minimum of the inner product of every token with every other token - we do this on the encoder side when creating self-attention (taking the inner-products). So with just 1 layer, we are able to give out the above information as far as a regular transformer is concerned. Now consider the case of sparse attention models (not just BigB but all the X-mers). They simply cannot do this with 1 layer (in sub-quadratic time). This is simply because they don't have all the inner products (that is what makes them sparse). We would have to add layers on top to get what we want (viz the minimum of all the possible inner-products). Adding layers increases time and in the end we are forced to do it in quadratic time. The BB paper for some reason goes into extreme details of this - giving mathematical proofs and interpretations and conjectures and lemmas and what not. Instead it should have focussed more on what makes BigB different from the other transformers which have already introduced the concepts of global tokens and sliding window tokens. For e.g. a little more details on Random tokens, their possible configuration, their tuning - what settings work what dont, the intuitive aspects behind why it gives better results. In fact there are two versions of BigB and the one without the random tokens is the one used in most of the benchmarks (Hugginface has the other one I think) In that case, what is new that BigB offers and why is it doing better. It would have been good if the paper had focussed more on these aspects instead of going into great details into proving limitations that all sparse transformers have.
\nSorry if this came out as a bit of a rant. The folks have done some really good work. It would really help to know why their model is doing so good. As of now there are no good blogs or discussions on BigB other than the original paper itself. I am sure there would be lots of follow-up studies and interesting discussions ahead.
", "votes": null, "replies": [] } ] }, { "id": 1259464, "author_name": "cpmpml", "author_url": "", "post_date": "04/01/2021 12:32:41", "content": "It is now available in huggingfaces master….
", "votes": null, "replies": [ { "id": 1259486, "author_name": "sirishks", "author_url": "", "post_date": "04/01/2021 13:01:33", "content": "HuggingFace: How to use BigBird Transformer
\nBigBird Colab notebook
\nHuggingFace BigBird documentation
\nHuggingFace also has BigBird alternatives such as Longformer and Reformer
@cpmpml Good to know, Huggingface's implementation should be more convenient to use than Google's.
", "votes": null, "replies": [] }, { "id": 1259569, "author_name": "cpmpml", "author_url": "", "post_date": "04/01/2021 14:08:16", "content": "Esp if you use pytorch ;)
\nMaybe time to move to real modeling here…
", "votes": null, "replies": [] }, { "id": 1259675, "author_name": "serigne", "author_url": "", "post_date": "04/01/2021 15:17:29", "content": "I looked for BigBirdForConditionalGeneration. It doesn't exist yet.
\nHow are supposed to use it for finetuning so ?
\nFor instance I use Model_nameForConditionalGeneration to finetune generative models like T5, MT5, Bart etc.
", "votes": null, "replies": [] } ] }, { "id": 1259639, "author_name": "ibtesama", "author_url": "", "post_date": "04/01/2021 14:57:45", "content": "Cannot use Huggingface's BigBird Transformer in the kernel environment, I guess the transformers library needs to be updated.
", "votes": null, "replies": [ { "id": 1274204, "author_name": "prettyprettyboy", "author_url": "", "post_date": "04/15/2021 05:30:06", "content": "I download the bigbird-base-trivia-itc model but can`t find its vocabulary file.
", "votes": null, "replies": [] }, { "id": 1290321, "author_name": "hannes82", "author_url": "", "post_date": "05/01/2021 19:57:12", "content": "@prettyprettyboy you could load the tokenizer in a notebook with tokenizer = BigBird.from_pretrained('google/bigbird-base-trivia-itc') and then save it with tokenizer.save_pretrained(path_to_directory) and commit the notebook.
\nJust not sure what is the best way to access the classes, for instance BigBirdModel in a notebook without internet. @ibtesama did you find a solution?
", "votes": null, "replies": [] } ] }, { "id": 1260416, "author_name": "wochidadonggua", "author_url": "", "post_date": "04/02/2021 05:35:14", "content": "According to the memory of the previous comp, the update of Huggingface is still very fast, and sometimes even two releases can be experienced within a three-month competition. So I think maybe it's better to wait for a stable version of it🙈
", "votes": null, "replies": [] }, { "id": 1262224, "author_name": "cdeotte", "author_url": "", "post_date": "04/04/2021 01:47:44", "content": "In RIIID comp, the winning solution by Keetar also addressed longer sequences by changing Transformer complexity from O(n^2) to O(n). See post here
", "votes": null, "replies": [ { "id": 1284612, "author_name": "allohvk", "author_url": "", "post_date": "04/26/2021 06:24:05", "content": "Hmm…It is interesting. Going by the discussions, while some of the folks in the comp seems to have intuitively guessed that it was only the last question that mattered and used that info in their versions of code, looks like they didn't actually update the transformer implementation (1 line change) and hence stuck at quadratic time which limited the tokens.
\nI am just trying to imagine why this worked. I guess this is somewhat similar to the case like a longformer where we use global tokens but in Riiid case the number of global tokens is just 1….which is the last token in the series. It attends to every other token and its representation contains relevant information from all the previous tokens. So if we have a 1000 questions in history and we are required to guess the answer of the 1001'st question, then we can base this answer just on the outcome of the 1000th question (AS LONG AS the 1000th question EMBEDDING contains relevant info of all the relevant questions from 1-999). I guess this works because (a) this is not a tradition seq-seq transformation. It is more like a time-series and we are supposed to guess only the answer to the 1001th question (b) there is an LSTM on top of this encoder which anyway brings in captures of sequence and relationships to past questions…so rather than looking at this as the encoder-side of a transformer architecture it may be more appropriate to look at it like a traditional LSTM implementation on the encoder side where the embeddings of the last Q have more contextual information
\nCan this approach be generalized to time-series? That is an interesting thought as well…
\nAnyways thanks @cdeotte for sharing this wonderful link. Do guide in case I have been wrong anywhere
", "votes": null, "replies": [] } ] }, { "id": 1273173, "author_name": "zavodrobotov", "author_url": "", "post_date": "04/14/2021 06:26:13", "content": "Good idea!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1256994": "One of the challenges of this competition is the text length. Most sections are longer than 512 tokens, which is the limit for classical transformers.\nThe attention being quadratic, it is already quite long to train a model with a maximum text length of 512 tokens.\n\nGoogle recently released [Big Bird](https://github.com/google-research/bigbird) which allows for the use of longer sequences : its attention mechanism is **sparse** and of **linear complexity**.\n\nAccording to the [Long Range Arena](https://arxiv.org/pdf/2011.04006.pdf) benchmark, Big Bird is the only model (so far) that reduces the memory cost without any performance loss (see bellow). But keep in mind that the benchmark is made by Google as well ...\n\n\n\n> Paper : https://arxiv.org/abs/2007.14062\n> Code : https://github.com/google-research/bigbird", "1257037": "Big Bird is worth looking into.\n\nHowever, keep in mind that support for longer sequences is only upto 8x length (compared to previous models).\nAnother limitation of BigBird is that moving to sparse-attention may require polynomially more layers, because of Orthogonal Vector Conjecture (OVC) problem.", "1259464": "It is now available in huggingfaces master....", "1259486": "HuggingFace: How to [use BigBird Transformer](https://github.com/huggingface/transformers/blob/master/templates/adding_a_new_model/open_model_proposals/ADD_BIG_BIRD.md)\n\nBigBird [Colab notebook](https://colab.research.google.com/drive/1DVOm1VHjW0eKCayFq1N2GpY6GR9M4tJP?usp=sharing)\n\nHuggingFace [BigBird documentation](https://huggingface.co/transformers/master/model_doc/bigbird.html)\nHuggingFace also has BigBird alternatives such as [Longformer](https://huggingface.co/transformers/model_doc/longformer.html) and [Reformer](https://huggingface.co/transformers/model_doc/reformer.html)", "1259508": "cpmpml Good to know, Huggingface's implementation should be more convenient to use than Google's.", "1259569": "Esp if you use pytorch ;)\n\nMaybe time to move to real modeling here...", "1259639": "Cannot use Huggingface's BigBird Transformer in the kernel environment, I guess the transformers library needs to be updated.", "1259675": "I looked for BigBirdForConditionalGeneration. It doesn't exist yet. \n\nHow are supposed to use it for finetuning so ? \n\nFor instance I use Model_nameForConditionalGeneration to finetune generative models like T5, MT5, Bart etc.", "1260416": "According to the memory of the previous comp, the update of Huggingface is still very fast, and sometimes even two releases can be experienced within a three-month competition. So I think maybe it's better to wait for a stable version of it🙈", "1262224": "In RIIID comp, the winning solution by Keetar also addressed longer sequences by changing Transformer complexity from O(n^2) to O(n). See post [here][1]\n\n[1]: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/218318", "1263528": "What is the OVC problem?", "1273173": "Good idea!", "1274204": "I download the bigbird-base-trivia-itc model but can`t find its vocabulary file.", "1284602": "chewzy , in the conventional transformer every token 'knows' every other token, wheras in the sparser versions (BigBird, Longformer, most X-mers etc) this is not the case. So while token 'A' has info pertaining to Token 'M' in conventional transformer and just 1 layer is needed to extract this info, in the sparser versions, we may have to build another few layers on top to achieve this effect. If 'A' knows 'H' and 'H' knows 'M' then it will take at least 1 additional layer to connect 'A' to 'M'. This is somewhat like convolutions where the field of vision keeps getting bigger and bigger as one adds layer upon layer (the visibility cone originating from the topmost layer expands to cover more tokens at the bottom-most layer as the number of layers increase). I believe this is what @sirishks was implying and I agree with him. He also brings out an imp point that after all this gimmicks and tricks, the improvement is 8X instead of X (with a slightly lower accuracy due to the sparse tokens) and we need to see how well it applies to the current context (show US the data)...\n\ncoming to the OVC problem itself, in simple words it states the above intuition. Let us take a simpler problem of trying to find the most dis-similar token for each token in the given input sequence. This simply boils down to taking the minimum of the inner product of every token with every other token - we do this on the encoder side when creating self-attention (taking the inner-products). So with just 1 layer, we are able to give out the above information as far as a regular transformer is concerned. Now consider the case of sparse attention models (not just BigB but all the X-mers). They simply cannot do this with 1 layer (in sub-quadratic time). This is simply because they don't have all the inner products (that is what makes them sparse). We would have to add layers on top to get what we want (viz the minimum of all the possible inner-products). Adding layers increases time and in the end we are forced to do it in quadratic time. The BB paper for some reason goes into extreme details of this - giving mathematical proofs and interpretations and conjectures and lemmas and what not. Instead it should have focussed more on what makes BigB different from the other transformers which have already introduced the concepts of global tokens and sliding window tokens. For e.g. a little more details on Random tokens, their possible configuration, their tuning - what settings work what dont, the intuitive aspects behind why it gives better results. In fact there are two versions of BigB and the one without the random tokens is the one used in most of the benchmarks (Hugginface has the other one I think) In that case, what is new that BigB offers and why is it doing better. It would have been good if the paper had focussed more on these aspects instead of going into great details into proving limitations that all sparse transformers have. \n\nSorry if this came out as a bit of a rant. The folks have done some really good work. It would really help to know why their model is doing so good. As of now there are no good blogs or discussions on BigB other than the original paper itself. I am sure there would be lots of follow-up studies and interesting discussions ahead.", "1284612": "Hmm...It is interesting. Going by the discussions, while some of the folks in the comp seems to have intuitively guessed that it was only the last question that mattered and used that info in their versions of code, looks like they didn't actually update the transformer implementation (1 line change) and hence stuck at quadratic time which limited the tokens.\n\nI am just trying to imagine why this worked. I guess this is somewhat similar to the case like a longformer where we use global tokens but in Riiid case the number of global tokens is just 1....which is the last token in the series. It attends to every other token and its representation contains relevant information from all the previous tokens. So if we have a 1000 questions in history and we are required to guess the answer of the 1001'st question, then we can base this answer just on the outcome of the 1000th question (AS LONG AS the 1000th question EMBEDDING contains relevant info of all the relevant questions from 1-999). I guess this works because (a) this is not a tradition seq-seq transformation. It is more like a time-series and we are supposed to guess only the answer to the 1001th question (b) there is an LSTM on top of this encoder which anyway brings in captures of sequence and relationships to past questions...so rather than looking at this as the encoder-side of a transformer architecture it may be more appropriate to look at it like a traditional LSTM implementation on the encoder side where the embeddings of the last Q have more contextual information\n\nCan this approach be generalized to time-series? That is an interesting thought as well...\n\nAnyways thanks @cdeotte for sharing this wonderful link. Do guide in case I have been wrong anywhere", "1290321": "prettyprettyboy you could load the tokenizer in a notebook with tokenizer = BigBird.from_pretrained('google/bigbird-base-trivia-itc') and then save it with tokenizer.save_pretrained(*path_to_directory*) and commit the notebook.\n\nJust not sure what is the best way to access the classes, for instance BigBirdModel in a notebook without internet. @ibtesama did you find a solution?" }, "source": "meta" }