{ "id": 229239, "title": "Long text and OOM solution", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/229239", "author_name": "", "post_date": "2021-03-29T06:39:54.763668700Z", "votes": 4, "comment_count": 2, "views": 0, "content": "

Long text:
\nYou can try key sentence extraction to choose TOP k sentences, such as BM25, ROCC.
\nOr you can use long-transformers , transformer-xl.
\nSliding windows is also a good idea, but it may not work here. The text is too long. But here is a plan to be selected

\n

OOM:
\nTorch-checkpoint

\n
\n

Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.
\n Mixed precision is a trustworthy choice FP-16/apex

\n
\n

Waiting for a wonderful bert model!😆😆😆

", "messages": [ { "id": "1255739", "postDate": "03/29/2021 06:39:54", "content": "

Long text:
\nYou can try key sentence extraction to choose TOP k sentences, such as BM25, ROCC.
\nOr you can use long-transformers , transformer-xl.
\nSliding windows is also a good idea, but it may not work here. The text is too long. But here is a plan to be selected

\n

OOM:
\nTorch-checkpoint

\n
\n

Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.
\n Mixed precision is a trustworthy choice FP-16/apex

\n
\n

Waiting for a wonderful bert model!😆😆😆

", "rawMarkdown": "Long text:\nYou can try key sentence extraction to choose TOP k sentences, such as [BM25](https://en.wikipedia.org/wiki/Okapi_BM25), [ROCC](https://github.com/vikas95/AutoROCC). \nOr you can use [long-transformers](https://github.com/allenai/longformer) , [transformer-xl](https://github.com/kimiyoung/transformer-xl). \nSliding windows is also a good idea, but it may not work here. The text is too long. But [here](https://github.com/canjiali/PARADE) is a plan to be selected\n\nOOM:\n[Torch-checkpoint](https://arxiv.org/abs/1604.06174)\n> Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.\nMixed precision is a trustworthy choice [FP-16/apex](https://github.com/NVIDIA/apex)\n\n\nWaiting for a wonderful bert model!😆😆😆", "votes": null }, { "id": "1259571", "postDate": "04/01/2021 14:08:44", "content": "

👍……………………………….

", "rawMarkdown": "👍.....................................", "votes": null }, { "id": "1262365", "postDate": "04/04/2021 06:53:09", "content": "

Longformer maxes out the memory, as soon as you feed the first input.

", "rawMarkdown": "Longformer maxes out the memory, as soon as you feed the first input.", "votes": null } ], "comments": [ { "id": 1259571, "author_name": "udaygurugubelli", "author_url": "", "post_date": "04/01/2021 14:08:44", "content": "

👍……………………………….

", "votes": null, "replies": [] }, { "id": 1262365, "author_name": "ibtesama", "author_url": "", "post_date": "04/04/2021 06:53:09", "content": "

Longformer maxes out the memory, as soon as you feed the first input.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1255739": "Long text:\nYou can try key sentence extraction to choose TOP k sentences, such as [BM25](https://en.wikipedia.org/wiki/Okapi_BM25), [ROCC](https://github.com/vikas95/AutoROCC). \nOr you can use [long-transformers](https://github.com/allenai/longformer) , [transformer-xl](https://github.com/kimiyoung/transformer-xl). \nSliding windows is also a good idea, but it may not work here. The text is too long. But [here](https://github.com/canjiali/PARADE) is a plan to be selected\n\nOOM:\n[Torch-checkpoint](https://arxiv.org/abs/1604.06174)\n> Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.\nMixed precision is a trustworthy choice [FP-16/apex](https://github.com/NVIDIA/apex)\n\n\nWaiting for a wonderful bert model!😆😆😆", "1259571": "👍.....................................", "1262365": "Longformer maxes out the memory, as soon as you feed the first input." }, "source": "meta" }