I'm new to NLP and transformers so I also face these issues for the first time.
\nUsing transformers fast tokenizers. Most of the tokenizer from transformers have a fast version which is implemented by rust. Use the fast version will significantly speed up training.
Load all the data in memory and keep a smaller number work workers. There are no disk IO so the DataLoader processes are comuting dense now, and most of the computing are coming from the tokenize. If using fast version of tokenizers, I believe the implementation is multithreaded(without the GIL limitation) by it self. In my case, there are no benefit when the number of workers is larger than 2.
Out of Memory issue. At first there will always be a OOM error when I train on my training set. I thought there are \"memory leak\" in my code. But turned out that the Tokenizer will eat more memory each time it meets some new words. So to reduce the num of workers will significant lower the memory usage. If still not enough memory to train a whole epoch, we could consider split the training set to subsets, and for each epoch train one subset of the dataset.
Update:
\nencode all the dataset and save to disk will completely solve the memory issue and boost training speed.
", "messages": [ { "id": "1804554", "postDate": "05/29/2022 06:30:39", "content": "I'm new to NLP and transformers so I also face these issues for the first time.
\nUsing transformers fast tokenizers. Most of the tokenizer from transformers have a fast version which is implemented by rust. Use the fast version will significantly speed up training.
Load all the data in memory and keep a smaller number work workers. There are no disk IO so the DataLoader processes are comuting dense now, and most of the computing are coming from the tokenize. If using fast version of tokenizers, I believe the implementation is multithreaded(without the GIL limitation) by it self. In my case, there are no benefit when the number of workers is larger than 2.
Out of Memory issue. At first there will always be a OOM error when I train on my training set. I thought there are \"memory leak\" in my code. But turned out that the Tokenizer will eat more memory each time it meets some new words. So to reduce the num of workers will significant lower the memory usage. If still not enough memory to train a whole epoch, we could consider split the training set to subsets, and for each epoch train one subset of the dataset.
Update:
\nencode all the dataset and save to disk will completely solve the memory issue and boost training speed.
", "rawMarkdown": "I'm new to NLP and transformers so I also face these issues for the first time.\n\n1. Using transformers `fast tokenizers`. Most of the tokenizer from `transformers` have a `fast` version which is implemented by `rust`. Use the fast version will significantly speed up training.\n\n2. Load all the data in memory and keep a smaller number work `workers`. There are no disk IO so the DataLoader processes are comuting dense now, and most of the computing are coming from the `tokenize`. If using fast version of tokenizers, I believe the implementation is multithreaded(without the GIL limitation) by it self. In my case, there are no benefit when the number of workers is larger than 2.\n\n3. Out of Memory issue. At first there will always be a OOM error when I train on my training set. I thought there are \"memory leak\" in my code. But turned out that the `Tokenizer` will eat more memory each time it meets some new words. So to reduce the num of workers will significant lower the memory usage. If still not enough memory to train a whole epoch, we could consider split the training set to subsets, and for each epoch train one subset of the dataset.\n\nUpdate:\n\nencode all the dataset and save to disk will completely solve the memory issue and boost training speed.", "votes": null }, { "id": "1811780", "postDate": "06/05/2022 06:06:10", "content": "I also wrote a blogpost on \"Preventing The CUDA Out Of Memory Error In PyTorch\". Hope it's helpful π
", "rawMarkdown": "I also wrote a blogpost on \"**[Preventing The CUDA Out Of Memory Error In PyTorch](https://wandb.ai/wandb_fc/tips/reports/Preventing-The-CUDA-Out-Of-Memory-Error-In-PyTorch--VmlldzoxNzU3NjA1)**\". Hope it's helpful π", "votes": null }, { "id": "1816837", "postDate": "06/10/2022 15:51:05", "content": "@snaker thanks for your valuable comments. Do you have any reference on item 3? Tokenizer size should not ideally grow during fine-tuning, but yes, there is definitely a steady increase in memory over time. Doesent look like a leak but it is happening.
", "rawMarkdown": "snaker thanks for your valuable comments. Do you have any reference on item 3? Tokenizer size should not ideally grow during fine-tuning, but yes, there is definitely a steady increase in memory over time. Doesent look like a leak but it is happening.", "votes": null }, { "id": "1817283", "postDate": "06/11/2022 06:11:51", "content": "I think some tokenizers are using unlimited cache: https://github.com/huggingface/transformers/issues/2827#issue-564095688
", "rawMarkdown": "I think some tokenizers are using unlimited cache: https://github.com/huggingface/transformers/issues/2827#issue-564095688", "votes": null }, { "id": "1817304", "postDate": "06/11/2022 06:35:17", "content": "Very interesting. Thank you for the infoβ¦
", "rawMarkdown": "Very interesting. Thank you for the info...", "votes": null }, { "id": "1817907", "postDate": "06/12/2022 02:56:54", "content": "If you do not need dynamic encoding, consider encode all the dataset and save to disk. This wll completely solve the memory issue had heavily boost the training speed.
", "rawMarkdown": "If you do not need dynamic encoding, consider encode all the dataset and save to disk. This wll completely solve the memory issue had heavily boost the training speed.", "votes": null }, { "id": "1820876", "postDate": "06/15/2022 04:34:14", "content": "Thank you for such a nice Post! Is it okay if I translate the blog post into my native language and share it?ππ @sauravmaheshkar
", "rawMarkdown": "Thank you for such a nice Post! Is it okay if I translate the blog post into my native language and share it?ππ @sauravmaheshkar", "votes": null }, { "id": "1820931", "postDate": "06/15/2022 05:36:22", "content": "If you are running low on VRAM, it's also worth trying gradient checkpointing to save VRAM at the cost of extra calculation. For BERT it may be ~25% slower but would use ~30-40% less VRAM.
\nA few other good pieces of advice from π€ https://huggingface.co/docs/transformers/performance
", "rawMarkdown": "If you are running low on VRAM, it's also worth trying gradient checkpointing to save VRAM at the cost of extra calculation. For BERT it may be ~25% slower but would use ~30-40% less VRAM.\n\nA few other good pieces of advice from π€ https://huggingface.co/docs/transformers/performance", "votes": null } ], "comments": [ { "id": 1811780, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "06/05/2022 06:06:10", "content": "I also wrote a blogpost on \"Preventing The CUDA Out Of Memory Error In PyTorch\". Hope it's helpful π
", "votes": null, "replies": [ { "id": 1820876, "author_name": "heuingttiang", "author_url": "", "post_date": "06/15/2022 04:34:14", "content": "Thank you for such a nice Post! Is it okay if I translate the blog post into my native language and share it?ππ @sauravmaheshkar
", "votes": null, "replies": [] } ] }, { "id": 1816837, "author_name": "allohvk", "author_url": "", "post_date": "06/10/2022 15:51:05", "content": "@snaker thanks for your valuable comments. Do you have any reference on item 3? Tokenizer size should not ideally grow during fine-tuning, but yes, there is definitely a steady increase in memory over time. Doesent look like a leak but it is happening.
", "votes": null, "replies": [ { "id": 1817283, "author_name": "mrchair2170", "author_url": "", "post_date": "06/11/2022 06:11:51", "content": "I think some tokenizers are using unlimited cache: https://github.com/huggingface/transformers/issues/2827#issue-564095688
", "votes": null, "replies": [] }, { "id": 1817304, "author_name": "allohvk", "author_url": "", "post_date": "06/11/2022 06:35:17", "content": "Very interesting. Thank you for the infoβ¦
", "votes": null, "replies": [] }, { "id": 1817907, "author_name": "snaker", "author_url": "", "post_date": "06/12/2022 02:56:54", "content": "If you do not need dynamic encoding, consider encode all the dataset and save to disk. This wll completely solve the memory issue had heavily boost the training speed.
", "votes": null, "replies": [] } ] }, { "id": 1820931, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "06/15/2022 05:36:22", "content": "If you are running low on VRAM, it's also worth trying gradient checkpointing to save VRAM at the cost of extra calculation. For BERT it may be ~25% slower but would use ~30-40% less VRAM.
\nA few other good pieces of advice from π€ https://huggingface.co/docs/transformers/performance
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1804554": "I'm new to NLP and transformers so I also face these issues for the first time.\n\n1. Using transformers `fast tokenizers`. Most of the tokenizer from `transformers` have a `fast` version which is implemented by `rust`. Use the fast version will significantly speed up training.\n\n2. Load all the data in memory and keep a smaller number work `workers`. There are no disk IO so the DataLoader processes are comuting dense now, and most of the computing are coming from the `tokenize`. If using fast version of tokenizers, I believe the implementation is multithreaded(without the GIL limitation) by it self. In my case, there are no benefit when the number of workers is larger than 2.\n\n3. Out of Memory issue. At first there will always be a OOM error when I train on my training set. I thought there are \"memory leak\" in my code. But turned out that the `Tokenizer` will eat more memory each time it meets some new words. So to reduce the num of workers will significant lower the memory usage. If still not enough memory to train a whole epoch, we could consider split the training set to subsets, and for each epoch train one subset of the dataset.\n\nUpdate:\n\nencode all the dataset and save to disk will completely solve the memory issue and boost training speed.", "1811780": "I also wrote a blogpost on \"**[Preventing The CUDA Out Of Memory Error In PyTorch](https://wandb.ai/wandb_fc/tips/reports/Preventing-The-CUDA-Out-Of-Memory-Error-In-PyTorch--VmlldzoxNzU3NjA1)**\". Hope it's helpful π", "1816837": "snaker thanks for your valuable comments. Do you have any reference on item 3? Tokenizer size should not ideally grow during fine-tuning, but yes, there is definitely a steady increase in memory over time. Doesent look like a leak but it is happening.", "1817283": "I think some tokenizers are using unlimited cache: https://github.com/huggingface/transformers/issues/2827#issue-564095688", "1817304": "Very interesting. Thank you for the info...", "1817907": "If you do not need dynamic encoding, consider encode all the dataset and save to disk. This wll completely solve the memory issue had heavily boost the training speed.", "1820876": "Thank you for such a nice Post! Is it okay if I translate the blog post into my native language and share it?ππ @sauravmaheshkar", "1820931": "If you are running low on VRAM, it's also worth trying gradient checkpointing to save VRAM at the cost of extra calculation. For BERT it may be ~25% slower but would use ~30-40% less VRAM.\n\nA few other good pieces of advice from π€ https://huggingface.co/docs/transformers/performance" }, "source": "meta" }