The last couple of weeks were full of blockers for me. I thought I had a good model working on 10% sample data, but am now unable to get it to work on complete data. Since the comp is anyway over for me, I thought of documenting few blockers faced/their solutions since I didn’t find them online (well, in the limited time I searched). These solutions might be helpful for others facing similar situation in future competitions..
\nNOTE - No AI4C specific insights are shared
\nBLOCKER 1
\nI was trying to get HF trainer to work. Documentation is sparse especially if we use custom models when things don't work seamlessly. Below is one open bug which can be painful to debug. Assume we have a custom trainer like the below:
class CustomTrainer(Trainer):\n def compute_loss(self, model, inputs, return_outputs=False):\n outputs = model(inputs[\"ids\"], inputs[\"mask\"], inputs[\"fts\"])\n loss = criterion(outputs, inputs[\"labels\"])\n\n ##Doesen’t work - Simple tuple\n return (loss, outputs) if return_outputs else loss\n\n ##This works - a tuple within a tuple: \n return (loss, (loss, outputs)) if return_outputs else loss\nWHY: First statement fails for the last batch which is of a different size usually. This is a bug in the HF trainer.
\nSOLUTION:If you drop the last batch it works. Else as an alternative, one can infer with a batch-size of 1 (per_device_eval_batch_size=1,) as done here which is nice reference kernel for HF trainer in general - https://github.com/ubamba98/feedback-prize. However, we need larger batch sizes in this comp and overriding the behaviour to return a tuple within a tuple makes it work for any batch_size.
\n
BLOCKER 2
\nIf we use fast tokenisers and take this (repeating and somewhat irritating) message seriously -
\n\n\"The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks…To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\"
\n
you might be tempted to set explicitly set the environment variable TOKENIZERS_PARALLELISM=True. DONT! Doing so will cause your training to freeze without any error message (in certain version of transformers).
\nWHY: The root cause is in the data loader which doesn't iterate if we use a combination of num_workers>0 (multithreading at Python level) and TOKENIZERS_PARALLELISM=True (multi-threading at Rust level).
\nSOLUTION:There are some simple solutions. I have documented this in a bit detail here - https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996
\n
BLOCKER 3 (Imp one)
\nTraining code gets 'Kill'ed every few hours by Linux OOM killer. Reason: cgroup ran out of memory and Linux killer process came in to sacrifice one process based on some internal algorithm. Initially it started killing one of the data loader threads thereby crashing the main program. Reducing num_workers to 2, delayed the killing but didn’t help. The source seems to be a Leaking CPU memory (not GPU mem). Note - this has nothing to do with tokenizer or code leaks because I had pre-tokenised everything and stored in dicts. So my dataloader/datasets were very simple actually without any scope for leaks.
PARTIALLY SOLVED ISSUE: Changed num_workers = 0 based on this: https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
\nThe error seems to be due to triggering of copy-on-write for python native datatypes like dicts, list etc which causes causing shared memory in getitem() of the data loader to be duplicated in all child processes over time. I could literally see this in action where we had 30GB+ VRAM, taken by each of the 7 child threads. Replacing lists by single refcount objects like numpy or Panda objects will fix this. Alternatively, simply set num_workers=0 (for this comp, itdidnt affect the training time)
But even then the main thread got killed after 6 hours or so. I tried restarting from the last checkpoint, but HF trainer checkpoint takes its job very seriously and restores even the memory state to pre-crash levels (doesn’t start from clean memory state), so there is crash almost immediately after restarting from checkpoint.
\nCOMPLETELY SOLVED ISSUE: I changed 2 things - Did tokenising on the fly and stopped using HF trainer which seems to fix the issue. Initially I blamed HF trainer for this leak (it is easy to blame one person for everything, no), but now that I have given up on the competition, I was carrying out some experiments which make me believe otherwise.
\nLIKELY source of error:
\nBased on the cloud provider, cgroup memory is allocated accordingly. In my case my CPU was 32GB memory(per core..there were 7 cores). The moment my code used to overshoot this, the Linux process forced a crash. I thought CPU memory was shared across cores leveraging memory controllers which should ideally give 32*7 GB, but looks like it is not the case here. So I had to make do with 32GB. But why was my program taking up this 32 GB?
The reason for the leak seems to be the ever-dependable Python dict object. Turns out there is a reason why it is super fast. While the implementation details are outside of python specifications, I believe in most cases they are stored on heap memory. Normally this does not impact anything but in cases like AI4C where we have 10's of millions of records it starts to clutter up heap memory very fast. On top of it we have inner dicts to store individual code cells. Something bad happens there in terms of memory management. Possibly it may still have worked with some careful coding (e.g. using dict.iterkeys instead of dict.keys etc) but I didn't have time for that due to comp deadline). Pretty soon the dict object starts gulping away RAM and the OS starts swapping between disk and RAM. You can notice this phase when swap memory becomes full and the training slows down. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. My cloud provider had just 2GB swap space. This got overloaded pretty quickly and this is where the Linux OOM killer steps in. Game over :)
\nIt would have probably worked with a non-dict object (even a list) with a couple of workers at the dataloader. The list object is much slower than dict but this where the multi-threading would help.
\nNote: The above findings are after optimising memory with standard items like mixed precision, AMP, gradient accumulation, batch_size reduction etc.
", "messages": [ { "id": 1894298, "postDate": "2022-08-11T11:43:07.990Z", "content": "The last couple of weeks were full of blockers for me. I thought I had a good model working on 10% sample data, but am now unable to get it to work on complete data. Since the comp is anyway over for me, I thought of documenting few blockers faced/their solutions since I didn’t find them online (well, in the limited time I searched). These solutions might be helpful for others facing similar situation in future competitions..
\nNOTE - No AI4C specific insights are shared
\nBLOCKER 1
\nI was trying to get HF trainer to work. Documentation is sparse especially if we use custom models when things don't work seamlessly. Below is one open bug which can be painful to debug. Assume we have a custom trainer like the below:
class CustomTrainer(Trainer):\n def compute_loss(self, model, inputs, return_outputs=False):\n outputs = model(inputs[\"ids\"], inputs[\"mask\"], inputs[\"fts\"])\n loss = criterion(outputs, inputs[\"labels\"])\n\n ##Doesen’t work - Simple tuple\n return (loss, outputs) if return_outputs else loss\n\n ##This works - a tuple within a tuple: \n return (loss, (loss, outputs)) if return_outputs else loss\nWHY: First statement fails for the last batch which is of a different size usually. This is a bug in the HF trainer.
\nSOLUTION:If you drop the last batch it works. Else as an alternative, one can infer with a batch-size of 1 (per_device_eval_batch_size=1,) as done here which is nice reference kernel for HF trainer in general - https://github.com/ubamba98/feedback-prize. However, we need larger batch sizes in this comp and overriding the behaviour to return a tuple within a tuple makes it work for any batch_size.
\n
BLOCKER 2
\nIf we use fast tokenisers and take this (repeating and somewhat irritating) message seriously -
\n\n\"The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks…To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\"
\n
you might be tempted to set explicitly set the environment variable TOKENIZERS_PARALLELISM=True. DONT! Doing so will cause your training to freeze without any error message (in certain version of transformers).
\nWHY: The root cause is in the data loader which doesn't iterate if we use a combination of num_workers>0 (multithreading at Python level) and TOKENIZERS_PARALLELISM=True (multi-threading at Rust level).
\nSOLUTION:There are some simple solutions. I have documented this in a bit detail here - https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996
\n
BLOCKER 3 (Imp one)
\nTraining code gets 'Kill'ed every few hours by Linux OOM killer. Reason: cgroup ran out of memory and Linux killer process came in to sacrifice one process based on some internal algorithm. Initially it started killing one of the data loader threads thereby crashing the main program. Reducing num_workers to 2, delayed the killing but didn’t help. The source seems to be a Leaking CPU memory (not GPU mem). Note - this has nothing to do with tokenizer or code leaks because I had pre-tokenised everything and stored in dicts. So my dataloader/datasets were very simple actually without any scope for leaks.
PARTIALLY SOLVED ISSUE: Changed num_workers = 0 based on this: https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
\nThe error seems to be due to triggering of copy-on-write for python native datatypes like dicts, list etc which causes causing shared memory in getitem() of the data loader to be duplicated in all child processes over time. I could literally see this in action where we had 30GB+ VRAM, taken by each of the 7 child threads. Replacing lists by single refcount objects like numpy or Panda objects will fix this. Alternatively, simply set num_workers=0 (for this comp, itdidnt affect the training time)
But even then the main thread got killed after 6 hours or so. I tried restarting from the last checkpoint, but HF trainer checkpoint takes its job very seriously and restores even the memory state to pre-crash levels (doesn’t start from clean memory state), so there is crash almost immediately after restarting from checkpoint.
\nCOMPLETELY SOLVED ISSUE: I changed 2 things - Did tokenising on the fly and stopped using HF trainer which seems to fix the issue. Initially I blamed HF trainer for this leak (it is easy to blame one person for everything, no), but now that I have given up on the competition, I was carrying out some experiments which make me believe otherwise.
\nLIKELY source of error:
\nBased on the cloud provider, cgroup memory is allocated accordingly. In my case my CPU was 32GB memory(per core..there were 7 cores). The moment my code used to overshoot this, the Linux process forced a crash. I thought CPU memory was shared across cores leveraging memory controllers which should ideally give 32*7 GB, but looks like it is not the case here. So I had to make do with 32GB. But why was my program taking up this 32 GB?
The reason for the leak seems to be the ever-dependable Python dict object. Turns out there is a reason why it is super fast. While the implementation details are outside of python specifications, I believe in most cases they are stored on heap memory. Normally this does not impact anything but in cases like AI4C where we have 10's of millions of records it starts to clutter up heap memory very fast. On top of it we have inner dicts to store individual code cells. Something bad happens there in terms of memory management. Possibly it may still have worked with some careful coding (e.g. using dict.iterkeys instead of dict.keys etc) but I didn't have time for that due to comp deadline). Pretty soon the dict object starts gulping away RAM and the OS starts swapping between disk and RAM. You can notice this phase when swap memory becomes full and the training slows down. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. My cloud provider had just 2GB swap space. This got overloaded pretty quickly and this is where the Linux OOM killer steps in. Game over :)
\nIt would have probably worked with a non-dict object (even a list) with a couple of workers at the dataloader. The list object is much slower than dict but this where the multi-threading would help.
\nNote: The above findings are after optimising memory with standard items like mixed precision, AMP, gradient accumulation, batch_size reduction etc.
", "rawMarkdown": "The last couple of weeks were full of blockers for me. I thought I had a good model working on 10% sample data, but am now unable to get it to work on complete data. Since the comp is anyway over for me, I thought of documenting few blockers faced/their solutions since I didn’t find them online (well, in the limited time I searched). These solutions might be helpful for others facing similar situation in future competitions.. \n\nNOTE - No AI4C specific insights are shared\n\n**BLOCKER 1**\nI was trying to get HF trainer to work. Documentation is sparse especially if we use custom models when things don't work seamlessly. Below is one open bug which can be painful to debug. Assume we have a custom trainer like the below:\n\n```\nclass CustomTrainer(Trainer):\n def compute_loss(self, model, inputs, return_outputs=False):\n outputs = model(inputs[\"ids\"], inputs[\"mask\"], inputs[\"fts\"])\n loss = criterion(outputs, inputs[\"labels\"])\n\n ##Doesen’t work - Simple tuple\n\t\treturn (loss, outputs) if return_outputs else loss\n\n ##This works - a tuple within a tuple: \n return (loss, (loss, outputs)) if return_outputs else loss\n\n```\nWHY: First statement fails for the last batch which is of a different size usually. This is a bug in the HF trainer.\n\nSOLUTION:If you drop the last batch it works. Else as an alternative, one can infer with a batch-size of 1 (per_device_eval_batch_size=1,) as done here which is nice reference kernel for HF trainer in general - https://github.com/ubamba98/feedback-prize. However, we need larger batch sizes in this comp and overriding the behaviour to return a tuple within a tuple makes it work for any batch_size.\n \n\n\n**BLOCKER 2**\nIf we use fast tokenisers and take this (repeating and somewhat irritating) message seriously - \n> \"The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\"\n\nyou might be tempted to set explicitly set the environment variable TOKENIZERS_PARALLELISM=True. DONT! Doing so will cause your training to freeze without any error message (in certain version of transformers). \n\nWHY: The root cause is in the data loader which doesn't iterate if we use a combination of num_workers>0 (multithreading at Python level) and TOKENIZERS_PARALLELISM=True (multi-threading at Rust level). \n\nSOLUTION:There are some simple solutions. I have documented this in a bit detail here - https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996\n \n\n\n**BLOCKER 3** (Imp one)\nTraining code gets 'Kill'ed every few hours by Linux OOM killer. Reason: cgroup ran out of memory and Linux killer process came in to sacrifice one process based on some internal algorithm. Initially it started killing one of the data loader threads thereby crashing the main program. Reducing num_workers to 2, delayed the killing but didn’t help. The source seems to be a Leaking CPU memory (not GPU mem). Note - this has nothing to do with tokenizer or code leaks because I had pre-tokenised everything and stored in dicts. So my dataloader/datasets were very simple actually without any scope for leaks.\n\nPARTIALLY SOLVED ISSUE: Changed num_workers = 0 based on this: https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662\nThe error seems to be due to triggering of copy-on-write for python native datatypes like dicts, list etc which causes causing shared memory in getitem() of the data loader to be duplicated in all child processes over time. I could literally see this in action where we had 30GB+ VRAM, taken by each of the 7 child threads. Replacing lists by single refcount objects like numpy or Panda objects will fix this. Alternatively, simply set num_workers=0 (for this comp, itdidnt affect the training time)\n\nBut even then the main thread got killed after 6 hours or so. I tried restarting from the last checkpoint, but HF trainer checkpoint takes its job very seriously and restores even the memory state to pre-crash levels (doesn’t start from clean memory state), so there is crash almost immediately after restarting from checkpoint.\n\nCOMPLETELY SOLVED ISSUE: I changed 2 things - Did tokenising on the fly and stopped using HF trainer which seems to fix the issue. Initially I blamed HF trainer for this leak (it is easy to blame one person for everything, no), but now that I have given up on the competition, I was carrying out some experiments which make me believe otherwise.\n\nLIKELY source of error: \nBased on the cloud provider, cgroup memory is allocated accordingly. In my case my CPU was 32GB memory(per core..there were 7 cores). The moment my code used to overshoot this, the Linux process forced a crash. I thought CPU memory was shared across cores leveraging memory controllers which should ideally give 32*7 GB, but looks like it is not the case here. So I had to make do with 32GB. But why was my program taking up this 32 GB?\n\nThe reason for the leak seems to be the ever-dependable Python dict object. Turns out there is a reason why it is super fast. While the implementation details are outside of python specifications, I believe in most cases they are stored on heap memory. Normally this does not impact anything but in cases like AI4C where we have 10's of millions of records it starts to clutter up heap memory very fast. On top of it we have inner dicts to store individual code cells. Something bad happens there in terms of memory management. Possibly it may still have worked with some careful coding (e.g. using dict.iterkeys instead of dict.keys etc) but I didn't have time for that due to comp deadline). Pretty soon the dict object starts gulping away RAM and the OS starts swapping between disk and RAM. You can notice this phase when swap memory becomes full and the training slows down. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. My cloud provider had just 2GB swap space. This got overloaded pretty quickly and this is where the Linux OOM killer steps in. Game over :)\n\nIt would have probably worked with a non-dict object (even a list) with a couple of workers at the dataloader. The list object is much slower than dict but this where the multi-threading would help. \n\nNote: The above findings are after optimising memory with standard items like mixed precision, AMP, gradient accumulation, batch_size reduction etc.", "votes": 5 }, { "id": 1894392, "postDate": "2022-08-11T13:14:51.373Z", "content": "\n\nDocumentation is sparse especially if we use custom models when things don't work seamlessly.
\n
Hi @allohvk , I have shown how to write a custom model which play nicely and is consistent with the HF trainer in my recent blog post here. Although it is a lot of work to write a custom model, I found this to be the only effective way to write custom models for HF trainer. Hope this might help you in the future.
", "rawMarkdown": "> Documentation is sparse especially if we use custom models when things don't work seamlessly.\n\nHi @allohvk , I have shown how to write a custom model which play nicely and is consistent with the HF trainer in my recent blog post [here](https://jarvislabs.ai/blogs/hf-getting-started/#custom-model-implementation). Although it is a lot of work to write a custom model, I found this to be the only effective way to write custom models for HF trainer. Hope this might help you in the future.", "votes": 3, "replies": [ { "id": 1894464, "postDate": "2022-08-11T14:11:15.953Z", "content": "Interesting, this statement didn't work for me:
\n return (loss, outputs) if return_outputs else loss\nMaybe you ran with drop_last as True, or your last batch had same size as any other batch or (most likely) you used the default option of return_outputs=False. Below should fix the same in case one wants to use return_outputs=True
\n return (loss, (loss, outputs)) if return_outputs else loss\nBut otherwise it is a superb piece of writing. Thanks for publishing the same. Very, very informative and I only wish I had found your blog earlier.
", "rawMarkdown": "Interesting, this statement didn't work for me:\n```\n return (loss, outputs) if return_outputs else loss\n```\n\nMaybe you ran with drop_last as True, or your last batch had same size as any other batch or (most likely) you used the default option of return_outputs=False. Below should fix the same in case one wants to use return_outputs=True\n\n```\n return (loss, (loss, outputs)) if return_outputs else loss\n\n```\nBut otherwise it is a superb piece of writing. Thanks for publishing the same. Very, very informative and I only wish I had found your blog earlier.", "votes": 1 } ] }, { "id": 1894732, "postDate": "2022-08-11T17:15:34.193Z", "content": "When I use the Trainer, I usually define my loss within the model rather than in the compute_loss function. Did you try that?
The output of the forward function is (loss, logits)
For some reason, I had to create a custom_loss. I forgot why. Maybe it was not needed in hindsight.
\nFor regular flows, HF trainer works like a charm. It is awesome.
\nAlso as you mentioned, the normal output is (loss, logits) but for above case code works only if we do a (loss, (loss, logits)). The HF trainer seems to reads the second entry of the output tuple and throws an error if it is not of format (loss, logits). This condition gets simulated only in the last batch (if its size is not equal to regular batch-size).
", "rawMarkdown": "For some reason, I had to create a custom_loss. I forgot why. Maybe it was not needed in hindsight.\n\nFor regular flows, HF trainer works like a charm. It is awesome.\n\nAlso as you mentioned, the normal output is (loss, logits) but for above case code works only if we do a (loss, (loss, logits)). The HF trainer seems to reads the second entry of the output tuple and throws an error if it is not of format (loss, logits). This condition gets simulated only in the last batch (if its size is not equal to regular batch-size).", "votes": 1 } ] }, { "id": 1894371, "postDate": "2022-08-11T12:53:19.803Z", "content": "Great notes!
", "rawMarkdown": "Great notes!", "votes": 1, "replies": [ { "id": 1894389, "postDate": "2022-08-11T13:11:30.317Z", "content": "Thanks Mark. All the best for the comp!!
", "rawMarkdown": "Thanks Mark. All the best for the comp!!", "votes": 1 } ] }, { "id": 1894424, "postDate": "2022-08-11T13:49:44.080Z", "content": "I faced similar problems, has to restart from latest saved epoch checkpoint. I have never faced this using tensorfow and tfrecords before but for the more flexibility I have to use torch which seems unstable for large datasets. Set pin_memory to False help a lot but still has chance to crash.
", "rawMarkdown": "I faced similar problems, has to restart from latest saved epoch checkpoint. I have never faced this using tensorfow and tfrecords before but for the more flexibility I have to use torch which seems unstable for large datasets. Set pin_memory to False help a lot but still has chance to crash.", "votes": 2, "replies": [ { "id": 1894481, "postDate": "2022-08-11T14:23:32.117Z", "content": "Agree, pin_memory = True leads to some strage behaviour
\nUnfortuntely in my case, I had left everything for last minute. The delays meant I had only around 36 hours to train and I had to cut down my model size from 1536 to 768…and base model to small version…which is still running as I write this :(
", "rawMarkdown": "Agree, pin_memory = True leads to some strage behaviour\n\nUnfortuntely in my case, I had left everything for last minute. The delays meant I had only around 36 hours to train and I had to cut down my model size from 1536 to 768...and base model to small version...which is still running as I write this :(" }, { "id": 1894556, "postDate": "2022-08-11T15:13:56.783Z", "content": "Yes , training need long time. I have found one serious bug in the final days, I could only finetune from checkpoint with mistakes, could not afford to retrain from scratch💔
", "rawMarkdown": "Yes , training need long time. I have found one serious bug in the final days, I could only finetune from checkpoint with mistakes, could not afford to retrain from scratch💔", "votes": 1 } ] }, { "id": 1894462, "postDate": "2022-08-11T14:10:05.083Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1894392, "author_name": "Atharva Ingle", "author_url": "", "post_date": "2022-08-11T13:14:51.373000", "content": "\n\nDocumentation is sparse especially if we use custom models when things don't work seamlessly.
\n
Hi @allohvk , I have shown how to write a custom model which play nicely and is consistent with the HF trainer in my recent blog post here. Although it is a lot of work to write a custom model, I found this to be the only effective way to write custom models for HF trainer. Hope this might help you in the future.
", "votes": 3, "replies": [ { "id": 1894464, "author_name": "Allohvk", "author_url": "", "post_date": "2022-08-11T14:11:15.953000", "content": "Interesting, this statement didn't work for me:
\n return (loss, outputs) if return_outputs else loss\nMaybe you ran with drop_last as True, or your last batch had same size as any other batch or (most likely) you used the default option of return_outputs=False. Below should fix the same in case one wants to use return_outputs=True
\n return (loss, (loss, outputs)) if return_outputs else loss\nBut otherwise it is a superb piece of writing. Thanks for publishing the same. Very, very informative and I only wish I had found your blog earlier.
", "votes": 1, "replies": [] } ] }, { "id": 1894732, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-08-11T17:15:34.193000", "content": "When I use the Trainer, I usually define my loss within the model rather than in the compute_loss function. Did you try that?
The output of the forward function is (loss, logits)
For some reason, I had to create a custom_loss. I forgot why. Maybe it was not needed in hindsight.
\nFor regular flows, HF trainer works like a charm. It is awesome.
\nAlso as you mentioned, the normal output is (loss, logits) but for above case code works only if we do a (loss, (loss, logits)). The HF trainer seems to reads the second entry of the output tuple and throws an error if it is not of format (loss, logits). This condition gets simulated only in the last batch (if its size is not equal to regular batch-size).
", "votes": 1, "replies": [] } ] }, { "id": 1894371, "author_name": "Mark Baushenko", "author_url": "", "post_date": "2022-08-11T12:53:19.803000", "content": "Great notes!
", "votes": 1, "replies": [ { "id": 1894389, "author_name": "Allohvk", "author_url": "", "post_date": "2022-08-11T13:11:30.317000", "content": "Thanks Mark. All the best for the comp!!
", "votes": 1, "replies": [] } ] }, { "id": 1894424, "author_name": "gezi", "author_url": "", "post_date": "2022-08-11T13:49:44.080000", "content": "I faced similar problems, has to restart from latest saved epoch checkpoint. I have never faced this using tensorfow and tfrecords before but for the more flexibility I have to use torch which seems unstable for large datasets. Set pin_memory to False help a lot but still has chance to crash.
", "votes": 2, "replies": [ { "id": 1894481, "author_name": "Allohvk", "author_url": "", "post_date": "2022-08-11T14:23:32.117000", "content": "Agree, pin_memory = True leads to some strage behaviour
\nUnfortuntely in my case, I had left everything for last minute. The delays meant I had only around 36 hours to train and I had to cut down my model size from 1536 to 768…and base model to small version…which is still running as I write this :(
", "votes": 0, "replies": [] }, { "id": 1894556, "author_name": "gezi", "author_url": "", "post_date": "2022-08-11T15:13:56.783000", "content": "Yes , training need long time. I have found one serious bug in the final days, I could only finetune from checkpoint with mistakes, could not afford to retrain from scratch💔
", "votes": 1, "replies": [] } ] }, { "id": 1894462, "author_name": "", "author_url": "", "post_date": "2022-08-11T14:10:05.083000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1894298": "The last couple of weeks were full of blockers for me. I thought I had a good model working on 10% sample data, but am now unable to get it to work on complete data. Since the comp is anyway over for me, I thought of documenting few blockers faced/their solutions since I didn’t find them online (well, in the limited time I searched). These solutions might be helpful for others facing similar situation in future competitions.. \n\nNOTE - No AI4C specific insights are shared\n\n**BLOCKER 1**\nI was trying to get HF trainer to work. Documentation is sparse especially if we use custom models when things don't work seamlessly. Below is one open bug which can be painful to debug. Assume we have a custom trainer like the below:\n\n```\nclass CustomTrainer(Trainer):\n def compute_loss(self, model, inputs, return_outputs=False):\n outputs = model(inputs[\"ids\"], inputs[\"mask\"], inputs[\"fts\"])\n loss = criterion(outputs, inputs[\"labels\"])\n\n ##Doesen’t work - Simple tuple\n\t\treturn (loss, outputs) if return_outputs else loss\n\n ##This works - a tuple within a tuple: \n return (loss, (loss, outputs)) if return_outputs else loss\n\n```\nWHY: First statement fails for the last batch which is of a different size usually. This is a bug in the HF trainer.\n\nSOLUTION:If you drop the last batch it works. Else as an alternative, one can infer with a batch-size of 1 (per_device_eval_batch_size=1,) as done here which is nice reference kernel for HF trainer in general - https://github.com/ubamba98/feedback-prize. However, we need larger batch sizes in this comp and overriding the behaviour to return a tuple within a tuple makes it work for any batch_size.\n \n\n\n**BLOCKER 2**\nIf we use fast tokenisers and take this (repeating and somewhat irritating) message seriously - \n> \"The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\"\n\nyou might be tempted to set explicitly set the environment variable TOKENIZERS_PARALLELISM=True. DONT! Doing so will cause your training to freeze without any error message (in certain version of transformers). \n\nWHY: The root cause is in the data loader which doesn't iterate if we use a combination of num_workers>0 (multithreading at Python level) and TOKENIZERS_PARALLELISM=True (multi-threading at Rust level). \n\nSOLUTION:There are some simple solutions. I have documented this in a bit detail here - https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996\n \n\n\n**BLOCKER 3** (Imp one)\nTraining code gets 'Kill'ed every few hours by Linux OOM killer. Reason: cgroup ran out of memory and Linux killer process came in to sacrifice one process based on some internal algorithm. Initially it started killing one of the data loader threads thereby crashing the main program. Reducing num_workers to 2, delayed the killing but didn’t help. The source seems to be a Leaking CPU memory (not GPU mem). Note - this has nothing to do with tokenizer or code leaks because I had pre-tokenised everything and stored in dicts. So my dataloader/datasets were very simple actually without any scope for leaks.\n\nPARTIALLY SOLVED ISSUE: Changed num_workers = 0 based on this: https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662\nThe error seems to be due to triggering of copy-on-write for python native datatypes like dicts, list etc which causes causing shared memory in getitem() of the data loader to be duplicated in all child processes over time. I could literally see this in action where we had 30GB+ VRAM, taken by each of the 7 child threads. Replacing lists by single refcount objects like numpy or Panda objects will fix this. Alternatively, simply set num_workers=0 (for this comp, itdidnt affect the training time)\n\nBut even then the main thread got killed after 6 hours or so. I tried restarting from the last checkpoint, but HF trainer checkpoint takes its job very seriously and restores even the memory state to pre-crash levels (doesn’t start from clean memory state), so there is crash almost immediately after restarting from checkpoint.\n\nCOMPLETELY SOLVED ISSUE: I changed 2 things - Did tokenising on the fly and stopped using HF trainer which seems to fix the issue. Initially I blamed HF trainer for this leak (it is easy to blame one person for everything, no), but now that I have given up on the competition, I was carrying out some experiments which make me believe otherwise.\n\nLIKELY source of error: \nBased on the cloud provider, cgroup memory is allocated accordingly. In my case my CPU was 32GB memory(per core..there were 7 cores). The moment my code used to overshoot this, the Linux process forced a crash. I thought CPU memory was shared across cores leveraging memory controllers which should ideally give 32*7 GB, but looks like it is not the case here. So I had to make do with 32GB. But why was my program taking up this 32 GB?\n\nThe reason for the leak seems to be the ever-dependable Python dict object. Turns out there is a reason why it is super fast. While the implementation details are outside of python specifications, I believe in most cases they are stored on heap memory. Normally this does not impact anything but in cases like AI4C where we have 10's of millions of records it starts to clutter up heap memory very fast. On top of it we have inner dicts to store individual code cells. Something bad happens there in terms of memory management. Possibly it may still have worked with some careful coding (e.g. using dict.iterkeys instead of dict.keys etc) but I didn't have time for that due to comp deadline). Pretty soon the dict object starts gulping away RAM and the OS starts swapping between disk and RAM. You can notice this phase when swap memory becomes full and the training slows down. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. My cloud provider had just 2GB swap space. This got overloaded pretty quickly and this is where the Linux OOM killer steps in. Game over :)\n\nIt would have probably worked with a non-dict object (even a list) with a couple of workers at the dataloader. The list object is much slower than dict but this where the multi-threading would help. \n\nNote: The above findings are after optimising memory with standard items like mixed precision, AMP, gradient accumulation, batch_size reduction etc.", "1894392": "> Documentation is sparse especially if we use custom models when things don't work seamlessly.\n\nHi @allohvk , I have shown how to write a custom model which play nicely and is consistent with the HF trainer in my recent blog post [here](https://jarvislabs.ai/blogs/hf-getting-started/#custom-model-implementation). Although it is a lot of work to write a custom model, I found this to be the only effective way to write custom models for HF trainer. Hope this might help you in the future.", "1894732": "When I use the Trainer, I usually define my loss within the model rather than in the `compute_loss` function. Did you try that? \n\nThe output of the `forward` function is (loss, logits)", "1894371": "Great notes!", "1894424": "I faced similar problems, has to restart from latest saved epoch checkpoint. I have never faced this using tensorfow and tfrecords before but for the more flexibility I have to use torch which seems unstable for large datasets. Set pin_memory to False help a lot but still has chance to crash.", "1894462": "" } }