Is anyone else using Colabs Pro and running into disconnection issues? \nSometimes it runs fun and I'll get through 300 epochs running YOLOv5 but there are many cases where it'll hit 30 or 40 epochs and time out and not recover and I look and Colabs tab is red and the kernel is dead. Not quite sure what is going on but it seemed a lot more stable using the free version than Pro.
\n\nJust curious if it's just me and if you are using High-Ram or if you had issues how you may have fixed those.
", "messages": [ { "id": "912961", "postDate": "07/02/2020 21:15:24", "content": "Is anyone else using Colabs Pro and running into disconnection issues? \nSometimes it runs fun and I'll get through 300 epochs running YOLOv5 but there are many cases where it'll hit 30 or 40 epochs and time out and not recover and I look and Colabs tab is red and the kernel is dead. Not quite sure what is going on but it seemed a lot more stable using the free version than Pro.
\n\nJust curious if it's just me and if you are using High-Ram or if you had issues how you may have fixed those.
", "rawMarkdown": "Is anyone else using Colabs Pro and running into disconnection issues? \nSometimes it runs fun and I'll get through 300 epochs running YOLOv5 but there are many cases where it'll hit 30 or 40 epochs and time out and not recover and I look and Colabs tab is red and the kernel is dead. Not quite sure what is going on but it seemed a lot more stable using the free version than Pro. \n\nJust curious if it's just me and if you are using High-Ram or if you had issues how you may have fixed those.", "votes": null }, { "id": "912980", "postDate": "07/02/2020 21:34:23", "content": "I don't think I've ever ran into that issue on Colab Pro. You are running that javascript thing to stop colab from disconnecting right?
", "rawMarkdown": "I don't think I've ever ran into that issue on Colab Pro. You are running that javascript thing to stop colab from disconnecting right?", "votes": null }, { "id": "912998", "postDate": "07/02/2020 21:53:46", "content": "@jonykarki I'm not using the javascript because people said it didn't work anymore.. that's why I bumped up to Pro. It does continue the epoch most of the time if it stalls but I have no idea why it's even stalling in the first place. Sometimes I'm working on my computer and it's running in another tab. With free I'd run it all night for 6-7 hours with no issues.
\n\nAre you running the javascript console thing?
", "rawMarkdown": "jonykarki I'm not using the javascript because people said it didn't work anymore.. that's why I bumped up to Pro. It does continue the epoch most of the time if it stalls but I have no idea why it's even stalling in the first place. Sometimes I'm working on my computer and it's running in another tab. With free I'd run it all night for 6-7 hours with no issues. \n\nAre you running the javascript console thing?", "votes": null }, { "id": "913016", "postDate": "07/02/2020 22:15:29", "content": "Yes, I am actually. I'm not sure what exactly the problem is in your case, but I always run the following javascript code and it seems to work for me.
\n\ntotalTime = 0\nfunction ClickConnect() {\n totalTime += (60000/1000)/60\n console.log('Working')\n console.log(\"Total time passed \" + totalTime)\n document\n .querySelector('#top-toolbar > colab-connect-button')\n .shadowRoot.querySelector('#connect')\n .click()\n}
\n\nsetInterval(ClickConnect, 60000)
", "rawMarkdown": "Yes, I am actually. I'm not sure what exactly the problem is in your case, but I always run the following javascript code and it seems to work for me. \n\ntotalTime = 0\nfunction ClickConnect() {\n totalTime += (60000/1000)/60\n console.log('Working')\n console.log(\"Total time passed \" + totalTime)\n document\n .querySelector('#top-toolbar > colab-connect-button')\n .shadowRoot.querySelector('#connect')\n .click()\n}\n\nsetInterval(ClickConnect, 60000)", "votes": null }, { "id": "913028", "postDate": "07/02/2020 22:35:04", "content": "Awesome @jonykarki This worked great!
", "rawMarkdown": "Awesome @jonykarki This worked great!", "votes": null }, { "id": "913138", "postDate": "07/03/2020 02:39:31", "content": "Have anyone used Colab Pro TPUs or you guys just using GPUs?
", "rawMarkdown": "Have anyone used Colab Pro TPUs or you guys just using GPUs?", "votes": null }, { "id": "913144", "postDate": "07/03/2020 02:47:42", "content": "I assume most people are using GPUs because there aren't many detection frameworks that are TPU compatible
", "rawMarkdown": "I assume most people are using GPUs because there aren't many detection frameworks that are TPU compatible", "votes": null }, { "id": "913153", "postDate": "07/03/2020 02:57:16", "content": "So @jonykarki solution worked great. But I hit 67 epochs running YOLOv5 and got this error:\n\"Buffered data was truncated after reaching the output size limit\"
\n\nThis is a new error I hadn't seen in colabs. Anyone have any thoughts?
", "rawMarkdown": "So @jonykarki solution worked great. But I hit 67 epochs running YOLOv5 and got this error:\n\"Buffered data was truncated after reaching the output size limit\"\n\nThis is a new error I hadn't seen in colabs. Anyone have any thoughts?", "votes": null }, { "id": "913746", "postDate": "07/03/2020 11:40:22", "content": "It's a problem of colab. Colab only allows limited output size in a cell, so when the cell outputs too much, the output text will be truncated. If you want to monitor the training process all time, modify the training code to output less, like updating the progress bar line for less times. If you don' t care about it, just read result.txt, it will be updated every epoch.\nSo in conclusion, it's just a warning, not an error. It will not stop your training, but you can't monitor the training process in the notebook after this happens.
@wasupandceacar Thank you! Wish I knew that last night before I stopped executing the kernel. Oh well. Live and learn. :)
", "rawMarkdown": "wasupandceacar Thank you! Wish I knew that last night before I stopped executing the kernel. Oh well. Live and learn. :)", "votes": null }, { "id": "914424", "postDate": "07/03/2020 21:13:01", "content": "Can someone please help me out and tell me why I got a downvote? Should I have posted this elsewhere?
", "rawMarkdown": "Can someone please help me out and tell me why I got a downvote? Should I have posted this elsewhere?", "votes": null } ], "comments": [ { "id": 912980, "author_name": "jonykarki", "author_url": "", "post_date": "07/02/2020 21:34:23", "content": "I don't think I've ever ran into that issue on Colab Pro. You are running that javascript thing to stop colab from disconnecting right?
", "votes": null, "replies": [ { "id": 912998, "author_name": "crained", "author_url": "", "post_date": "07/02/2020 21:53:46", "content": "@jonykarki I'm not using the javascript because people said it didn't work anymore.. that's why I bumped up to Pro. It does continue the epoch most of the time if it stalls but I have no idea why it's even stalling in the first place. Sometimes I'm working on my computer and it's running in another tab. With free I'd run it all night for 6-7 hours with no issues.
\n\nAre you running the javascript console thing?
", "votes": null, "replies": [] }, { "id": 913016, "author_name": "jonykarki", "author_url": "", "post_date": "07/02/2020 22:15:29", "content": "Yes, I am actually. I'm not sure what exactly the problem is in your case, but I always run the following javascript code and it seems to work for me.
\n\ntotalTime = 0\nfunction ClickConnect() {\n totalTime += (60000/1000)/60\n console.log('Working')\n console.log(\"Total time passed \" + totalTime)\n document\n .querySelector('#top-toolbar > colab-connect-button')\n .shadowRoot.querySelector('#connect')\n .click()\n}
\n\nsetInterval(ClickConnect, 60000)
", "votes": null, "replies": [] }, { "id": 913028, "author_name": "crained", "author_url": "", "post_date": "07/02/2020 22:35:04", "content": "Awesome @jonykarki This worked great!
", "votes": null, "replies": [] } ] }, { "id": 913138, "author_name": "bopengiowa", "author_url": "", "post_date": "07/03/2020 02:39:31", "content": "Have anyone used Colab Pro TPUs or you guys just using GPUs?
", "votes": null, "replies": [ { "id": 913144, "author_name": "matthewmasters", "author_url": "", "post_date": "07/03/2020 02:47:42", "content": "I assume most people are using GPUs because there aren't many detection frameworks that are TPU compatible
", "votes": null, "replies": [] } ] }, { "id": 913153, "author_name": "crained", "author_url": "", "post_date": "07/03/2020 02:57:16", "content": "So @jonykarki solution worked great. But I hit 67 epochs running YOLOv5 and got this error:\n\"Buffered data was truncated after reaching the output size limit\"
\n\nThis is a new error I hadn't seen in colabs. Anyone have any thoughts?
", "votes": null, "replies": [ { "id": 913746, "author_name": "wasupandceacar", "author_url": "", "post_date": "07/03/2020 11:40:22", "content": "It's a problem of colab. Colab only allows limited output size in a cell, so when the cell outputs too much, the output text will be truncated. If you want to monitor the training process all time, modify the training code to output less, like updating the progress bar line for less times. If you don' t care about it, just read result.txt, it will be updated every epoch.\nSo in conclusion, it's just a warning, not an error. It will not stop your training, but you can't monitor the training process in the notebook after this happens.
@wasupandceacar Thank you! Wish I knew that last night before I stopped executing the kernel. Oh well. Live and learn. :)
", "votes": null, "replies": [] } ] }, { "id": 914424, "author_name": "crained", "author_url": "", "post_date": "07/03/2020 21:13:01", "content": "Can someone please help me out and tell me why I got a downvote? Should I have posted this elsewhere?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "912961": "Is anyone else using Colabs Pro and running into disconnection issues? \nSometimes it runs fun and I'll get through 300 epochs running YOLOv5 but there are many cases where it'll hit 30 or 40 epochs and time out and not recover and I look and Colabs tab is red and the kernel is dead. Not quite sure what is going on but it seemed a lot more stable using the free version than Pro. \n\nJust curious if it's just me and if you are using High-Ram or if you had issues how you may have fixed those.", "912980": "I don't think I've ever ran into that issue on Colab Pro. You are running that javascript thing to stop colab from disconnecting right?", "912998": "jonykarki I'm not using the javascript because people said it didn't work anymore.. that's why I bumped up to Pro. It does continue the epoch most of the time if it stalls but I have no idea why it's even stalling in the first place. Sometimes I'm working on my computer and it's running in another tab. With free I'd run it all night for 6-7 hours with no issues. \n\nAre you running the javascript console thing?", "913016": "Yes, I am actually. I'm not sure what exactly the problem is in your case, but I always run the following javascript code and it seems to work for me. \n\ntotalTime = 0\nfunction ClickConnect() {\n totalTime += (60000/1000)/60\n console.log('Working')\n console.log(\"Total time passed \" + totalTime)\n document\n .querySelector('#top-toolbar > colab-connect-button')\n .shadowRoot.querySelector('#connect')\n .click()\n}\n\nsetInterval(ClickConnect, 60000)", "913028": "Awesome @jonykarki This worked great!", "913138": "Have anyone used Colab Pro TPUs or you guys just using GPUs?", "913144": "I assume most people are using GPUs because there aren't many detection frameworks that are TPU compatible", "913153": "So @jonykarki solution worked great. But I hit 67 epochs running YOLOv5 and got this error:\n\"Buffered data was truncated after reaching the output size limit\"\n\nThis is a new error I hadn't seen in colabs. Anyone have any thoughts?", "913746": "It's a problem of colab. Colab only allows limited output size in a cell, so when the cell outputs too much, the output text will be truncated. If you want to monitor the training process all time, modify the training code to output less, like updating the progress bar line for less times. If you don' t care about it, just read `result.txt`, it will be updated every epoch.\nSo in conclusion, it's just a warning, not an error. It will not stop your training, but you can't monitor the training process in the notebook after this happens.", "913862": "wasupandceacar Thank you! Wish I knew that last night before I stopped executing the kernel. Oh well. Live and learn. :)", "914424": "Can someone please help me out and tell me why I got a downvote? Should I have posted this elsewhere?" }, "source": "meta" }