is it normal for an epoch to take 52 minutes in colab while activating gpu with apex?
", "messages": [ { "id": 926276, "postDate": "2020-07-12T16:05:10.403Z", "content": "I am facing the same issue. It seems that colab assigns gpu randomly, and the training time, various from around 10 mins to 50 mins, is related to the machine that colab assigns.
", "rawMarkdown": "I am facing the same issue. It seems that colab assigns gpu randomly, and the training time, various from around 10 mins to 50 mins, is related to the machine that colab assigns.", "votes": 1, "replies": [ { "id": 926321, "postDate": "2020-07-12T16:39:54.700Z", "content": "yes thought so \nit makes me feel like this competition is more of a computational resources show off than anything else even the other competitions this summer are all CV with datasets size ranging from 30gb-300gb+
\nit makes me wanna give up on winning medals this summer
LOL. It's really not normal for an epoch to take 52 minutes. For me, it took about 10 minutes per epoch on P100 GPU with about 5-6 GBs of mem usage with apex.
", "rawMarkdown": "LOL. It's really not normal for an epoch to take 52 minutes. For me, it took about 10 minutes per epoch on P100 GPU with about 5-6 GBs of mem usage with apex." }, { "id": 924335, "postDate": "2020-07-11T11:32:33.593Z", "content": "Depends on the model, image size I guess - but seems a bit on the long side... How are you loading your data?
", "rawMarkdown": "Depends on the model, image size I guess - but seems a bit on the long side... How are you loading your data?", "replies": [ { "id": 924642, "postDate": "2020-07-11T14:56:27.950Z", "content": "from google drive but what I have noticed is that colab is giving me tesla k80 12gb while giving tesla p100 16gb for others I thought this might be the reason.I forgot to say its a yolov5 with original image size 1024
", "rawMarkdown": "from google drive but what I have noticed is that colab is giving me tesla k80 12gb while giving tesla p100 16gb for others I thought this might be the reason.I forgot to say its a yolov5 with original image size 1024\n" }, { "id": 926192, "postDate": "2020-07-12T15:10:53.833Z", "content": "if you are mounting the data-set from google drive, I think you should transfer all the data into your local working folder on colab before using the data-loader for model training. Loading the data directly from drive will add quite a bit of over-head.
", "rawMarkdown": "if you are mounting the data-set from google drive, I think you should transfer all the data into your local working folder on colab before using the data-loader for model training. Loading the data directly from drive will add quite a bit of over-head." } ] }, { "id": 924257, "postDate": "2020-07-11T10:55:37.750Z", "content": "is it normal for an epoch to take 52 minutes in colab while activating gpu with apex?
", "rawMarkdown": "is it normal for an epoch to take 52 minutes in colab while activating gpu with apex?" }, { "id": 924339, "postDate": "2020-07-11T11:35:38.460Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 926276, "author_name": "RoyZ", "author_url": "", "post_date": "2020-07-12T16:05:10.403000", "content": "I am facing the same issue. It seems that colab assigns gpu randomly, and the training time, various from around 10 mins to 50 mins, is related to the machine that colab assigns.
", "votes": 1, "replies": [ { "id": 926321, "author_name": "mohaned abid", "author_url": "", "post_date": "2020-07-12T16:39:54.700000", "content": "yes thought so \nit makes me feel like this competition is more of a computational resources show off than anything else even the other competitions this summer are all CV with datasets size ranging from 30gb-300gb+
\nit makes me wanna give up on winning medals this summer
LOL. It's really not normal for an epoch to take 52 minutes. For me, it took about 10 minutes per epoch on P100 GPU with about 5-6 GBs of mem usage with apex.
", "votes": 0, "replies": [] }, { "id": 924335, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2020-07-11T11:32:33.593000", "content": "Depends on the model, image size I guess - but seems a bit on the long side... How are you loading your data?
", "votes": 0, "replies": [ { "id": 924642, "author_name": "mohaned abid", "author_url": "", "post_date": "2020-07-11T14:56:27.950000", "content": "from google drive but what I have noticed is that colab is giving me tesla k80 12gb while giving tesla p100 16gb for others I thought this might be the reason.I forgot to say its a yolov5 with original image size 1024
", "votes": 0, "replies": [] }, { "id": 926192, "author_name": "Kenneth Rithvik", "author_url": "", "post_date": "2020-07-12T15:10:53.833000", "content": "if you are mounting the data-set from google drive, I think you should transfer all the data into your local working folder on colab before using the data-loader for model training. Loading the data directly from drive will add quite a bit of over-head.
", "votes": 0, "replies": [] } ] }, { "id": 924339, "author_name": "", "author_url": "", "post_date": "2020-07-11T11:35:38.460000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "926276": "I am facing the same issue. It seems that colab assigns gpu randomly, and the training time, various from around 10 mins to 50 mins, is related to the machine that colab assigns.", "924870": "LOL. It's really not normal for an epoch to take 52 minutes. For me, it took about 10 minutes per epoch on P100 GPU with about 5-6 GBs of mem usage with apex.", "924335": "Depends on the model, image size I guess - but seems a bit on the long side... How are you loading your data?", "924257": "is it normal for an epoch to take 52 minutes in colab while activating gpu with apex?", "924339": "" } }