{ "id": 52983, "title": "Problem with training deepsense.ai U-Net", "url": "/competitions/data-science-bowl-2018/discussion/52983", "author_name": "", "post_date": "2018-03-26T01:43:33.158109300Z", "votes": null, "comment_count": 7, "views": 0, "content": "

I followed all the steps the deepsense.ai repository here to start training a U-Net with weighted loss and morphological postprocessing but after running the training script locally like this

\n\n

!python main.py -- train_evaluate_predict_pipeline --pipeline_name unet_multitask

\n\n

returns this

\n\n

\nneptune: Executing in Offline Mode.
\nneptune: Executing in Offline Mode.
\n2018-03-25 23-56-30 dsb-2018 >>> training
\nneptune: Executing in Offline Mode.
\n2018-03-25 23-56-30 steps >>> step reader_train adapting inputs
\n2018-03-25 23-56-30 steps >>> step reader_train fitting and transforming...
\n100%|████████████████████████████████████████| 540/540 [00:03<00:00, 146.20it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 789.82it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 753.27it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 903.94it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 863.26it/s]
\n2018-03-25 23-56-36 steps >>> step reader_train saving transformer...
\n2018-03-25 23-56-36 steps >>> step reader_train caching outputs...
\n2018-03-25 23-56-42 steps >>> step reader_train saving outputs...
\n2018-03-25 23-56-49 steps >>> step reader_inference adapting inputs
\n2018-03-25 23-56-49 steps >>> step reader_inference fitting and transforming...
\n100%|████████████████████████████████████████| 130/130 [00:00<00:00, 314.88it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1190.10it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1136.10it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1279.12it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1243.90it/s]
\n2018-03-25 23-56-50 steps >>> step reader_inference saving transformer...
\n2018-03-25 23-56-50 steps >>> step reader_inference caching outputs...
\n2018-03-25 23-56-51 steps >>> step reader_inference saving outputs...
\n2018-03-25 23-56-52 steps >>> step loader adapting inputs
\n2018-03-25 23-56-52 steps >>> step loader fitting and transforming...
\n2018-03-25 23-56-52 steps >>> step loader saving transformer...
\n2018-03-25 23-56-52 steps >>> step loader caching outputs...
\n2018-03-25 23-57-00 steps >>> step unet_multitask unpacking inputs
\n2018-03-25 23-57-00 steps >>> step unet_multitask fitting and transforming...
\n2018-03-25 23-57-00 steps >>> initializing model weights...
\n2018-03-25 23-57-03 steps >>> starting training...
\n2018-03-25 23-57-03 steps >>> initial lr: 0.0001
\n2018-03-25 23-57-03 steps >>> epoch 0 ...
\n

\n\n

in a Google Colaboratory notebook with a K80 gpu, the model never finishes training epoch 0 even after waiting for more than an hour.

\n\n

Has this happened to anyone else and do you know why this might be happening?

", "messages": [ { "id": "303316", "postDate": "03/26/2018 01:43:33", "content": "

I followed all the steps the deepsense.ai repository here to start training a U-Net with weighted loss and morphological postprocessing but after running the training script locally like this

\n\n

!python main.py -- train_evaluate_predict_pipeline --pipeline_name unet_multitask

\n\n

returns this

\n\n

\nneptune: Executing in Offline Mode.
\nneptune: Executing in Offline Mode.
\n2018-03-25 23-56-30 dsb-2018 >>> training
\nneptune: Executing in Offline Mode.
\n2018-03-25 23-56-30 steps >>> step reader_train adapting inputs
\n2018-03-25 23-56-30 steps >>> step reader_train fitting and transforming...
\n100%|████████████████████████████████████████| 540/540 [00:03<00:00, 146.20it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 789.82it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 753.27it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 903.94it/s]
\n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 863.26it/s]
\n2018-03-25 23-56-36 steps >>> step reader_train saving transformer...
\n2018-03-25 23-56-36 steps >>> step reader_train caching outputs...
\n2018-03-25 23-56-42 steps >>> step reader_train saving outputs...
\n2018-03-25 23-56-49 steps >>> step reader_inference adapting inputs
\n2018-03-25 23-56-49 steps >>> step reader_inference fitting and transforming...
\n100%|████████████████████████████████████████| 130/130 [00:00<00:00, 314.88it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1190.10it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1136.10it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1279.12it/s]
\n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1243.90it/s]
\n2018-03-25 23-56-50 steps >>> step reader_inference saving transformer...
\n2018-03-25 23-56-50 steps >>> step reader_inference caching outputs...
\n2018-03-25 23-56-51 steps >>> step reader_inference saving outputs...
\n2018-03-25 23-56-52 steps >>> step loader adapting inputs
\n2018-03-25 23-56-52 steps >>> step loader fitting and transforming...
\n2018-03-25 23-56-52 steps >>> step loader saving transformer...
\n2018-03-25 23-56-52 steps >>> step loader caching outputs...
\n2018-03-25 23-57-00 steps >>> step unet_multitask unpacking inputs
\n2018-03-25 23-57-00 steps >>> step unet_multitask fitting and transforming...
\n2018-03-25 23-57-00 steps >>> initializing model weights...
\n2018-03-25 23-57-03 steps >>> starting training...
\n2018-03-25 23-57-03 steps >>> initial lr: 0.0001
\n2018-03-25 23-57-03 steps >>> epoch 0 ...
\n

\n\n

in a Google Colaboratory notebook with a K80 gpu, the model never finishes training epoch 0 even after waiting for more than an hour.

\n\n

Has this happened to anyone else and do you know why this might be happening?

", "rawMarkdown": "I followed all the steps the deepsense.ai repository [here][1] to start training a U-Net with weighted loss and morphological postprocessing but after running the training script locally like this \n\n`!python main.py -- train_evaluate_predict_pipeline --pipeline_name unet_multitask`\n\nreturns this \n\n```\nneptune: Executing in Offline Mode. \nneptune: Executing in Offline Mode. \n2018-03-25 23-56-30 dsb-2018 >>> training \nneptune: Executing in Offline Mode. \n2018-03-25 23-56-30 steps >>> step reader_train adapting inputs \n2018-03-25 23-56-30 steps >>> step reader_train fitting and transforming... \n100%|████████████████████████████████████████| 540/540 [00:03<00:00, 146.20it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 789.82it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 753.27it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 903.94it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 863.26it/s] \n2018-03-25 23-56-36 steps >>> step reader_train saving transformer... \n2018-03-25 23-56-36 steps >>> step reader_train caching outputs... \n2018-03-25 23-56-42 steps >>> step reader_train saving outputs... \n2018-03-25 23-56-49 steps >>> step reader_inference adapting inputs \n2018-03-25 23-56-49 steps >>> step reader_inference fitting and transforming... \n100%|████████████████████████████████████████| 130/130 [00:00<00:00, 314.88it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1190.10it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1136.10it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1279.12it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1243.90it/s] \n2018-03-25 23-56-50 steps >>> step reader_inference saving transformer... \n2018-03-25 23-56-50 steps >>> step reader_inference caching outputs... \n2018-03-25 23-56-51 steps >>> step reader_inference saving outputs... \n2018-03-25 23-56-52 steps >>> step loader adapting inputs \n2018-03-25 23-56-52 steps >>> step loader fitting and transforming... \n2018-03-25 23-56-52 steps >>> step loader saving transformer... \n2018-03-25 23-56-52 steps >>> step loader caching outputs... \n2018-03-25 23-57-00 steps >>> step unet_multitask unpacking inputs \n2018-03-25 23-57-00 steps >>> step unet_multitask fitting and transforming... \n2018-03-25 23-57-00 steps >>> initializing model weights... \n2018-03-25 23-57-03 steps >>> starting training... \n2018-03-25 23-57-03 steps >>> initial lr: 0.0001 \n2018-03-25 23-57-03 steps >>> epoch 0 ... \n```\n\nin a Google Colaboratory notebook with a K80 gpu, the model never finishes training epoch 0 even after waiting for more than an hour.\n\nHas this happened to anyone else and do you know why this might be happening?\n\n [1]: https://github.com/neptune-ml/data-science-bowl-2018", "votes": null }, { "id": "303765", "postDate": "03/26/2018 16:26:57", "content": "

Try smaller height and width like 256.

", "rawMarkdown": "Try smaller height and width like 256.", "votes": null }, { "id": "304166", "postDate": "03/27/2018 05:55:06", "content": "

Hi bkKaggle,

\n\n

Do you have access to the GPU? What is you batch size and images size? These parameters may contribute seriously to the single epoch time. If there is no error, I assume that single epoch takes a lot of time.

\n\n

For the Google Colaboratory case: how did you upload the data? Is you Internet connection stable? In case you lost connection to the Google Colaboratory, you may have to restart an experiment.

\n\n

Best,

\n\n

Kamil

", "rawMarkdown": "Hi bkKaggle,\n\nDo you have access to the GPU? What is you batch size and images size? These parameters may contribute seriously to the single epoch time. If there is no error, I assume that single epoch takes a lot of time.\n\nFor the Google Colaboratory case: how did you upload the data? Is you Internet connection stable? In case you lost connection to the Google Colaboratory, you may have to restart an experiment.\n\nBest,\n\nKamil", "votes": null }, { "id": "304168", "postDate": "03/27/2018 05:55:22", "content": "

Yes, this may help ;-)

", "rawMarkdown": "Yes, this may help ;-)", "votes": null }, { "id": "304580", "postDate": "03/27/2018 19:06:35", "content": "

Hi,

\n\n

Reducing the image size to 256x256 and using a batch size of 1 with a gpu solved the problem of the model never being able to finish training epoch 0 . However, after the model finishes training, and starts the prediction part of the pipeline, it gives the following error.

\n\n

\n2018-03-27 18-46-54 steps >>> step contour_resize saving transformer...
\n2018-03-27 18-46-54 steps >>> step contour_resize caching outputs...
\n2018-03-27 18-46-58 steps >>> step contour_resize saving outputs...
\n2018-03-27 18-47-03 steps >>> step detached adapting inputs
\n2018-03-27 18-47-03 steps >>> step detached fitting and transforming...
\n0it [00:00, ?it/s]tcmalloc: large alloc 23176101888 bytes == 0x7f6130686000 @ 0x7f6707786107 0x7f66a9e924be 0x7f66a9e8c671 0x7f66a9e7f3ea 0x4c4a3d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6\nKilled\n

\n\n

Does anyone know why this is happening and how to fix it?

", "rawMarkdown": "Hi,\n\nReducing the image size to 256x256 and using a batch size of 1 with a gpu solved the problem of the model never being able to finish training epoch 0 . However, after the model finishes training, and starts the prediction part of the pipeline, it gives the following error. \n\n```\n2018-03-27 18-46-54 steps >>> step contour_resize saving transformer... \n2018-03-27 18-46-54 steps >>> step contour_resize caching outputs... \n2018-03-27 18-46-58 steps >>> step contour_resize saving outputs... \n2018-03-27 18-47-03 steps >>> step detached adapting inputs \n2018-03-27 18-47-03 steps >>> step detached fitting and transforming... \n0it [00:00, ?it/s]tcmalloc: large alloc 23176101888 bytes == 0x7f6130686000 @ 0x7f6707786107 0x7f66a9e924be 0x7f66a9e8c671 0x7f66a9e7f3ea 0x4c4a3d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6\nKilled\n```\n\nDoes anyone know why this is happening and how to fix it?", "votes": null }, { "id": "304998", "postDate": "03/28/2018 10:39:34", "content": "

Hi, that's good -> we are moving forward :-)

\n\n

Give me more details about your experiment configuration: machine, gpu, memory, etc.

\n\n

In general malloc function is responsible for memory allocation, whereas TCMalloc is a fast implementation of it. You process has been killed due to large memory allocation: 23176101888 bytes in your case.

\n\n

Quick fix:

\n\n
    \n
  1. decrease img size OR decrease batch_size_inference in neptune.yaml:L65.
    2. \n
  2. re-run your experiment with overwrite: 0 in the neptune.yaml:L44.
    3. \n
\n\n

Best,

\n\n

Kamil

", "rawMarkdown": "Hi, that's good -> we are moving forward :-)\n\nGive me more details about your experiment configuration: machine, gpu, memory, etc.\n\nIn general `malloc` function is responsible for memory allocation, whereas `TCMalloc` is a fast implementation of it. You process has been killed due to large memory allocation: `23176101888 bytes` in your case.\n\nQuick fix:\n\n1. decrease img size OR decrease `batch_size_inference` in [neptune.yaml:L65](https://github.com/neptune-ml/data-science-bowl-2018/blob/master/neptune.yaml#L65).\n2. re-run your experiment with `overwrite: 0` in the [neptune.yaml:L44](https://github.com/neptune-ml/data-science-bowl-2018/blob/master/neptune.yaml#L44).\n\nBest,\n\nKamil", "votes": null }, { "id": "305925", "postDate": "03/29/2018 16:23:40", "content": "

Hi,

\n\n

Running my code on a machine with 52 gb of ram fixes the problem of large memory allocation. However, when the model reaches

\n\n

2018-03-29 15-19-40 steps >>> step detached fitting and transforming..., it returns a MemoryError pointing to

\n\n

\nFile \"/home/[USER]/data-science-bowl-2018/postprocessing.py\", line 214, in
\n m_padded = morph.binary_closing(m_padded, selem=struct_el)\n

\n\n

It looks to me that postprocessing the masks uses up all of my RAM. Is this true? Is it possible to postprocess the masks without the model returning these errors?

", "rawMarkdown": "Hi, \n\nRunning my code on a machine with 52 gb of ram fixes the problem of large memory allocation. However, when the model reaches \n\n```2018-03-29 15-19-40 steps >>> step detached fitting and transforming...```, it returns a ```MemoryError``` pointing to \n\n```\nFile \"/home/[USER]/data-science-bowl-2018/postprocessing.py\", line 214, in \n m_padded = morph.binary_closing(m_padded, selem=struct_el)\n```\n\nIt looks to me that postprocessing the masks uses up all of my RAM. Is this true? Is it possible to postprocess the masks without the model returning these errors?", "votes": null }, { "id": "308456", "postDate": "04/03/2018 14:39:03", "content": "

Hi @bkKaggle,

\n\n
    \n
  1. Please upload entire error message.
    2. \n
  2. Let me know about your machine and setup: code branch, running command, CPU, GPU, memory.
    3. \n
\n\n

Recent postprocessing does not use a lot of memory. New routines introduced in the dev-patching branch are more memory expensive.

\n\n

Do you have other memory-expensive jobs running on this machine?

\n\n

Best,

\n\n

Kamil

", "rawMarkdown": "Hi @bkKaggle,\n\n1. Please upload entire error message.\n2. Let me know about your machine and setup: code branch, running command, CPU, GPU, memory.\n\nRecent postprocessing does not use a lot of memory. New routines introduced in the `dev-patching` branch are more memory expensive.\n\nDo you have other memory-expensive jobs running on this machine?\n\nBest,\n\nKamil", "votes": null } ], "comments": [ { "id": 303765, "author_name": "shujian", "author_url": "", "post_date": "03/26/2018 16:26:57", "content": "

Try smaller height and width like 256.

", "votes": null, "replies": [ { "id": 304168, "author_name": "kkaczmarek", "author_url": "", "post_date": "03/27/2018 05:55:22", "content": "

Yes, this may help ;-)

", "votes": null, "replies": [] } ] }, { "id": 304166, "author_name": "kkaczmarek", "author_url": "", "post_date": "03/27/2018 05:55:06", "content": "

Hi bkKaggle,

\n\n

Do you have access to the GPU? What is you batch size and images size? These parameters may contribute seriously to the single epoch time. If there is no error, I assume that single epoch takes a lot of time.

\n\n

For the Google Colaboratory case: how did you upload the data? Is you Internet connection stable? In case you lost connection to the Google Colaboratory, you may have to restart an experiment.

\n\n

Best,

\n\n

Kamil

", "votes": null, "replies": [ { "id": 304580, "author_name": "bkkaggle", "author_url": "", "post_date": "03/27/2018 19:06:35", "content": "

Hi,

\n\n

Reducing the image size to 256x256 and using a batch size of 1 with a gpu solved the problem of the model never being able to finish training epoch 0 . However, after the model finishes training, and starts the prediction part of the pipeline, it gives the following error.

\n\n

\n2018-03-27 18-46-54 steps >>> step contour_resize saving transformer...
\n2018-03-27 18-46-54 steps >>> step contour_resize caching outputs...
\n2018-03-27 18-46-58 steps >>> step contour_resize saving outputs...
\n2018-03-27 18-47-03 steps >>> step detached adapting inputs
\n2018-03-27 18-47-03 steps >>> step detached fitting and transforming...
\n0it [00:00, ?it/s]tcmalloc: large alloc 23176101888 bytes == 0x7f6130686000 @ 0x7f6707786107 0x7f66a9e924be 0x7f66a9e8c671 0x7f66a9e7f3ea 0x4c4a3d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6\nKilled\n

\n\n

Does anyone know why this is happening and how to fix it?

", "votes": null, "replies": [] }, { "id": 304998, "author_name": "kkaczmarek", "author_url": "", "post_date": "03/28/2018 10:39:34", "content": "

Hi, that's good -> we are moving forward :-)

\n\n

Give me more details about your experiment configuration: machine, gpu, memory, etc.

\n\n

In general malloc function is responsible for memory allocation, whereas TCMalloc is a fast implementation of it. You process has been killed due to large memory allocation: 23176101888 bytes in your case.

\n\n

Quick fix:

\n\n
    \n
  1. decrease img size OR decrease batch_size_inference in neptune.yaml:L65.
    2. \n
  2. re-run your experiment with overwrite: 0 in the neptune.yaml:L44.
    3. \n
\n\n

Best,

\n\n

Kamil

", "votes": null, "replies": [] }, { "id": 305925, "author_name": "bkkaggle", "author_url": "", "post_date": "03/29/2018 16:23:40", "content": "

Hi,

\n\n

Running my code on a machine with 52 gb of ram fixes the problem of large memory allocation. However, when the model reaches

\n\n

2018-03-29 15-19-40 steps >>> step detached fitting and transforming..., it returns a MemoryError pointing to

\n\n

\nFile \"/home/[USER]/data-science-bowl-2018/postprocessing.py\", line 214, in
\n m_padded = morph.binary_closing(m_padded, selem=struct_el)\n

\n\n

It looks to me that postprocessing the masks uses up all of my RAM. Is this true? Is it possible to postprocess the masks without the model returning these errors?

", "votes": null, "replies": [] }, { "id": 308456, "author_name": "kkaczmarek", "author_url": "", "post_date": "04/03/2018 14:39:03", "content": "

Hi @bkKaggle,

\n\n
    \n
  1. Please upload entire error message.
    2. \n
  2. Let me know about your machine and setup: code branch, running command, CPU, GPU, memory.
    3. \n
\n\n

Recent postprocessing does not use a lot of memory. New routines introduced in the dev-patching branch are more memory expensive.

\n\n

Do you have other memory-expensive jobs running on this machine?

\n\n

Best,

\n\n

Kamil

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "303316": "I followed all the steps the deepsense.ai repository [here][1] to start training a U-Net with weighted loss and morphological postprocessing but after running the training script locally like this \n\n`!python main.py -- train_evaluate_predict_pipeline --pipeline_name unet_multitask`\n\nreturns this \n\n```\nneptune: Executing in Offline Mode. \nneptune: Executing in Offline Mode. \n2018-03-25 23-56-30 dsb-2018 >>> training \nneptune: Executing in Offline Mode. \n2018-03-25 23-56-30 steps >>> step reader_train adapting inputs \n2018-03-25 23-56-30 steps >>> step reader_train fitting and transforming... \n100%|████████████████████████████████████████| 540/540 [00:03<00:00, 146.20it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 789.82it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 753.27it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 903.94it/s] \n100%|████████████████████████████████████████| 540/540 [00:00<00:00, 863.26it/s] \n2018-03-25 23-56-36 steps >>> step reader_train saving transformer... \n2018-03-25 23-56-36 steps >>> step reader_train caching outputs... \n2018-03-25 23-56-42 steps >>> step reader_train saving outputs... \n2018-03-25 23-56-49 steps >>> step reader_inference adapting inputs \n2018-03-25 23-56-49 steps >>> step reader_inference fitting and transforming... \n100%|████████████████████████████████████████| 130/130 [00:00<00:00, 314.88it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1190.10it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1136.10it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1279.12it/s] \n100%|███████████████████████████████████████| 130/130 [00:00<00:00, 1243.90it/s] \n2018-03-25 23-56-50 steps >>> step reader_inference saving transformer... \n2018-03-25 23-56-50 steps >>> step reader_inference caching outputs... \n2018-03-25 23-56-51 steps >>> step reader_inference saving outputs... \n2018-03-25 23-56-52 steps >>> step loader adapting inputs \n2018-03-25 23-56-52 steps >>> step loader fitting and transforming... \n2018-03-25 23-56-52 steps >>> step loader saving transformer... \n2018-03-25 23-56-52 steps >>> step loader caching outputs... \n2018-03-25 23-57-00 steps >>> step unet_multitask unpacking inputs \n2018-03-25 23-57-00 steps >>> step unet_multitask fitting and transforming... \n2018-03-25 23-57-00 steps >>> initializing model weights... \n2018-03-25 23-57-03 steps >>> starting training... \n2018-03-25 23-57-03 steps >>> initial lr: 0.0001 \n2018-03-25 23-57-03 steps >>> epoch 0 ... \n```\n\nin a Google Colaboratory notebook with a K80 gpu, the model never finishes training epoch 0 even after waiting for more than an hour.\n\nHas this happened to anyone else and do you know why this might be happening?\n\n [1]: https://github.com/neptune-ml/data-science-bowl-2018", "303765": "Try smaller height and width like 256.", "304166": "Hi bkKaggle,\n\nDo you have access to the GPU? What is you batch size and images size? These parameters may contribute seriously to the single epoch time. If there is no error, I assume that single epoch takes a lot of time.\n\nFor the Google Colaboratory case: how did you upload the data? Is you Internet connection stable? In case you lost connection to the Google Colaboratory, you may have to restart an experiment.\n\nBest,\n\nKamil", "304168": "Yes, this may help ;-)", "304580": "Hi,\n\nReducing the image size to 256x256 and using a batch size of 1 with a gpu solved the problem of the model never being able to finish training epoch 0 . However, after the model finishes training, and starts the prediction part of the pipeline, it gives the following error. \n\n```\n2018-03-27 18-46-54 steps >>> step contour_resize saving transformer... \n2018-03-27 18-46-54 steps >>> step contour_resize caching outputs... \n2018-03-27 18-46-58 steps >>> step contour_resize saving outputs... \n2018-03-27 18-47-03 steps >>> step detached adapting inputs \n2018-03-27 18-47-03 steps >>> step detached fitting and transforming... \n0it [00:00, ?it/s]tcmalloc: large alloc 23176101888 bytes == 0x7f6130686000 @ 0x7f6707786107 0x7f66a9e924be 0x7f66a9e8c671 0x7f66a9e7f3ea 0x4c4a3d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54ffee 0x48b86d 0x45969e 0x552029 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6\nKilled\n```\n\nDoes anyone know why this is happening and how to fix it?", "304998": "Hi, that's good -> we are moving forward :-)\n\nGive me more details about your experiment configuration: machine, gpu, memory, etc.\n\nIn general `malloc` function is responsible for memory allocation, whereas `TCMalloc` is a fast implementation of it. You process has been killed due to large memory allocation: `23176101888 bytes` in your case.\n\nQuick fix:\n\n1. decrease img size OR decrease `batch_size_inference` in [neptune.yaml:L65](https://github.com/neptune-ml/data-science-bowl-2018/blob/master/neptune.yaml#L65).\n2. re-run your experiment with `overwrite: 0` in the [neptune.yaml:L44](https://github.com/neptune-ml/data-science-bowl-2018/blob/master/neptune.yaml#L44).\n\nBest,\n\nKamil", "305925": "Hi, \n\nRunning my code on a machine with 52 gb of ram fixes the problem of large memory allocation. However, when the model reaches \n\n```2018-03-29 15-19-40 steps >>> step detached fitting and transforming...```, it returns a ```MemoryError``` pointing to \n\n```\nFile \"/home/[USER]/data-science-bowl-2018/postprocessing.py\", line 214, in \n m_padded = morph.binary_closing(m_padded, selem=struct_el)\n```\n\nIt looks to me that postprocessing the masks uses up all of my RAM. Is this true? Is it possible to postprocess the masks without the model returning these errors?", "308456": "Hi @bkKaggle,\n\n1. Please upload entire error message.\n2. Let me know about your machine and setup: code branch, running command, CPU, GPU, memory.\n\nRecent postprocessing does not use a lot of memory. New routines introduced in the `dev-patching` branch are more memory expensive.\n\nDo you have other memory-expensive jobs running on this machine?\n\nBest,\n\nKamil" }, "source": "meta" }