{
  "id": 117763,
  "title": "fit on GPU ends up with Process finished with exit code -1073741819 (0xC0000005)",
  "url": "/competitions/ashrae-energy-prediction/discussion/117763",
  "author_name": "",
  "post_date": "2019-11-17T18:33:17.039938700Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi all,\nwhen trying fit the NN model with this big train data (2,2 Gb) on my local machine, it ends up with out-of-memmory error <strong><em>Process finished with exit code -1073741819 (0xC0000005)</em></strong>\nPython print me some hint about using OOM code, so I have added in <code>comile.model</code> this parameter <br>\n <code>run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)</code> \nNope, still crashed when fitting\nUse Win10+PyCharm with 32Gb RAM and NVIDIA G970X with 4 Gb VRAM. It is a long side story how I combined CUDA and TF version, but now for datasets smaller than 1 Gb it works with CUDA 10.0 and TF 1.14.0 verquick and nice. Have anyone got the same error and know how to fix it? Upgrade to TF 2.0 and do some grim magic with CUDA?\nPS.Before getting 32 Gb RAM I used memReduction scripts posted on Kaggle, but it is only data loading process not fitting</p>",
  "messages": [
    {
      "id": "675200",
      "postDate": "11/17/2019 18:33:17",
      "content": "<p>Hi all,\nwhen trying fit the NN model with this big train data (2,2 Gb) on my local machine, it ends up with out-of-memmory error <strong><em>Process finished with exit code -1073741819 (0xC0000005)</em></strong>\nPython print me some hint about using OOM code, so I have added in <code>comile.model</code> this parameter <br>\n <code>run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)</code> \nNope, still crashed when fitting\nUse Win10+PyCharm with 32Gb RAM and NVIDIA G970X with 4 Gb VRAM. It is a long side story how I combined CUDA and TF version, but now for datasets smaller than 1 Gb it works with CUDA 10.0 and TF 1.14.0 verquick and nice. Have anyone got the same error and know how to fix it? Upgrade to TF 2.0 and do some grim magic with CUDA?\nPS.Before getting 32 Gb RAM I used memReduction scripts posted on Kaggle, but it is only data loading process not fitting</p>",
      "rawMarkdown": "Hi all,\nwhen trying fit the NN model with this big train data (2,2 Gb) on my local machine, it ends up with out-of-memmory error ***Process finished with exit code -1073741819 (0xC0000005)***\nPython print me some hint about using OOM code, so I have added in `comile.model` this parameter                                                   \n `run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)` \nNope, still crashed when fitting\nUse Win10+PyCharm with 32Gb RAM and NVIDIA G970X with 4 Gb VRAM. It is a long side story how I combined CUDA and TF version, but now for datasets smaller than 1 Gb it works with CUDA 10.0 and TF 1.14.0 verquick and nice. Have anyone got the same error and know how to fix it? Upgrade to TF 2.0 and do some grim magic with CUDA?\nPS.Before getting 32 Gb RAM I used memReduction scripts posted on Kaggle, but it is only data loading process not fitting",
      "votes": null
    },
    {
      "id": "679920",
      "postDate": "11/23/2019 16:57:41",
      "content": "<p>well, fix this problem \nSolution #1 - install TensorFlow 2.0.0 and my GPU stop working at model computing, LOL\nnice, but slow\nSolution #2 - befor GPU initialization put<a href=\"https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth\"> this</a> code for <code>limiting_gpu_memory_growth</code>, with downgrading to the tf 1.14 and disabled <code>run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)</code></p>",
      "rawMarkdown": "well, fix this problem \nSolution #1 - install TensorFlow 2.0.0 and my GPU stop working at model computing, LOL\nnice, but slow\nSolution #2 - befor GPU initialization put[ this](https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth) code for `limiting_gpu_memory_growth`, with downgrading to the tf 1.14 and disabled `run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)`",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 679920,
      "author_name": "stenkinvlad",
      "author_url": "",
      "post_date": "11/23/2019 16:57:41",
      "content": "<p>well, fix this problem \nSolution #1 - install TensorFlow 2.0.0 and my GPU stop working at model computing, LOL\nnice, but slow\nSolution #2 - befor GPU initialization put<a href=\"https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth\"> this</a> code for <code>limiting_gpu_memory_growth</code>, with downgrading to the tf 1.14 and disabled <code>run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)</code></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "675200": "Hi all,\nwhen trying fit the NN model with this big train data (2,2 Gb) on my local machine, it ends up with out-of-memmory error ***Process finished with exit code -1073741819 (0xC0000005)***\nPython print me some hint about using OOM code, so I have added in `comile.model` this parameter                                                   \n `run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)` \nNope, still crashed when fitting\nUse Win10+PyCharm with 32Gb RAM and NVIDIA G970X with 4 Gb VRAM. It is a long side story how I combined CUDA and TF version, but now for datasets smaller than 1 Gb it works with CUDA 10.0 and TF 1.14.0 verquick and nice. Have anyone got the same error and know how to fix it? Upgrade to TF 2.0 and do some grim magic with CUDA?\nPS.Before getting 32 Gb RAM I used memReduction scripts posted on Kaggle, but it is only data loading process not fitting",
    "679920": "well, fix this problem \nSolution #1 - install TensorFlow 2.0.0 and my GPU stop working at model computing, LOL\nnice, but slow\nSolution #2 - befor GPU initialization put[ this](https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth) code for `limiting_gpu_memory_growth`, with downgrading to the tf 1.14 and disabled `run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)`"
  },
  "source": "meta"
}