{
  "id": 98294,
  "title": "Runtime limit",
  "url": "/competitions/generative-dog-images/discussion/98294",
  "author_name": "",
  "post_date": "2019-07-02T17:26:23.210204900Z",
  "votes": null,
  "comment_count": 8,
  "views": 0,
  "content": "<p>My model stops executing after 1hour more or less and i only can run 36 epochs aprox, I saw that the kernel runtime max is 9h so i dont understand  this situation. Anybody has the samw problem?</p>",
  "messages": [
    {
      "id": "566864",
      "postDate": "07/02/2019 17:26:23",
      "content": "<p>My model stops executing after 1hour more or less and i only can run 36 epochs aprox, I saw that the kernel runtime max is 9h so i dont understand  this situation. Anybody has the samw problem?</p>",
      "rawMarkdown": "My model stops executing after 1hour more or less and i only can run 36 epochs aprox, I saw that the kernel runtime max is 9h so i dont understand  this situation. Anybody has the samw problem?",
      "votes": null
    },
    {
      "id": "566895",
      "postDate": "07/02/2019 18:40:59",
      "content": "<p>The biggest reason I've observed of kernels stopping is memory problems. I'd suggest that you make sure you are not exceeding CPU memory nor GPU memory.</p>",
      "rawMarkdown": "The biggest reason I've observed of kernels stopping is memory problems. I'd suggest that you make sure you are not exceeding CPU memory nor GPU memory.",
      "votes": null
    },
    {
      "id": "566939",
      "postDate": "07/02/2019 20:08:40",
      "content": "<p>I experienced the same when I tried larger minibatches (32 works, 64 doesn't work). I agree with Chris, try to lower your memory needs.</p>",
      "rawMarkdown": "I experienced the same when I tried larger minibatches (32 works, 64 doesn't work). I agree with Chris, try to lower your memory needs.",
      "votes": null
    },
    {
      "id": "567267",
      "postDate": "07/03/2019 08:56:56",
      "content": "<p>Hm, now i got a strange error:\n<img src=\"https://i.imgur.com/vbOGjv6.png\" alt=\"not running\"></p>\n\n<p>The model misteriously stopped after 9h with a runtime of 0 seconds. \nA rerun on the exact same model worked...</p>",
      "rawMarkdown": "Hm, now i got a strange error:\n![not running](https://i.imgur.com/vbOGjv6.png)\n\nThe model misteriously stopped after 9h with a runtime of 0 seconds. \nA rerun on the exact same model worked...",
      "votes": null
    },
    {
      "id": "567543",
      "postDate": "07/03/2019 16:58:05",
      "content": "<p>Okey thanks Chris i think my mistake is to load all the images in memory before training instead of only loading the batxh im processing</p>",
      "rawMarkdown": "Okey thanks Chris i think my mistake is to load all the images in memory before training instead of only loading the batxh im processing",
      "votes": null
    },
    {
      "id": "567550",
      "postDate": "07/03/2019 17:12:02",
      "content": "<p>Ok. Additionally, you can leave them on the hard drive and read them from disk with <code>ImageDataGenerator().flow_from_directory()</code> from <code>keras.preprocessing.image</code></p>",
      "rawMarkdown": "Ok. Additionally, you can leave them on the hard drive and read them from disk with `ImageDataGenerator().flow_from_directory()` from `keras.preprocessing.image`",
      "votes": null
    },
    {
      "id": "567586",
      "postDate": "07/03/2019 18:19:54",
      "content": "<p>Thanks! i will take in consideration</p>",
      "rawMarkdown": "Thanks! i will take in consideration",
      "votes": null
    },
    {
      "id": "567893",
      "postDate": "07/04/2019 06:19:04",
      "content": "<p>I implemented the data generator and im still struggling with the same problem, i can only run 30 epochs, it takes 50 mins so i dont understand it, i had run cell by cell the kernel while monitoring the resources and my RAM never gets over 3gb so i dont know what im doing bad :(</p>",
      "rawMarkdown": "I implemented the data generator and im still struggling with the same problem, i can only run 30 epochs, it takes 50 mins so i dont understand it, i had run cell by cell the kernel while monitoring the resources and my RAM never gets over 3gb so i dont know what im doing bad :(",
      "votes": null
    },
    {
      "id": "567898",
      "postDate": "07/04/2019 06:24:39",
      "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F936560%2F70fc7f0bc6c3a69ef4d02f5d5b5b80d6%2FCaptura.PNG?generation=1562221401018337&amp;alt=media\" alt=\"\"></p>\n\n<p>My CPU memory is high but this happened on many kernels, is it a problem? also im using package tqdm to iterate over each batch can this package be the reason? Many thanks</p>",
      "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F936560%2F70fc7f0bc6c3a69ef4d02f5d5b5b80d6%2FCaptura.PNG?generation=1562221401018337&amp;alt=media)\n\nMy CPU memory is high but this happened on many kernels, is it a problem? also im using package tqdm to iterate over each batch can this package be the reason? Many thanks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 566895,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "07/02/2019 18:40:59",
      "content": "<p>The biggest reason I've observed of kernels stopping is memory problems. I'd suggest that you make sure you are not exceeding CPU memory nor GPU memory.</p>",
      "votes": null,
      "replies": [
        {
          "id": 567543,
          "author_name": "enric1296",
          "author_url": "",
          "post_date": "07/03/2019 16:58:05",
          "content": "<p>Okey thanks Chris i think my mistake is to load all the images in memory before training instead of only loading the batxh im processing</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 567550,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "07/03/2019 17:12:02",
          "content": "<p>Ok. Additionally, you can leave them on the hard drive and read them from disk with <code>ImageDataGenerator().flow_from_directory()</code> from <code>keras.preprocessing.image</code></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 566939,
      "author_name": "ilu000",
      "author_url": "",
      "post_date": "07/02/2019 20:08:40",
      "content": "<p>I experienced the same when I tried larger minibatches (32 works, 64 doesn't work). I agree with Chris, try to lower your memory needs.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 567267,
      "author_name": "ilu000",
      "author_url": "",
      "post_date": "07/03/2019 08:56:56",
      "content": "<p>Hm, now i got a strange error:\n<img src=\"https://i.imgur.com/vbOGjv6.png\" alt=\"not running\"></p>\n\n<p>The model misteriously stopped after 9h with a runtime of 0 seconds. \nA rerun on the exact same model worked...</p>",
      "votes": null,
      "replies": [
        {
          "id": 567586,
          "author_name": "enric1296",
          "author_url": "",
          "post_date": "07/03/2019 18:19:54",
          "content": "<p>Thanks! i will take in consideration</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 567893,
      "author_name": "enric1296",
      "author_url": "",
      "post_date": "07/04/2019 06:19:04",
      "content": "<p>I implemented the data generator and im still struggling with the same problem, i can only run 30 epochs, it takes 50 mins so i dont understand it, i had run cell by cell the kernel while monitoring the resources and my RAM never gets over 3gb so i dont know what im doing bad :(</p>",
      "votes": null,
      "replies": [
        {
          "id": 567898,
          "author_name": "enric1296",
          "author_url": "",
          "post_date": "07/04/2019 06:24:39",
          "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F936560%2F70fc7f0bc6c3a69ef4d02f5d5b5b80d6%2FCaptura.PNG?generation=1562221401018337&amp;alt=media\" alt=\"\"></p>\n\n<p>My CPU memory is high but this happened on many kernels, is it a problem? also im using package tqdm to iterate over each batch can this package be the reason? Many thanks</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "566864": "My model stops executing after 1hour more or less and i only can run 36 epochs aprox, I saw that the kernel runtime max is 9h so i dont understand  this situation. Anybody has the samw problem?",
    "566895": "The biggest reason I've observed of kernels stopping is memory problems. I'd suggest that you make sure you are not exceeding CPU memory nor GPU memory.",
    "566939": "I experienced the same when I tried larger minibatches (32 works, 64 doesn't work). I agree with Chris, try to lower your memory needs.",
    "567267": "Hm, now i got a strange error:\n![not running](https://i.imgur.com/vbOGjv6.png)\n\nThe model misteriously stopped after 9h with a runtime of 0 seconds. \nA rerun on the exact same model worked...",
    "567543": "Okey thanks Chris i think my mistake is to load all the images in memory before training instead of only loading the batxh im processing",
    "567550": "Ok. Additionally, you can leave them on the hard drive and read them from disk with `ImageDataGenerator().flow_from_directory()` from `keras.preprocessing.image`",
    "567586": "Thanks! i will take in consideration",
    "567893": "I implemented the data generator and im still struggling with the same problem, i can only run 30 epochs, it takes 50 mins so i dont understand it, i had run cell by cell the kernel while monitoring the resources and my RAM never gets over 3gb so i dont know what im doing bad :(",
    "567898": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F936560%2F70fc7f0bc6c3a69ef4d02f5d5b5b80d6%2FCaptura.PNG?generation=1562221401018337&amp;alt=media)\n\nMy CPU memory is high but this happened on many kernels, is it a problem? also im using package tqdm to iterate over each batch can this package be the reason? Many thanks"
  },
  "source": "meta"
}