{
  "id": 105893,
  "title": "GPU kernel failed..",
  "url": "/competitions/champs-scalar-coupling/discussion/105893",
  "author_name": "",
  "post_date": "2019-08-27T03:27:06.472612300Z",
  "votes": 14,
  "comment_count": 24,
  "views": 0,
  "content": "<p>From yesterday, I cannot succeed commit with this error... \nAnyone is happened same thing ??</p>\n\n<p>very hard in the last 2days...</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1305588%2F545a3394c763cd885476b7a45d93cced%2F2019-08-27%2010.14.56.png?generation=1566876116391964&amp;alt=media\" alt=\"ErrorScreenShot\"></p>",
  "messages": [
    {
      "id": "608637",
      "postDate": "08/27/2019 03:27:06",
      "content": "<p>From yesterday, I cannot succeed commit with this error... \nAnyone is happened same thing ??</p>\n\n<p>very hard in the last 2days...</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1305588%2F545a3394c763cd885476b7a45d93cced%2F2019-08-27%2010.14.56.png?generation=1566876116391964&amp;alt=media\" alt=\"ErrorScreenShot\"></p>",
      "rawMarkdown": "From yesterday, I cannot succeed commit with this error... \nAnyone is happened same thing ??\n\nvery hard in the last 2days...\n\n![ErrorScreenShot](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1305588%2F545a3394c763cd885476b7a45d93cced%2F2019-08-27%2010.14.56.png?generation=1566876116391964&amp;alt=media)",
      "votes": null
    },
    {
      "id": "608740",
      "postDate": "08/27/2019 06:01:08",
      "content": "<p>I had errors too :(</p>",
      "rawMarkdown": "I had errors too :(",
      "votes": null
    },
    {
      "id": "608781",
      "postDate": "08/27/2019 07:03:09",
      "content": "<p>Have you seen this discussion? I hope it works.</p>\n\n<p><a href=\"https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105889#608660\">https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105889#608660</a></p>\n\n<blockquote>\n  <p>But after 5 unsuccessful attempts to commit my kernel, I changed docker version from latest to (versions 1 to 33) and it worked.</p>\n</blockquote>",
      "rawMarkdown": "Have you seen this discussion? I hope it works.\n\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105889#608660\n\n&gt; But after 5 unsuccessful attempts to commit my kernel, I changed docker version from latest to (versions 1 to 33) and it worked.",
      "votes": null
    },
    {
      "id": "608788",
      "postDate": "08/27/2019 07:05:45",
      "content": "<p>Same. Yesterday during the day everything was fine. During the night I had this very issues - no log, runtime 0, commit unsuccesful (1 GPU running at a time, so new rules about GPU is not the reason).</p>",
      "rawMarkdown": "Same. Yesterday during the day everything was fine. During the night I had this very issues - no log, runtime 0, commit unsuccesful (1 GPU running at a time, so new rules about GPU is not the reason).",
      "votes": null
    },
    {
      "id": "608802",
      "postDate": "08/27/2019 07:16:21",
      "content": "<p>I had encountered the same issue too. Bad timing on GPU issues as many competitors try to improve their model/s using this tool.</p>",
      "rawMarkdown": "I had encountered the same issue too. Bad timing on GPU issues as many competitors try to improve their model/s using this tool.",
      "votes": null
    },
    {
      "id": "608810",
      "postDate": "08/27/2019 07:23:15",
      "content": "<p>same here. My kernel ran all the way through but it doesn't produce the notebook at the end. So annoying after waiting 5 hours</p>",
      "rawMarkdown": "same here. My kernel ran all the way through but it doesn't produce the notebook at the end. So annoying after waiting 5 hours",
      "votes": null
    },
    {
      "id": "608813",
      "postDate": "08/27/2019 07:26:53",
      "content": "<p>Seems the same here... ran for 8-9 hours and got nothing😭 </p>",
      "rawMarkdown": "Seems the same here... ran for 8-9 hours and got nothing😭",
      "votes": null
    },
    {
      "id": "608886",
      "postDate": "08/27/2019 08:47:40",
      "content": "<p>Same here, I've been having problems for the past couple of days. Errors with no logs available and my output is missing.</p>",
      "rawMarkdown": "Same here, I've been having problems for the past couple of days. Errors with no logs available and my output is missing.",
      "votes": null
    },
    {
      "id": "608898",
      "postDate": "08/27/2019 08:53:15",
      "content": "<p>Thank you for good information!! \nDo you know how to change docker version? (I cannot change it..)</p>",
      "rawMarkdown": "Thank you for good information!! \nDo you know how to change docker version? (I cannot change it..)",
      "votes": null
    },
    {
      "id": "609008",
      "postDate": "08/27/2019 10:32:01",
      "content": "<p>total fail in the last 24 hrs for my GPU kernels. the timing is very unfortunate.\nthe docker image version can be changed to the previous version in the settings on the right of the interactive session. i saw this option earlier today but not now.</p>\n\n<p>earlier today I could run the interactive session but this too fails now.</p>",
      "rawMarkdown": "total fail in the last 24 hrs for my GPU kernels. the timing is very unfortunate.\nthe docker image version can be changed to the previous version in the settings on the right of the interactive session. i saw this option earlier today but not now.\n\nearlier today I could run the interactive session but this too fails now.",
      "votes": null
    },
    {
      "id": "609050",
      "postDate": "08/27/2019 11:27:49",
      "content": "<p>Same here. I'm running the same GPU commit code that worked a few days ago (versions 1 and 2 below). But last night both GPU commits failed. They both show a runtime of 0 seconds and a red error circle. Both use a timer and should run for 8.5 hours and then stop execution. Also my teammate submitted two and both of his failed.</p>\n\n<p><img src=\"http://playagricola.com/Kaggle/error82719.png\" alt=\"image\"></p>",
      "rawMarkdown": "Same here. I'm running the same GPU commit code that worked a few days ago (versions 1 and 2 below). But last night both GPU commits failed. They both show a runtime of 0 seconds and a red error circle. Both use a timer and should run for 8.5 hours and then stop execution. Also my teammate submitted two and both of his failed.\n  \n![image](http://playagricola.com/Kaggle/error82719.png)",
      "votes": null
    },
    {
      "id": "609062",
      "postDate": "08/27/2019 11:43:56",
      "content": "<p>I can change docker image by touching 'Setting' and 'Docker' few times and it worked</p>",
      "rawMarkdown": "I can change docker image by touching 'Setting' and 'Docker' few times and it worked",
      "votes": null
    },
    {
      "id": "609067",
      "postDate": "08/27/2019 11:47:45",
      "content": "<p>I'm seeing failures too, but only in saving the output files...</p>\n\n<p>I push two GPU model training kernels at the same time. I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine. The call home to say 'complete' also happens but then the kernel status (<code>kaggle kernels status</code> command) is \"error\". When I fetch the output, (<code>kaggle kernels output</code> command) the normal console log file is available and downloaded (with correct date/time), but none of the output files...</p>",
      "rawMarkdown": "I'm seeing failures too, but only in saving the output files...\n\nI push two GPU model training kernels at the same time. I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine. The call home to say 'complete' also happens but then the kernel status (`kaggle kernels status` command) is \"error\". When I fetch the output, (`kaggle kernels output` command) the normal console log file is available and downloaded (with correct date/time), but none of the output files...",
      "votes": null
    },
    {
      "id": "609118",
      "postDate": "08/27/2019 12:28:13",
      "content": "<p>They might update some software for CUDA. Whan I update CUDA or CUDNN, I'm sometimes forced to reinstall Python packages.</p>",
      "rawMarkdown": "They might update some software for CUDA. Whan I update CUDA or CUDNN, I'm sometimes forced to reinstall Python packages.",
      "votes": null
    },
    {
      "id": "609240",
      "postDate": "08/27/2019 14:22:28",
      "content": "<p>Thanks for reporting this everyone, we found and fixed this issue. Let us know if you experience any after effects.</p>",
      "rawMarkdown": "Thanks for reporting this everyone, we found and fixed this issue. Let us know if you experience any after effects.",
      "votes": null
    },
    {
      "id": "609246",
      "postDate": "08/27/2019 14:25:14",
      "content": "<p>Other bugs: Sometimes after a few commits my notebook contents delete itself, only the first block remains, and I have to fork the last version to continue editing.</p>\n\n<p>Also sometimes uncommitted kernels show and sometimes don't?</p>",
      "rawMarkdown": "Other bugs: Sometimes after a few commits my notebook contents delete itself, only the first block remains, and I have to fork the last version to continue editing.\n\nAlso sometimes uncommitted kernels show and sometimes don't?",
      "votes": null
    },
    {
      "id": "609394",
      "postDate": "08/27/2019 16:56:17",
      "content": "<p>i have not seen notebooks self-deleting their contents. have you tried re-fleshing the page?</p>",
      "rawMarkdown": "i have not seen notebooks self-deleting their contents. have you tried re-fleshing the page?",
      "votes": null
    },
    {
      "id": "609401",
      "postDate": "08/27/2019 17:05:08",
      "content": "<p>\"I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine.\"</p>\n\n<p>This is a great idea.  I should really code up something like this... </p>",
      "rawMarkdown": "\"I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine.\"\n\nThis is a great idea.  I should really code up something like this...",
      "votes": null
    },
    {
      "id": "609481",
      "postDate": "08/27/2019 19:12:58",
      "content": "<p>Yes I have, has happened to me more than once. Checked on multiple PCs also.</p>",
      "rawMarkdown": "Yes I have, has happened to me more than once. Checked on multiple PCs also.",
      "votes": null
    },
    {
      "id": "609489",
      "postDate": "08/27/2019 19:26:21",
      "content": "<p>I experienced this self-deleting bug, too. I tried to refresh it, close and reopen the notebook, but nothing helped, I had to fork my previous commit. Kind of annoying if you already made some changes to the notebook and your work is lost.</p>",
      "rawMarkdown": "I experienced this self-deleting bug, too. I tried to refresh it, close and reopen the notebook, but nothing helped, I had to fork my previous commit. Kind of annoying if you already made some changes to the notebook and your work is lost.",
      "votes": null
    },
    {
      "id": "609536",
      "postDate": "08/27/2019 20:28:05",
      "content": "<p>Thanks <a href=\"/abendorf\">@abendorf</a> , it's really made a big difference to kernels usability for me...</p>\n\n<p>With Python a really quick implementation would be:</p>\n\n<pre><code>import requests\n\ndef report_http(d):\n    ip, port = '1.2.3.4', 8899  # your ip, port\n    try:\n        p = '&amp;'.join([f'{k}={v}' for k,v in d.items()])\n        r = requests.get(f'http://{ip}:{port}/rep?{p}')\n    except:\n        pass\n</code></pre>\n\n<p>Then in a Keras callback:</p>\n\n<pre><code>class MyKerasCallback(tf.keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs={}):\n        d = dict(fold=FOLD, epoch=epoch)\n        d.update(logs)\n        report_http(d)\n</code></pre>\n\n<p>You just need to ensure the port is forwarded to some machine in your local network. For a simple (and perhaps insecure) server you can create an empty directory, then start a http server (this is for Python 2, on a Raspberry PI for me):</p>\n\n<pre><code>nohup python -m SimpleHTTPServer 8899 &amp;  \n</code></pre>\n\n<p>then watch the log file: </p>\n\n<pre><code>tail -f nohup.out\n</code></pre>\n\n<p>You don't even have to implement the '/rep' URL, you can just look at the 404 log messages ;)</p>\n\n<p><code>[27/Aug/2019 14:11:19] \"GET /rep?fold=2&amp;epoch=89&amp;mae=0.04454391077160835&amp;val_mae=0.05135239288210869&amp;lr=0.00010040000051958486 HTTP/1.1\" 404 -</code></p>",
      "rawMarkdown": "Thanks @abendorf , it's really made a big difference to kernels usability for me...\n\nWith Python a really quick implementation would be:\n\n    import requests\n\n    def report_http(d):\n        ip, port = '1.2.3.4', 8899  # your ip, port\n        try:\n            p = '&amp;'.join([f'{k}={v}' for k,v in d.items()])\n            r = requests.get(f'http://{ip}:{port}/rep?{p}')\n        except:\n            pass\n\n\nThen in a Keras callback:\n\n    class MyKerasCallback(tf.keras.callbacks.Callback):\n        def on_epoch_end(self, epoch, logs={}):\n            d = dict(fold=FOLD, epoch=epoch)\n            d.update(logs)\n            report_http(d)\n\n\nYou just need to ensure the port is forwarded to some machine in your local network. For a simple (and perhaps insecure) server you can create an empty directory, then start a http server (this is for Python 2, on a Raspberry PI for me):\n\n    nohup python -m SimpleHTTPServer 8899 &amp;  \n\nthen watch the log file: \n\n    tail -f nohup.out\n\nYou don't even have to implement the '/rep' URL, you can just look at the 404 log messages ;)\n\n`[27/Aug/2019 14:11:19] \"GET /rep?fold=2&amp;epoch=89&amp;mae=0.04454391077160835&amp;val_mae=0.05135239288210869&amp;lr=0.00010040000051958486 HTTP/1.1\" 404 -`",
      "votes": null
    },
    {
      "id": "609540",
      "postDate": "08/27/2019 20:39:57",
      "content": "<p>Fantastic!  Thanks a bunch for the code!  I'll give this a try.</p>",
      "rawMarkdown": "Fantastic!  Thanks a bunch for the code!  I'll give this a try.",
      "votes": null
    },
    {
      "id": "609551",
      "postDate": "08/27/2019 20:58:18",
      "content": "<p><a href=\"/julianmukaj\">@julianmukaj</a> have u tried different browsers as well? I experienced it a few times in the past with Google Chrome but when opened the notebook in Mozilla it worked. </p>",
      "rawMarkdown": "julianmukaj have u tried different browsers as well? I experienced it a few times in the past with Google Chrome but when opened the notebook in Mozilla it worked.",
      "votes": null
    },
    {
      "id": "609840",
      "postDate": "08/28/2019 06:45:05",
      "content": "<p>Thank you for information\nIn my kernel docker button , there are only latest choice. (I don't know why)\nBut this issue is solved as Dustin said! </p>",
      "rawMarkdown": "Thank you for information\nIn my kernel docker button , there are only latest choice. (I don't know why)\nBut this issue is solved as Dustin said!",
      "votes": null
    },
    {
      "id": "609977",
      "postDate": "08/28/2019 09:32:25",
      "content": "<p><a href=\"/mhviraf\">@mhviraf</a> Yes, just re-tried in Microsoft Edge also, contents show as deleted still.</p>",
      "rawMarkdown": "mhviraf Yes, just re-tried in Microsoft Edge also, contents show as deleted still.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 608740,
      "author_name": "dmintry",
      "author_url": "",
      "post_date": "08/27/2019 06:01:08",
      "content": "<p>I had errors too :(</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 608781,
      "author_name": "pridegoodmusic",
      "author_url": "",
      "post_date": "08/27/2019 07:03:09",
      "content": "<p>Have you seen this discussion? I hope it works.</p>\n\n<p><a href=\"https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105889#608660\">https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105889#608660</a></p>\n\n<blockquote>\n  <p>But after 5 unsuccessful attempts to commit my kernel, I changed docker version from latest to (versions 1 to 33) and it worked.</p>\n</blockquote>",
      "votes": null,
      "replies": [
        {
          "id": 608898,
          "author_name": "go5kuramubon",
          "author_url": "",
          "post_date": "08/27/2019 08:53:15",
          "content": "<p>Thank you for good information!! \nDo you know how to change docker version? (I cannot change it..)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609062,
          "author_name": "yamsam",
          "author_url": "",
          "post_date": "08/27/2019 11:43:56",
          "content": "<p>I can change docker image by touching 'Setting' and 'Docker' few times and it worked</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609840,
          "author_name": "go5kuramubon",
          "author_url": "",
          "post_date": "08/28/2019 06:45:05",
          "content": "<p>Thank you for information\nIn my kernel docker button , there are only latest choice. (I don't know why)\nBut this issue is solved as Dustin said! </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 608788,
      "author_name": "stanislavblinov",
      "author_url": "",
      "post_date": "08/27/2019 07:05:45",
      "content": "<p>Same. Yesterday during the day everything was fine. During the night I had this very issues - no log, runtime 0, commit unsuccesful (1 GPU running at a time, so new rules about GPU is not the reason).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 608802,
      "author_name": "projdev",
      "author_url": "",
      "post_date": "08/27/2019 07:16:21",
      "content": "<p>I had encountered the same issue too. Bad timing on GPU issues as many competitors try to improve their model/s using this tool.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 608810,
      "author_name": "chechir",
      "author_url": "",
      "post_date": "08/27/2019 07:23:15",
      "content": "<p>same here. My kernel ran all the way through but it doesn't produce the notebook at the end. So annoying after waiting 5 hours</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 608813,
      "author_name": "gumuxiansheng",
      "author_url": "",
      "post_date": "08/27/2019 07:26:53",
      "content": "<p>Seems the same here... ran for 8-9 hours and got nothing😭 </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 608886,
      "author_name": "taniaj",
      "author_url": "",
      "post_date": "08/27/2019 08:47:40",
      "content": "<p>Same here, I've been having problems for the past couple of days. Errors with no logs available and my output is missing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 609008,
      "author_name": "stocks",
      "author_url": "",
      "post_date": "08/27/2019 10:32:01",
      "content": "<p>total fail in the last 24 hrs for my GPU kernels. the timing is very unfortunate.\nthe docker image version can be changed to the previous version in the settings on the right of the interactive session. i saw this option earlier today but not now.</p>\n\n<p>earlier today I could run the interactive session but this too fails now.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 609050,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "08/27/2019 11:27:49",
      "content": "<p>Same here. I'm running the same GPU commit code that worked a few days ago (versions 1 and 2 below). But last night both GPU commits failed. They both show a runtime of 0 seconds and a red error circle. Both use a timer and should run for 8.5 hours and then stop execution. Also my teammate submitted two and both of his failed.</p>\n\n<p><img src=\"http://playagricola.com/Kaggle/error82719.png\" alt=\"image\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 609067,
      "author_name": "jtrotman",
      "author_url": "",
      "post_date": "08/27/2019 11:47:45",
      "content": "<p>I'm seeing failures too, but only in saving the output files...</p>\n\n<p>I push two GPU model training kernels at the same time. I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine. The call home to say 'complete' also happens but then the kernel status (<code>kaggle kernels status</code> command) is \"error\". When I fetch the output, (<code>kaggle kernels output</code> command) the normal console log file is available and downloaded (with correct date/time), but none of the output files...</p>",
      "votes": null,
      "replies": [
        {
          "id": 609401,
          "author_name": "abendorf",
          "author_url": "",
          "post_date": "08/27/2019 17:05:08",
          "content": "<p>\"I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine.\"</p>\n\n<p>This is a great idea.  I should really code up something like this... </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609536,
          "author_name": "jtrotman",
          "author_url": "",
          "post_date": "08/27/2019 20:28:05",
          "content": "<p>Thanks <a href=\"/abendorf\">@abendorf</a> , it's really made a big difference to kernels usability for me...</p>\n\n<p>With Python a really quick implementation would be:</p>\n\n<pre><code>import requests\n\ndef report_http(d):\n    ip, port = '1.2.3.4', 8899  # your ip, port\n    try:\n        p = '&amp;'.join([f'{k}={v}' for k,v in d.items()])\n        r = requests.get(f'http://{ip}:{port}/rep?{p}')\n    except:\n        pass\n</code></pre>\n\n<p>Then in a Keras callback:</p>\n\n<pre><code>class MyKerasCallback(tf.keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs={}):\n        d = dict(fold=FOLD, epoch=epoch)\n        d.update(logs)\n        report_http(d)\n</code></pre>\n\n<p>You just need to ensure the port is forwarded to some machine in your local network. For a simple (and perhaps insecure) server you can create an empty directory, then start a http server (this is for Python 2, on a Raspberry PI for me):</p>\n\n<pre><code>nohup python -m SimpleHTTPServer 8899 &amp;  \n</code></pre>\n\n<p>then watch the log file: </p>\n\n<pre><code>tail -f nohup.out\n</code></pre>\n\n<p>You don't even have to implement the '/rep' URL, you can just look at the 404 log messages ;)</p>\n\n<p><code>[27/Aug/2019 14:11:19] \"GET /rep?fold=2&amp;epoch=89&amp;mae=0.04454391077160835&amp;val_mae=0.05135239288210869&amp;lr=0.00010040000051958486 HTTP/1.1\" 404 -</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609540,
          "author_name": "abendorf",
          "author_url": "",
          "post_date": "08/27/2019 20:39:57",
          "content": "<p>Fantastic!  Thanks a bunch for the code!  I'll give this a try.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 609118,
      "author_name": "toshik",
      "author_url": "",
      "post_date": "08/27/2019 12:28:13",
      "content": "<p>They might update some software for CUDA. Whan I update CUDA or CUDNN, I'm sometimes forced to reinstall Python packages.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 609240,
      "author_name": "herbison",
      "author_url": "",
      "post_date": "08/27/2019 14:22:28",
      "content": "<p>Thanks for reporting this everyone, we found and fixed this issue. Let us know if you experience any after effects.</p>",
      "votes": null,
      "replies": [
        {
          "id": 609246,
          "author_name": "julianmukaj",
          "author_url": "",
          "post_date": "08/27/2019 14:25:14",
          "content": "<p>Other bugs: Sometimes after a few commits my notebook contents delete itself, only the first block remains, and I have to fork the last version to continue editing.</p>\n\n<p>Also sometimes uncommitted kernels show and sometimes don't?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609394,
          "author_name": "stocks",
          "author_url": "",
          "post_date": "08/27/2019 16:56:17",
          "content": "<p>i have not seen notebooks self-deleting their contents. have you tried re-fleshing the page?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609481,
          "author_name": "julianmukaj",
          "author_url": "",
          "post_date": "08/27/2019 19:12:58",
          "content": "<p>Yes I have, has happened to me more than once. Checked on multiple PCs also.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609489,
          "author_name": "mankyfej",
          "author_url": "",
          "post_date": "08/27/2019 19:26:21",
          "content": "<p>I experienced this self-deleting bug, too. I tried to refresh it, close and reopen the notebook, but nothing helped, I had to fork my previous commit. Kind of annoying if you already made some changes to the notebook and your work is lost.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609551,
          "author_name": "mhviraf",
          "author_url": "",
          "post_date": "08/27/2019 20:58:18",
          "content": "<p><a href=\"/julianmukaj\">@julianmukaj</a> have u tried different browsers as well? I experienced it a few times in the past with Google Chrome but when opened the notebook in Mozilla it worked. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609977,
          "author_name": "julianmukaj",
          "author_url": "",
          "post_date": "08/28/2019 09:32:25",
          "content": "<p><a href=\"/mhviraf\">@mhviraf</a> Yes, just re-tried in Microsoft Edge also, contents show as deleted still.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "608637": "From yesterday, I cannot succeed commit with this error... \nAnyone is happened same thing ??\n\nvery hard in the last 2days...\n\n![ErrorScreenShot](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1305588%2F545a3394c763cd885476b7a45d93cced%2F2019-08-27%2010.14.56.png?generation=1566876116391964&amp;alt=media)",
    "608740": "I had errors too :(",
    "608781": "Have you seen this discussion? I hope it works.\n\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105889#608660\n\n&gt; But after 5 unsuccessful attempts to commit my kernel, I changed docker version from latest to (versions 1 to 33) and it worked.",
    "608788": "Same. Yesterday during the day everything was fine. During the night I had this very issues - no log, runtime 0, commit unsuccesful (1 GPU running at a time, so new rules about GPU is not the reason).",
    "608802": "I had encountered the same issue too. Bad timing on GPU issues as many competitors try to improve their model/s using this tool.",
    "608810": "same here. My kernel ran all the way through but it doesn't produce the notebook at the end. So annoying after waiting 5 hours",
    "608813": "Seems the same here... ran for 8-9 hours and got nothing😭",
    "608886": "Same here, I've been having problems for the past couple of days. Errors with no logs available and my output is missing.",
    "608898": "Thank you for good information!! \nDo you know how to change docker version? (I cannot change it..)",
    "609008": "total fail in the last 24 hrs for my GPU kernels. the timing is very unfortunate.\nthe docker image version can be changed to the previous version in the settings on the right of the interactive session. i saw this option earlier today but not now.\n\nearlier today I could run the interactive session but this too fails now.",
    "609050": "Same here. I'm running the same GPU commit code that worked a few days ago (versions 1 and 2 below). But last night both GPU commits failed. They both show a runtime of 0 seconds and a red error circle. Both use a timer and should run for 8.5 hours and then stop execution. Also my teammate submitted two and both of his failed.\n  \n![image](http://playagricola.com/Kaggle/error82719.png)",
    "609062": "I can change docker image by touching 'Setting' and 'Docker' few times and it worked",
    "609067": "I'm seeing failures too, but only in saving the output files...\n\nI push two GPU model training kernels at the same time. I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine. The call home to say 'complete' also happens but then the kernel status (`kaggle kernels status` command) is \"error\". When I fetch the output, (`kaggle kernels output` command) the normal console log file is available and downloaded (with correct date/time), but none of the output files...",
    "609118": "They might update some software for CUDA. Whan I update CUDA or CUDNN, I'm sometimes forced to reinstall Python packages.",
    "609240": "Thanks for reporting this everyone, we found and fixed this issue. Let us know if you experience any after effects.",
    "609246": "Other bugs: Sometimes after a few commits my notebook contents delete itself, only the first block remains, and I have to fork the last version to continue editing.\n\nAlso sometimes uncommitted kernels show and sometimes don't?",
    "609394": "i have not seen notebooks self-deleting their contents. have you tried re-fleshing the page?",
    "609401": "\"I monitor them with http requests - they send me back the training/validation score progress for every epoch, so I know they're running just fine.\"\n\nThis is a great idea.  I should really code up something like this...",
    "609481": "Yes I have, has happened to me more than once. Checked on multiple PCs also.",
    "609489": "I experienced this self-deleting bug, too. I tried to refresh it, close and reopen the notebook, but nothing helped, I had to fork my previous commit. Kind of annoying if you already made some changes to the notebook and your work is lost.",
    "609536": "Thanks @abendorf , it's really made a big difference to kernels usability for me...\n\nWith Python a really quick implementation would be:\n\n    import requests\n\n    def report_http(d):\n        ip, port = '1.2.3.4', 8899  # your ip, port\n        try:\n            p = '&amp;'.join([f'{k}={v}' for k,v in d.items()])\n            r = requests.get(f'http://{ip}:{port}/rep?{p}')\n        except:\n            pass\n\n\nThen in a Keras callback:\n\n    class MyKerasCallback(tf.keras.callbacks.Callback):\n        def on_epoch_end(self, epoch, logs={}):\n            d = dict(fold=FOLD, epoch=epoch)\n            d.update(logs)\n            report_http(d)\n\n\nYou just need to ensure the port is forwarded to some machine in your local network. For a simple (and perhaps insecure) server you can create an empty directory, then start a http server (this is for Python 2, on a Raspberry PI for me):\n\n    nohup python -m SimpleHTTPServer 8899 &amp;  \n\nthen watch the log file: \n\n    tail -f nohup.out\n\nYou don't even have to implement the '/rep' URL, you can just look at the 404 log messages ;)\n\n`[27/Aug/2019 14:11:19] \"GET /rep?fold=2&amp;epoch=89&amp;mae=0.04454391077160835&amp;val_mae=0.05135239288210869&amp;lr=0.00010040000051958486 HTTP/1.1\" 404 -`",
    "609540": "Fantastic!  Thanks a bunch for the code!  I'll give this a try.",
    "609551": "julianmukaj have u tried different browsers as well? I experienced it a few times in the past with Google Chrome but when opened the notebook in Mozilla it worked.",
    "609840": "Thank you for information\nIn my kernel docker button , there are only latest choice. (I don't know why)\nBut this issue is solved as Dustin said!",
    "609977": "mhviraf Yes, just re-tried in Microsoft Edge also, contents show as deleted still."
  },
  "source": "meta"
}