![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F28727454%2F100156461e24da05f19f9cf3ccfc6a5e%2F2026-03-22%20072928.png?generation=1774135803681306&alt=media\")
4m 17s · GPU T4 x2 LB timeout？\nHi everyone, I'm running into a confusing timeout issue and hoping someone can help explain what's happening.\n10 days ago, I had a submission that ran in 4m 17s on GPU P100, passed successfully, and scored within about 6 hours — no issues at all.\nToday, I submitted a new notebook with a different model, and also utilized dual T4 GPUs in parallel. The runtime showed either 5m 10s on GPU P100 or 4m 17s on GPU T4 x2, but the submission came back as notebook timeout. I'm confused because the logged runtime doesn't seem dramatically different from before, yet it's now failing.\nA few questions I have:
\nHas anyone else experienced a sudden increase in timeout errors recently?
\nAny insights would be greatly appreciated. Thanks!
", "messages": [ { "id": 3426065, "postDate": "2026-03-21T20:17:19.277Z", "content": "Update: CUDA Error on Old Submission + Garbled Output from Both Models
\nI have some new findings that make this even more confusing.\nWhen I tried to re-run my original submission (the one that passed 10 days ago), it now throws:\nCUDA error: no kernel image is available for execution on the device\nMy current submission fixes this CUDA issue and also has use_cache=True, but it still times out as mentioned before.\nOn top of that, I manually checked the inference output of both models:
\n/kaggle/input/models/mattiaangeli/byt5-akkadian-mbr-v2/pytorch/default/1
\n/kaggle/input/datasets/assiaben/final-byt5/byt5-akkadian-optimized-34x
\nBoth are producing completely garbled, unreadable output — not just low-quality translations, but incoherent garbage that makes no sense at all.\nSo it seems like something has changed on Kaggle's end — possibly the GPU hardware was upgraded (which would explain the CUDA kernel error on the old submission), and this hardware change may also be affecting how the models run. Could the hosts or anyone from Kaggle clarify what happened? Has the GPU environment changed recently?
", "rawMarkdown": "Update: CUDA Error on Old Submission + Garbled Output from Both Models\n\nI have some new findings that make this even more confusing.\nWhen I tried to re-run my original submission (the one that passed 10 days ago), it now throws:\nCUDA error: no kernel image is available for execution on the device\nMy current submission fixes this CUDA issue and also has use_cache=True, but it still times out as mentioned before.\nOn top of that, I manually checked the inference output of both models:\n\n/kaggle/input/models/mattiaangeli/byt5-akkadian-mbr-v2/pytorch/default/1\n\n/kaggle/input/datasets/assiaben/final-byt5/byt5-akkadian-optimized-34x\n\nBoth are producing completely garbled, unreadable output — not just low-quality translations, but incoherent garbage that makes no sense at all.\nSo it seems like something has changed on Kaggle's end — possibly the GPU hardware was upgraded (which would explain the CUDA kernel error on the old submission), and this hardware change may also be affecting how the models run. Could the hosts or anyone from Kaggle clarify what happened? Has the GPU environment changed recently?", "votes": 1, "replies": [ { "id": 3426072, "postDate": "2026-03-21T20:26:47.257Z", "content": "It's module updates, try pinning your notebook to original environment or look at other posts from today about how to reinstall modules. For me I had to force transformers==5.2.0 for the garbled output, but you might have other issues as well.
", "rawMarkdown": "It's module updates, try pinning your notebook to original environment or look at other posts from today about how to reinstall modules. For me I had to force transformers==5.2.0 for the garbled output, but you might have other issues as well.", "votes": 1 } ] }, { "id": 3426029, "postDate": "2026-03-21T18:10:29.227Z", "content": "One way to debug is to run your script on a file bigger than test.csv. Because test.csv is so small before submission the differences in run time are not very significant.\nBe aware that many packages have been updated recently so if your notebook isn't pinned to an environment, it might affect how it runs.
", "rawMarkdown": "One way to debug is to run your script on a file bigger than test.csv. Because test.csv is so small before submission the differences in run time are not very significant.\nBe aware that many packages have been updated recently so if your notebook isn't pinned to an environment, it might affect how it runs.", "votes": 1 }, { "id": 3425918, "postDate": "2026-03-21T14:29:45.223Z", "content": "4m 17s · GPU T4 x2 LB timeout？\nHi everyone, I'm running into a confusing timeout issue and hoping someone can help explain what's happening.\n10 days ago, I had a submission that ran in 4m 17s on GPU P100, passed successfully, and scored within about 6 hours — no issues at all.\nToday, I submitted a new notebook with a different model, and also utilized dual T4 GPUs in parallel. The runtime showed either 5m 10s on GPU P100 or 4m 17s on GPU T4 x2, but the submission came back as notebook timeout. I'm confused because the logged runtime doesn't seem dramatically different from before, yet it's now failing.\nA few questions I have:
\nHas anyone else experienced a sudden increase in timeout errors recently?
\nAny insights would be greatly appreciated. Thanks!
", "rawMarkdown": "4m 17s · GPU T4 x2 LB timeout？\nHi everyone, I'm running into a confusing timeout issue and hoping someone can help explain what's happening.\n**10 days ago**, I had a submission that ran in **4m 17s on GPU P100**, **passed successfully**, and scored within about 6 hours — no issues at all.\nToday, I submitted a new notebook with a different model, and also utilized dual T4 GPUs in parallel. The runtime showed either **5m 10s on GPU P100** or **4m 17s on GPU T4 x2**, but the submission came back as **notebook timeout**. I'm confused because the logged runtime doesn't seem dramatically different from before, yet it's now failing.\nA few questions I have:\n\nHas anyone else experienced a sudden increase in timeout errors recently?\n1. Could the hosts have increased the size of the test set or the data used for metric calculation on the backend? That would explain why the scoring step is taking longer even if the notebook runtime looks similar.\n2. Is there a known difference in how P100 vs T4 x2 handles inference at submission time that might cause this?\n3. Also, I'm wondering — **will previously accepted submissions be affected by this change as well**? Or are those scores already finalized and safe?\n\nAny insights would be greatly appreciated. Thanks!", "votes": 2 }, { "id": 3426334, "postDate": "2026-03-22T11:35:50.100Z", "content": "I too am getting this error. I must admit I panicked a lot when I realised that a notebook I ran 2 days ago is now getting a runtime error out of nowhere. No changes from my side, no changes to the models; nothing. I shall try the transformer fix.\nIf author manages to fix this problem, can you please share how you did it here?\nFIX: Pin to Original Environment\nSession Options>Environment> Change from \"Always use latest\" to \"Pin to original environment\"
", "rawMarkdown": "I too am getting this error. I must admit I panicked a lot when I realised that a notebook I ran 2 days ago is now getting a runtime error out of nowhere. No changes from my side, no changes to the models; nothing. I shall try the transformer fix.\nIf author manages to fix this problem, can you please share how you did it here?\nFIX: Pin to Original Environment\nSession Options>Environment> Change from \"Always use latest\" to \"Pin to original environment\" ", "replies": [ { "id": 3426336, "postDate": "2026-03-22T11:36:50.263Z", "content": "I have found the fix; just pin to original environment and you should be good to go
", "rawMarkdown": "I have found the fix; just pin to original environment and you should be good to go" } ] }, { "id": 3426137, "postDate": "2026-03-21T23:30:07.067Z", "content": "Further Update: P100 GPU No Longer Supported by Current PyTorch Version\nI have another finding to share. After further investigation, I can confirm that the GPU assigned is a Tesla P100-PCIE-16GB with CUDA capability 6.0. However, the current PyTorch version installed in the Kaggle environment only supports architectures from sm_70 to sm_120 (CUDA capability 7.0 and above)\n
both Environment Preference is the same situation
", "rawMarkdown": "both **Environment Preference** is the same situation", "replies": [ { "id": 3426139, "postDate": "2026-03-21T23:33:43.390Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 3425925, "postDate": "2026-03-21T14:45:49.610Z", "content": "One possibility is that you set use_cache: False during training and the generation_config.json containing that arg overrides the use_cache: True in the inference notebook.
Or maybe the transformer version is different.
", "rawMarkdown": "One possibility is that you set `use_cache: False` during training and the `generation_config.json` containing that arg overrides the `use_cache: True` in the inference notebook.\n\nOr maybe the transformer version is different.", "replies": [ { "id": 3426006, "postDate": "2026-03-21T17:26:21.983Z", "content": "thank you！I‘ll try
", "rawMarkdown": "thank you！I‘ll try" } ] } ], "comments": [ { "id": 3426065, "author_name": "Cat D", "author_url": "", "post_date": "2026-03-21T20:17:19.277000", "content": "Update: CUDA Error on Old Submission + Garbled Output from Both Models
\nI have some new findings that make this even more confusing.\nWhen I tried to re-run my original submission (the one that passed 10 days ago), it now throws:\nCUDA error: no kernel image is available for execution on the device\nMy current submission fixes this CUDA issue and also has use_cache=True, but it still times out as mentioned before.\nOn top of that, I manually checked the inference output of both models:
\n/kaggle/input/models/mattiaangeli/byt5-akkadian-mbr-v2/pytorch/default/1
\n/kaggle/input/datasets/assiaben/final-byt5/byt5-akkadian-optimized-34x
\nBoth are producing completely garbled, unreadable output — not just low-quality translations, but incoherent garbage that makes no sense at all.\nSo it seems like something has changed on Kaggle's end — possibly the GPU hardware was upgraded (which would explain the CUDA kernel error on the old submission), and this hardware change may also be affecting how the models run. Could the hosts or anyone from Kaggle clarify what happened? Has the GPU environment changed recently?
", "votes": 1, "replies": [ { "id": 3426072, "author_name": "tennogh", "author_url": "", "post_date": "2026-03-21T20:26:47.257000", "content": "It's module updates, try pinning your notebook to original environment or look at other posts from today about how to reinstall modules. For me I had to force transformers==5.2.0 for the garbled output, but you might have other issues as well.
", "votes": 1, "replies": [] } ] }, { "id": 3426029, "author_name": "tennogh", "author_url": "", "post_date": "2026-03-21T18:10:29.227000", "content": "One way to debug is to run your script on a file bigger than test.csv. Because test.csv is so small before submission the differences in run time are not very significant.\nBe aware that many packages have been updated recently so if your notebook isn't pinned to an environment, it might affect how it runs.
", "votes": 1, "replies": [] }, { "id": 3426334, "author_name": "DeathlyHallows1239", "author_url": "", "post_date": "2026-03-22T11:35:50.100000", "content": "I too am getting this error. I must admit I panicked a lot when I realised that a notebook I ran 2 days ago is now getting a runtime error out of nowhere. No changes from my side, no changes to the models; nothing. I shall try the transformer fix.\nIf author manages to fix this problem, can you please share how you did it here?\nFIX: Pin to Original Environment\nSession Options>Environment> Change from \"Always use latest\" to \"Pin to original environment\"
", "votes": 0, "replies": [ { "id": 3426336, "author_name": "DeathlyHallows1239", "author_url": "", "post_date": "2026-03-22T11:36:50.263000", "content": "I have found the fix; just pin to original environment and you should be good to go
", "votes": 0, "replies": [] } ] }, { "id": 3426137, "author_name": "Cat D", "author_url": "", "post_date": "2026-03-21T23:30:07.067000", "content": "Further Update: P100 GPU No Longer Supported by Current PyTorch Version\nI have another finding to share. After further investigation, I can confirm that the GPU assigned is a Tesla P100-PCIE-16GB with CUDA capability 6.0. However, the current PyTorch version installed in the Kaggle environment only supports architectures from sm_70 to sm_120 (CUDA capability 7.0 and above)\n
both Environment Preference is the same situation
", "votes": 0, "replies": [ { "id": 3426139, "author_name": "", "author_url": "", "post_date": "2026-03-21T23:33:43.390000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3425925, "author_name": "Tomorin", "author_url": "", "post_date": "2026-03-21T14:45:49.610000", "content": "One possibility is that you set use_cache: False during training and the generation_config.json containing that arg overrides the use_cache: True in the inference notebook.
Or maybe the transformer version is different.
", "votes": 0, "replies": [ { "id": 3426006, "author_name": "Cat D", "author_url": "", "post_date": "2026-03-21T17:26:21.983000", "content": "thank you！I‘ll try
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3426065": "Update: CUDA Error on Old Submission + Garbled Output from Both Models\n\nI have some new findings that make this even more confusing.\nWhen I tried to re-run my original submission (the one that passed 10 days ago), it now throws:\nCUDA error: no kernel image is available for execution on the device\nMy current submission fixes this CUDA issue and also has use_cache=True, but it still times out as mentioned before.\nOn top of that, I manually checked the inference output of both models:\n\n/kaggle/input/models/mattiaangeli/byt5-akkadian-mbr-v2/pytorch/default/1\n\n/kaggle/input/datasets/assiaben/final-byt5/byt5-akkadian-optimized-34x\n\nBoth are producing completely garbled, unreadable output — not just low-quality translations, but incoherent garbage that makes no sense at all.\nSo it seems like something has changed on Kaggle's end — possibly the GPU hardware was upgraded (which would explain the CUDA kernel error on the old submission), and this hardware change may also be affecting how the models run. Could the hosts or anyone from Kaggle clarify what happened? Has the GPU environment changed recently?", "3426029": "One way to debug is to run your script on a file bigger than test.csv. Because test.csv is so small before submission the differences in run time are not very significant.\nBe aware that many packages have been updated recently so if your notebook isn't pinned to an environment, it might affect how it runs.", "3425918": "4m 17s · GPU T4 x2 LB timeout？\nHi everyone, I'm running into a confusing timeout issue and hoping someone can help explain what's happening.\n**10 days ago**, I had a submission that ran in **4m 17s on GPU P100**, **passed successfully**, and scored within about 6 hours — no issues at all.\nToday, I submitted a new notebook with a different model, and also utilized dual T4 GPUs in parallel. The runtime showed either **5m 10s on GPU P100** or **4m 17s on GPU T4 x2**, but the submission came back as **notebook timeout**. I'm confused because the logged runtime doesn't seem dramatically different from before, yet it's now failing.\nA few questions I have:\n\nHas anyone else experienced a sudden increase in timeout errors recently?\n1. Could the hosts have increased the size of the test set or the data used for metric calculation on the backend? That would explain why the scoring step is taking longer even if the notebook runtime looks similar.\n2. Is there a known difference in how P100 vs T4 x2 handles inference at submission time that might cause this?\n3. Also, I'm wondering — **will previously accepted submissions be affected by this change as well**? Or are those scores already finalized and safe?\n\nAny insights would be greatly appreciated. Thanks!", "3426334": "I too am getting this error. I must admit I panicked a lot when I realised that a notebook I ran 2 days ago is now getting a runtime error out of nowhere. No changes from my side, no changes to the models; nothing. I shall try the transformer fix.\nIf author manages to fix this problem, can you please share how you did it here?\nFIX: Pin to Original Environment\nSession Options>Environment> Change from \"Always use latest\" to \"Pin to original environment\" ", "3426137": "Further Update: P100 GPU No Longer Supported by Current PyTorch Version\nI have another finding to share. After further investigation, I can confirm that the GPU assigned is a Tesla P100-PCIE-16GB with CUDA capability 6.0. However, the current PyTorch version installed in the Kaggle environment only supports architectures from sm_70 to sm_120 (CUDA capability 7.0 and above)\n", "3425925": "One possibility is that you set `use_cache: False` during training and the `generation_config.json` containing that arg overrides the `use_cache: True` in the inference notebook.\n\nOr maybe the transformer version is different." } }