{
  "id": 330403,
  "title": "'Notebook Exceeded Allowed Compute'",
  "url": "/competitions/AI4Code/discussion/330403",
  "author_name": "",
  "post_date": "2022-06-12T05:24:41.194427600Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Why my kenel can't the pass requirement? Is there anyone who can help me?</p>\n<p>spending time resource : 2914.7s - GPU</p>\n<p>Is there any constraints for submission?…</p>\n<p><a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/cafelatte1/embedding-cnn-with-pretrained-tokenizers</a></p>",
  "messages": [
    {
      "id": "1817985",
      "postDate": "06/12/2022 05:24:41",
      "content": "<p>Why my kenel can't the pass requirement? Is there anyone who can help me?</p>\n<p>spending time resource : 2914.7s - GPU</p>\n<p>Is there any constraints for submission?…</p>\n<p><a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/cafelatte1/embedding-cnn-with-pretrained-tokenizers</a></p>",
      "rawMarkdown": "Why my kenel can't the pass requirement? Is there anyone who can help me?\n\nspending time resource : 2914.7s - GPU\n\nIs there any constraints for submission?...\n\n[https://www.kaggle.com/code/cafelatte1/embedding-cnn-with-pretrained-tokenizers](url)",
      "votes": null
    },
    {
      "id": "1818285",
      "postDate": "06/12/2022 13:17:01",
      "content": "<p>unfortunately code submissions can be very painful to debug. For e.g.<br>\n<a href=\"https://www.kaggle.com/competitions/birdclef-2022/discussion/322092\" target=\"_blank\">https://www.kaggle.com/competitions/birdclef-2022/discussion/322092</a></p>\n<p>Expand your test _df to num of records expected at inference time (host has shared this somewhere in writeup). You can run this now with debug statements and profiler and see what is happening. If there is no issue, then the error is due to some data format in inference-time dateset which you may not have considered. </p>",
      "rawMarkdown": "unfortunately code submissions can be very painful to debug. For e.g.\nhttps://www.kaggle.com/competitions/birdclef-2022/discussion/322092\n\nExpand your test _df to num of records expected at inference time (host has shared this somewhere in writeup). You can run this now with debug statements and profiler and see what is happening. If there is no issue, then the error is due to some data format in inference-time dateset which you may not have considered.",
      "votes": null
    },
    {
      "id": "1818890",
      "postDate": "06/13/2022 08:54:51",
      "content": "<p>Thanks for your help! I will check it up (maybe the output files are the cause of the problem)</p>",
      "rawMarkdown": "Thanks for your help! I will check it up (maybe the output files are the cause of the problem)",
      "votes": null
    },
    {
      "id": "1818896",
      "postDate": "06/13/2022 09:06:35",
      "content": "<p>You are welcome. In this comp we dont generate more than 500 o/p files, so that may not be issue. Best way to debug is to go in foll order: 1) increase testsize to simulate actual data presented at  inference and run the code with error messages/profilers. 2) 80% of time above should solve the issue. If not there is problem with inference data text which you may have not handled..carefully need to go thru' all the pre-processing as well as rest of code.  3)in maybe 0.001% of time it is some weird Kaggle error or limitation. it is very difficult to debug these type error in inference mode</p>",
      "rawMarkdown": "You are welcome. In this comp we dont generate more than 500 o/p files, so that may not be issue. Best way to debug is to go in foll order: 1) increase testsize to simulate actual data presented at  inference and run the code with error messages/profilers. 2) 80% of time above should solve the issue. If not there is problem with inference data text which you may have not handled..carefully need to go thru' all the pre-processing as well as rest of code.  3)in maybe 0.001% of time it is some weird Kaggle error or limitation. it is very difficult to debug these type error in inference mode",
      "votes": null
    },
    {
      "id": "1819583",
      "postDate": "06/14/2022 00:47:42",
      "content": "<p>Feed consistent-size batches into the GPU - and instead of repeating the test set, point your code at the training set to get ~20k unique files.</p>\n<p>In my case, feeding it one notebooks' markdown cells at a time took too long… (and then I hit another bug because I was setting only by cell_id, not [id, cell_id] and that eventually hits a conflict)</p>",
      "rawMarkdown": "Feed consistent-size batches into the GPU - and instead of repeating the test set, point your code at the training set to get ~20k unique files.\n\nIn my case, feeding it one notebooks' markdown cells at a time took too long... (and then I hit another bug because I was setting only by cell_id, not [id, cell_id] and that eventually hits a conflict)",
      "votes": null
    },
    {
      "id": "1819869",
      "postDate": "06/14/2022 07:48:58",
      "content": "<p>It was something trivial… like the below case.<br>\nTest data was changed once the function is executed.</p>\n<p>`</p>\n<h1>error code</h1>\n<p>do in function:<br>\n    tmp_x = df_test<br>\n    tmp_x = … something </p>\n<h1>correct code</h1>\n<p>do in function:<br>\n    tmp_x = df_test.copy()<br>\n    tmp_x = … something <br>\n`</p>\n<p>Thanks to your help, I can recheck my preprocessing code.</p>",
      "rawMarkdown": "It was something trivial... like the below case.\nTest data was changed once the function is executed.\n\n`\n# error code\ndo in function:\n    tmp_x = df_test\n    tmp_x = ... something \n\n# correct code\ndo in function:\n    tmp_x = df_test.copy()\n    tmp_x = ... something \n`\n\nThanks to your help, I can recheck my preprocessing code.",
      "votes": null
    },
    {
      "id": "1819870",
      "postDate": "06/14/2022 07:49:46",
      "content": "<p>Thanks for the tips, I checked the code about preprocessing and found it !</p>",
      "rawMarkdown": "Thanks for the tips, I checked the code about preprocessing and found it !",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1818285,
      "author_name": "allohvk",
      "author_url": "",
      "post_date": "06/12/2022 13:17:01",
      "content": "<p>unfortunately code submissions can be very painful to debug. For e.g.<br>\n<a href=\"https://www.kaggle.com/competitions/birdclef-2022/discussion/322092\" target=\"_blank\">https://www.kaggle.com/competitions/birdclef-2022/discussion/322092</a></p>\n<p>Expand your test _df to num of records expected at inference time (host has shared this somewhere in writeup). You can run this now with debug statements and profiler and see what is happening. If there is no issue, then the error is due to some data format in inference-time dateset which you may not have considered. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1818890,
          "author_name": "cafelatte1",
          "author_url": "",
          "post_date": "06/13/2022 08:54:51",
          "content": "<p>Thanks for your help! I will check it up (maybe the output files are the cause of the problem)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1818896,
          "author_name": "allohvk",
          "author_url": "",
          "post_date": "06/13/2022 09:06:35",
          "content": "<p>You are welcome. In this comp we dont generate more than 500 o/p files, so that may not be issue. Best way to debug is to go in foll order: 1) increase testsize to simulate actual data presented at  inference and run the code with error messages/profilers. 2) 80% of time above should solve the issue. If not there is problem with inference data text which you may have not handled..carefully need to go thru' all the pre-processing as well as rest of code.  3)in maybe 0.001% of time it is some weird Kaggle error or limitation. it is very difficult to debug these type error in inference mode</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1819869,
          "author_name": "cafelatte1",
          "author_url": "",
          "post_date": "06/14/2022 07:48:58",
          "content": "<p>It was something trivial… like the below case.<br>\nTest data was changed once the function is executed.</p>\n<p>`</p>\n<h1>error code</h1>\n<p>do in function:<br>\n    tmp_x = df_test<br>\n    tmp_x = … something </p>\n<h1>correct code</h1>\n<p>do in function:<br>\n    tmp_x = df_test.copy()<br>\n    tmp_x = … something <br>\n`</p>\n<p>Thanks to your help, I can recheck my preprocessing code.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1819583,
      "author_name": "happycube",
      "author_url": "",
      "post_date": "06/14/2022 00:47:42",
      "content": "<p>Feed consistent-size batches into the GPU - and instead of repeating the test set, point your code at the training set to get ~20k unique files.</p>\n<p>In my case, feeding it one notebooks' markdown cells at a time took too long… (and then I hit another bug because I was setting only by cell_id, not [id, cell_id] and that eventually hits a conflict)</p>",
      "votes": null,
      "replies": [
        {
          "id": 1819870,
          "author_name": "cafelatte1",
          "author_url": "",
          "post_date": "06/14/2022 07:49:46",
          "content": "<p>Thanks for the tips, I checked the code about preprocessing and found it !</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1817985": "Why my kenel can't the pass requirement? Is there anyone who can help me?\n\nspending time resource : 2914.7s - GPU\n\nIs there any constraints for submission?...\n\n[https://www.kaggle.com/code/cafelatte1/embedding-cnn-with-pretrained-tokenizers](url)",
    "1818285": "unfortunately code submissions can be very painful to debug. For e.g.\nhttps://www.kaggle.com/competitions/birdclef-2022/discussion/322092\n\nExpand your test _df to num of records expected at inference time (host has shared this somewhere in writeup). You can run this now with debug statements and profiler and see what is happening. If there is no issue, then the error is due to some data format in inference-time dateset which you may not have considered.",
    "1818890": "Thanks for your help! I will check it up (maybe the output files are the cause of the problem)",
    "1818896": "You are welcome. In this comp we dont generate more than 500 o/p files, so that may not be issue. Best way to debug is to go in foll order: 1) increase testsize to simulate actual data presented at  inference and run the code with error messages/profilers. 2) 80% of time above should solve the issue. If not there is problem with inference data text which you may have not handled..carefully need to go thru' all the pre-processing as well as rest of code.  3)in maybe 0.001% of time it is some weird Kaggle error or limitation. it is very difficult to debug these type error in inference mode",
    "1819583": "Feed consistent-size batches into the GPU - and instead of repeating the test set, point your code at the training set to get ~20k unique files.\n\nIn my case, feeding it one notebooks' markdown cells at a time took too long... (and then I hit another bug because I was setting only by cell_id, not [id, cell_id] and that eventually hits a conflict)",
    "1819869": "It was something trivial... like the below case.\nTest data was changed once the function is executed.\n\n`\n# error code\ndo in function:\n    tmp_x = df_test\n    tmp_x = ... something \n\n# correct code\ndo in function:\n    tmp_x = df_test.copy()\n    tmp_x = ... something \n`\n\nThanks to your help, I can recheck my preprocessing code.",
    "1819870": "Thanks for the tips, I checked the code about preprocessing and found it !"
  },
  "source": "meta"
}