{
  "id": 585143,
  "title": "GatewayRuntimeError: Metadata size exceeds hard limit during inference",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/585143",
  "author_name": "",
  "post_date": "2025-06-18T08:25:07.150233Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>I'm currently encountering the following error when running run_local_gateway() in the CMI - Detect Behavior with Sensor Data competition:<br>\n<code>RuntimeWarning:  \n991 seconds elapsed before server startup.  \nThis exceeds the startup time limit of 900 seconds that the gateway will enforce during the rerun on the hidden test set.\n</code></p>\n<p>Then, the inference fails with this error:<br>\n<code>GatewayRuntimeError: (&lt;GatewayRuntimeErrorType.SERVER_CONNECTION_FAILED: 2&gt;,  \n'&lt;_InactiveRpcError of RPC that terminated with:  \nstatus = StatusCode.RESOURCE_EXHAUSTED  \ndetails = \"received metadata size exceeds hard limit (value length 85593 vs. 16384)\"  \n...')\n</code></p>\n<p>My model runs fine during training and local validation, but it fails when evaluated via the official kaggle_evaluation template.</p>\n<p>Has anyone else encountered this issue?<br>\nI'd appreciate any suggestions regarding:</p>\n<ul>\n<li>How to reduce the metadata size (e.g., maybe something wrong with my return values from predict)</li>\n<li>Ways to avoid slow server startup (e.g., moving heavy initialization before inference_server.serve())</li>\n</ul>\n<p>Thanks in advance for any help!</p>",
  "messages": [
    {
      "id": "3226903",
      "postDate": "06/18/2025 08:25:07",
      "content": "<p>Hi everyone,</p>\n<p>I'm currently encountering the following error when running run_local_gateway() in the CMI - Detect Behavior with Sensor Data competition:<br>\n<code>RuntimeWarning:  \n991 seconds elapsed before server startup.  \nThis exceeds the startup time limit of 900 seconds that the gateway will enforce during the rerun on the hidden test set.\n</code></p>\n<p>Then, the inference fails with this error:<br>\n<code>GatewayRuntimeError: (&lt;GatewayRuntimeErrorType.SERVER_CONNECTION_FAILED: 2&gt;,  \n'&lt;_InactiveRpcError of RPC that terminated with:  \nstatus = StatusCode.RESOURCE_EXHAUSTED  \ndetails = \"received metadata size exceeds hard limit (value length 85593 vs. 16384)\"  \n...')\n</code></p>\n<p>My model runs fine during training and local validation, but it fails when evaluated via the official kaggle_evaluation template.</p>\n<p>Has anyone else encountered this issue?<br>\nI'd appreciate any suggestions regarding:</p>\n<ul>\n<li>How to reduce the metadata size (e.g., maybe something wrong with my return values from predict)</li>\n<li>Ways to avoid slow server startup (e.g., moving heavy initialization before inference_server.serve())</li>\n</ul>\n<p>Thanks in advance for any help!</p>",
      "rawMarkdown": "Hi everyone,\n\nI'm currently encountering the following error when running run_local_gateway() in the CMI - Detect Behavior with Sensor Data competition:\n`RuntimeWarning:  \n991 seconds elapsed before server startup.  \nThis exceeds the startup time limit of 900 seconds that the gateway will enforce during the rerun on the hidden test set.\n`\n\nThen, the inference fails with this error:\n`GatewayRuntimeError: (<GatewayRuntimeErrorType.SERVER_CONNECTION_FAILED: 2>,  \n'<_InactiveRpcError of RPC that terminated with:  \nstatus = StatusCode.RESOURCE_EXHAUSTED  \ndetails = \"received metadata size exceeds hard limit (value length 85593 vs. 16384)\"  \n...')\n`\n\nMy model runs fine during training and local validation, but it fails when evaluated via the official kaggle_evaluation template.\n\nHas anyone else encountered this issue?\nI'd appreciate any suggestions regarding:\n- How to reduce the metadata size (e.g., maybe something wrong with my return values from predict)\n- Ways to avoid slow server startup (e.g., moving heavy initialization before inference_server.serve())\n\nThanks in advance for any help!",
      "votes": null
    },
    {
      "id": "3227086",
      "postDate": "06/18/2025 13:30:36",
      "content": "<p>You should load the pretrained model to infer</p>",
      "rawMarkdown": "You should load the pretrained model to infer",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3227086,
      "author_name": "shanzhong8",
      "author_url": "",
      "post_date": "06/18/2025 13:30:36",
      "content": "<p>You should load the pretrained model to infer</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3226903": "Hi everyone,\n\nI'm currently encountering the following error when running run_local_gateway() in the CMI - Detect Behavior with Sensor Data competition:\n`RuntimeWarning:  \n991 seconds elapsed before server startup.  \nThis exceeds the startup time limit of 900 seconds that the gateway will enforce during the rerun on the hidden test set.\n`\n\nThen, the inference fails with this error:\n`GatewayRuntimeError: (<GatewayRuntimeErrorType.SERVER_CONNECTION_FAILED: 2>,  \n'<_InactiveRpcError of RPC that terminated with:  \nstatus = StatusCode.RESOURCE_EXHAUSTED  \ndetails = \"received metadata size exceeds hard limit (value length 85593 vs. 16384)\"  \n...')\n`\n\nMy model runs fine during training and local validation, but it fails when evaluated via the official kaggle_evaluation template.\n\nHas anyone else encountered this issue?\nI'd appreciate any suggestions regarding:\n- How to reduce the metadata size (e.g., maybe something wrong with my return values from predict)\n- Ways to avoid slow server startup (e.g., moving heavy initialization before inference_server.serve())\n\nThanks in advance for any help!",
    "3227086": "You should load the pretrained model to infer"
  },
  "source": "meta"
}