{
  "id": 586435,
  "title": "Hitchhikers guide to submission ",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/586435",
  "author_name": "",
  "post_date": "2025-06-26T17:22:39.006297100Z",
  "votes": 6,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Here are some points I learned so far that I want to share with you:</p>\n<ul>\n<li><p>The submission process mandates running the submission server with the predict function. </p></li>\n<li><p>The predict function will be called many times during scoring, one time for each sequence. </p></li>\n<li><p>if your predict function produces one wrong response, then the submission will be considered invalid.  </p></li>\n<li><p>The Leaderboard score is not calculated using the text.csv data, the data remains completely hidden.</p></li>\n<li><p>Any print statements during submission will not be visible in your output or logs. </p></li>\n<li><p>During scoring the submission server is not running in your environment. Every file you access must be imported as \"Input\" </p></li>\n<li><p>There is a 900 sec timer for starting your submission server after the notebook was activated. </p></li>\n<li><p>Another timer ensures the response to the first sequence is received within 20 minutes(?)</p></li>\n<li><p>The submission itself can actually run a long time; I have submissions running for several hours before they finished and created a new result.  Note that the notebook run is already completed at that time.   </p>\n<p>please let me know we need to add anything. </p></li>\n</ul>",
  "messages": [
    {
      "id": "3233330",
      "postDate": "06/26/2025 17:22:39",
      "content": "<p>Here are some points I learned so far that I want to share with you:</p>\n<ul>\n<li><p>The submission process mandates running the submission server with the predict function. </p></li>\n<li><p>The predict function will be called many times during scoring, one time for each sequence. </p></li>\n<li><p>if your predict function produces one wrong response, then the submission will be considered invalid.  </p></li>\n<li><p>The Leaderboard score is not calculated using the text.csv data, the data remains completely hidden.</p></li>\n<li><p>Any print statements during submission will not be visible in your output or logs. </p></li>\n<li><p>During scoring the submission server is not running in your environment. Every file you access must be imported as \"Input\" </p></li>\n<li><p>There is a 900 sec timer for starting your submission server after the notebook was activated. </p></li>\n<li><p>Another timer ensures the response to the first sequence is received within 20 minutes(?)</p></li>\n<li><p>The submission itself can actually run a long time; I have submissions running for several hours before they finished and created a new result.  Note that the notebook run is already completed at that time.   </p>\n<p>please let me know we need to add anything. </p></li>\n</ul>",
      "rawMarkdown": "Here are some points I learned so far that I want to share with you:\n\n- The submission process mandates running the submission server with the predict function. \n-  The predict function will be called many times during scoring, one time for each sequence. \n- if your predict function produces one wrong response, then the submission will be considered invalid.  \n- The Leaderboard score is not calculated using the text.csv data, the data remains completely hidden.\n- Any print statements during submission will not be visible in your output or logs. \n- During scoring the submission server is not running in your environment. Every file you access must be imported as \"Input\" \n-  There is a 900 sec timer for starting your submission server after the notebook was activated. \n-  Another timer ensures the response to the first sequence is received within 20 minutes(?)\n- The submission itself can actually run a long time; I have submissions running for several hours before they finished and created a new result.  Note that the notebook run is already completed at that time.   \n \n  please let me know we need to add anything.",
      "votes": null
    },
    {
      "id": "3233457",
      "postDate": "06/26/2025 19:45:50",
      "content": "<p>Thanks for the reminder, for the submission time, do you upload the pretrained model and it still take a few hours?</p>",
      "rawMarkdown": "Thanks for the reminder, for the submission time, do you upload the pretrained model and it still take a few hours?",
      "votes": null
    },
    {
      "id": "3233462",
      "postDate": "06/26/2025 19:57:44",
      "content": "<p>yes, it does. The good news is that it still completes as long as the first response is received within the time window.</p>",
      "rawMarkdown": "yes, it does. The good news is that it still completes as long as the first response is received within the time window.",
      "votes": null
    },
    {
      "id": "3236140",
      "postDate": "06/30/2025 02:40:10",
      "content": "<p>Thanks for the tips.  In the evaluation API, we have</p>\n<p><code>def predict(sequence: pl.DataFrame, demographics: pl.DataFrame)</code></p>\n<p>Is it true that </p>\n<p>(1) for every call of <code>predict</code>, the inputs <code>sequence</code> and <code>demographics</code> are structurally the same as the sample test.csv and test_demographics.csv provided, but just in Polars dataframe format?</p>\n<p>(2) the input <code>sequence</code> contains exactly one (not more, not less) unique sequence ID whose subject person is guaranteed to be found in the input <code>demographics</code>?</p>\n<p>Thank you.</p>",
      "rawMarkdown": "Thanks for the tips.  In the evaluation API, we have\n\n`def predict(sequence: pl.DataFrame, demographics: pl.DataFrame)`\n\nIs it true that \n\n(1) for every call of `predict`, the inputs `sequence` and `demographics` are structurally the same as the sample test.csv and test_demographics.csv provided, but just in Polars dataframe format?\n\n(2) the input `sequence` contains exactly one (not more, not less) unique sequence ID whose subject person is guaranteed to be found in the input `demographics`?\n\nThank you.",
      "votes": null
    },
    {
      "id": "3236147",
      "postDate": "06/30/2025 02:49:27",
      "content": "<p>Yes and yes</p>",
      "rawMarkdown": "Yes and yes",
      "votes": null
    },
    {
      "id": "3251841",
      "postDate": "07/21/2025 16:47:19",
      "content": "<p>Hello Frederic.  </p>\n<p>I'm new to this kind of challenges. I have a question.</p>\n<p>Does my notebook run for every sequence that runs on the inference server?</p>",
      "rawMarkdown": "Hello Frederic.  \n\nI'm new to this kind of challenges. I have a question.\n\nDoes my notebook run for every sequence that runs on the inference server?",
      "votes": null
    },
    {
      "id": "3252364",
      "postDate": "07/22/2025 14:46:40",
      "content": "<p>no it runs one time. once it is finished, the predict function is called several thousand times. </p>",
      "rawMarkdown": "no it runs one time. once it is finished, the predict function is called several thousand times.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3233457,
      "author_name": "georgewang73",
      "author_url": "",
      "post_date": "06/26/2025 19:45:50",
      "content": "<p>Thanks for the reminder, for the submission time, do you upload the pretrained model and it still take a few hours?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3233462,
          "author_name": "fredericnicholson",
          "author_url": "",
          "post_date": "06/26/2025 19:57:44",
          "content": "<p>yes, it does. The good news is that it still completes as long as the first response is received within the time window.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3236140,
      "author_name": "revealer",
      "author_url": "",
      "post_date": "06/30/2025 02:40:10",
      "content": "<p>Thanks for the tips.  In the evaluation API, we have</p>\n<p><code>def predict(sequence: pl.DataFrame, demographics: pl.DataFrame)</code></p>\n<p>Is it true that </p>\n<p>(1) for every call of <code>predict</code>, the inputs <code>sequence</code> and <code>demographics</code> are structurally the same as the sample test.csv and test_demographics.csv provided, but just in Polars dataframe format?</p>\n<p>(2) the input <code>sequence</code> contains exactly one (not more, not less) unique sequence ID whose subject person is guaranteed to be found in the input <code>demographics</code>?</p>\n<p>Thank you.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3236147,
          "author_name": "jackvd",
          "author_url": "",
          "post_date": "06/30/2025 02:49:27",
          "content": "<p>Yes and yes</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3251841,
      "author_name": "pluceroo",
      "author_url": "",
      "post_date": "07/21/2025 16:47:19",
      "content": "<p>Hello Frederic.  </p>\n<p>I'm new to this kind of challenges. I have a question.</p>\n<p>Does my notebook run for every sequence that runs on the inference server?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3252364,
          "author_name": "fredericnicholson",
          "author_url": "",
          "post_date": "07/22/2025 14:46:40",
          "content": "<p>no it runs one time. once it is finished, the predict function is called several thousand times. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3233330": "Here are some points I learned so far that I want to share with you:\n\n- The submission process mandates running the submission server with the predict function. \n-  The predict function will be called many times during scoring, one time for each sequence. \n- if your predict function produces one wrong response, then the submission will be considered invalid.  \n- The Leaderboard score is not calculated using the text.csv data, the data remains completely hidden.\n- Any print statements during submission will not be visible in your output or logs. \n- During scoring the submission server is not running in your environment. Every file you access must be imported as \"Input\" \n-  There is a 900 sec timer for starting your submission server after the notebook was activated. \n-  Another timer ensures the response to the first sequence is received within 20 minutes(?)\n- The submission itself can actually run a long time; I have submissions running for several hours before they finished and created a new result.  Note that the notebook run is already completed at that time.   \n \n  please let me know we need to add anything.",
    "3233457": "Thanks for the reminder, for the submission time, do you upload the pretrained model and it still take a few hours?",
    "3233462": "yes, it does. The good news is that it still completes as long as the first response is received within the time window.",
    "3236140": "Thanks for the tips.  In the evaluation API, we have\n\n`def predict(sequence: pl.DataFrame, demographics: pl.DataFrame)`\n\nIs it true that \n\n(1) for every call of `predict`, the inputs `sequence` and `demographics` are structurally the same as the sample test.csv and test_demographics.csv provided, but just in Polars dataframe format?\n\n(2) the input `sequence` contains exactly one (not more, not less) unique sequence ID whose subject person is guaranteed to be found in the input `demographics`?\n\nThank you.",
    "3236147": "Yes and yes",
    "3251841": "Hello Frederic.  \n\nI'm new to this kind of challenges. I have a question.\n\nDoes my notebook run for every sequence that runs on the inference server?",
    "3252364": "no it runs one time. once it is finished, the predict function is called several thousand times."
  },
  "source": "meta"
}