{
  "id": 441370,
  "title": "I keep encountering a \"Submission scoring error\"!",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/441370",
  "author_name": "",
  "post_date": "2023-09-18T13:20:35.401219600Z",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I wanted to test a simple LSTM model, so I inferred using the LSTM model and got the results. I submitted a submission.csv file with 336 rows, which I inferred to have events as stated from the test data,</p>\n<table>\n<thead>\n<tr>\n<th>row_id</th>\n<th>series_id</th>\n<th>step</th>\n<th>event</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>10</td>\n<td>038441c925bb</td>\n<td>10</td>\n<td>wakeup</td>\n<td>0.60284287</td>\n</tr>\n<tr>\n<td>12</td>\n<td>038441c925bb</td>\n<td>12</td>\n<td>onset</td>\n<td>0.69078106</td>\n</tr>\n<tr>\n<td>14</td>\n<td>038441c925bb</td>\n<td>14</td>\n<td>wakeup</td>\n<td>0.5251346</td>\n</tr>\n<tr>\n<td>17</td>\n<td>038441c925bb</td>\n<td>17</td>\n<td>wakeup</td>\n<td>0.44773376</td>\n</tr>\n</tbody>\n</table>\n<p>but I encountered a \"Submission scoring error\" message. I thought the issue might be due to the number of rows not matching the test data, so I created a submission file with the same number of rows as the test data, leaving the event value empty if no event was inferred. </p>\n<table>\n<thead>\n<tr>\n<th>row_id</th>\n<th>series_id</th>\n<th>step</th>\n<th>event</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>11</td>\n<td>038441c925bb</td>\n<td>11</td>\n<td></td>\n<td>0.9871346</td>\n</tr>\n<tr>\n<td>12</td>\n<td>038441c925bb</td>\n<td>12</td>\n<td>onset</td>\n<td>0.77261764</td>\n</tr>\n<tr>\n<td>13</td>\n<td>038441c925bb</td>\n<td>13</td>\n<td></td>\n<td>0.9830816</td>\n</tr>\n<tr>\n<td>14</td>\n<td>038441c925bb</td>\n<td>14</td>\n<td>wakeup</td>\n<td>0.5745813</td>\n</tr>\n</tbody>\n</table>\n<p>However, I still keep getting the error. Do you have any idea why this might be happening?</p>",
  "messages": [
    {
      "id": "2444769",
      "postDate": "09/18/2023 13:20:35",
      "content": "<p>I wanted to test a simple LSTM model, so I inferred using the LSTM model and got the results. I submitted a submission.csv file with 336 rows, which I inferred to have events as stated from the test data,</p>\n<table>\n<thead>\n<tr>\n<th>row_id</th>\n<th>series_id</th>\n<th>step</th>\n<th>event</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>10</td>\n<td>038441c925bb</td>\n<td>10</td>\n<td>wakeup</td>\n<td>0.60284287</td>\n</tr>\n<tr>\n<td>12</td>\n<td>038441c925bb</td>\n<td>12</td>\n<td>onset</td>\n<td>0.69078106</td>\n</tr>\n<tr>\n<td>14</td>\n<td>038441c925bb</td>\n<td>14</td>\n<td>wakeup</td>\n<td>0.5251346</td>\n</tr>\n<tr>\n<td>17</td>\n<td>038441c925bb</td>\n<td>17</td>\n<td>wakeup</td>\n<td>0.44773376</td>\n</tr>\n</tbody>\n</table>\n<p>but I encountered a \"Submission scoring error\" message. I thought the issue might be due to the number of rows not matching the test data, so I created a submission file with the same number of rows as the test data, leaving the event value empty if no event was inferred. </p>\n<table>\n<thead>\n<tr>\n<th>row_id</th>\n<th>series_id</th>\n<th>step</th>\n<th>event</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>11</td>\n<td>038441c925bb</td>\n<td>11</td>\n<td></td>\n<td>0.9871346</td>\n</tr>\n<tr>\n<td>12</td>\n<td>038441c925bb</td>\n<td>12</td>\n<td>onset</td>\n<td>0.77261764</td>\n</tr>\n<tr>\n<td>13</td>\n<td>038441c925bb</td>\n<td>13</td>\n<td></td>\n<td>0.9830816</td>\n</tr>\n<tr>\n<td>14</td>\n<td>038441c925bb</td>\n<td>14</td>\n<td>wakeup</td>\n<td>0.5745813</td>\n</tr>\n</tbody>\n</table>\n<p>However, I still keep getting the error. Do you have any idea why this might be happening?</p>",
      "rawMarkdown": "I wanted to test a simple LSTM model, so I inferred using the LSTM model and got the results. I submitted a submission.csv file with 336 rows, which I inferred to have events as stated from the test data,\n\n\n|row_id|series_id|step|event|score|\n| --- | --- | --- | --- | --- | \n|10|038441c925bb|10|wakeup|0.60284287|\n|12|038441c925bb|12|onset|0.69078106|\n|14|038441c925bb|14|wakeup|0.5251346|\n|17|038441c925bb|17|wakeup|0.44773376|\n\nbut I encountered a \"Submission scoring error\" message. I thought the issue might be due to the number of rows not matching the test data, so I created a submission file with the same number of rows as the test data, leaving the event value empty if no event was inferred. \n\n|row_id|series_id|step|event|score|\n| --- | --- | --- | --- | --- | \n|11|038441c925bb\t|11|\t|0.9871346|\n|12|038441c925bb|12|onset\t|0.77261764|\n|13|038441c925bb|13| |0.9830816|\n|14|038441c925bb|14|wakeup|0.5745813\n\nHowever, I still keep getting the error. Do you have any idea why this might be happening?",
      "votes": null
    },
    {
      "id": "2444849",
      "postDate": "09/18/2023 14:09:28",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/skihyeon\" target=\"_blank\">@skihyeon</a> based on my similar experience, I got the error message when my submission contains null. In general, I think the cause is everything that makes the event detection AP function call in competition metric run unsuccessfully, such as submission contains null, unmatched number of rows, etc CMIIW.</p>",
      "rawMarkdown": "Hi @skihyeon based on my similar experience, I got the error message when my submission contains null. In general, I think the cause is everything that makes the event detection AP function call in competition metric run unsuccessfully, such as submission contains null, unmatched number of rows, etc CMIIW.",
      "votes": null
    },
    {
      "id": "2444912",
      "postDate": "09/18/2023 14:39:01",
      "content": "<p>To eliminate null values, I initially removed rows with null events. If I understood <a href=\"https://www.kaggle.com/Rimba\" target=\"_blank\">@Rimba</a> Erlangga's answer correctly, the number of rows in the submission should match the test series, and there shouldn't be any null values. Does this mean that every event in the test series has to be either 'onset' or 'wakeup'?</p>",
      "rawMarkdown": "To eliminate null values, I initially removed rows with null events. If I understood @Rimba Erlangga's answer correctly, the number of rows in the submission should match the test series, and there shouldn't be any null values. Does this mean that every event in the test series has to be either 'onset' or 'wakeup'?",
      "votes": null
    },
    {
      "id": "2444943",
      "postDate": "09/18/2023 14:49:23",
      "content": "<p><a href=\"https://www.kaggle.com/skihyeon\" target=\"_blank\">@skihyeon</a> I revised my answer: unmatched number of rows may or may not be the cause of this. For your second case (leaving event name as empty) might cause the <code>event_detection_ap(solution, submission, ...)</code> call during your submission throw an error.</p>\n<p>Again, in general, try experimenting with the competition metric notebook in the overview page, to get information why your solution throws a submission scoring error.</p>",
      "rawMarkdown": "skihyeon I revised my answer: unmatched number of rows may or may not be the cause of this. For your second case (leaving event name as empty) might cause the `event_detection_ap(solution, submission, ...)` call during your submission throw an error.\n\nAgain, in general, try experimenting with the competition metric notebook in the overview page, to get information why your solution throws a submission scoring error.",
      "votes": null
    },
    {
      "id": "2444973",
      "postDate": "09/18/2023 15:05:24",
      "content": "<p>Thank you for the advice! I'll try various experiments.</p>",
      "rawMarkdown": "Thank you for the advice! I'll try various experiments.",
      "votes": null
    },
    {
      "id": "2445071",
      "postDate": "09/18/2023 15:52:38",
      "content": "<p>I think you are submitting predictions for the test file provided as a part of dataset, but to get your submission scored,  your code should also have inference logic. Your code is then executed on private data which has the same name as public test set and then scored. That is the reason you are having a mismatch in number of rows.</p>",
      "rawMarkdown": "I think you are submitting predictions for the test file provided as a part of dataset, but to get your submission scored,  your code should also have inference logic. Your code is then executed on private data which has the same name as public test set and then scored. That is the reason you are having a mismatch in number of rows.",
      "votes": null
    },
    {
      "id": "2445334",
      "postDate": "09/18/2023 18:32:54",
      "content": "<p>I see now. In my notebook, I was importing the provided test_series.parquet with a different name. I'll try a different approach. Thank you.</p>",
      "rawMarkdown": "I see now. In my notebook, I was importing the provided test_series.parquet with a different name. I'll try a different approach. Thank you.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2444849,
      "author_name": "rimbax",
      "author_url": "",
      "post_date": "09/18/2023 14:09:28",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/skihyeon\" target=\"_blank\">@skihyeon</a> based on my similar experience, I got the error message when my submission contains null. In general, I think the cause is everything that makes the event detection AP function call in competition metric run unsuccessfully, such as submission contains null, unmatched number of rows, etc CMIIW.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2444912,
          "author_name": "skihyeon",
          "author_url": "",
          "post_date": "09/18/2023 14:39:01",
          "content": "<p>To eliminate null values, I initially removed rows with null events. If I understood <a href=\"https://www.kaggle.com/Rimba\" target=\"_blank\">@Rimba</a> Erlangga's answer correctly, the number of rows in the submission should match the test series, and there shouldn't be any null values. Does this mean that every event in the test series has to be either 'onset' or 'wakeup'?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2444943,
          "author_name": "rimbax",
          "author_url": "",
          "post_date": "09/18/2023 14:49:23",
          "content": "<p><a href=\"https://www.kaggle.com/skihyeon\" target=\"_blank\">@skihyeon</a> I revised my answer: unmatched number of rows may or may not be the cause of this. For your second case (leaving event name as empty) might cause the <code>event_detection_ap(solution, submission, ...)</code> call during your submission throw an error.</p>\n<p>Again, in general, try experimenting with the competition metric notebook in the overview page, to get information why your solution throws a submission scoring error.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2444973,
              "author_name": "skihyeon",
              "author_url": "",
              "post_date": "09/18/2023 15:05:24",
              "content": "<p>Thank you for the advice! I'll try various experiments.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2445071,
      "author_name": "anilreddyvv",
      "author_url": "",
      "post_date": "09/18/2023 15:52:38",
      "content": "<p>I think you are submitting predictions for the test file provided as a part of dataset, but to get your submission scored,  your code should also have inference logic. Your code is then executed on private data which has the same name as public test set and then scored. That is the reason you are having a mismatch in number of rows.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2445334,
          "author_name": "skihyeon",
          "author_url": "",
          "post_date": "09/18/2023 18:32:54",
          "content": "<p>I see now. In my notebook, I was importing the provided test_series.parquet with a different name. I'll try a different approach. Thank you.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2444769": "I wanted to test a simple LSTM model, so I inferred using the LSTM model and got the results. I submitted a submission.csv file with 336 rows, which I inferred to have events as stated from the test data,\n\n\n|row_id|series_id|step|event|score|\n| --- | --- | --- | --- | --- | \n|10|038441c925bb|10|wakeup|0.60284287|\n|12|038441c925bb|12|onset|0.69078106|\n|14|038441c925bb|14|wakeup|0.5251346|\n|17|038441c925bb|17|wakeup|0.44773376|\n\nbut I encountered a \"Submission scoring error\" message. I thought the issue might be due to the number of rows not matching the test data, so I created a submission file with the same number of rows as the test data, leaving the event value empty if no event was inferred. \n\n|row_id|series_id|step|event|score|\n| --- | --- | --- | --- | --- | \n|11|038441c925bb\t|11|\t|0.9871346|\n|12|038441c925bb|12|onset\t|0.77261764|\n|13|038441c925bb|13| |0.9830816|\n|14|038441c925bb|14|wakeup|0.5745813\n\nHowever, I still keep getting the error. Do you have any idea why this might be happening?",
    "2444849": "Hi @skihyeon based on my similar experience, I got the error message when my submission contains null. In general, I think the cause is everything that makes the event detection AP function call in competition metric run unsuccessfully, such as submission contains null, unmatched number of rows, etc CMIIW.",
    "2444912": "To eliminate null values, I initially removed rows with null events. If I understood @Rimba Erlangga's answer correctly, the number of rows in the submission should match the test series, and there shouldn't be any null values. Does this mean that every event in the test series has to be either 'onset' or 'wakeup'?",
    "2444943": "skihyeon I revised my answer: unmatched number of rows may or may not be the cause of this. For your second case (leaving event name as empty) might cause the `event_detection_ap(solution, submission, ...)` call during your submission throw an error.\n\nAgain, in general, try experimenting with the competition metric notebook in the overview page, to get information why your solution throws a submission scoring error.",
    "2444973": "Thank you for the advice! I'll try various experiments.",
    "2445071": "I think you are submitting predictions for the test file provided as a part of dataset, but to get your submission scored,  your code should also have inference logic. Your code is then executed on private data which has the same name as public test set and then scored. That is the reason you are having a mismatch in number of rows.",
    "2445334": "I see now. In my notebook, I was importing the provided test_series.parquet with a different name. I'll try a different approach. Thank you."
  },
  "source": "meta"
}