{ "id": 437209, "title": "Code for evaluation ", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/437209", "author_name": "", "post_date": "2023-09-05T21:48:07.453121600Z", "votes": 16, "comment_count": 7, "views": 0, "content": "

Hi,

\n
   May I ask  we can   code  evaluating  prediction? The evaluation description doesn't seem very clear  . I\n\n    In particular, I'm  sure what  we should predict. It seems like  score  within  range [,]   sample submission,  's  clearly stated. In addition, what    confuses  a bit.\n\n  I think  may be helpful  provide us  a code template  how our predictions are evaluated.\n\n  Thanks!\n
", "messages": [ { "id": "2425409", "postDate": "09/05/2023 21:48:07", "content": "

Hi,

\n
   May I ask  we can   code  evaluating  prediction? The evaluation description doesn't seem very clear  . I\n\n    In particular, I'm  sure what  we should predict. It seems like  score  within  range [,]   sample submission,  's  clearly stated. In addition, what    confuses  a bit.\n\n  I think  may be helpful  provide us  a code template  how our predictions are evaluated.\n\n  Thanks!\n
", "rawMarkdown": "Hi,\n\n May I ask if we can get the code for evaluating the prediction? The evaluation description doesn't seem very clear to me. I\n\n In particular, I'm not sure what \"score\" we should predict. It seems like the score is within the range [0,1] from the sample submission, but it's not clearly stated. In addition, what the \"error threshold\" is confuses me a bit.\n\n I think it may be helpful to provide us with a code template of how our predictions are evaluated.\n\n Thanks!", "votes": null }, { "id": "2425698", "postDate": "09/06/2023 05:47:54", "content": "

Dear @wentinglu

\n

The score could well correspond to our classifier probability for awake/asleep transition?

\n

Allthe best,
\ncarl

", "rawMarkdown": "Dear @wentinglu \n\nThe `score` could well correspond to our classifier probability for awake/asleep transition?\n\nAllthe best,\ncarl", "votes": null }, { "id": "2426289", "postDate": "09/06/2023 14:30:08", "content": "

I second this! Ofc, we can all recreate it individually, but why spend time on an activity that brings exactly zero value to the organizers?

", "rawMarkdown": "I second this! Ofc, we can all recreate it individually, but why spend time on an activity that brings exactly zero value to the organizers?", "votes": null }, { "id": "2426311", "postDate": "09/06/2023 14:42:40", "content": "

Hi @wentinglu, I just added a link to the code for the metric to the Evaluation documentation. The code is here: Event Detection AP. For reference, this metric is nearly the same as that used in the DFL competition.

\n

The score is a confidence score, just like used in the average precision and aucroc metrics in sklearn. The metric only considers the ordering of the scores, but you could think about it as a probability in [0, 1], if you like.

", "rawMarkdown": "Hi @wentinglu, I just added a link to the code for the metric to the Evaluation documentation. The code is here: [**Event Detection AP**](https://www.kaggle.com/code/metric/event-detection-ap/notebook). For reference, this metric is nearly the same as that used in the [DFL competition](https://www.kaggle.com/c/dfl-bundesliga-data-shootout).\n\nThe `score` is a confidence score, just like used in the average precision and aucroc metrics in sklearn. The metric only considers the ordering of the scores, but you could think about it as a probability in [0, 1], if you like.", "votes": null }, { "id": "2429748", "postDate": "09/08/2023 19:14:14", "content": "

Dear @ryanholbrook

\n

\n

All the best,
\ncarl

\n

Update: I have just learnt that one can attach notebooks to notebooks

", "rawMarkdown": "Dear @ryanholbrook \n\n~~Would it be possible to perhaps add the code from the [Event Detection AP](https://www.kaggle.com/code/metric/event-detection-ap/notebook) to the competition data as, say, the file `Event_Detection_AP.py` that we could then `import` and then use for our local testing without adding all of the (rather long) metric source to the actual testing script?~~\n\nAll the best,\ncarl \n\n**Update:** [I have just learnt that one can attach notebooks to notebooks](https://www.kaggle.com/discussions/product-feedback/438846#2435227)", "votes": null }, { "id": "2429779", "postDate": "09/08/2023 19:48:16", "content": "

Thanks a lot! It really helps!

", "rawMarkdown": "Thanks a lot! It really helps!", "votes": null }, { "id": "2435345", "postDate": "09/12/2023 21:43:07", "content": "

Hi, so in sklearn as far as I know the \"score\" is the probability of the positive class but in the example you guys gave us both \"onset\" and \"wakeup\" are give a score of \"0\" which does not make sense. Hope you can clear this up for us.

", "rawMarkdown": "Hi, so in sklearn as far as I know the \"score\" is the probability of the positive class but in the example you guys gave us both \"onset\" and \"wakeup\" are give a score of \"0\" which does not make sense. Hope you can clear this up for us.", "votes": null }, { "id": "2479891", "postDate": "10/13/2023 01:24:33", "content": "

Hello! Сan you please tell me in what form you need to send an answer, what should the code look like? I'm trying to download and it gives errors. Can you please send the code that gives the answer (you can just a piece that writes a csv file) and the file itself that turns out, the answers can be made incorrect, but so that the format is such that kaggle accepts this code

", "rawMarkdown": "Hello! Сan you please tell me in what form you need to send an answer, what should the code look like? I'm trying to download and it gives errors. Can you please send the code that gives the answer (you can just a piece that writes a csv file) and the file itself that turns out, the answers can be made incorrect, but so that the format is such that kaggle accepts this code", "votes": null } ], "comments": [ { "id": 2425698, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "09/06/2023 05:47:54", "content": "

Dear @wentinglu

\n

The score could well correspond to our classifier probability for awake/asleep transition?

\n

Allthe best,
\ncarl

", "votes": null, "replies": [] }, { "id": 2426289, "author_name": "narsil", "author_url": "", "post_date": "09/06/2023 14:30:08", "content": "

I second this! Ofc, we can all recreate it individually, but why spend time on an activity that brings exactly zero value to the organizers?

", "votes": null, "replies": [] }, { "id": 2426311, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/06/2023 14:42:40", "content": "

Hi @wentinglu, I just added a link to the code for the metric to the Evaluation documentation. The code is here: Event Detection AP. For reference, this metric is nearly the same as that used in the DFL competition.

\n

The score is a confidence score, just like used in the average precision and aucroc metrics in sklearn. The metric only considers the ordering of the scores, but you could think about it as a probability in [0, 1], if you like.

", "votes": null, "replies": [ { "id": 2429748, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "09/08/2023 19:14:14", "content": "

Dear @ryanholbrook

\n

\n

All the best,
\ncarl

\n

Update: I have just learnt that one can attach notebooks to notebooks

", "votes": null, "replies": [] }, { "id": 2429779, "author_name": "wentinglu", "author_url": "", "post_date": "09/08/2023 19:48:16", "content": "

Thanks a lot! It really helps!

", "votes": null, "replies": [] }, { "id": 2435345, "author_name": "g0lski", "author_url": "", "post_date": "09/12/2023 21:43:07", "content": "

Hi, so in sklearn as far as I know the \"score\" is the probability of the positive class but in the example you guys gave us both \"onset\" and \"wakeup\" are give a score of \"0\" which does not make sense. Hope you can clear this up for us.

", "votes": null, "replies": [] }, { "id": 2479891, "author_name": "rinatbayanov", "author_url": "", "post_date": "10/13/2023 01:24:33", "content": "

Hello! Сan you please tell me in what form you need to send an answer, what should the code look like? I'm trying to download and it gives errors. Can you please send the code that gives the answer (you can just a piece that writes a csv file) and the file itself that turns out, the answers can be made incorrect, but so that the format is such that kaggle accepts this code

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2425409": "Hi,\n\n May I ask if we can get the code for evaluating the prediction? The evaluation description doesn't seem very clear to me. I\n\n In particular, I'm not sure what \"score\" we should predict. It seems like the score is within the range [0,1] from the sample submission, but it's not clearly stated. In addition, what the \"error threshold\" is confuses me a bit.\n\n I think it may be helpful to provide us with a code template of how our predictions are evaluated.\n\n Thanks!", "2425698": "Dear @wentinglu \n\nThe `score` could well correspond to our classifier probability for awake/asleep transition?\n\nAllthe best,\ncarl", "2426289": "I second this! Ofc, we can all recreate it individually, but why spend time on an activity that brings exactly zero value to the organizers?", "2426311": "Hi @wentinglu, I just added a link to the code for the metric to the Evaluation documentation. The code is here: [**Event Detection AP**](https://www.kaggle.com/code/metric/event-detection-ap/notebook). For reference, this metric is nearly the same as that used in the [DFL competition](https://www.kaggle.com/c/dfl-bundesliga-data-shootout).\n\nThe `score` is a confidence score, just like used in the average precision and aucroc metrics in sklearn. The metric only considers the ordering of the scores, but you could think about it as a probability in [0, 1], if you like.", "2429748": "Dear @ryanholbrook \n\n~~Would it be possible to perhaps add the code from the [Event Detection AP](https://www.kaggle.com/code/metric/event-detection-ap/notebook) to the competition data as, say, the file `Event_Detection_AP.py` that we could then `import` and then use for our local testing without adding all of the (rather long) metric source to the actual testing script?~~\n\nAll the best,\ncarl \n\n**Update:** [I have just learnt that one can attach notebooks to notebooks](https://www.kaggle.com/discussions/product-feedback/438846#2435227)", "2429779": "Thanks a lot! It really helps!", "2435345": "Hi, so in sklearn as far as I know the \"score\" is the probability of the positive class but in the example you guys gave us both \"onset\" and \"wakeup\" are give a score of \"0\" which does not make sense. Hope you can clear this up for us.", "2479891": "Hello! Сan you please tell me in what form you need to send an answer, what should the code look like? I'm trying to download and it gives errors. Can you please send the code that gives the answer (you can just a piece that writes a csv file) and the file itself that turns out, the answers can be made incorrect, but so that the format is such that kaggle accepts this code" }, "source": "meta" }