{ "id": 243170, "title": "Doubt regarding private leaderboard testing", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/243170", "author_name": "", "post_date": "2021-06-01T12:19:53.664643800Z", "votes": null, "comment_count": 4, "views": 0, "content": "

@ernestogimeno
\n@philculliton
\n@haritzpuerto

\n

I am confused that whether the final 88% of test data will have the same 9hrs running limit during evaluation. Can somebody help me out with this?

", "messages": [ { "id": "1331357", "postDate": "06/01/2021 12:19:53", "content": "

@ernestogimeno
\n@philculliton
\n@haritzpuerto

\n

I am confused that whether the final 88% of test data will have the same 9hrs running limit during evaluation. Can somebody help me out with this?

", "rawMarkdown": "ernestogimeno \n@philculliton \n@haritzpuerto\n\n I am confused that whether the final 88% of test data will have the same 9hrs running limit during evaluation. Can somebody help me out with this?", "votes": null }, { "id": "1331395", "postDate": "06/01/2021 12:44:09", "content": "

9h limit is for the whole test set (public and private).
\nGood thing is when you submit, if you receive a valid public score, it means that you are within limits and you do also already have a valid private score that will be disclosed at the end of the competition. In some extreme circumstances, this direct relationship (having a public score => having a valid PB score) might not be valid but, overall this should be valid.

\n

A small hint on if you will be within limits: Total test size (private and public) is a little above 8000 publications. This gives around 4 seconds processing time for each publication (9 hours x 60 min x 60 sec / 8000). So for example for the sample test set of 4 publications, if your algorithm can process those 4 pubs. in less than 16 (upto around 20) sec, you should be OK.

", "rawMarkdown": "9h limit is for the whole test set (public and private).\nGood thing is when you submit, if you receive a valid public score, it means that you are within limits and you do also already have a valid private score that will be disclosed at the end of the competition. In some extreme circumstances, this direct relationship (having a public score => having a valid PB score) might not be valid but, overall this should be valid.\n\n**A small hint on if you will be within limits:** Total test size (private and public) is a little above 8000 publications. This gives around 4 seconds processing time for each publication (9 hours x 60 min x 60 sec / 8000). So for example for the sample test set of 4 publications, if your algorithm can process those 4 pubs. in less than 16 (upto around 20) sec, you should be OK.", "votes": null }, { "id": "1331430", "postDate": "06/01/2021 13:15:09", "content": "

@abdulkadirguner thanks for the help

", "rawMarkdown": "abdulkadirguner thanks for the help", "votes": null }, { "id": "1342505", "postDate": "06/09/2021 14:07:25", "content": "

@abdulkadirguner Do you mean, after submitting the code runs on both private and public datasets! In other words, once the code is submitted, it runs on both public and private datasets, and the final time which is shown is the total time it is taking, which must be less than 9hours?
\nPls verify, @ernestogimeno
\n@philculliton
\n@haritzpuerto
\n@julialane
\n@maggiemd

", "rawMarkdown": "abdulkadirguner Do you mean, after submitting the code runs on both private and public datasets! In other words, once the code is submitted, it runs on both public and private datasets, and the final time which is shown is the total time it is taking, which must be less than 9hours?\nPls verify, @ernestogimeno\n@philculliton\n@haritzpuerto \n@julialane\n@maggiemd", "votes": null }, { "id": "1342612", "postDate": "06/09/2021 15:36:53", "content": "

Yes, when you 'SUBMIT', the test folder is replaced with the real test folder with all the test files (exact mechanisms might vary but, that is what basically happens). And if you successfully get a valid 'public leader board score' you can be 99 percent safe that your code has run on all test data and you have also a private leader board score.
\nIn submission, 'and unlike run/commit stage', there is no cue about time unless you keep track of when you have submitted,, and when you see the score in your 'my submissions' page.

", "rawMarkdown": "Yes, when you 'SUBMIT', the test folder is replaced with the real test folder with all the test files (exact mechanisms might vary but, that is what basically happens). And **if you successfully get a valid 'public leader board score'** you can be 99 percent safe that your code has run on all test data and you have also a private leader board score. \nIn submission, 'and unlike run/commit stage', there is no cue about time unless you keep track of when you have submitted,, and when you see the score in your 'my submissions' page.", "votes": null } ], "comments": [ { "id": 1331395, "author_name": "abdulkadirguner", "author_url": "", "post_date": "06/01/2021 12:44:09", "content": "

9h limit is for the whole test set (public and private).
\nGood thing is when you submit, if you receive a valid public score, it means that you are within limits and you do also already have a valid private score that will be disclosed at the end of the competition. In some extreme circumstances, this direct relationship (having a public score => having a valid PB score) might not be valid but, overall this should be valid.

\n

A small hint on if you will be within limits: Total test size (private and public) is a little above 8000 publications. This gives around 4 seconds processing time for each publication (9 hours x 60 min x 60 sec / 8000). So for example for the sample test set of 4 publications, if your algorithm can process those 4 pubs. in less than 16 (upto around 20) sec, you should be OK.

", "votes": null, "replies": [] }, { "id": 1331430, "author_name": "dark17", "author_url": "", "post_date": "06/01/2021 13:15:09", "content": "

@abdulkadirguner thanks for the help

", "votes": null, "replies": [] }, { "id": 1342505, "author_name": "sandeepkumariitp", "author_url": "", "post_date": "06/09/2021 14:07:25", "content": "

@abdulkadirguner Do you mean, after submitting the code runs on both private and public datasets! In other words, once the code is submitted, it runs on both public and private datasets, and the final time which is shown is the total time it is taking, which must be less than 9hours?
\nPls verify, @ernestogimeno
\n@philculliton
\n@haritzpuerto
\n@julialane
\n@maggiemd

", "votes": null, "replies": [ { "id": 1342612, "author_name": "abdulkadirguner", "author_url": "", "post_date": "06/09/2021 15:36:53", "content": "

Yes, when you 'SUBMIT', the test folder is replaced with the real test folder with all the test files (exact mechanisms might vary but, that is what basically happens). And if you successfully get a valid 'public leader board score' you can be 99 percent safe that your code has run on all test data and you have also a private leader board score.
\nIn submission, 'and unlike run/commit stage', there is no cue about time unless you keep track of when you have submitted,, and when you see the score in your 'my submissions' page.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1331357": "ernestogimeno \n@philculliton \n@haritzpuerto\n\n I am confused that whether the final 88% of test data will have the same 9hrs running limit during evaluation. Can somebody help me out with this?", "1331395": "9h limit is for the whole test set (public and private).\nGood thing is when you submit, if you receive a valid public score, it means that you are within limits and you do also already have a valid private score that will be disclosed at the end of the competition. In some extreme circumstances, this direct relationship (having a public score => having a valid PB score) might not be valid but, overall this should be valid.\n\n**A small hint on if you will be within limits:** Total test size (private and public) is a little above 8000 publications. This gives around 4 seconds processing time for each publication (9 hours x 60 min x 60 sec / 8000). So for example for the sample test set of 4 publications, if your algorithm can process those 4 pubs. in less than 16 (upto around 20) sec, you should be OK.", "1331430": "abdulkadirguner thanks for the help", "1342505": "abdulkadirguner Do you mean, after submitting the code runs on both private and public datasets! In other words, once the code is submitted, it runs on both public and private datasets, and the final time which is shown is the total time it is taking, which must be less than 9hours?\nPls verify, @ernestogimeno\n@philculliton\n@haritzpuerto \n@julialane\n@maggiemd", "1342612": "Yes, when you 'SUBMIT', the test folder is replaced with the real test folder with all the test files (exact mechanisms might vary but, that is what basically happens). And **if you successfully get a valid 'public leader board score'** you can be 99 percent safe that your code has run on all test data and you have also a private leader board score. \nIn submission, 'and unlike run/commit stage', there is no cue about time unless you keep track of when you have submitted,, and when you see the score in your 'my submissions' page." }, "source": "meta" }