Like many people, I've seen both my selected submissions mysteriously generate \"Timeout exceeded\" errors. This is really disappointing, since I don't believe either submission came close to the time limit when originally submitted. The .tsv files are respectively 1.78 and 1.96 GB, which is large but I don't think either particularly unusual or unreasonable for this competition. However, this isn't just about my two somewhat hacked-together submissions, but about research groups who invested substantially more time and effort than I did and whose professional standing in the protein function prediction field is up for assessment in this competition. This isn't just a Kaggle competition, it's a scientific assessment of the current state of the protein function prediction field. The competition can hardly achieve that if entries are unpredictably and arbitrarily excluded for taking too long to score a test set whose size is unknown and not easily predictable.
\nIf the new test dataset is larger than the previous one, I would ask the organisers to increase the time limit commensurately with dataset size to give submissions that succeded previously the opportunity to be scored. I also note that this is not the final leaderboard, and there are more updates to come. Hopefully the experience of the current difficulties will make things easier for the organisers at subsequent updates (though it would still be good to see this one put right).
\nFinally, I understand that organising a competition like this is a pretty thankless task. I am very much aware that it takes a great deal of effort, and whatever my frustrations about the current situation, I do want to acknowledge the hard work that has gone in and continues to go into running this competition. I fully realise that the zeroing of so many submissions in this update wasn't foreseen or intended.
", "messages": [ { "id": "3432311", "postDate": "03/31/2026 11:56:24", "content": "Like many people, I've seen both my selected submissions mysteriously generate \"Timeout exceeded\" errors. This is really disappointing, since I don't believe either submission came close to the time limit when originally submitted. The .tsv files are respectively 1.78 and 1.96 GB, which is large but I don't think either particularly unusual or unreasonable for this competition. However, this isn't just about my two somewhat hacked-together submissions, but about research groups who invested substantially more time and effort than I did and whose professional standing in the protein function prediction field is up for assessment in this competition. This isn't just a Kaggle competition, it's a scientific assessment of the current state of the protein function prediction field. The competition can hardly achieve that if entries are unpredictably and arbitrarily excluded for taking too long to score a test set whose size is unknown and not easily predictable.
\nIf the new test dataset is larger than the previous one, I would ask the organisers to increase the time limit commensurately with dataset size to give submissions that succeded previously the opportunity to be scored. I also note that this is not the final leaderboard, and there are more updates to come. Hopefully the experience of the current difficulties will make things easier for the organisers at subsequent updates (though it would still be good to see this one put right).
\nFinally, I understand that organising a competition like this is a pretty thankless task. I am very much aware that it takes a great deal of effort, and whatever my frustrations about the current situation, I do want to acknowledge the hard work that has gone in and continues to go into running this competition. I fully realise that the zeroing of so many submissions in this update wasn't foreseen or intended.
", "rawMarkdown": "Like many people, I've seen both my selected submissions mysteriously generate \"Timeout exceeded\" errors. This is really disappointing, since I don't believe either submission came close to the time limit when originally submitted. The .tsv files are respectively 1.78 and 1.96 GB, which is large but I don't think either particularly unusual or unreasonable for this competition. However, this isn't just about my two somewhat hacked-together submissions, but about research groups who invested substantially more time and effort than I did and whose professional standing in the protein function prediction field is up for assessment in this competition. This isn't just a Kaggle competition, it's a scientific assessment of the current state of the protein function prediction field. The competition can hardly achieve that if entries are unpredictably and arbitrarily excluded for taking too long to score a test set whose size is unknown and not easily predictable.\n\nIf the new test dataset is larger than the previous one, I would ask the organisers to increase the time limit commensurately with dataset size to give submissions that succeded previously the opportunity to be scored. I also note that this is not the final leaderboard, and there are more updates to come. Hopefully the experience of the current difficulties will make things easier for the organisers at subsequent updates (though it would still be good to see this one put right).\n\nFinally, I understand that organising a competition like this is a pretty thankless task. I am very much aware that it takes a great deal of effort, and whatever my frustrations about the current situation, I do want to acknowledge the hard work that has gone in and continues to go into running this competition. I fully realise that the zeroing of so many submissions in this update wasn't foreseen or intended.", "votes": null }, { "id": "3432353", "postDate": "03/31/2026 13:35:28", "content": "Hi John, we understand the frustration. Please hang tight since the rescoring is still in progress. We acknowledge that a large number of submissions are getting timed out, so the Kaggle engineers are working on it. We also consider switching back to 0.01 threshold step to reduce computation time (our plan was to do 0.001 threshold step for the current update).
\nThe new test dataset has more than 2,000 proteins in the ground truth, which is substantially larger than the previous test set, hence the longer evaluation time.
", "rawMarkdown": "Hi John, we understand the frustration. Please hang tight since the rescoring is still in progress. We acknowledge that a large number of submissions are getting timed out, so the Kaggle engineers are working on it. We also consider switching back to 0.01 threshold step to reduce computation time (our plan was to do 0.001 threshold step for the current update). \n\nThe new test dataset has more than 2,000 proteins in the ground truth, which is substantially larger than the previous test set, hence the longer evaluation time.", "votes": null }, { "id": "3438336", "postDate": "04/09/2026 00:20:59", "content": "UPDATE (April 8, 9:25 PM ET): Mostly all submissions were restored. Now showing 0.370 and 387 entries. Thanks to An Phan and the Kaggle engineers for the fix. Leaving the original report below in case it helps debug similar cases.
\n--
\n\n", "rawMarkdown": "UPDATE (April 8, 9:25 PM ET): Mostly all submissions were restored. Now showing 0.370 and 387 entries. Thanks to An Phan and the Kaggle engineers for the fix. Leaving the original report below in case it helps debug similar cases. \n\n--\n>Just adding a data point to John's thread since I'm seeing a different failure mode that may help the engineers. I ended up position #62, public score 0.388, 394 valid submissions over the competition. Majority were TSV, some authored notebooks, some forked and edited. After rescoring, every single submission in my history it now shows \"Internal scoring error,\" not \"Timeout exceeded.\" My best rescored value came back at 0.016 (down from 0.388) before even that disappeared, and I'm now completely absent from the leaderboard. Not #62, not even last, I am not ranked. Flagging because \"Internal scoring error\" across my 394 submissions looks like a different bug than the timeout issue John is describing, even if they're both downstream of the same rescoring pass. If it helps, all my TSVs were submitted at 0.01 threshold step, so the 0.001→0.01 switch An Phan mentioned wouldn't change my case. Happy to provide submission IDs or collaborate in any way we can help. Thank you.", "votes": null }, { "id": "3438605", "postDate": "04/09/2026 12:23:49", "content": "Just adding a data point to John's thread since I'm seeing a different failure mode that may help the engineers. I ended up position #62, public score 0.388, 394 valid submissions over the competition. Majority were TSV, some authored notebooks, some forked and edited. After rescoring, every single submission in my history it now shows \"Internal scoring error,\" not \"Timeout exceeded.\" My best rescored value came back at 0.016 (down from 0.388) before even that disappeared, and I'm now completely absent from the leaderboard. Not #62, not even last, I am not ranked. Flagging because \"Internal scoring error\" across my 394 submissions looks like a different bug than the timeout issue John is describing, even if they're both downstream of the same rescoring pass. If it helps, all my TSVs were submitted at 0.01 threshold step, so the 0.001→0.01 switch An Phan mentioned wouldn't change my case. Happy to provide submission IDs or collaborate in any way we can help. Thank you.
\n
Thanks very much to the organisers for fixing this! The LB is looking much healthier now with more than 2000 teams having valid scores.
", "rawMarkdown": "Thanks very much to the organisers for fixing this! The LB is looking much healthier now with more than 2000 teams having valid scores.", "votes": null } ], "comments": [ { "id": 3432353, "author_name": "ahphan", "author_url": "", "post_date": "03/31/2026 13:35:28", "content": "Hi John, we understand the frustration. Please hang tight since the rescoring is still in progress. We acknowledge that a large number of submissions are getting timed out, so the Kaggle engineers are working on it. We also consider switching back to 0.01 threshold step to reduce computation time (our plan was to do 0.001 threshold step for the current update).
\nThe new test dataset has more than 2,000 proteins in the ground truth, which is substantially larger than the previous test set, hence the longer evaluation time.
", "votes": null, "replies": [ { "id": 3438336, "author_name": "santmani", "author_url": "", "post_date": "04/09/2026 00:20:59", "content": "UPDATE (April 8, 9:25 PM ET): Mostly all submissions were restored. Now showing 0.370 and 387 entries. Thanks to An Phan and the Kaggle engineers for the fix. Leaving the original report below in case it helps debug similar cases.
\n--
\n\n", "votes": null, "replies": [] } ] }, { "id": 3438605, "author_name": "jbomitchell", "author_url": "", "post_date": "04/09/2026 12:23:49", "content": "Just adding a data point to John's thread since I'm seeing a different failure mode that may help the engineers. I ended up position #62, public score 0.388, 394 valid submissions over the competition. Majority were TSV, some authored notebooks, some forked and edited. After rescoring, every single submission in my history it now shows \"Internal scoring error,\" not \"Timeout exceeded.\" My best rescored value came back at 0.016 (down from 0.388) before even that disappeared, and I'm now completely absent from the leaderboard. Not #62, not even last, I am not ranked. Flagging because \"Internal scoring error\" across my 394 submissions looks like a different bug than the timeout issue John is describing, even if they're both downstream of the same rescoring pass. If it helps, all my TSVs were submitted at 0.01 threshold step, so the 0.001→0.01 switch An Phan mentioned wouldn't change my case. Happy to provide submission IDs or collaborate in any way we can help. Thank you.
\n
Thanks very much to the organisers for fixing this! The LB is looking much healthier now with more than 2000 teams having valid scores.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3432311": "Like many people, I've seen both my selected submissions mysteriously generate \"Timeout exceeded\" errors. This is really disappointing, since I don't believe either submission came close to the time limit when originally submitted. The .tsv files are respectively 1.78 and 1.96 GB, which is large but I don't think either particularly unusual or unreasonable for this competition. However, this isn't just about my two somewhat hacked-together submissions, but about research groups who invested substantially more time and effort than I did and whose professional standing in the protein function prediction field is up for assessment in this competition. This isn't just a Kaggle competition, it's a scientific assessment of the current state of the protein function prediction field. The competition can hardly achieve that if entries are unpredictably and arbitrarily excluded for taking too long to score a test set whose size is unknown and not easily predictable.\n\nIf the new test dataset is larger than the previous one, I would ask the organisers to increase the time limit commensurately with dataset size to give submissions that succeded previously the opportunity to be scored. I also note that this is not the final leaderboard, and there are more updates to come. Hopefully the experience of the current difficulties will make things easier for the organisers at subsequent updates (though it would still be good to see this one put right).\n\nFinally, I understand that organising a competition like this is a pretty thankless task. I am very much aware that it takes a great deal of effort, and whatever my frustrations about the current situation, I do want to acknowledge the hard work that has gone in and continues to go into running this competition. I fully realise that the zeroing of so many submissions in this update wasn't foreseen or intended.", "3432353": "Hi John, we understand the frustration. Please hang tight since the rescoring is still in progress. We acknowledge that a large number of submissions are getting timed out, so the Kaggle engineers are working on it. We also consider switching back to 0.01 threshold step to reduce computation time (our plan was to do 0.001 threshold step for the current update). \n\nThe new test dataset has more than 2,000 proteins in the ground truth, which is substantially larger than the previous test set, hence the longer evaluation time.", "3438336": "UPDATE (April 8, 9:25 PM ET): Mostly all submissions were restored. Now showing 0.370 and 387 entries. Thanks to An Phan and the Kaggle engineers for the fix. Leaving the original report below in case it helps debug similar cases. \n\n--\n>Just adding a data point to John's thread since I'm seeing a different failure mode that may help the engineers. I ended up position #62, public score 0.388, 394 valid submissions over the competition. Majority were TSV, some authored notebooks, some forked and edited. After rescoring, every single submission in my history it now shows \"Internal scoring error,\" not \"Timeout exceeded.\" My best rescored value came back at 0.016 (down from 0.388) before even that disappeared, and I'm now completely absent from the leaderboard. Not #62, not even last, I am not ranked. Flagging because \"Internal scoring error\" across my 394 submissions looks like a different bug than the timeout issue John is describing, even if they're both downstream of the same rescoring pass. If it helps, all my TSVs were submitted at 0.01 threshold step, so the 0.001→0.01 switch An Phan mentioned wouldn't change my case. Happy to provide submission IDs or collaborate in any way we can help. Thank you.", "3438605": "Thanks very much to the organisers for fixing this! The LB is looking much healthier now with more than 2000 teams having valid scores." }, "source": "meta" }