![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3847086%2Fe5c029b401e26371ce010928b0e2e3fd%2FScreenshot%20(421).png?generation=1698904825980754&alt=media\")
This was the First update so I will do a analysis on that.
\nFirst of all, there were still quite a few shake-ups. For example, Team: @its7171 jumped from Rank: 647 -> 273, which is quite an impressive jump for this short time period!
Team: @svyatoslavpln
\nRank: 497 -> 346
\nScore: 0.55982 -> 0.55982
Team: @chronobcelp
\nRank: 151 -> 91
\nScore: 0.58771 -> 0.58771
Team: @alexcei Latskov
\nRank: 98 -> 69
\nScore: 0.58771 -> 0.58771
Team: @alloyshine
\nRank: 49 -> 37
\nScore: 0.59144 -> 0.59144
Team: Synthetic Goose
\nRank: 10 -> 7
\nScore: 0.62109 -> 0.62109
Team: Synthetic Goose
\nRank: 10 -> 7
\nScore: 0.62109 -> 0.62109
Team: @its7171
\nRank: 647 -> 30
\nScore: 0.55892 -> 0.56671
Team: @aschamp
\nRank: 413 -> 188
\nScore: 0.56014 -> 0.5527
Team: @luojq2019
\nRank: 181 -> 41
\nScore: 0.58158 -> 0.56334
Team: @drchenb
\nRank: 96 -> 20
\nScore: 0.58771 -> 0.57134
Team: [Risk-ZaloPay] Aggressive
\nRank: 40 -> 14
\nScore: 0.59271 -> 0.57444
Team: xc-ant
\nRank: 19 -> 11
\nScore: 0.60995 -> 0.57847
Team: Synthetic Goose
\nRank: 10 -> 4
\nScore: 0.62109 -> 0.59308
This was the First update so I will do a analysis on that.
\nFirst of all, there were still quite a few shake-ups. For example, Team: @its7171 jumped from Rank: 647 -> 273, which is quite an impressive jump for this short time period!
Team: @svyatoslavpln
\nRank: 497 -> 346
\nScore: 0.55982 -> 0.55982
Team: @chronobcelp
\nRank: 151 -> 91
\nScore: 0.58771 -> 0.58771
Team: @alexcei Latskov
\nRank: 98 -> 69
\nScore: 0.58771 -> 0.58771
Team: @alloyshine
\nRank: 49 -> 37
\nScore: 0.59144 -> 0.59144
Team: Synthetic Goose
\nRank: 10 -> 7
\nScore: 0.62109 -> 0.62109
Team: Synthetic Goose
\nRank: 10 -> 7
\nScore: 0.62109 -> 0.62109
Team: @its7171
\nRank: 647 -> 30
\nScore: 0.55892 -> 0.56671
Team: @aschamp
\nRank: 413 -> 188
\nScore: 0.56014 -> 0.5527
Team: @luojq2019
\nRank: 181 -> 41
\nScore: 0.58158 -> 0.56334
Team: @drchenb
\nRank: 96 -> 20
\nScore: 0.58771 -> 0.57134
Team: [Risk-ZaloPay] Aggressive
\nRank: 40 -> 14
\nScore: 0.59271 -> 0.57444
Team: xc-ant
\nRank: 19 -> 11
\nScore: 0.60995 -> 0.57847
Team: Synthetic Goose
\nRank: 10 -> 4
\nScore: 0.62109 -> 0.59308
Thanks for sharing. Just for some reason \"Maximum top 10 rank improvement\" is mentioned twice.
", "rawMarkdown": "Thanks for sharing. Just for some reason \"Maximum top 10 rank improvement\" is mentioned twice.", "votes": null }, { "id": "2509220", "postDate": "11/02/2023 09:19:17", "content": "I believe what happened is that the selected submissions were re-evaluated on the new data, and their score was replaced with the new score obtained on this data. Then, the leaderboard was reordered. So, the teams that maintain the same score are because they did not select the submission with the best score in the public and therefore it is maintained (or because they had 2 submissions with the same score). On the other hand, the case of @its7171 or people who improved their score could be because the score obtained in the private is higher than what they had in the public. It's just a hypothesis.
", "rawMarkdown": "I believe what happened is that the selected submissions were re-evaluated on the new data, and their score was replaced with the new score obtained on this data. Then, the leaderboard was reordered. So, the teams that maintain the same score are because they did not select the submission with the best score in the public and therefore it is maintained (or because they had 2 submissions with the same score). On the other hand, the case of @its7171 or people who improved their score could be because the score obtained in the private is higher than what they had in the public. It's just a hypothesis.", "votes": null }, { "id": "2509311", "postDate": "11/02/2023 10:12:05", "content": "My hypothesis is that leaderboard is currently using max(old_public_score, new_public_score) because for myself I see a message \"Your submission scored 0.56034, which is not an improvement of your previous score\".
But there are people who have a worse score than before, which couldn't be possible if its max(old_public_score, new_public_score).
", "rawMarkdown": "But there are people who have a worse score than before, which couldn't be possible if its max(old_public_score, new_public_score).", "votes": null }, { "id": "2509330", "postDate": "11/02/2023 10:22:40", "content": "Yeah, I'm not sure what has happened to those people, but for me that's definitely the case… When you look at LB, do you also see a message \"Your submission scored…\"?
", "rawMarkdown": "Yeah, I'm not sure what has happened to those people, but for me that's definitely the case... When you look at LB, do you also see a message \"Your submission scored...\"?", "votes": null }, { "id": "2509341", "postDate": "11/02/2023 10:30:04", "content": "Yes, thats the message on my last submission score.
\nMy last submissions are these:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13599107%2F9644629bd5516e8821ae41daee725e57%2FCaptura.PNG?generation=1698920729366176&alt=media\")
\nSo my selected submissions were updated on new data (because before update version 117 that now scores 0.566 had an score of 0.62155 like 120) and the unselected submissions were no updated.
\nSo I think that I keep 0.62155 in the public lb because I had another submission with 0.62155 that I did not select which is 120
Right, but then I don't understand the point of showing unselected submissions on LB… Anyways, it seems that the final results may be different not just because of the new data, but also because of some finalization tricks.
", "rawMarkdown": "Right, but then I don't understand the point of showing unselected submissions on LB... Anyways, it seems that the final results may be different not just because of the new data, but also because of some finalization tricks.", "votes": null }, { "id": "2509362", "postDate": "11/02/2023 10:40:03", "content": "Yes, me neither
", "rawMarkdown": "Yes, me neither", "votes": null }, { "id": "2509459", "postDate": "11/02/2023 11:38:11", "content": "Like you guys, I'm trying to make sense of what has happened here. It looks like the visible scores of my team's selected submissions have changed, but those of our unselected submissions have not. Because we didn't select our highest public scoring submission (assumed to be overfitted), our visible score is unchanged from before.
\nHowever, I'm also confused about public v private. The 'new' data are post-competition-deadline protein functional annotations. My expectation is therefore that they belong amongst the private LB data, but the new scores are presented as public and no private LB is visible.
", "rawMarkdown": "Like you guys, I'm trying to make sense of what has happened here. It looks like the visible scores of my team's *selected* submissions have changed, but those of our unselected submissions have not. Because we didn't select our highest public scoring submission (assumed to be overfitted), our visible score is unchanged from before.\n\nHowever, I'm also confused about public v private. The 'new' data are post-competition-deadline protein functional annotations. My expectation is therefore that they belong amongst the *private* LB data, but the new scores are presented as *public* and no private LB is visible.", "votes": null }, { "id": "2509537", "postDate": "11/02/2023 12:36:06", "content": "\n\nIt looks like the visible scores of my team's selected submissions have changed, but those of our unselected submissions have not
\n
They never re-compute unselected submissions that would be a massive waste of CPU/GPU.
\n\n\nMy expectation is therefore that they belong amongst the private LB data, but the new scores are presented as public and no private LB is visible.
\n
https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/452325
\nIt is not the final private LB hence they updated the RANKING on public LB but not showing the new scores I believe apart from on your selected submissions
", "rawMarkdown": ">It looks like the visible scores of my team's selected submissions have changed, but those of our unselected submissions have not\n\nThey never re-compute unselected submissions that would be a massive waste of CPU/GPU.\n\n>My expectation is therefore that they belong amongst the private LB data, but the new scores are presented as public and no private LB is visible.\n\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/452325 \n\nIt is not the final private LB hence they updated the RANKING on public LB but not showing the new scores I believe apart from on your selected submissions", "votes": null }, { "id": "2509550", "postDate": "11/02/2023 12:44:06", "content": "Agreed about what the process has been, but the ranking we now see (which is strictly ordered by firstly visible score, then submission time) is not very informative if the scores are no longer all derived on the same basis. Other competitions have handled interim updates in more informative ways.
", "rawMarkdown": "Agreed about what the process has been, but the ranking we now see (which is strictly ordered by firstly visible score, then submission time) is not very informative if the scores are no longer all derived on the same basis. Other competitions have handled interim updates in more informative ways.", "votes": null }, { "id": "2509718", "postDate": "11/02/2023 14:23:03", "content": "So it seems that the current public LB score is calculated as follows.
\nmax(new_selected_score1, new_selected_score2, old_not_selected_scores)
\nSince most new recalculated scores seem to be much worse, current LB not have much meaningful information. 🤔
I hope that LB is calculated by following like private LB,
\nmax(new_selected_score1, new_selected_score2)
\nor recalculate all submissions:
\nmax(new_selected_score1, new_selected_score2, new_not_selected_scores)
Agree its very confusing, but if we waited this long we can wait once more for the final private LB result.
", "rawMarkdown": "Agree its very confusing, but if we waited this long we can wait once more for the final private LB result.", "votes": null }, { "id": "2512391", "postDate": "11/04/2023 13:28:12", "content": "Ignoring the ranking, did any of the scores improve in the new leaderboard? Or even stay the same? Staying the same on a new data set would seem highly improbable, though. How much did the currently top ranked method change in terms of score? Annoying that I can't check myself now that the public board has been overwritten (such bad science to do that).
", "rawMarkdown": "Ignoring the ranking, did any of the scores improve in the new leaderboard? Or even stay the same? Staying the same on a new data set would seem highly improbable, though. How much did the currently top ranked method change in terms of score? Annoying that I can't check myself now that the public board has been overwritten (such bad science to do that).", "votes": null } ], "comments": [ { "id": 2509048, "author_name": "kononenko", "author_url": "", "post_date": "11/02/2023 06:13:05", "content": "Thanks for sharing. Just for some reason \"Maximum top 10 rank improvement\" is mentioned twice.
", "votes": null, "replies": [] }, { "id": 2509220, "author_name": "mpp1998", "author_url": "", "post_date": "11/02/2023 09:19:17", "content": "I believe what happened is that the selected submissions were re-evaluated on the new data, and their score was replaced with the new score obtained on this data. Then, the leaderboard was reordered. So, the teams that maintain the same score are because they did not select the submission with the best score in the public and therefore it is maintained (or because they had 2 submissions with the same score). On the other hand, the case of @its7171 or people who improved their score could be because the score obtained in the private is higher than what they had in the public. It's just a hypothesis.
", "votes": null, "replies": [ { "id": 2509311, "author_name": "kononenko", "author_url": "", "post_date": "11/02/2023 10:12:05", "content": "My hypothesis is that leaderboard is currently using max(old_public_score, new_public_score) because for myself I see a message \"Your submission scored 0.56034, which is not an improvement of your previous score\".
But there are people who have a worse score than before, which couldn't be possible if its max(old_public_score, new_public_score).
", "votes": null, "replies": [ { "id": 2509330, "author_name": "kononenko", "author_url": "", "post_date": "11/02/2023 10:22:40", "content": "Yeah, I'm not sure what has happened to those people, but for me that's definitely the case… When you look at LB, do you also see a message \"Your submission scored…\"?
", "votes": null, "replies": [ { "id": 2509341, "author_name": "mpp1998", "author_url": "", "post_date": "11/02/2023 10:30:04", "content": "Yes, thats the message on my last submission score.
\nMy last submissions are these:
\n
\nSo my selected submissions were updated on new data (because before update version 117 that now scores 0.566 had an score of 0.62155 like 120) and the unselected submissions were no updated.
\nSo I think that I keep 0.62155 in the public lb because I had another submission with 0.62155 that I did not select which is 120
Right, but then I don't understand the point of showing unselected submissions on LB… Anyways, it seems that the final results may be different not just because of the new data, but also because of some finalization tricks.
", "votes": null, "replies": [ { "id": 2509362, "author_name": "mpp1998", "author_url": "", "post_date": "11/02/2023 10:40:03", "content": "Yes, me neither
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 2509459, "author_name": "jbomitchell", "author_url": "", "post_date": "11/02/2023 11:38:11", "content": "Like you guys, I'm trying to make sense of what has happened here. It looks like the visible scores of my team's selected submissions have changed, but those of our unselected submissions have not. Because we didn't select our highest public scoring submission (assumed to be overfitted), our visible score is unchanged from before.
\nHowever, I'm also confused about public v private. The 'new' data are post-competition-deadline protein functional annotations. My expectation is therefore that they belong amongst the private LB data, but the new scores are presented as public and no private LB is visible.
", "votes": null, "replies": [ { "id": 2509537, "author_name": "julianmukaj", "author_url": "", "post_date": "11/02/2023 12:36:06", "content": "\n\nIt looks like the visible scores of my team's selected submissions have changed, but those of our unselected submissions have not
\n
They never re-compute unselected submissions that would be a massive waste of CPU/GPU.
\n\n\nMy expectation is therefore that they belong amongst the private LB data, but the new scores are presented as public and no private LB is visible.
\n
https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/452325
\nIt is not the final private LB hence they updated the RANKING on public LB but not showing the new scores I believe apart from on your selected submissions
", "votes": null, "replies": [ { "id": 2509550, "author_name": "jbomitchell", "author_url": "", "post_date": "11/02/2023 12:44:06", "content": "Agreed about what the process has been, but the ranking we now see (which is strictly ordered by firstly visible score, then submission time) is not very informative if the scores are no longer all derived on the same basis. Other competitions have handled interim updates in more informative ways.
", "votes": null, "replies": [ { "id": 2510285, "author_name": "julianmukaj", "author_url": "", "post_date": "11/02/2023 21:18:52", "content": "Agree its very confusing, but if we waited this long we can wait once more for the final private LB result.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2509718, "author_name": "its7171", "author_url": "", "post_date": "11/02/2023 14:23:03", "content": "So it seems that the current public LB score is calculated as follows.
\nmax(new_selected_score1, new_selected_score2, old_not_selected_scores)
\nSince most new recalculated scores seem to be much worse, current LB not have much meaningful information. 🤔
I hope that LB is calculated by following like private LB,
\nmax(new_selected_score1, new_selected_score2)
\nor recalculate all submissions:
\nmax(new_selected_score1, new_selected_score2, new_not_selected_scores)
Ignoring the ranking, did any of the scores improve in the new leaderboard? Or even stay the same? Staying the same on a new data set would seem highly improbable, though. How much did the currently top ranked method change in terms of score? Annoying that I can't check myself now that the public board has been overwritten (such bad science to do that).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2509038": "This was the First update so I will do a analysis on that.\nFirst of all, there were still quite a few shake-ups. For example, Team: @its7171 jumped from Rank: 647 -> 273, which is quite an impressive jump for this short time period!\n\n### Maximum top 500 rank improvement:\nTeam: @svyatoslavpln \nRank: 497 -> 346\nScore: 0.55982 -> 0.55982\n\n### Maximum top 200 rank improvement:\nTeam: @chronobcelp \nRank: 151 -> 91\nScore: 0.58771 -> 0.58771\n\n### Maximum top 100 rank improvement:\nTeam: @alexcei Latskov\nRank: 98 -> 69\nScore: 0.58771 -> 0.58771\n\n### Maximum top 50 rank improvement:\nTeam: @alloyshine \nRank: 49 -> 37\nScore: 0.59144 -> 0.59144\n\n### Maximum top 20 rank improvement:\nTeam: Synthetic Goose\nRank: 10 -> 7\nScore: 0.62109 -> 0.62109\n\n### Maximum top 10 rank improvement:\nTeam: Synthetic Goose\nRank: 10 -> 7\nScore: 0.62109 -> 0.62109\n\n# Fixing UPDATED STATS\n\n### Maximum overall rank improvement:\nTeam: @its7171\nRank: 647 -> 30\nScore: 0.55892 -> 0.56671\n\n### Maximum top 500 rank improvement:\nTeam: @aschamp \nRank: 413 -> 188\nScore: 0.56014 -> 0.5527\n\n### Maximum top 200 rank improvement:\nTeam: @luojq2019 \nRank: 181 -> 41\nScore: 0.58158 -> 0.56334\n\n### Maximum top 100 rank improvement:\nTeam: @drchenb \nRank: 96 -> 20\nScore: 0.58771 -> 0.57134\n\n### Maximum top 50 rank improvement:\nTeam: [Risk-ZaloPay] Aggressive\nRank: 40 -> 14\nScore: 0.59271 -> 0.57444\n\n### Maximum top 20 rank improvement:\nTeam: xc-ant\nRank: 19 -> 11\nScore: 0.60995 -> 0.57847\n\n### Maximum top 10 rank improvement:\nTeam: Synthetic Goose\nRank: 10 -> 4\nScore: 0.62109 -> 0.59308\n\n.png?generation=1698904825980754&alt=media)", "2509048": "Thanks for sharing. Just for some reason \"Maximum top 10 rank improvement\" is mentioned twice.", "2509220": "I believe what happened is that the selected submissions were re-evaluated on the new data, and their score was replaced with the new score obtained on this data. Then, the leaderboard was reordered. So, the teams that maintain the same score are because they did not select the submission with the best score in the public and therefore it is maintained (or because they had 2 submissions with the same score). On the other hand, the case of @its7171 or people who improved their score could be because the score obtained in the private is higher than what they had in the public. It's just a hypothesis.", "2509311": "My hypothesis is that leaderboard is currently using `max(old_public_score, new_public_score)` because for myself I see a message \"Your submission scored 0.56034, which is not an improvement of your previous score\".", "2509322": "But there are people who have a worse score than before, which couldn't be possible if its max(old_public_score, new_public_score).", "2509330": "Yeah, I'm not sure what has happened to those people, but for me that's definitely the case... When you look at LB, do you also see a message \"Your submission scored...\"?", "2509341": "Yes, thats the message on my last submission score.\nMy last submissions are these:\n\nSo my selected submissions were updated on new data (because before update version 117 that now scores 0.566 had an score of 0.62155 like 120) and the unselected submissions were no updated.\nSo I think that I keep 0.62155 in the public lb because I had another submission with 0.62155 that I did not select which is 120", "2509355": "Right, but then I don't understand the point of showing unselected submissions on LB... Anyways, it seems that the final results may be different not just because of the new data, but also because of some finalization tricks.", "2509362": "Yes, me neither", "2509459": "Like you guys, I'm trying to make sense of what has happened here. It looks like the visible scores of my team's *selected* submissions have changed, but those of our unselected submissions have not. Because we didn't select our highest public scoring submission (assumed to be overfitted), our visible score is unchanged from before.\n\nHowever, I'm also confused about public v private. The 'new' data are post-competition-deadline protein functional annotations. My expectation is therefore that they belong amongst the *private* LB data, but the new scores are presented as *public* and no private LB is visible.", "2509537": ">It looks like the visible scores of my team's selected submissions have changed, but those of our unselected submissions have not\n\nThey never re-compute unselected submissions that would be a massive waste of CPU/GPU.\n\n>My expectation is therefore that they belong amongst the private LB data, but the new scores are presented as public and no private LB is visible.\n\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/452325 \n\nIt is not the final private LB hence they updated the RANKING on public LB but not showing the new scores I believe apart from on your selected submissions", "2509550": "Agreed about what the process has been, but the ranking we now see (which is strictly ordered by firstly visible score, then submission time) is not very informative if the scores are no longer all derived on the same basis. Other competitions have handled interim updates in more informative ways.", "2509718": "So it seems that the current public LB score is calculated as follows.\n`max(new_selected_score1, new_selected_score2, old_not_selected_scores)`\nSince most new recalculated scores seem to be much worse, current LB not have much meaningful information. 🤔\n\nI hope that LB is calculated by following like private LB,\n`max(new_selected_score1, new_selected_score2)`\nor recalculate all submissions:\n`max(new_selected_score1, new_selected_score2, new_not_selected_scores)`", "2510285": "Agree its very confusing, but if we waited this long we can wait once more for the final private LB result.", "2512391": "Ignoring the ranking, did any of the scores improve in the new leaderboard? Or even stay the same? Staying the same on a new data set would seem highly improbable, though. How much did the currently top ranked method change in terms of score? Annoying that I can't check myself now that the public board has been overwritten (such bad science to do that)." }, "source": "meta" }