Let's see what happens if we do not select our final 2 submissions.
\n\nThe submission page says: If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.
\n\nIn this stage 2, the public LB score is 0 for all submissions. Any of our submission is the best submissions based on the public LB score. As we don't know how the selection code breaks ties we can safely assume that the selection is random here.
\n\nThe conclusion is clear: better select your to submissions yourself than rely on a random thing ;)
\n\nEdit: there is no hidden public LB score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734
\n\nWe also know from a previous competition that:
\n\n\n\n\nIf you do not select, it picks the best based on public score (here, a\n meaningless number) and breaks ties by earliest submitted. There is no\n good reason not to manually select.
\n
Thanks to Oscar Takeshita for finding this.
", "messages": [ { "id": "313634", "postDate": "04/13/2018 15:53:36", "content": "Let's see what happens if we do not select our final 2 submissions.
\n\nThe submission page says: If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.
\n\nIn this stage 2, the public LB score is 0 for all submissions. Any of our submission is the best submissions based on the public LB score. As we don't know how the selection code breaks ties we can safely assume that the selection is random here.
\n\nThe conclusion is clear: better select your to submissions yourself than rely on a random thing ;)
\n\nEdit: there is no hidden public LB score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734
\n\nWe also know from a previous competition that:
\n\n\n\n\nIf you do not select, it picks the best based on public score (here, a\n meaningless number) and breaks ties by earliest submitted. There is no\n good reason not to manually select.
\n
Thanks to Oscar Takeshita for finding this.
", "rawMarkdown": "Let's see what happens if we do not select our final 2 submissions.\n\nThe submission page says: *If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.*\n\nIn this stage 2, the public LB score is 0 for all submissions. Any of our submission is the best submissions based on the public LB score. As we don't know how the selection code breaks ties we can safely assume that the selection is random here. \n\nThe conclusion is clear: better select your to submissions yourself than rely on a random thing ;)\n\nEdit: there is no hidden public LB score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734 \n\nWe also know from a [previous competition][1] that:\n\n> If you do not select, it picks the best based on public score (here, a\n> meaningless number) and breaks ties by earliest submitted. There is no\n> good reason not to manually select.\n\nThanks to Oscar Takeshita for finding this.\n\n\n [1]: https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39356#220604", "votes": null }, { "id": "313635", "postDate": "04/13/2018 15:57:05", "content": "It's clear to me now. Thanks! :)
", "rawMarkdown": "It's clear to me now. Thanks! :)", "votes": null }, { "id": "313650", "postDate": "04/13/2018 16:22:59", "content": "From the other point of view it's possible that public LB scores are 0 only for us but in fact the real scores are just hidden behind. In that case it will not be random. So if you are ready to play - one of the options is to choose one and let the system choose the second
", "rawMarkdown": "From the other point of view it's possible that public LB scores are 0 only for us but in fact the real scores are just hidden behind. In that case it will not be random. So if you are ready to play - one of the options is to choose one and let the system choose the second", "votes": null }, { "id": "313727", "postDate": "04/13/2018 19:02:42", "content": "I rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.
\n\nThink of it. Why would Kaggle team collect images for stage2 if it is not for using them? And why would they compute a public score that they don't show?
", "rawMarkdown": "I rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.\n\nThink of it. Why would Kaggle team collect images for stage2 if it is not for using them? And why would they compute a public score that they don't show?", "votes": null }, { "id": "313733", "postDate": "04/13/2018 19:10:59", "content": "can't argue, looks quite logical and way easier, but doesn't it contradict with \"This leaderboard is identical to the private leaderboard in this competition.\" on LB tab?
", "rawMarkdown": "can't argue, looks quite logical and way easier, but doesn't it contradict with \"This leaderboard is identical to the private leaderboard in this competition.\" on LB tab?", "votes": null }, { "id": "313740", "postDate": "04/13/2018 19:18:07", "content": "@bobbqe My hunch is that if you don't select your options in this competition, Kaggle will simply pick your first two earliest second stage chronological submissions. That's how they break ties with identical public scores (0.000 for this competition). If you really would like to pick one of your options at random (no really good reason for that as @CPMP wrote) I'd pick one randomly yourself.
", "rawMarkdown": "bobbqe My hunch is that if you don't select your options in this competition, Kaggle will simply pick your first two earliest second stage chronological submissions. That's how they break ties with identical public scores (0.000 for this competition). If you really would like to pick one of your options at random (no really good reason for that as @CPMP wrote) I'd pick one randomly yourself.", "votes": null }, { "id": "313801", "postDate": "04/13/2018 21:05:31", "content": "\n\n\nI rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.
\n
I was right, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734
", "rawMarkdown": "> I rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.\n\nI was right, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734", "votes": null }, { "id": "313815", "postDate": "04/13/2018 21:35:36", "content": "Yes, I think you are right for the public score. I think the private will be scored not over the entire test set though. My first image competition (iceberg) was like that. Images that were adversary generated were excluded of scoring.
", "rawMarkdown": "Yes, I think you are right for the public score. I think the private will be scored not over the entire test set though. My first image competition (iceberg) was like that. Images that were adversary generated were excluded of scoring.", "votes": null }, { "id": "313942", "postDate": "04/14/2018 05:45:20", "content": "great! still not clear with the auto selection of 2 subs
", "rawMarkdown": "great! still not clear with the auto selection of 2 subs", "votes": null }, { "id": "313952", "postDate": "04/14/2018 06:20:16", "content": "This post may be helpful. It was for a different competition but has my question answered officially by Kaggle.
", "rawMarkdown": "[**This post**](https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39356) may be helpful. It was for a different competition but has my question answered officially by Kaggle.", "votes": null }, { "id": "313996", "postDate": "04/14/2018 10:05:27", "content": "Thanks Oscar! We now know this:
\n\nIf you do not select, it picks the best based on public score (here, a meaningless number) and breaks ties by earliest submitted. There is no good reason not to manually select.
", "rawMarkdown": "Thanks Oscar! We now know this:\n\nIf you do not select, it picks the best based on public score (here, a meaningless number) and breaks ties by earliest submitted. There is no good reason not to manually select.", "votes": null }, { "id": "314025", "postDate": "04/14/2018 12:33:30", "content": "I think this is a very serious issue here, can we get a confirmation from Kaggle team, that if two not selected will pick the first two based on submission time!? Many thanks!
", "rawMarkdown": "I think this is a very serious issue here, can we get a confirmation from Kaggle team, that if two not selected will pick the first two based on submission time!? Many thanks!", "votes": null }, { "id": "314060", "postDate": "04/14/2018 14:29:53", "content": "Oscar found the answer to that question, just read Kaggle staff answer to it.
", "rawMarkdown": "Oscar found the answer to that question, just read Kaggle staff answer to it.", "votes": null }, { "id": "314082", "postDate": "04/14/2018 14:58:37", "content": "You are welcome @CPMP. It also means that if we are trully not sure about which two ones we want to select and decide to roll a dice, we should roll our own dice. We are still on the drivers' seat this way. Kaggle's auto-selection won't be random because it picks the two earliest submission. One can argue, that we are often right in our first guess... Is it so? In which case, Kaggle is being smart ;)
\n\nIf many is using the exact same package producing the exact same solution, the earliest is indeed the best!
", "rawMarkdown": "You are welcome @CPMP. It also means that if we are trully not sure about which two ones we want to select and decide to roll a dice, we should roll our own dice. We are still on the drivers' seat this way. Kaggle's auto-selection won't be random because it picks the two earliest submission. One can argue, that we are often right in our first guess... Is it so? In which case, Kaggle is being smart ;) \n\nIf many is using the exact same package producing the exact same solution, the earliest is indeed the best!", "votes": null } ], "comments": [ { "id": 313635, "author_name": "vijaybj", "author_url": "", "post_date": "04/13/2018 15:57:05", "content": "It's clear to me now. Thanks! :)
", "votes": null, "replies": [] }, { "id": 313650, "author_name": "vladimirgroza", "author_url": "", "post_date": "04/13/2018 16:22:59", "content": "From the other point of view it's possible that public LB scores are 0 only for us but in fact the real scores are just hidden behind. In that case it will not be random. So if you are ready to play - one of the options is to choose one and let the system choose the second
", "votes": null, "replies": [ { "id": 313727, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 19:02:42", "content": "I rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.
\n\nThink of it. Why would Kaggle team collect images for stage2 if it is not for using them? And why would they compute a public score that they don't show?
", "votes": null, "replies": [] }, { "id": 313733, "author_name": "vladimirgroza", "author_url": "", "post_date": "04/13/2018 19:10:59", "content": "can't argue, looks quite logical and way easier, but doesn't it contradict with \"This leaderboard is identical to the private leaderboard in this competition.\" on LB tab?
", "votes": null, "replies": [] }, { "id": 313740, "author_name": "pliptor", "author_url": "", "post_date": "04/13/2018 19:18:07", "content": "@bobbqe My hunch is that if you don't select your options in this competition, Kaggle will simply pick your first two earliest second stage chronological submissions. That's how they break ties with identical public scores (0.000 for this competition). If you really would like to pick one of your options at random (no really good reason for that as @CPMP wrote) I'd pick one randomly yourself.
", "votes": null, "replies": [] }, { "id": 313801, "author_name": "cpmpml", "author_url": "", "post_date": "04/13/2018 21:05:31", "content": "\n\n\nI rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.
\n
I was right, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734
", "votes": null, "replies": [] }, { "id": 313815, "author_name": "pliptor", "author_url": "", "post_date": "04/13/2018 21:35:36", "content": "Yes, I think you are right for the public score. I think the private will be scored not over the entire test set though. My first image competition (iceberg) was like that. Images that were adversary generated were excluded of scoring.
", "votes": null, "replies": [] }, { "id": 313942, "author_name": "vladimirgroza", "author_url": "", "post_date": "04/14/2018 05:45:20", "content": "great! still not clear with the auto selection of 2 subs
", "votes": null, "replies": [] }, { "id": 313952, "author_name": "pliptor", "author_url": "", "post_date": "04/14/2018 06:20:16", "content": "This post may be helpful. It was for a different competition but has my question answered officially by Kaggle.
", "votes": null, "replies": [] }, { "id": 313996, "author_name": "cpmpml", "author_url": "", "post_date": "04/14/2018 10:05:27", "content": "Thanks Oscar! We now know this:
\n\nIf you do not select, it picks the best based on public score (here, a meaningless number) and breaks ties by earliest submitted. There is no good reason not to manually select.
", "votes": null, "replies": [] }, { "id": 314082, "author_name": "pliptor", "author_url": "", "post_date": "04/14/2018 14:58:37", "content": "You are welcome @CPMP. It also means that if we are trully not sure about which two ones we want to select and decide to roll a dice, we should roll our own dice. We are still on the drivers' seat this way. Kaggle's auto-selection won't be random because it picks the two earliest submission. One can argue, that we are often right in our first guess... Is it so? In which case, Kaggle is being smart ;)
\n\nIf many is using the exact same package producing the exact same solution, the earliest is indeed the best!
", "votes": null, "replies": [] } ] }, { "id": 314025, "author_name": "h938454", "author_url": "", "post_date": "04/14/2018 12:33:30", "content": "I think this is a very serious issue here, can we get a confirmation from Kaggle team, that if two not selected will pick the first two based on submission time!? Many thanks!
", "votes": null, "replies": [ { "id": 314060, "author_name": "cpmpml", "author_url": "", "post_date": "04/14/2018 14:29:53", "content": "Oscar found the answer to that question, just read Kaggle staff answer to it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "313634": "Let's see what happens if we do not select our final 2 submissions.\n\nThe submission page says: *If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.*\n\nIn this stage 2, the public LB score is 0 for all submissions. Any of our submission is the best submissions based on the public LB score. As we don't know how the selection code breaks ties we can safely assume that the selection is random here. \n\nThe conclusion is clear: better select your to submissions yourself than rely on a random thing ;)\n\nEdit: there is no hidden public LB score, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734 \n\nWe also know from a [previous competition][1] that:\n\n> If you do not select, it picks the best based on public score (here, a\n> meaningless number) and breaks ties by earliest submitted. There is no\n> good reason not to manually select.\n\nThanks to Oscar Takeshita for finding this.\n\n\n [1]: https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39356#220604", "313635": "It's clear to me now. Thanks! :)", "313650": "From the other point of view it's possible that public LB scores are 0 only for us but in fact the real scores are just hidden behind. In that case it will not be random. So if you are ready to play - one of the options is to choose one and let the system choose the second", "313727": "I rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.\n\nThink of it. Why would Kaggle team collect images for stage2 if it is not for using them? And why would they compute a public score that they don't show?", "313733": "can't argue, looks quite logical and way easier, but doesn't it contradict with \"This leaderboard is identical to the private leaderboard in this competition.\" on LB tab?", "313740": "bobbqe My hunch is that if you don't select your options in this competition, Kaggle will simply pick your first two earliest second stage chronological submissions. That's how they break ties with identical public scores (0.000 for this competition). If you really would like to pick one of your options at random (no really good reason for that as @CPMP wrote) I'd pick one randomly yourself.", "313801": "> I rather think that all stage2 test images are used for private LB scoring. In that case there is no image left for public LB, hence the arbitrary 0 score.\n\nI was right, see https://www.kaggle.com/c/data-science-bowl-2018/discussion/54432#313734", "313815": "Yes, I think you are right for the public score. I think the private will be scored not over the entire test set though. My first image competition (iceberg) was like that. Images that were adversary generated were excluded of scoring.", "313942": "great! still not clear with the auto selection of 2 subs", "313952": "[**This post**](https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39356) may be helpful. It was for a different competition but has my question answered officially by Kaggle.", "313996": "Thanks Oscar! We now know this:\n\nIf you do not select, it picks the best based on public score (here, a meaningless number) and breaks ties by earliest submitted. There is no good reason not to manually select.", "314025": "I think this is a very serious issue here, can we get a confirmation from Kaggle team, that if two not selected will pick the first two based on submission time!? Many thanks!", "314060": "Oscar found the answer to that question, just read Kaggle staff answer to it.", "314082": "You are welcome @CPMP. It also means that if we are trully not sure about which two ones we want to select and decide to roll a dice, we should roll our own dice. We are still on the drivers' seat this way. Kaggle's auto-selection won't be random because it picks the two earliest submission. One can argue, that we are often right in our first guess... Is it so? In which case, Kaggle is being smart ;) \n\nIf many is using the exact same package producing the exact same solution, the earliest is indeed the best!" }, "source": "meta" }