Dear Organizers, to whom it may concern,
\n\nI might spent less time on the forums than on the images and my information might not be complete but I see quite a few unclear points you may specify.\nI am posting them here, please can you help or clarify.
\n\nWhat is the reason/logic having 0 score on the leaderboard and until when will it be like this? I have no criticism against it (probably it is a very cool idea) but I have very bad feelings that it was not communicated to 3600 teams.
Is stage1 position influences the final ranking?
There are images (in my opinion), that have no nuclei. I might be wrong here but than I can trash the 12 years I spent on mic images. As far as I calculate the maximum score on an empty image is 0 (it can be NaN). Can you pls specify what happens in this case, how you expect us to handle this?
What will be the final score? Do you evaluate all the submissions of stage2? Is it the best one on stage2?
I appreciate your answers or pinpointing me to where these are specified.\nPeter
", "messages": [ { "id": 313356, "postDate": "2018-04-13T06:36:16.787Z", "content": "Dear Organizers, to whom it may concern,
\n\nI might spent less time on the forums than on the images and my information might not be complete but I see quite a few unclear points you may specify.\nI am posting them here, please can you help or clarify.
\n\nWhat is the reason/logic having 0 score on the leaderboard and until when will it be like this? I have no criticism against it (probably it is a very cool idea) but I have very bad feelings that it was not communicated to 3600 teams.
Is stage1 position influences the final ranking?
There are images (in my opinion), that have no nuclei. I might be wrong here but than I can trash the 12 years I spent on mic images. As far as I calculate the maximum score on an empty image is 0 (it can be NaN). Can you pls specify what happens in this case, how you expect us to handle this?
What will be the final score? Do you evaluate all the submissions of stage2? Is it the best one on stage2?
I appreciate your answers or pinpointing me to where these are specified.\nPeter
", "rawMarkdown": "Dear Organizers, to whom it may concern,\n\nI might spent less time on the forums than on the images and my information might not be complete but I see quite a few unclear points you may specify.\nI am posting them here, please can you help or clarify.\n\n1. What is the reason/logic having 0 score on the leaderboard and until when will it be like this? I have no criticism against it (probably it is a very cool idea) but I have very bad feelings that it was not communicated to 3600 teams.\n\n2. Is stage1 position influences the final ranking?\n\n3. There are images (in my opinion), that have no nuclei. I might be wrong here but than I can trash the 12 years I spent on mic images. As far as I calculate the maximum score on an empty image is 0 (it can be NaN). Can you pls specify what happens in this case, how you expect us to handle this?\n\n4. What will be the final score? Do you evaluate all the submissions of stage2? Is it the best one on stage2?\n\nI appreciate your answers or pinpointing me to where these are specified.\nPeter", "votes": 35 }, { "id": 313734, "postDate": "2018-04-13T19:11:18.400Z", "content": "In order to prevent probing, all the images in stage 2 test set are either in private leaderboard or ignored. Therefore your public leaderboard scores will always be 0.
No. Final ranking is determined by your private leaderboard ranking.
An empty string will work
It's the same as other competitions, where the private leaderboard evaluates on your two selected submissions (My Submissions-> Choose \"Use for final score\"). We'll have the system auto-select for you if you don't select.
Wendy, Does the auto selection takes the submission with best score for stage 2 data or does it select the last two (or first two) submission. Also appreciate if you could post a pinned FAQ thread about;
\n\nThere may be more questions from other and I thin it is easier to keep track of all those since we don't have much time until the end of the competition.
", "rawMarkdown": "Wendy, Does the auto selection takes the submission with best score for stage 2 data or does it select the last two (or first two) submission. Also appreciate if you could post a pinned FAQ thread about;\n\n 1. Retraining with stage1 test data. \n 2. Retraining with data sets that were not a part of our submitted model (without changing any hyper parameters of the model) . \n 3. Are we allowed to label the stage 2 data with a model and then use those labels to train another model. \n\nThere may be more questions from other and I thin it is easier to keep track of all those since we don't have much time until the end of the competition. \n" }, { "id": 314094, "postDate": "2018-04-14T15:42:46.597Z", "content": "Wendy, PLEASE have a separate post for all kind of official information.
", "rawMarkdown": "Wendy, PLEASE have a separate post for all kind of official information.", "votes": 2 }, { "id": 314127, "postDate": "2018-04-14T17:01:37.470Z", "content": "Hi, @wendy, @Peter, regarding the 3rd question that peter raise, it is actually an important point. Based on the evaluation metric, if the ground truth is no nuclei, then what we submit doesn't matter, it will always be 0, so there is no point of putting images with no nuclei in final private evaluation.
\n\nA bigger questions is whether we are rewarded by identifying the images with no nuclei, i.e, rewarded by providing empty string when the ground truth is no nuclei. Currently, I don't think that is the case based on the evaluation metric. (And I don't think changing the evaluation metric or rules at the last minute would be a good idea either).
", "rawMarkdown": "Hi, @wendy, @Peter, regarding the 3rd question that peter raise, it is actually an important point. Based on the evaluation metric, if the ground truth is no nuclei, then what we submit doesn't matter, it will always be 0, so there is no point of putting images with no nuclei in final private evaluation. \n\nA bigger questions is whether we are rewarded by identifying the images with no nuclei, i.e, rewarded by providing empty string when the ground truth is no nuclei. Currently, I don't think that is the case based on the evaluation metric. (And I don't think changing the evaluation metric or rules at the last minute would be a good idea either)." }, { "id": 314134, "postDate": "2018-04-14T17:12:04.353Z", "content": "Unfortunately it is much more difficult than you explain. Imagine that you detect 0 and there is 0 on the image. If you are not rewarded for this, than it is absolutely meaningless to submit it as empty, it is better to assume (speculate) that you missed and you put random objects. In this case, if there was indeed zero objects you anyway get 0 score, BUT if there were objects (and your classifier missed them) and you accidentally hit them, you get extra scores. So if the evaluation metric is so that it gives 0 for empty images, I would fill images that my classier found empty with random number of random objects and with a small probability you get more than 0 score.\nYou see? That is why I am pushing this question.
", "rawMarkdown": "Unfortunately it is much more difficult than you explain. Imagine that you detect 0 and there is 0 on the image. If you are not rewarded for this, than it is absolutely meaningless to submit it as empty, it is better to assume (speculate) that you missed and you put random objects. In this case, if there was indeed zero objects you anyway get 0 score, BUT if there were objects (and your classifier missed them) and you accidentally hit them, you get extra scores. So if the evaluation metric is so that it gives 0 for empty images, I would fill images that my classier found empty with random number of random objects and with a small probability you get more than 0 score.\nYou see? That is why I am pushing this question.", "votes": 2 }, { "id": 314136, "postDate": "2018-04-14T17:15:51.737Z", "content": "\n\n\nBased on the evaluation metric, if the ground truth is no nuclei, then what we submit doesn't matter, it will always be 0
\n
Not necessarily, the evaluation metric will always be 0 if we predict some nuclei, but if our prediction is correct (no nuclei), then it will be 0/0 (undefined), which in this case logically should be evaluated to 1 since our prediction is perfect. Of course, it would have been better if this case were explicitly described in the rules. Hopefully there are no such images in the private set.
", "rawMarkdown": "> Based on the evaluation metric, if the ground truth is no nuclei, then what we submit doesn't matter, it will always be 0\n\nNot necessarily, the evaluation metric will always be 0 if we predict some nuclei, but if our prediction is correct (no nuclei), then it will be 0/0 (undefined), which in this case logically should be evaluated to 1 since our prediction is perfect. Of course, it would have been better if this case were explicitly described in the rules. Hopefully there are no such images in the private set.", "votes": 1 }, { "id": 314678, "postDate": "2018-04-16T05:04:48.043Z", "content": "Hi @Wendy,
\n\nSo, if there is no nucleus in one image, I delete the id from the .csv. Is this right?
\n\nThanks.
", "rawMarkdown": "Hi @Wendy,\n\nSo, if there is no nucleus in one image, I delete the id from the .csv. Is this right?\n\nThanks." }, { "id": 315084, "postDate": "2018-04-16T18:20:41.160Z", "content": "@Bohdan, no, you must include all image ids in the file, but you can predict an empty string to indicate there's no nuclei.
", "rawMarkdown": "@Bohdan, no, you must include all image ids in the file, but you can predict an empty string to indicate there's no nuclei. " }, { "id": 315087, "postDate": "2018-04-16T18:23:03.807Z", "content": "@Wendy,\nI include like this:
\n\nid1,1 1 1 1 .....\nid2 \nid3\nid4\nit seems get wrong.....(if id2,3,4 have no nuclei)\nBut if I delete line id2 id3 id4 , it can be submitted
", "rawMarkdown": "@Wendy,\nI include like this:\n\n id1,1 1 1 1 .....\n id2 \n id3\n id4\n\nit seems get wrong.....(if id2,3,4 have no nuclei)\nBut if I delete line id2 id3 id4 , it can be submitted" }, { "id": 315115, "postDate": "2018-04-16T19:06:13.900Z", "content": "You still need a comma
\n\nid1,1 1 1 1 .....\nid2,\nid3,\nid4,\nI try to submit my results again.\nthanks a lot...
", "rawMarkdown": "I try to submit my results again.\nthanks a lot..." } ] }, { "id": 317232, "postDate": "2018-04-21T03:01:23.177Z", "content": "why change the rule of submit model and didn't inform us immediately and formally? We didnt know about the rule of \"We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.\" At the very start, we just know that :
\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n\nprize position means top5 in this competition.
\n\nI think kaggle should give us an explanation and solution to solve this misleading.
", "rawMarkdown": "why change the rule of submit model and didn't inform us immediately and formally? We didnt know about the rule of \"We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.\" At the very start, we just know that :\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\nYes. However, if you do not upload a model and **finish in a prize position**, your team will be removed from the competition standings entirely.\n\nprize position means top5 in this competition.\n\nI think kaggle should give us an explanation and solution to solve this misleading.", "votes": 9 }, { "id": 317278, "postDate": "2018-04-21T05:19:04.910Z", "content": "I might won't be popular with this remark regarding model upload, but for example in our case we had a very heterogeneous model, 3 deep nets, and in addition some python and some matlab code. For us it was a great effort to properly document and make a tested standalone script that works and can be shared with public. This was surely more than 1-2 days worktime of my colleague. Besides, we, as most of the others who uploaded the model agreed to make our 2 months work and potentially long lasting valuable ideas public.
", "rawMarkdown": "I might won't be popular with this remark regarding model upload, but for example in our case we had a very heterogeneous model, 3 deep nets, and in addition some python and some matlab code. For us it was a great effort to properly document and make a tested standalone script that works and can be shared with public. This was surely more than 1-2 days worktime of my colleague. Besides, we, as most of the others who uploaded the model agreed to make our 2 months work and potentially long lasting valuable ideas public. ", "votes": 8 }, { "id": 315283, "postDate": "2018-04-17T00:42:02.873Z", "content": "Are the medals for this competition calculated based only on the number of people who made a stage 2 submission? I thought it was supposed to be based on all 3,600 or whatever participants.
", "rawMarkdown": "Are the medals for this competition calculated based only on the number of people who made a stage 2 submission? I thought it was supposed to be based on all 3,600 or whatever participants.", "votes": 3, "replies": [ { "id": 315306, "postDate": "2018-04-17T01:03:13.663Z", "content": "Don't worry, it will be adjusted.
", "rawMarkdown": "Don't worry, it will be adjusted.", "votes": 1 }, { "id": 315328, "postDate": "2018-04-17T01:24:16.823Z", "content": "PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points.
\n\nBased on above notification, will they change this rule?
", "rawMarkdown": "**PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points.** \n\nBased on above notification, will they change this rule?", "votes": -1 }, { "id": 315331, "postDate": "2018-04-17T01:30:25.053Z", "content": "May I ask how do you know there will be an adjustment?
", "rawMarkdown": "May I ask how do you know there will be an adjustment?" }, { "id": 315337, "postDate": "2018-04-17T01:43:01.220Z", "content": "What as was talking about is this rule for 2-stage competitions: For the purposes of medals and points calculations, we will use the number of participants in the first stage to calculate points and medals.
\n\nObviously, those who do not submit in the second stage will not get any medals or prizes.
\n\nHe probably knows about the adjustment from his previous experience with 2-stage competitions.
", "rawMarkdown": "What as was talking about is this rule for 2-stage competitions: **For the purposes of medals and points calculations, we will use the number of participants in the first stage to calculate points and medals.** \n\nObviously, those who do not submit in the second stage will not get any medals or prizes.\n\nHe probably knows about the adjustment from his previous experience with 2-stage competitions.", "votes": 1 } ] }, { "id": 314096, "postDate": "2018-04-14T15:44:35.367Z", "content": "We are pinning this message to the top as it has some good information on Stage 2.
", "rawMarkdown": "We are pinning this message to the top as it has some good information on Stage 2. ", "votes": 2, "replies": [ { "id": 314098, "postDate": "2018-04-14T15:50:35.597Z", "content": "Thanks, Maggie!
", "rawMarkdown": "Thanks, Maggie!" } ] }, { "id": 314143, "postDate": "2018-04-14T17:55:37.487Z", "content": "I must say I am still confused by point 3.
\n\n1) In the submission, each row should correspond to one nuclei. Thus, a row with no run-length encoding would mean that you detected a nuclei but cannot specify where exactly it is (which is perfectly acceptable in case of nuclei without clear boundaries).
\n\n2) If images without nuclei are used in the evaluation, the evaluation should result in a ascore of Undefined due to division with 0 if predicted correctly (this is not a problem if we are forced to submit at least one nuclei for each image even if the nuclei mask area is 0). This would either (a) make the evaluation meaningless, (b) mean that the evaluation rules are not described corresponding to actual algorithm, (c) the nucleiless images are not actually used for evaluation (that is, they do not belong to the \"solution set\"), or (d) the nucleiless images are not in the \"solution set\" and will only penalize the score (even with area 0 nuclei). So, which case is the correct one?
\n\n3) I noticed that in stage 1 I would get an error when missing some ImageId-s. The same does not seem to be the case for stage 2. So should I add rows with missing ImageId-s or can I expect the submission with missing ImageId-s to be evaluated to a numerical score?
", "rawMarkdown": "I must say I am still confused by point 3.\n\n1) In the submission, each row should correspond to one nuclei. Thus, a row with no run-length encoding would mean that you detected a nuclei but cannot specify where exactly it is (which is perfectly acceptable in case of nuclei without clear boundaries).\n\n2) If images without nuclei are used in the evaluation, the evaluation should result in a ascore of Undefined due to division with 0 if predicted correctly (this is not a problem if we are forced to submit at least one nuclei for each image even if the nuclei mask area is 0). This would either (a) make the evaluation meaningless, (b) mean that the evaluation rules are not described corresponding to actual algorithm, (c) the nucleiless images are not actually used for evaluation (that is, they do not belong to the \"solution set\"), or (d) the nucleiless images are not in the \"solution set\" and will only penalize the score (even with area 0 nuclei). So, which case is the correct one?\n\n3) I noticed that in stage 1 I would get an error when missing some ImageId-s. The same does not seem to be the case for stage 2. So should I add rows with missing ImageId-s or can I expect the submission with missing ImageId-s to be evaluated to a numerical score?", "votes": 2, "replies": [ { "id": 314445, "postDate": "2018-04-15T15:33:33.067Z", "content": "Same question on 3). Can we just ignore that ImageId (missing ImageId) if we did not detect any nuclei in our final submission?
", "rawMarkdown": "Same question on 3). Can we just ignore that ImageId (missing ImageId) if we did not detect any nuclei in our final submission?" } ] }, { "id": 313767, "postDate": "2018-04-13T20:01:30.033Z", "content": "On number 4, how will the system auto-select the submissions? If it is by the highest score on the private leaderboard, any team selecting their submissions manually would be doing that to their own detriment as far as the competition is concerned.
", "rawMarkdown": "On number 4, how will the system auto-select the submissions? If it is by the highest score on the private leaderboard, any team selecting their submissions manually would be doing that to their own detriment as far as the competition is concerned.", "votes": 2, "replies": [ { "id": 314092, "postDate": "2018-04-14T15:34:23.143Z", "content": "You should make your own selection of your two models, our auto-select addresses ties and will select the earliest submission.
", "rawMarkdown": "You should make your own selection of your two models, our auto-select addresses ties and will select the earliest submission. ", "votes": 1 } ] }, { "id": 317154, "postDate": "2018-04-20T23:13:52.500Z", "content": "Thanks for all of the feedback around this 2-stage competition. We have an FAQ that we are updating now. The rules for this competition state that you need to upload your model before the end of Stage 1 in order to participate in Round 2. We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.
", "rawMarkdown": "Thanks for all of the feedback around this 2-stage competition. We have an FAQ that we are updating now. The rules for this competition state that you need to upload your model before the end of Stage 1 in order to participate in Round 2. We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model. ", "votes": -7, "replies": [ { "id": 317156, "postDate": "2018-04-20T23:18:51.160Z", "content": "Hi Maggie. Thank you for your feedback. I uploaded the model at 6:05 CT on April 10th yet the LB ranking has been removed. Could you please advise. Thank you.
", "rawMarkdown": "Hi Maggie. Thank you for your feedback. I uploaded the model at 6:05 CT on April 10th yet the LB ranking has been removed. Could you please advise. Thank you. ", "votes": 2 }, { "id": 317272, "postDate": "2018-04-21T05:08:43.390Z", "content": "I uploaded sha256sum 2 days before stage 1 deadline and still removed. Could you double check?
", "rawMarkdown": "I uploaded sha256sum 2 days before stage 1 deadline and still removed. Could you double check?", "votes": -1 }, { "id": 317304, "postDate": "2018-04-21T06:58:24.597Z", "content": "Thanks for your feedback. But I only saw the original rule:
\n\n\n\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
Because we didn't think our team can achieve prize position, so I decided to use our time to continue testing our model. Make entire code well documented and public will cost us more than 2 days.
\n\nBy the way, if you persist in your new rule, that means every uploaded code are indeed models. What if they just uploaded an empty file or a \"Hello World\"? You must check every uploaded models for all teams, make sure their model codes work and can get the same LB score as they shown in the LB now. I think this is impossible!
", "rawMarkdown": "Thanks for your feedback. But I only saw the original rule:\n\n> :Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a **prize position**, your team will be removed from the competition standings entirely.\n\nBecause we didn't think our team can achieve **prize position**, so I decided to use our time to continue testing our model. Make entire code well documented and public will cost us more than 2 days.\n\nBy the way, if you persist in your **new rule**, that means every uploaded code are indeed models. **What if they just uploaded an empty file or a \"Hello World\"?** You must check every uploaded models for all teams, make sure their model codes work and **can get the same LB score as they shown in the LB now**. I think this is impossible!\n", "votes": 2 }, { "id": 317307, "postDate": "2018-04-21T07:22:00.433Z", "content": "I had uploaded the code before the deadline time limit, still missing on the LB. not sure why it was removed.
", "rawMarkdown": "I had uploaded the code before the deadline time limit, still missing on the LB. not sure why it was removed. " }, { "id": 317313, "postDate": "2018-04-21T07:42:44.950Z", "content": "@Maggie Hi Maggie. You didn't tell us \" We will remove any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model. \" before stage1 deadline. That is unfair. You can't just change the rules without notifying all the competitors. You also didn't explain what is the \"prize posistion\". Does prize position refer to where the money can be obtained or where the medal can be obtained?
\n\nApart from this, I didn't know that there are so many noise pictures on stage2 that my model didn't produce any predictions. It just ignored them. If I submit this csv file, I will get a class id error. So it's very necessary to change the code so that they can generate forecasts with empty values when my model meet those noise images. I didn't meet this problem when I was in stage1. Only in that case will I not encounter this kind of errors. Is this behavior also counted as a violation of the rules? And i also uploaded my model before stage1 deadline.
\n\nAll my solutions can be find in this link: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54838#316229. You can reproduce my result if you follow my steps. So i am not cheating. I spent six weeks without a day off doing this competition. My score just removed without any reasonable explanation. This behavior will make us and those people who have devoted much effort to this competition very disappointed. That is cheating.
\n\nI hope official staff can give me an explanation about this issues.
\n\nThanks Zheng
", "rawMarkdown": "@Maggie Hi Maggie. You didn't tell us \" We will remove any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model. \" before stage1 deadline. That is unfair. You can't just change the rules without notifying all the competitors. You also didn't explain what is the \"prize posistion\". Does prize position refer to where the money can be obtained or where the medal can be obtained?\n\nApart from this, I didn't know that there are so many noise pictures on stage2 that my model didn't produce any predictions. It just ignored them. If I submit this csv file, I will get a class id error. So it's very necessary to change the code so that they can generate forecasts with empty values when my model meet those noise images. I didn't meet this problem when I was in stage1. Only in that case will I not encounter this kind of errors. Is this behavior also counted as a violation of the rules? And i also uploaded my model before stage1 deadline.\n\nAll my solutions can be find in this link: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54838#316229. You can reproduce my result if you follow my steps. So i am not cheating. I spent six weeks without a day off doing this competition. My score just removed without any reasonable explanation. This behavior will make us and those people who have devoted much effort to this competition very disappointed. **That is cheating.**\n\nI hope official staff can give me an explanation about this issues.\n\nThanks Zheng", "votes": 8 }, { "id": 317316, "postDate": "2018-04-21T07:55:11.993Z", "content": "@Maggie where can we appeal? And please give us a process where we can formally express our disagreement about the consistency of your decisions, so that Kaggle management can evaluate each admin's performance. Besides since our effort is not being respected, and you change the rules retrospectively then your job as admins should also be evaluated by both Kaggle and the sponsors .
", "rawMarkdown": "@Maggie where can we appeal? And please give us a process where we can formally express our disagreement about the consistency of your decisions, so that Kaggle management can evaluate each admin's performance. Besides since our effort is not being respected, and you change the rules retrospectively then your job as admins should also be evaluated by both Kaggle and the sponsors .", "votes": 4 }, { "id": 317326, "postDate": "2018-04-21T08:07:30.130Z", "content": "@gavrand Can't agree any more.
", "rawMarkdown": "@gavrand Can't agree any more.", "votes": 2 }, { "id": 317336, "postDate": "2018-04-21T08:27:07.023Z", "content": "Hi @Maggie. I need to know. I had uploaded my model and old weights before the Stage1 deadline. Howewer during stage2 I found some non-scientific bugs in my code. I had it fixed, finetune my model (none of hyperparameters were changed) and make a stage2 submission. What should I do now? Should I upload the bug-fixed model? Or should I wait for your (orgsanizers) request? Or all is ok and I have no need to worry about?
", "rawMarkdown": "Hi @Maggie. I need to know. I had uploaded my model and old weights before the Stage1 deadline. Howewer during stage2 I found some non-scientific bugs in my code. I had it fixed, finetune my model (none of hyperparameters were changed) and make a stage2 submission. What should I do now? Should I upload the bug-fixed model? Or should I wait for your (orgsanizers) request? Or all is ok and I have no need to worry about?", "votes": 2 }, { "id": 317410, "postDate": "2018-04-21T13:13:21.450Z", "content": "@Maggie may I ask where exactly in the rules https://www.kaggle.com/c/data-science-bowl-2018/rules is this rule you are referring to?
", "rawMarkdown": "@Maggie may I ask where exactly in the rules [https://www.kaggle.com/c/data-science-bowl-2018/rules][1] is this rule you are referring to? \n\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/rules", "votes": 3 }, { "id": 317443, "postDate": "2018-04-21T15:48:41.350Z", "content": "This is outrageous. I worked for months on this competition. I thought uploading was only for people who wanted to be eligible for prizes.
", "rawMarkdown": "This is outrageous. I worked for months on this competition. I thought uploading was only for people who wanted to be eligible for prizes.", "votes": 2 } ] }, { "id": 318105, "postDate": "2018-04-23T07:36:12.557Z", "content": "I was removed from the final competition step. Did it happen because I did not submit my model before April 11? Could you help me with links or instruction how I should have done it?
", "rawMarkdown": "I was removed from the final competition step. Did it happen because I did not submit my model before April 11? Could you help me with links or instruction how I should have done it? " }, { "id": 317582, "postDate": "2018-04-21T22:46:27.810Z", "content": "Our team was ranked #599 a few days ago on the private leaderboard, and I came back to check today for updates and no longer see our team on the leaderboard. We did upload 2 submissions and put check marks next to them. Why did everything disappear? It now looks like we never participated at all, no records, nothing. It's our first competition - could someone at least tell us what we did wrong - so that we can get it right next time? We'd appreciate if we can get the records back for having participated and being ranked. Thanks!
", "rawMarkdown": "Our team was ranked #599 a few days ago on the private leaderboard, and I came back to check today for updates and no longer see our team on the leaderboard. We did upload 2 submissions and put check marks next to them. Why did everything disappear? It now looks like we never participated at all, no records, nothing. It's our first competition - could someone at least tell us what we did wrong - so that we can get it right next time? We'd appreciate if we can get the records back for having participated and being ranked. Thanks!" }, { "id": 317347, "postDate": "2018-04-21T08:46:43.117Z", "content": "Is there a method for us to see our private scores?
", "rawMarkdown": "Is there a method for us to see our private scores? " }, { "id": 314301, "postDate": "2018-04-15T07:36:20.580Z", "content": "very good post.
", "rawMarkdown": "very good post. " }, { "id": 314189, "postDate": "2018-04-14T20:07:01.107Z", "content": "I understand that you can retrain your model with stage 1 test data without tuning hyperparameters. However, are we allowed to change \"prediction\" portion of our code at all? For instance, say that I was using ensemble prediction in stage 1. Can I get rid of the ensemble prediction and just predict with the pre-trained model from stage 1?
", "rawMarkdown": "I understand that you can retrain your model with stage 1 test data without tuning hyperparameters. However, are we allowed to change \"prediction\" portion of our code at all? For instance, say that I was using ensemble prediction in stage 1. Can I get rid of the ensemble prediction and just predict with the pre-trained model from stage 1? ", "replies": [ { "id": 314222, "postDate": "2018-04-14T22:34:18.470Z", "content": "It was said by Kaggle staff that no 'scientific' modifications are allowed, hence what you propose is not allowed. This is my understanding however, you may prefer to wait for Kaggle staff answer.
", "rawMarkdown": "It was said by Kaggle staff that no 'scientific' modifications are allowed, hence what you propose is not allowed. This is my understanding however, you may prefer to wait for Kaggle staff answer.", "votes": 4 } ] }, { "id": 313804, "postDate": "2018-04-13T21:07:44.133Z", "content": "Many thanks @Wendy!!
\n\nFor #3 you may specify a little more yet. \nBy definition if there is no cell on the image and:\na) we predict 0-> the score is 0/0=NaN, althoug the prediciton was good. Are you giving 0 or 1 score for this\nb) we predict more than zero than the score 0/x = 0.
\n\nIf you give a 0 for a) than it is better (or at least equal) to predict false positives as being correct. \nWhat is the score for case a).?
", "rawMarkdown": "Many thanks @Wendy!!\n\nFor #3 you may specify a little more yet. \nBy definition if there is no cell on the image and:\na) we predict 0-> the score is 0/0=NaN, althoug the prediciton was good. Are you giving 0 or 1 score for this\nb) we predict more than zero than the score 0/x = 0. \n\nIf you give a 0 for a) than it is better (or at least equal) to predict false positives as being correct. \nWhat is the score for case a).?", "replies": [ { "id": 314095, "postDate": "2018-04-14T15:43:44.520Z", "content": "@Peter At this point I figure there's not much that can be done. Wendy did say that an empty string would work.
\n\nI know I had a bunch of null outputs and a few that should have been null but had small false positives. I could fix the latter with a few lines of code in post processing but as I understand the rules that wouldn't be allowed.
\n\nI'm assuming we get full disclosure about the scoring (empty images, which images were actually included in final score, etc.) at the end of the competition.
", "rawMarkdown": "@Peter At this point I figure there's not much that can be done. Wendy did say that an empty string would work. \n\nI know I had a bunch of null outputs and a few that should have been null but had small false positives. I could fix the latter with a few lines of code in post processing but as I understand the rules that wouldn't be allowed.\n\n I'm assuming we get full disclosure about the scoring (empty images, which images were actually included in final score, etc.) at the end of the competition. " }, { "id": 315099, "postDate": "2018-04-16T18:38:12.763Z", "content": "We add an Epsilon to the denominator so it won't be NaN.
Note that precision is defined as tp/(tp+fp+fn), so, whatever you predict there, tp will always be 0, so the score will be the same for everyone anyway. You know what to do here.
You ask great questions, and I'm looking for Kaggle staff answers. However, there is one question that can be answered right now:
\n\n\n\n\nIs stage1 position influences the final ranking?
\n
Absolutely not.
", "rawMarkdown": "You ask great questions, and I'm looking for Kaggle staff answers. However, there is one question that can be answered right now:\n\n> Is stage1 position influences the final ranking?\n\nAbsolutely not." }, { "id": 317193, "postDate": "2018-04-21T01:25:42.940Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 313734, "author_name": "Wendy Kan", "author_url": "", "post_date": "2018-04-13T19:11:18.400000", "content": "In order to prevent probing, all the images in stage 2 test set are either in private leaderboard or ignored. Therefore your public leaderboard scores will always be 0.
No. Final ranking is determined by your private leaderboard ranking.
An empty string will work
It's the same as other competitions, where the private leaderboard evaluates on your two selected submissions (My Submissions-> Choose \"Use for final score\"). We'll have the system auto-select for you if you don't select.
Wendy, Does the auto selection takes the submission with best score for stage 2 data or does it select the last two (or first two) submission. Also appreciate if you could post a pinned FAQ thread about;
\n\nThere may be more questions from other and I thin it is easier to keep track of all those since we don't have much time until the end of the competition.
", "votes": 0, "replies": [] }, { "id": 314094, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-04-14T15:42:46.597000", "content": "Wendy, PLEASE have a separate post for all kind of official information.
", "votes": 2, "replies": [] }, { "id": 314127, "author_name": "Rui Li", "author_url": "", "post_date": "2018-04-14T17:01:37.470000", "content": "Hi, @wendy, @Peter, regarding the 3rd question that peter raise, it is actually an important point. Based on the evaluation metric, if the ground truth is no nuclei, then what we submit doesn't matter, it will always be 0, so there is no point of putting images with no nuclei in final private evaluation.
\n\nA bigger questions is whether we are rewarded by identifying the images with no nuclei, i.e, rewarded by providing empty string when the ground truth is no nuclei. Currently, I don't think that is the case based on the evaluation metric. (And I don't think changing the evaluation metric or rules at the last minute would be a good idea either).
", "votes": 0, "replies": [] }, { "id": 314134, "author_name": "Peter Horvath", "author_url": "", "post_date": "2018-04-14T17:12:04.353000", "content": "Unfortunately it is much more difficult than you explain. Imagine that you detect 0 and there is 0 on the image. If you are not rewarded for this, than it is absolutely meaningless to submit it as empty, it is better to assume (speculate) that you missed and you put random objects. In this case, if there was indeed zero objects you anyway get 0 score, BUT if there were objects (and your classifier missed them) and you accidentally hit them, you get extra scores. So if the evaluation metric is so that it gives 0 for empty images, I would fill images that my classier found empty with random number of random objects and with a small probability you get more than 0 score.\nYou see? That is why I am pushing this question.
", "votes": 2, "replies": [] }, { "id": 314136, "author_name": "Stan Kriventsov", "author_url": "", "post_date": "2018-04-14T17:15:51.737000", "content": "\n\n\nBased on the evaluation metric, if the ground truth is no nuclei, then what we submit doesn't matter, it will always be 0
\n
Not necessarily, the evaluation metric will always be 0 if we predict some nuclei, but if our prediction is correct (no nuclei), then it will be 0/0 (undefined), which in this case logically should be evaluated to 1 since our prediction is perfect. Of course, it would have been better if this case were explicitly described in the rules. Hopefully there are no such images in the private set.
", "votes": 1, "replies": [] }, { "id": 314678, "author_name": "Bohdan", "author_url": "", "post_date": "2018-04-16T05:04:48.043000", "content": "Hi @Wendy,
\n\nSo, if there is no nucleus in one image, I delete the id from the .csv. Is this right?
\n\nThanks.
", "votes": 0, "replies": [] }, { "id": 315084, "author_name": "Wendy Kan", "author_url": "", "post_date": "2018-04-16T18:20:41.160000", "content": "@Bohdan, no, you must include all image ids in the file, but you can predict an empty string to indicate there's no nuclei.
", "votes": 0, "replies": [] }, { "id": 315087, "author_name": "Bohdan", "author_url": "", "post_date": "2018-04-16T18:23:03.807000", "content": "@Wendy,\nI include like this:
\n\nid1,1 1 1 1 .....\nid2 \nid3\nid4\nit seems get wrong.....(if id2,3,4 have no nuclei)\nBut if I delete line id2 id3 id4 , it can be submitted
", "votes": 0, "replies": [] }, { "id": 315115, "author_name": "Wendy Kan", "author_url": "", "post_date": "2018-04-16T19:06:13.900000", "content": "You still need a comma
\n\nid1,1 1 1 1 .....\nid2,\nid3,\nid4,\nI try to submit my results again.\nthanks a lot...
", "votes": 0, "replies": [] } ] }, { "id": 317232, "author_name": "databatman", "author_url": "", "post_date": "2018-04-21T03:01:23.177000", "content": "why change the rule of submit model and didn't inform us immediately and formally? We didnt know about the rule of \"We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.\" At the very start, we just know that :
\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\nYes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n\nprize position means top5 in this competition.
\n\nI think kaggle should give us an explanation and solution to solve this misleading.
", "votes": 9, "replies": [] }, { "id": 317278, "author_name": "Peter Horvath", "author_url": "", "post_date": "2018-04-21T05:19:04.910000", "content": "I might won't be popular with this remark regarding model upload, but for example in our case we had a very heterogeneous model, 3 deep nets, and in addition some python and some matlab code. For us it was a great effort to properly document and make a tested standalone script that works and can be shared with public. This was surely more than 1-2 days worktime of my colleague. Besides, we, as most of the others who uploaded the model agreed to make our 2 months work and potentially long lasting valuable ideas public.
", "votes": 8, "replies": [] }, { "id": 315283, "author_name": "Stan Kriventsov", "author_url": "", "post_date": "2018-04-17T00:42:02.873000", "content": "Are the medals for this competition calculated based only on the number of people who made a stage 2 submission? I thought it was supposed to be based on all 3,600 or whatever participants.
", "votes": 3, "replies": [ { "id": 315306, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-17T01:03:13.663000", "content": "Don't worry, it will be adjusted.
", "votes": 1, "replies": [] }, { "id": 315328, "author_name": "IAmLegend", "author_url": "", "post_date": "2018-04-17T01:24:16.823000", "content": "PLEASE NOTE: If you do not make a submission during the second stage of the competition, you will not appear on the final competition leaderboard and you will not receive competition ranking points.
\n\nBased on above notification, will they change this rule?
", "votes": -1, "replies": [] }, { "id": 315331, "author_name": "HaoyuWang", "author_url": "", "post_date": "2018-04-17T01:30:25.053000", "content": "May I ask how do you know there will be an adjustment?
", "votes": 0, "replies": [] }, { "id": 315337, "author_name": "Stan Kriventsov", "author_url": "", "post_date": "2018-04-17T01:43:01.220000", "content": "What as was talking about is this rule for 2-stage competitions: For the purposes of medals and points calculations, we will use the number of participants in the first stage to calculate points and medals.
\n\nObviously, those who do not submit in the second stage will not get any medals or prizes.
\n\nHe probably knows about the adjustment from his previous experience with 2-stage competitions.
", "votes": 1, "replies": [] } ] }, { "id": 314096, "author_name": "Maggie", "author_url": "", "post_date": "2018-04-14T15:44:35.367000", "content": "We are pinning this message to the top as it has some good information on Stage 2.
", "votes": 2, "replies": [ { "id": 314098, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-04-14T15:50:35.597000", "content": "Thanks, Maggie!
", "votes": 0, "replies": [] } ] }, { "id": 314143, "author_name": "theofour", "author_url": "", "post_date": "2018-04-14T17:55:37.487000", "content": "I must say I am still confused by point 3.
\n\n1) In the submission, each row should correspond to one nuclei. Thus, a row with no run-length encoding would mean that you detected a nuclei but cannot specify where exactly it is (which is perfectly acceptable in case of nuclei without clear boundaries).
\n\n2) If images without nuclei are used in the evaluation, the evaluation should result in a ascore of Undefined due to division with 0 if predicted correctly (this is not a problem if we are forced to submit at least one nuclei for each image even if the nuclei mask area is 0). This would either (a) make the evaluation meaningless, (b) mean that the evaluation rules are not described corresponding to actual algorithm, (c) the nucleiless images are not actually used for evaluation (that is, they do not belong to the \"solution set\"), or (d) the nucleiless images are not in the \"solution set\" and will only penalize the score (even with area 0 nuclei). So, which case is the correct one?
\n\n3) I noticed that in stage 1 I would get an error when missing some ImageId-s. The same does not seem to be the case for stage 2. So should I add rows with missing ImageId-s or can I expect the submission with missing ImageId-s to be evaluated to a numerical score?
", "votes": 2, "replies": [ { "id": 314445, "author_name": "TerenceLiu", "author_url": "", "post_date": "2018-04-15T15:33:33.067000", "content": "Same question on 3). Can we just ignore that ImageId (missing ImageId) if we did not detect any nuclei in our final submission?
", "votes": 0, "replies": [] } ] }, { "id": 313767, "author_name": "Stan Kriventsov", "author_url": "", "post_date": "2018-04-13T20:01:30.033000", "content": "On number 4, how will the system auto-select the submissions? If it is by the highest score on the private leaderboard, any team selecting their submissions manually would be doing that to their own detriment as far as the competition is concerned.
", "votes": 2, "replies": [ { "id": 314092, "author_name": "Maggie", "author_url": "", "post_date": "2018-04-14T15:34:23.143000", "content": "You should make your own selection of your two models, our auto-select addresses ties and will select the earliest submission.
", "votes": 1, "replies": [] } ] }, { "id": 317154, "author_name": "Maggie", "author_url": "", "post_date": "2018-04-20T23:13:52.500000", "content": "Thanks for all of the feedback around this 2-stage competition. We have an FAQ that we are updating now. The rules for this competition state that you need to upload your model before the end of Stage 1 in order to participate in Round 2. We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.
", "votes": -7, "replies": [ { "id": 317156, "author_name": "Leith Akkawi", "author_url": "", "post_date": "2018-04-20T23:18:51.160000", "content": "Hi Maggie. Thank you for your feedback. I uploaded the model at 6:05 CT on April 10th yet the LB ranking has been removed. Could you please advise. Thank you.
", "votes": 2, "replies": [] }, { "id": 317272, "author_name": "bsp2020", "author_url": "", "post_date": "2018-04-21T05:08:43.390000", "content": "I uploaded sha256sum 2 days before stage 1 deadline and still removed. Could you double check?
", "votes": -1, "replies": [] }, { "id": 317304, "author_name": "Brian Luo", "author_url": "", "post_date": "2018-04-21T06:58:24.597000", "content": "Thanks for your feedback. But I only saw the original rule:
\n\n\n\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage? Yes. However, if you do not upload a model and finish in a prize position, your team will be removed from the competition standings entirely.
\n
Because we didn't think our team can achieve prize position, so I decided to use our time to continue testing our model. Make entire code well documented and public will cost us more than 2 days.
\n\nBy the way, if you persist in your new rule, that means every uploaded code are indeed models. What if they just uploaded an empty file or a \"Hello World\"? You must check every uploaded models for all teams, make sure their model codes work and can get the same LB score as they shown in the LB now. I think this is impossible!
", "votes": 2, "replies": [] }, { "id": 317307, "author_name": "Kishore M", "author_url": "", "post_date": "2018-04-21T07:22:00.433000", "content": "I had uploaded the code before the deadline time limit, still missing on the LB. not sure why it was removed.
", "votes": 0, "replies": [] }, { "id": 317313, "author_name": "Zheng Li", "author_url": "", "post_date": "2018-04-21T07:42:44.950000", "content": "@Maggie Hi Maggie. You didn't tell us \" We will remove any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model. \" before stage1 deadline. That is unfair. You can't just change the rules without notifying all the competitors. You also didn't explain what is the \"prize posistion\". Does prize position refer to where the money can be obtained or where the medal can be obtained?
\n\nApart from this, I didn't know that there are so many noise pictures on stage2 that my model didn't produce any predictions. It just ignored them. If I submit this csv file, I will get a class id error. So it's very necessary to change the code so that they can generate forecasts with empty values when my model meet those noise images. I didn't meet this problem when I was in stage1. Only in that case will I not encounter this kind of errors. Is this behavior also counted as a violation of the rules? And i also uploaded my model before stage1 deadline.
\n\nAll my solutions can be find in this link: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54838#316229. You can reproduce my result if you follow my steps. So i am not cheating. I spent six weeks without a day off doing this competition. My score just removed without any reasonable explanation. This behavior will make us and those people who have devoted much effort to this competition very disappointed. That is cheating.
\n\nI hope official staff can give me an explanation about this issues.
\n\nThanks Zheng
", "votes": 8, "replies": [] }, { "id": 317316, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-21T07:55:11.993000", "content": "@Maggie where can we appeal? And please give us a process where we can formally express our disagreement about the consistency of your decisions, so that Kaggle management can evaluate each admin's performance. Besides since our effort is not being respected, and you change the rules retrospectively then your job as admins should also be evaluated by both Kaggle and the sponsors .
", "votes": 4, "replies": [] }, { "id": 317326, "author_name": "Zheng Li", "author_url": "", "post_date": "2018-04-21T08:07:30.130000", "content": "@gavrand Can't agree any more.
", "votes": 2, "replies": [] }, { "id": 317336, "author_name": "Ilya Dobrynin", "author_url": "", "post_date": "2018-04-21T08:27:07.023000", "content": "Hi @Maggie. I need to know. I had uploaded my model and old weights before the Stage1 deadline. Howewer during stage2 I found some non-scientific bugs in my code. I had it fixed, finetune my model (none of hyperparameters were changed) and make a stage2 submission. What should I do now? Should I upload the bug-fixed model? Or should I wait for your (orgsanizers) request? Or all is ok and I have no need to worry about?
", "votes": 2, "replies": [] }, { "id": 317410, "author_name": "gavrand", "author_url": "", "post_date": "2018-04-21T13:13:21.450000", "content": "@Maggie may I ask where exactly in the rules https://www.kaggle.com/c/data-science-bowl-2018/rules is this rule you are referring to?
", "votes": 3, "replies": [] }, { "id": 317443, "author_name": "JamesEverard", "author_url": "", "post_date": "2018-04-21T15:48:41.350000", "content": "This is outrageous. I worked for months on this competition. I thought uploading was only for people who wanted to be eligible for prizes.
", "votes": 2, "replies": [] } ] }, { "id": 318105, "author_name": "Timur Iskhakov", "author_url": "", "post_date": "2018-04-23T07:36:12.557000", "content": "I was removed from the final competition step. Did it happen because I did not submit my model before April 11? Could you help me with links or instruction how I should have done it?
", "votes": 0, "replies": [] }, { "id": 317582, "author_name": "photogazer", "author_url": "", "post_date": "2018-04-21T22:46:27.810000", "content": "Our team was ranked #599 a few days ago on the private leaderboard, and I came back to check today for updates and no longer see our team on the leaderboard. We did upload 2 submissions and put check marks next to them. Why did everything disappear? It now looks like we never participated at all, no records, nothing. It's our first competition - could someone at least tell us what we did wrong - so that we can get it right next time? We'd appreciate if we can get the records back for having participated and being ranked. Thanks!
", "votes": 0, "replies": [] }, { "id": 317347, "author_name": "Samrat Saha", "author_url": "", "post_date": "2018-04-21T08:46:43.117000", "content": "Is there a method for us to see our private scores?
", "votes": 0, "replies": [] }, { "id": 314301, "author_name": "", "author_url": "", "post_date": "2018-04-15T07:36:20.580000", "content": "very good post.
", "votes": 0, "replies": [] }, { "id": 314189, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-04-14T20:07:01.107000", "content": "I understand that you can retrain your model with stage 1 test data without tuning hyperparameters. However, are we allowed to change \"prediction\" portion of our code at all? For instance, say that I was using ensemble prediction in stage 1. Can I get rid of the ensemble prediction and just predict with the pre-trained model from stage 1?
", "votes": 0, "replies": [ { "id": 314222, "author_name": "CPMP", "author_url": "", "post_date": "2018-04-14T22:34:18.470000", "content": "It was said by Kaggle staff that no 'scientific' modifications are allowed, hence what you propose is not allowed. This is my understanding however, you may prefer to wait for Kaggle staff answer.
", "votes": 4, "replies": [] } ] }, { "id": 313804, "author_name": "Peter Horvath", "author_url": "", "post_date": "2018-04-13T21:07:44.133000", "content": "Many thanks @Wendy!!
\n\nFor #3 you may specify a little more yet. \nBy definition if there is no cell on the image and:\na) we predict 0-> the score is 0/0=NaN, althoug the prediciton was good. Are you giving 0 or 1 score for this\nb) we predict more than zero than the score 0/x = 0.
\n\nIf you give a 0 for a) than it is better (or at least equal) to predict false positives as being correct. \nWhat is the score for case a).?
", "votes": 0, "replies": [ { "id": 314095, "author_name": "David Ermer", "author_url": "", "post_date": "2018-04-14T15:43:44.520000", "content": "@Peter At this point I figure there's not much that can be done. Wendy did say that an empty string would work.
\n\nI know I had a bunch of null outputs and a few that should have been null but had small false positives. I could fix the latter with a few lines of code in post processing but as I understand the rules that wouldn't be allowed.
\n\nI'm assuming we get full disclosure about the scoring (empty images, which images were actually included in final score, etc.) at the end of the competition.
", "votes": 0, "replies": [] }, { "id": 315099, "author_name": "Wendy Kan", "author_url": "", "post_date": "2018-04-16T18:38:12.763000", "content": "We add an Epsilon to the denominator so it won't be NaN.
Note that precision is defined as tp/(tp+fp+fn), so, whatever you predict there, tp will always be 0, so the score will be the same for everyone anyway. You know what to do here.
You ask great questions, and I'm looking for Kaggle staff answers. However, there is one question that can be answered right now:
\n\n\n\n\nIs stage1 position influences the final ranking?
\n
Absolutely not.
", "votes": 0, "replies": [] }, { "id": 317193, "author_name": "", "author_url": "", "post_date": "2018-04-21T01:25:42.940000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "313356": "Dear Organizers, to whom it may concern,\n\nI might spent less time on the forums than on the images and my information might not be complete but I see quite a few unclear points you may specify.\nI am posting them here, please can you help or clarify.\n\n1. What is the reason/logic having 0 score on the leaderboard and until when will it be like this? I have no criticism against it (probably it is a very cool idea) but I have very bad feelings that it was not communicated to 3600 teams.\n\n2. Is stage1 position influences the final ranking?\n\n3. There are images (in my opinion), that have no nuclei. I might be wrong here but than I can trash the 12 years I spent on mic images. As far as I calculate the maximum score on an empty image is 0 (it can be NaN). Can you pls specify what happens in this case, how you expect us to handle this?\n\n4. What will be the final score? Do you evaluate all the submissions of stage2? Is it the best one on stage2?\n\nI appreciate your answers or pinpointing me to where these are specified.\nPeter", "313734": "1. In order to prevent probing, all the images in stage 2 test set are either in private leaderboard or ignored. Therefore your public leaderboard scores will always be 0.\n\n2. No. Final ranking is determined by your private leaderboard ranking. \n\n3. An empty string will work\n\n4. It's the same as other competitions, where the private leaderboard evaluates on your two selected submissions (My Submissions-> Choose \"Use for final score\"). We'll have the system auto-select for you if you don't select. \n\n", "317232": "why change the rule of submit model and didn't inform us immediately and formally? We didnt know about the rule of \"We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model.\" At the very start, we just know that :\n\n:Will I still be on the leaderboard if I don't upload a model but do submit in the second stage?\nYes. However, if you do not upload a model and **finish in a prize position**, your team will be removed from the competition standings entirely.\n\nprize position means top5 in this competition.\n\nI think kaggle should give us an explanation and solution to solve this misleading.", "317278": "I might won't be popular with this remark regarding model upload, but for example in our case we had a very heterogeneous model, 3 deep nets, and in addition some python and some matlab code. For us it was a great effort to properly document and make a tested standalone script that works and can be shared with public. This was surely more than 1-2 days worktime of my colleague. Besides, we, as most of the others who uploaded the model agreed to make our 2 months work and potentially long lasting valuable ideas public. ", "315283": "Are the medals for this competition calculated based only on the number of people who made a stage 2 submission? I thought it was supposed to be based on all 3,600 or whatever participants.", "314096": "We are pinning this message to the top as it has some good information on Stage 2. ", "314143": "I must say I am still confused by point 3.\n\n1) In the submission, each row should correspond to one nuclei. Thus, a row with no run-length encoding would mean that you detected a nuclei but cannot specify where exactly it is (which is perfectly acceptable in case of nuclei without clear boundaries).\n\n2) If images without nuclei are used in the evaluation, the evaluation should result in a ascore of Undefined due to division with 0 if predicted correctly (this is not a problem if we are forced to submit at least one nuclei for each image even if the nuclei mask area is 0). This would either (a) make the evaluation meaningless, (b) mean that the evaluation rules are not described corresponding to actual algorithm, (c) the nucleiless images are not actually used for evaluation (that is, they do not belong to the \"solution set\"), or (d) the nucleiless images are not in the \"solution set\" and will only penalize the score (even with area 0 nuclei). So, which case is the correct one?\n\n3) I noticed that in stage 1 I would get an error when missing some ImageId-s. The same does not seem to be the case for stage 2. So should I add rows with missing ImageId-s or can I expect the submission with missing ImageId-s to be evaluated to a numerical score?", "313767": "On number 4, how will the system auto-select the submissions? If it is by the highest score on the private leaderboard, any team selecting their submissions manually would be doing that to their own detriment as far as the competition is concerned.", "317154": "Thanks for all of the feedback around this 2-stage competition. We have an FAQ that we are updating now. The rules for this competition state that you need to upload your model before the end of Stage 1 in order to participate in Round 2. We have removed any teams from the leaderboard that either uploaded a model after the deadline or did not upload a model. ", "318105": "I was removed from the final competition step. Did it happen because I did not submit my model before April 11? Could you help me with links or instruction how I should have done it? ", "317582": "Our team was ranked #599 a few days ago on the private leaderboard, and I came back to check today for updates and no longer see our team on the leaderboard. We did upload 2 submissions and put check marks next to them. Why did everything disappear? It now looks like we never participated at all, no records, nothing. It's our first competition - could someone at least tell us what we did wrong - so that we can get it right next time? We'd appreciate if we can get the records back for having participated and being ranked. Thanks!", "317347": " - This was my first Kaggle competition, i was not aware of model\n uploading. \n - In the submission page i kept on uploading only the result. Big\n Lesson learned always upload the model.\n - I had a silver in second stage 2 rank 22 now it is removed. Is there\n a way i can get it back.\n - I can upload all models and code. Please provide the link..\n - Please let me know how can i have that LB score back. Admins please let me know if i can have my rank back..", "315116": "Is there a method for us to see our private scores? ", "314301": "very good post. ", "314189": "I understand that you can retrain your model with stage 1 test data without tuning hyperparameters. However, are we allowed to change \"prediction\" portion of our code at all? For instance, say that I was using ensemble prediction in stage 1. Can I get rid of the ensemble prediction and just predict with the pre-trained model from stage 1? ", "313804": "Many thanks @Wendy!!\n\nFor #3 you may specify a little more yet. \nBy definition if there is no cell on the image and:\na) we predict 0-> the score is 0/0=NaN, althoug the prediciton was good. Are you giving 0 or 1 score for this\nb) we predict more than zero than the score 0/x = 0. \n\nIf you give a 0 for a) than it is better (or at least equal) to predict false positives as being correct. \nWhat is the score for case a).?", "313509": "You ask great questions, and I'm looking for Kaggle staff answers. However, there is one question that can be answered right now:\n\n> Is stage1 position influences the final ranking?\n\nAbsolutely not.", "317193": "" } }