{ "id": 54295, "title": "The test2 images are indeed different ...", "url": "/competitions/data-science-bowl-2018/discussion/54295", "author_name": "", "post_date": "2018-04-12T00:27:22.592702600Z", "votes": 32, "comment_count": 80, "views": 0, "content": "

Looking at the test2 data, I think my results will be terrible .

\n\n

I note that there are some images without nuclei? Are there some nerve cells as well?

", "messages": [ { "id": "312531", "postDate": "04/12/2018 00:27:22", "content": "

Looking at the test2 data, I think my results will be terrible .

\n\n

I note that there are some images without nuclei? Are there some nerve cells as well?

", "rawMarkdown": "Looking at the test2 data, I think my results will be terrible .\n\nI note that there are some images without nuclei? Are there some nerve cells as well?", "votes": null }, { "id": "312540", "postDate": "04/12/2018 00:53:46", "content": "

If I remember correctly, there are some pictures not for scoring which are used to prevent pseudo labeling.

", "rawMarkdown": "If I remember correctly, there are some pictures not for scoring which are used to prevent pseudo labeling.", "votes": null }, { "id": "312541", "postDate": "04/12/2018 00:54:12", "content": "

Makes you wonder whether we should have prioritized learning microbiology and annotating different datasets rather than fine-tuning off-the-shelf instance segmentation architectures, or training regimes on stage 1 train...

", "rawMarkdown": "Makes you wonder whether we should have prioritized learning microbiology and annotating different datasets rather than fine-tuning off-the-shelf instance segmentation architectures, or training regimes on stage 1 train...", "votes": null }, { "id": "312543", "postDate": "04/12/2018 00:57:15", "content": "

Now i opted for this \"learning microbiology and annotating different datasets\". Actually i submitted a solution not with the highest public LB score. I submission one with LB=0.505 which i think work best on my downloaded cell images.

\n\n

but even that, my downloaded cell images are very different from what i saw in stage2 dataset

", "rawMarkdown": "Now i opted for this \"learning microbiology and annotating different datasets\". Actually i submitted a solution not with the highest public LB score. I submission one with LB=0.505 which i think work best on my downloaded cell images.\n\nbut even that, my downloaded cell images are very different from what i saw in stage2 dataset", "votes": null }, { "id": "312544", "postDate": "04/12/2018 00:59:44", "content": "

https://youtu.be/1g6Zb98KYJE

\n\n

You can’t say we didn’t warn you! Good luck everybody! 👍

", "rawMarkdown": "https://youtu.be/1g6Zb98KYJE\n\nYou can’t say we didn’t warn you! Good luck everybody! 👍", "votes": null }, { "id": "312545", "postDate": "04/12/2018 01:02:35", "content": "

Maybe! 😉

\n\n

We intentionally made it difficult for anybody to win by just overfitting ensembles of ensembles of ensembles. We wanted a generalization or robustness challenge.

", "rawMarkdown": "Maybe! 😉\n\nWe intentionally made it difficult for anybody to win by just overfitting ensembles of ensembles of ensembles. We wanted a generalization or robustness challenge.", "votes": null }, { "id": "312546", "postDate": "04/12/2018 01:06:11", "content": "

@Allen what is your score range for stage2, using the model you build from stage1? I recall you mentioned there is large overfitting on stage1 in your thread.

", "rawMarkdown": "Allen what is your score range for stage2, using the model you build from stage1? I recall you mentioned there is large overfitting on stage1 in your thread.", "votes": null }, { "id": "312547", "postDate": "04/12/2018 01:09:11", "content": "

Funny we attempted (ran out of time for 10 mins) to generate new images using a CycleGAN using our GT as reference. Images came out quite nice, but still since we were using GT from stage 1 the nuclei shapes are nothing like stage 2 so would not have helped anyway.

", "rawMarkdown": "Funny we attempted (ran out of time for 10 mins) to generate new images using a CycleGAN using our GT as reference. Images came out quite nice, but still since we were using GT from stage 1 the nuclei shapes are nothing like stage 2 so would not have helped anyway.", "votes": null }, { "id": "312551", "postDate": "04/12/2018 01:11:15", "content": "

@Heng My best performing model was in the low 40s. (I didn’t have time to check my other models but I suspect they would’ve done better. )

", "rawMarkdown": "Heng My best performing model was in the low 40s. (I didn’t have time to check my other models but I suspect they would’ve done better. )", "votes": null }, { "id": "312552", "postDate": "04/12/2018 01:12:28", "content": "

Thanks!

", "rawMarkdown": "Thanks!", "votes": null }, { "id": "312554", "postDate": "04/12/2018 01:16:33", "content": "

I'd like to candidly ask if for the purpose of getting a great winning model at the end of the competition it would have been better to include at least one instance of each type of test image on stage 1 test; and if not, what is the motivation from the competition organizer's point of view to come up with such vastly different test images?

", "rawMarkdown": "I'd like to candidly ask if for the purpose of getting a great winning model at the end of the competition it would have been better to include at least one instance of each type of test image on stage 1 test; and if not, what is the motivation from the competition organizer's point of view to come up with such vastly different test images?", "votes": null }, { "id": "312556", "postDate": "04/12/2018 01:22:57", "content": "

It’s a fair question. I’m one of many organizers, so I can only provide my personal perspective. I’m interested in accuracy (as described by the competition metric), but I think the far more exciting and far more useful metric is robustness. An accurate model is valuable, but a less accurate model that can adapt to new experimental conditions is truly groundbreaking.

\n\n

I should add that from a user’s perspective (and also a computer vision perspective) the images are more similar than other image segmentation competitions.

", "rawMarkdown": "It’s a fair question. I’m one of many organizers, so I can only provide my personal perspective. I’m interested in accuracy (as described by the competition metric), but I think the far more exciting and far more useful metric is robustness. An accurate model is valuable, but a less accurate model that can adapt to new experimental conditions is truly groundbreaking.\n\nI should add that from a user’s perspective (and also a computer vision perspective) the images are more similar than other image segmentation competitions.", "votes": null }, { "id": "312560", "postDate": "04/12/2018 01:25:45", "content": "

private lb will be really interesting

", "rawMarkdown": "private lb will be really interesting", "votes": null }, { "id": "312561", "postDate": "04/12/2018 01:28:36", "content": "

Yeah I was worried about this. The idea is to generalize but to do this the training examples need to 'span the space.' I trained on all the data that was posted in the extra data thread that was usable (good ground truth images) and then augmented that. I ended up training on ~27,000 images and after spot checking the 2nd round test, there are still types of images I didn't have to train on. The final results will be interesting and maybe a bit of a crap shoot.

", "rawMarkdown": "Yeah I was worried about this. The idea is to generalize but to do this the training examples need to 'span the space.' I trained on all the data that was posted in the extra data thread that was usable (good ground truth images) and then augmented that. I ended up training on ~27,000 images and after spot checking the 2nd round test, there are still types of images I didn't have to train on. The final results will be interesting and maybe a bit of a crap shoot.", "votes": null }, { "id": "312565", "postDate": "04/12/2018 01:43:36", "content": "

What does it means when the submission is still initializing? I am trying to make sure our team made the deadline.

", "rawMarkdown": "What does it means when the submission is still initializing? I am trying to make sure our team made the deadline.", "votes": null }, { "id": "312581", "postDate": "04/12/2018 02:08:35", "content": "

There are few images are just very strange and I don't know whether they are real....

\n\n

For example, the image starts with 0fd3ed... and the one starts with 0fd475a...

\n\n

They are just completely black image to me.... are they real cell images for scoring? I am not sure whether the submission will allow 0 masks for some images...

", "rawMarkdown": "There are few images are just very strange and I don't know whether they are real....\n\nFor example, the image starts with 0fd3ed... and the one starts with 0fd475a...\n\nThey are just completely black image to me.... are they real cell images for scoring? I am not sure whether the submission will allow 0 masks for some images...", "votes": null }, { "id": "312592", "postDate": "04/12/2018 02:39:09", "content": "

I just checked. They are not zeroed out. If you equalize them you'll see standard fluorescence noise. Just like with any other image, you need to carefully manage false positives.

", "rawMarkdown": "I just checked. They are not zeroed out. If you equalize them you'll see standard fluorescence noise. Just like with any other image, you need to carefully manage false positives.", "votes": null }, { "id": "312600", "postDate": "04/12/2018 02:57:12", "content": "

Yes indeed, but the imaging modalities used (brightfield & fluorescence) are the same as for stage 1, so the algorithms used by the contestants for Stage 1 should work fine (with the exception of a few tweaks, of course) ... 😉

", "rawMarkdown": "Yes indeed, but the imaging modalities used (brightfield & fluorescence) are the same as for stage 1, so the algorithms used by the contestants for Stage 1 should work fine (with the exception of a few tweaks, of course) ... 😉", "votes": null }, { "id": "312618", "postDate": "04/12/2018 03:49:50", "content": "

This is the best one:

\n\n

5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2

\n\n

It's straight out of a textbook. It even has the caption 'Fig 5.' still on it. Notably it was a tiff file even though it was saved with a .png extension. I wonder if it's in the stage 2 set by mistake. I posted a similar one in the pinned data quality thread.

", "rawMarkdown": "This is the best one:\n\n5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2\n\nIt's straight out of a textbook. It even has the caption 'Fig 5.' still on it. Notably it was a tiff file even though it was saved with a .png extension. I wonder if it's in the stage 2 set by mistake. I posted a similar one in the pinned data quality thread.", "votes": null }, { "id": "312736", "postDate": "04/12/2018 08:27:10", "content": "

I guess the organizer put that image on purpose. 'Fig 5.' should NOT be detected if a model REALLY learns what are nuclei cells or not.

", "rawMarkdown": "I guess the organizer put that image on purpose. 'Fig 5.' should NOT be detected if a model REALLY learns what are nuclei cells or not.", "votes": null }, { "id": "312779", "postDate": "04/12/2018 10:11:58", "content": "

The image 170bc41b2095177ccc..... is a jpeg, not a png.\nThe art style is similar to the tiff image mentioned above.

", "rawMarkdown": "The image 170bc41b2095177ccc..... is a jpeg, not a png.\nThe art style is similar to the tiff image mentioned above.", "votes": null }, { "id": "312784", "postDate": "04/12/2018 10:21:47", "content": "
\n

We intentionally made it difficult for anybody to win by just\n overfitting ensembles of ensembles of ensembles. We wanted a\n generalization or robustness challenge.

\n
\n\n

What you actually did, you failed on your part to prepare and provide diverse data needed to create robust models, wasted sponsor money and kagglers time, and turned erstwhile notorious ds bowl into joke.

", "rawMarkdown": "> We intentionally made it difficult for anybody to win by just\n> overfitting ensembles of ensembles of ensembles. We wanted a\n> generalization or robustness challenge.\n\nWhat you actually did, you failed on your part to prepare and provide diverse data needed to create robust models, wasted sponsor money and kagglers time, and turned erstwhile notorious ds bowl into joke.", "votes": null }, { "id": "312793", "postDate": "04/12/2018 10:46:38", "content": "

May I ask a question? Is there some images that actually don't have any nuclei? like this one?2f25c16e472f0b5f0c14b654a4e78f8f83ad7505f056d48349e1e12e0de98839.png

", "rawMarkdown": "May I ask a question? Is there some images that actually don't have any nuclei? like this one?2f25c16e472f0b5f0c14b654a4e78f8f83ad7505f056d48349e1e12e0de98839.png", "votes": null }, { "id": "312811", "postDate": "04/12/2018 11:31:27", "content": "

Hi, Allen. \nWe are not allowed to change source codes, but our networks predict zeros for some images. What should we put into submission file in case if we strictly want to submit zeros for some image? Current submission format prohibits excluding img_ids with zeros only but zeros only could not be encoded with rle. Thanks

", "rawMarkdown": "Hi, Allen. \nWe are not allowed to change source codes, but our networks predict zeros for some images. What should we put into submission file in case if we strictly want to submit zeros for some image? Current submission format prohibits excluding img_ids with zeros only but zeros only could not be encoded with rle. Thanks", "votes": null }, { "id": "312816", "postDate": "04/12/2018 11:45:03", "content": "

@Alexander Buslaev I have the same problem. This is what I am using. I just add id + comma and not 0 0 (seems to work):

\n\n
import pandas as pd\nimport argparse\nimport os\nimport shutil\n\n\ndef main():\n  parser = argparse.ArgumentParser()\n  parser.add_argument(\n      '--submission_fn',\n      type=str,\n      default=\"\",\n      help=\"\"\"\\\n      Path to the submission file.\\\n      \"\"\")\n  parser.add_argument(\n      '--sample_submission_fn',\n      type=str,\n      default=\"\",\n      help=\"\"\"\\\n      Path to the sample submission filename.\\\n      \"\"\")\n  args, unparsed = parser.parse_known_args()\n  assert os.path.exists(args.submission_fn)\n  assert os.path.exists(args.sample_submission_fn)\n  sub = pd.read_csv(args.submission_fn)\n  sub_ids = set(sub['ImageId'])\n  sample_sub = pd.read_csv(args.sample_submission_fn)\n  sample_sub_ids = set(sample_sub['ImageId'])\n  missing_ids = sample_sub_ids.difference(sub_ids)\n\n  print(\"Adding %d missing ids.\" % (len(missing_ids)))\n  out_fn = args.submission_fn.replace('.csv', '_all_ids.csv')\n  shutil.copy(args.submission_fn, out_fn)\n  with open(out_fn, 'a') as f:\n    for missing_id in missing_ids:\n      f.write('%s,\\n' % (missing_id))\n\n  print(\"Done! Wrote updated submission to %s.\" % (out_fn))\n\n\nif __name__ == '__main__':\n  main()\n
", "rawMarkdown": "Alexander Buslaev I have the same problem. This is what I am using. I just add id + comma and not 0 0 (seems to work):\n\n import pandas as pd\n import argparse\n import os\n import shutil\n \n \n def main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\n '--submission_fn',\n type=str,\n default=\"\",\n help=\"\"\"\\\n Path to the submission file.\\\n \"\"\")\n parser.add_argument(\n '--sample_submission_fn',\n type=str,\n default=\"\",\n help=\"\"\"\\\n Path to the sample submission filename.\\\n \"\"\")\n args, unparsed = parser.parse_known_args()\n assert os.path.exists(args.submission_fn)\n assert os.path.exists(args.sample_submission_fn)\n sub = pd.read_csv(args.submission_fn)\n sub_ids = set(sub['ImageId'])\n sample_sub = pd.read_csv(args.sample_submission_fn)\n sample_sub_ids = set(sample_sub['ImageId'])\n missing_ids = sample_sub_ids.difference(sub_ids)\n \n print(\"Adding %d missing ids.\" % (len(missing_ids)))\n out_fn = args.submission_fn.replace('.csv', '_all_ids.csv')\n shutil.copy(args.submission_fn, out_fn)\n with open(out_fn, 'a') as f:\n for missing_id in missing_ids:\n f.write('%s,\\n' % (missing_id))\n \n print(\"Done! Wrote updated submission to %s.\" % (out_fn))\n \n \n if __name__ == '__main__':\n main()", "votes": null }, { "id": "312817", "postDate": "04/12/2018 11:53:28", "content": "

Actually i don't really grasp the purpose of the challenge - having a generalizing and robust model is fine, it may even be groundbreaking, but part of the images in stage 2 test are just of bad quality, integration times are sometimes much too low or there is just some random noise. In practice, you can't work with these images because actually what you would do in case of fluorescence images was i.e. to stain with antibodies you want to quantify or to detect, see if certain cells/nuclei are positive for a certain antibody or to count the number of FISH probes per nucleus- you would never run and trust an analysis based on images like those...

", "rawMarkdown": "Actually i don't really grasp the purpose of the challenge - having a generalizing and robust model is fine, it may even be groundbreaking, but part of the images in stage 2 test are just of bad quality, integration times are sometimes much too low or there is just some random noise. In practice, you can't work with these images because actually what you would do in case of fluorescence images was i.e. to stain with antibodies you want to quantify or to detect, see if certain cells/nuclei are positive for a certain antibody or to count the number of FISH probes per nucleus- you would never run and trust an analysis based on images like those...", "votes": null }, { "id": "312819", "postDate": "04/12/2018 11:58:59", "content": "

I am trained in biology, and from that perspective, the stage 2 images are not drastically different from what I might have expected to see in my day-to-day work. That said, I focused on training for the range of features I saw in the combined stage 1 train and test images. I expected some zoom changes, but not IF images with cytoplasmic dyes, or radically different cell types (neurons). I was quite selective with my training sets, and kept them quite small.

\n\n

I do worry that the winner is essentially the team with the biggest GPU farm to process the largest and most diverse external dataset comprised of different bioimaging methods. I guess we will have to wait and see.

\n\n

Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.

", "rawMarkdown": "I am trained in biology, and from that perspective, the stage 2 images are not drastically different from what I might have expected to see in my day-to-day work. That said, I focused on training for the range of features I saw in the combined stage 1 train and test images. I expected some zoom changes, but not IF images with cytoplasmic dyes, or radically different cell types (neurons). I was quite selective with my training sets, and kept them quite small.\n\nI do worry that the winner is essentially the team with the biggest GPU farm to process the largest and most diverse external dataset comprised of different bioimaging methods. I guess we will have to wait and see.\n\nAlso not sure how to submit a 'no nuclei found in this image' RLE without getting errors.", "votes": null }, { "id": "312820", "postDate": "04/12/2018 12:01:25", "content": "

\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"

\n\n

just put image_id without rle (i.e leave it empty, like empty string '')

", "rawMarkdown": "\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"\n\njust put image_id without rle (i.e leave it empty, like empty string '')", "votes": null }, { "id": "312823", "postDate": "04/12/2018 12:12:19", "content": "

Which of the external datasets (of course mentioned in the external data thread, to be allowed here) do you think would have helped?

\n\n

IMO incrementing or enriching the external data for a competition makes total sense but on this one (and based on the external labelled datasets I checked on the external data thread) the effort required to get a realistic training set needed to get SOTA results on stage 2 date is orders of magnitude higher than in other competitions. I would love to be proven wrong.

", "rawMarkdown": "Which of the external datasets (of course mentioned in the external data thread, to be allowed here) do you think would have helped?\n\nIMO incrementing or enriching the external data for a competition makes total sense but on this one (and based on the external labelled datasets I checked on the external data thread) the effort required to get a realistic training set needed to get SOTA results on stage 2 date is orders of magnitude higher than in other competitions. I would love to be proven wrong.", "votes": null }, { "id": "312824", "postDate": "04/12/2018 12:12:43", "content": "

\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"

\n\n

just put image_id without rle (i.e leave it empty, like empty string '')

\n\n

We can use this:

\n\n

The submission process (for stage1 and stage2) is :

\n\n", "rawMarkdown": "\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"\n\njust put image_id without rle (i.e leave it empty, like empty string '')\n\n\nWe can use this:\n\nThe submission process (for stage1 and stage2) is :\n\n- run model on images and make csv file\n\n- for images not recorded in csv (i.e. zero detection), we fill in the csv file by hand.", "votes": null }, { "id": "312831", "postDate": "04/12/2018 12:35:56", "content": "

I don't disagree with this comment. It's a competitions with a bunch of gotchas and plot twists. However this is not the way you'd actually try to generate a practical solution to a real problem.

", "rawMarkdown": "I don't disagree with this comment. It's a competitions with a bunch of gotchas and plot twists. However this is not the way you'd actually try to generate a practical solution to a real problem.", "votes": null }, { "id": "312894", "postDate": "04/12/2018 14:05:47", "content": "

If some people had used relevant training data my point would probably stand, but you're right that there wasn't really anything posted there that would have made too much difference; so the playing field might have turned out pretty level in the end. I guess everyone was finding things that were similar to those provided in stage 1.

\n\n

Thanks for the tip about leaving RLE column blank. I didn't realise we could do that.

", "rawMarkdown": "If some people had used relevant training data my point would probably stand, but you're right that there wasn't really anything posted there that would have made too much difference; so the playing field might have turned out pretty level in the end. I guess everyone was finding things that were similar to those provided in stage 1.\n\nThanks for the tip about leaving RLE column blank. I didn't realise we could do that.", "votes": null }, { "id": "312897", "postDate": "04/12/2018 14:08:47", "content": "

@David and @rakhlin Will you feel this way if competitors post scores in the range (say ±0.1 AP) of the phase one leaderboard?

", "rawMarkdown": "David and @rakhlin Will you feel this way if competitors post scores in the range (say ±0.1 AP) of the phase one leaderboard?", "votes": null }, { "id": "312929", "postDate": "04/12/2018 14:44:05", "content": "

I'll stick by the statement that if you were trying to come up with a practical solution you wouldn't split up the data in the way it was split up. It will be interesting to see how the final scores come out. My feeling is that the leaders in the 2nd round will be different than in the first round. But we'll see what happens.

", "rawMarkdown": "I'll stick by the statement that if you were trying to come up with a practical solution you wouldn't split up the data in the way it was split up. It will be interesting to see how the final scores come out. My feeling is that the leaders in the 2nd round will be different than in the first round. But we'll see what happens.", "votes": null }, { "id": "312937", "postDate": "04/12/2018 14:50:47", "content": "

@David If a competitor came up with a solution that evaluated to, for example, 0.5 AP on both phase one AND phase two you wouldn’t consider that a practical solution?

", "rawMarkdown": "David If a competitor came up with a solution that evaluated to, for example, 0.5 AP on both phase one AND phase two you wouldn’t consider that a practical solution?", "votes": null }, { "id": "312942", "postDate": "04/12/2018 14:59:03", "content": "

What would the score have been if the data had been split up in a more reasonable manner? You're not making the right comparison. I don't know what the ultimate best score could be but I rather doubt that you can get to it by using the available data in the way it was used in the competition.

", "rawMarkdown": "What would the score have been if the data had been split up in a more reasonable manner? You're not making the right comparison. I don't know what the ultimate best score could be but I rather doubt that you can get to it by using the available data in the way it was used in the competition.", "votes": null }, { "id": "312944", "postDate": "04/12/2018 15:02:43", "content": "

One thing I don't really understand is why the fluorescent images with multiple dyes are merged in a single channel? There are some single channel phase images which make sense, but you wouldn't normally choose dyes with overlapping emission spectra to stain multiple cell compartments. Unless it is showing the distribution of a protein or chemical that can translocate between the nucleus and cytoplasm. In that case, this is a bad way to look for nuclei (which I thought was the purpose of the data).

", "rawMarkdown": "One thing I don't really understand is why the fluorescent images with multiple dyes are merged in a single channel? There are some single channel phase images which make sense, but you wouldn't normally choose dyes with overlapping emission spectra to stain multiple cell compartments. Unless it is showing the distribution of a protein or chemical that can translocate between the nucleus and cytoplasm. In that case, this is a bad way to look for nuclei (which I thought was the purpose of the data).", "votes": null }, { "id": "312945", "postDate": "04/12/2018 15:02:57", "content": "

@David The competition wasn’t to see who could most overfit their model (that’s a useful competition, but that wasn’t the purpose of this competition) to a specific set of experimental conditions. It was, plainly, to see who could build a model that best adapts to data with unseen experimental conditions.

", "rawMarkdown": "David The competition wasn’t to see who could most overfit their model (that’s a useful competition, but that wasn’t the purpose of *this* competition) to a specific set of experimental conditions. It was, plainly, to see who could build a model that best adapts to data with unseen experimental conditions.", "votes": null }, { "id": "312947", "postDate": "04/12/2018 15:06:20", "content": "

Now I understood why you \"facepalm\" :D

", "rawMarkdown": "Now I understood why you \"facepalm\" :D", "votes": null }, { "id": "312950", "postDate": "04/12/2018 15:11:30", "content": "

Okay the point of my comments is not to be overly critical of the competition. I'm just trying to point out that if you were trying to generate the best possible predictor from a given set of data you wouldn't purposefully exclude classes of data from the training set. That's not how you avoid over training and ensure generalization. That makes a good competition but not smart if you want the best possible solution to a problem.

", "rawMarkdown": "Okay the point of my comments is not to be overly critical of the competition. I'm just trying to point out that if you were trying to generate the best possible predictor from a given set of data you wouldn't purposefully exclude classes of data from the training set. That's not how you avoid over training and ensure generalization. That makes a good competition but not smart if you want the best possible solution to a problem.", "votes": null }, { "id": "312955", "postDate": "04/12/2018 15:19:24", "content": "

Yes, of course, but this is a competition. This isn’t a practical problem. You’re absolutely correct that, in the real world, it’d be silly to not use all available data. This was a competition to help unearth strategies to construct and train a model to solve practical problems. I don’t know how you’d design a competition without a challenging holdout that achieved that aim.

", "rawMarkdown": "Yes, of course, but this is a competition. This isn’t a practical problem. You’re absolutely correct that, in the real world, it’d be silly to not use all available data. This was a competition to help unearth *strategies* to construct and train a model to solve practical problems. I don’t know how you’d design a competition without a challenging holdout that achieved that aim.", "votes": null }, { "id": "312956", "postDate": "04/12/2018 15:21:42", "content": "

@Allen: Indeed, overfitting is (in most instances) an undesirable effect of constant retraining and fine-tuning of a model. However, the data set (in this context) should have been more practically split (with some of the Stage 2 images being introduced in Stage 1 for model training and validation). Introducing a starkly different dataset to an already previously-trained model & expecting a high percentage accuracy (and by high, I'm referring to a score of preferably 0.75++) would seemingly be a little over-ambitious, although this might not be so, if the model had been exposed to a more diverse training set from the start. This would be akin to human learning, where if you are (for example) a trained microbiologist, you would be an expert in a specified field of biology (say, microbial genetics), although if questions on advanced engineering mathematics were posed to you, you would probably be stumped and not be able to answer such questions (if you had no prior learning in this area), without consulting external references/resources. The same applies to a trained ML model, which might work pretty well on a test set similar (though not necessarily identical) to the training set, although the same model might fail drastically if required to predict the outcomes from a completely different test set.

", "rawMarkdown": "Allen: Indeed, overfitting is (in most instances) an undesirable effect of constant retraining and fine-tuning of a model. However, the data set (in this context) should have been more practically split (with some of the Stage 2 images being introduced in Stage 1 for model training and validation). Introducing a *starkly* different dataset to an already previously-trained model & expecting a high percentage accuracy (and by high, I'm referring to a score of preferably 0.75++) would seemingly be a little over-ambitious, although this might not be so, if the model had been exposed to a more diverse training set from the start. This would be akin to human learning, where if you are (for example) a trained microbiologist, you would be an expert in a specified field of biology (say, microbial genetics), although if questions on advanced engineering mathematics were posed to you, you would probably be stumped and not be able to answer such questions (if you had no prior learning in this area), without consulting external references/resources. The same applies to a trained ML model, which might work pretty well on a test set ***similar*** (though not necessarily *identical*) to the training set, although the same model might fail drastically if required to predict the outcomes from a completely different test set.", "votes": null }, { "id": "312964", "postDate": "04/12/2018 15:31:52", "content": "
\n

The same applies to a trained ML model, which might work pretty well\n on a test set similar (though not necessarily identical) to the\n training set, although the same model might fail drastically if\n required to predict the outcomes from a completely different test set.

\n
\n\n

Exactly. We’re interested in models that were able to adapt to these conditions. I suspect the winners of the competition will be people or teams that focused entirely on that particular issue.

\n\n

I should add that none of the organizing biologists would describe the images from phase two as “starkly different” than phase one. In fact, there were instances where they eliminated certain images for that reason.

", "rawMarkdown": "> The same applies to a trained ML model, which might work pretty well\n> on a test set similar (though not necessarily identical) to the\n> training set, although the same model might fail drastically if\n> required to predict the outcomes from a completely different test set.\n\nExactly. We’re interested in models that were able to adapt to these conditions. I suspect the winners of the competition will be people or teams that focused entirely on that particular issue.\n\nI should add that none of the organizing biologists would describe the images from phase two as “starkly different” than phase one. In fact, there were instances where they eliminated certain images for that reason.", "votes": null }, { "id": "312972", "postDate": "04/12/2018 15:49:47", "content": "

@Allen: Some of the images introduced in Stage 2 were drawings scanned from a textbook, and not even photomicrographs - wouldn't you consider those to be starkly different from phase One (where no drawings were present in both the training and test data sets)? And some of these drawings also had other organelles (such as mitochondria) which might seem confusing to a trained ML model, and be interpreted as nuclei, so unless the organizers (including yourself) had put in these images but did not intend them to be scored, low scores would be inevitable for Stage 2 (even for the eventual winners), which leads us to the initial point that your primary objective for organizing this Competition (to attain robust models for accurately predicting the ROI corresponding to a cell's nucleus) would be unmet.

", "rawMarkdown": "Allen: Some of the images introduced in Stage 2 were drawings scanned from a textbook, and not even photomicrographs - wouldn't you consider those to be starkly different from phase One (where no drawings were present in both the training and test data sets)? And some of these drawings also had other organelles (such as mitochondria) which might seem confusing to a trained ML model, and be interpreted as nuclei, so unless the organizers (including yourself) had put in these images but did not intend them to be scored, low scores would be inevitable for Stage 2 (even for the eventual winners), which leads us to the initial point that your primary objective for organizing this Competition (to attain robust models for accurately predicting the ROI corresponding to a cell's nucleus) would be unmet.", "votes": null }, { "id": "313002", "postDate": "04/12/2018 16:45:54", "content": "

My two cents on this thread, is that the organizers have put on a great contest! Engaging problem set!

\n\n

After putting the kind of time we all put into this its natural to be frustrated by the stage 2 data. But the real world is always dealing us new unknowns.

\n\n

The organizers are writing the checks and designed an approach to get what they wanted out of it. At the end of the day the goal of ML is to take known circumstances and translate the learning to new ones. While its hard to learn about circles when all of your data are squares, that is not even close to the situation here. Its a fair fight, and extremely fun to work on.

\n\n

I am still grinding away on the stage 2, and do not expect to be at the top of the leaderboard. But special thanks to @Allen for being a great sport, and a guiding light for all of us!

\n\n

Cheers in advance to the winners. You will have assisted in the solving of a critical problem for humankind.

", "rawMarkdown": "My two cents on this thread, is that the organizers have put on a great contest! Engaging problem set! \n\nAfter putting the kind of time we all put into this its natural to be frustrated by the stage 2 data. But the real world is always dealing us new unknowns.\n\nThe organizers are writing the checks and designed an approach to get what they wanted out of it. At the end of the day the goal of ML is to take known circumstances and translate the learning to new ones. While its hard to learn about circles when all of your data are squares, that is not even close to the situation here. Its a fair fight, and extremely fun to work on.\n\nI am still grinding away on the stage 2, and do not expect to be at the top of the leaderboard. But special thanks to @Allen for being a great sport, and a guiding light for all of us!\n\nCheers in advance to the winners. You will have assisted in the solving of a critical problem for humankind.", "votes": null }, { "id": "313011", "postDate": "04/12/2018 16:52:39", "content": "

I agree, it is fun, and we were forewarned; but I did think there might be at least one or two examples of each type of image in the training dataset, even if it was somewhat biased towards one image type or another. It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.

", "rawMarkdown": "I agree, it is fun, and we were forewarned; but I did think there might be at least one or two examples of each type of image in the training dataset, even if it was somewhat biased towards one image type or another. It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.", "votes": null }, { "id": "313012", "postDate": "04/12/2018 16:53:34", "content": "

I'd argue that a good model should not detect any nuclei since there is no real nuclei in that drawing (it's a drawing, not a picture). I suspect that it won't be used for the LB calculation.

", "rawMarkdown": "I'd argue that a good model should not detect any nuclei since there is no real nuclei in that drawing (it's a drawing, not a picture). I suspect that it won't be used for the LB calculation.", "votes": null }, { "id": "313014", "postDate": "04/12/2018 16:56:54", "content": "

\"enter

\n\n

A classic conundrum in computer vision (and philosophy)! (Biology too, maybe!)

", "rawMarkdown": "![enter image description here][1]\n\nA classic conundrum in computer vision (and philosophy)! (Biology too, maybe!)\n\n [1]: https://upload.wikimedia.org/wikipedia/en/b/b9/MagrittePipe.jpg", "votes": null }, { "id": "313017", "postDate": "04/12/2018 17:01:20", "content": "

Depends how good the drawing is... A camera just captures (a slightly more accurate) picture of the nuclei too, and normally indirectly via some dye or other. However, as I pointed out in another thread, if you are making a drawing of a H&E stained tissue, why not just take a picture under the scope? That would be more useful and objective.

", "rawMarkdown": "Depends how good the drawing is... A camera just captures (a slightly more accurate) picture of the nuclei too, and normally indirectly via some dye or other. However, as I pointed out in another thread, if you are making a drawing of a H&E stained tissue, why not just take a picture under the scope? That would be more useful and objective.", "votes": null }, { "id": "313018", "postDate": "04/12/2018 17:06:59", "content": "

@DStjhb I agree. 😉 I also agree it's probably poison.

\n\n

Nevertheless, it’s a cute image and fun to think about in context of the competition.

", "rawMarkdown": "DStjhb I agree. 😉 I also agree it's probably poison.\n\nNevertheless, it’s a cute image and fun to think about in context of the competition.", "votes": null }, { "id": "313020", "postDate": "04/12/2018 17:12:44", "content": "
\n

It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.

\n
\n\n

I love this. 👍

\n\n

A goal some of the organizers of the competition and I shared was preventing the competitor with the most resources to ensemble and optimize hyperparameters from dominating. While I ultimately think ensembling and hyperparameter optimization will be key (or even essential), I hope we reduced that problem.

", "rawMarkdown": "> It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.\n\nI love this. 👍\n\nA goal some of the organizers of the competition and I shared was preventing the competitor with the most resources to ensemble and optimize hyperparameters from dominating. While I ultimately think ensembling and hyperparameter optimization will be key (or even essential), I hope we reduced that problem.", "votes": null }, { "id": "313022", "postDate": "04/12/2018 17:13:28", "content": "
\n

I am still grinding away on the stage 2, and do not expect to be at\n the top of the leaderboard. But special thanks to @Allen for being a\n great sport, and a guiding light for all of us!

\n
\n\n

Thanks, dude! Good luck. ☺️

", "rawMarkdown": "> I am still grinding away on the stage 2, and do not expect to be at\n> the top of the leaderboard. But special thanks to @Allen for being a\n> great sport, and a guiding light for all of us!\n\nThanks, dude! Good luck. ☺️", "votes": null }, { "id": "313026", "postDate": "04/12/2018 17:18:06", "content": "

I'm just really hoping my model tries to segment the figure legend like See--'s. That will make this whole thing worth it whatever the outcome.

", "rawMarkdown": "I'm just really hoping my model tries to segment the figure legend like See--'s. That will make this whole thing worth it whatever the outcome.", "votes": null }, { "id": "313682", "postDate": "04/13/2018 17:33:33", "content": "

Yeah, a lot of images are like a combo of images from different stain spectra from BBBC06 dataset. Reguarding external data, the playing field is actually probably not turned in favour of those who used much of it. 99% of that data, almost all except BBBC06 can only hurt perf on tests. Unless, maybe, you totally knew what the data's gonna be and you preprocessed it accordingly before feeding it into learning. It's generally much too complicated. And if you also thought there's gonna be even more diversity and applied heavy augmentation, like color invertion, etc, you're screwed big time.

", "rawMarkdown": "Yeah, a lot of images are like a combo of images from different stain spectra from BBBC06 dataset. Reguarding external data, the playing field is actually probably not turned in favour of those who used much of it. 99% of that data, almost all except BBBC06 can only hurt perf on tests. Unless, maybe, you totally knew what the data's gonna be and you preprocessed it accordingly before feeding it into learning. It's generally much too complicated. And if you also thought there's gonna be even more diversity and applied heavy augmentation, like color invertion, etc, you're screwed big time.", "votes": null }, { "id": "313691", "postDate": "04/13/2018 17:52:09", "content": "

I think what you guys should have actually done if you wanted to find out the best solution is you would provide, like DStjhb said, at least some pics with the types of datasets that the models are gonna be tested on and just get a really big and diverse - to your understanding of the level of diversity that you actually expect - not to our guesses - set of data - and do that in a more or less consistent manner as to what % of data in the training set is gonna correspond to what % of data in the testing set. You guys didn't actually include a lot of diverse data and I think the models that are gonna win are probably mostly just gonna show great results on some data similar to the training and testing data with a bonus to any people who, for example, only included BBBC06 dataset into their training loop, but not any other datasets. Because, anyone who actually expected the unexpected had most likely hurt his perf by much, including due to time wastage on chasing the wind.

", "rawMarkdown": "I think what you guys should have actually done if you wanted to find out the best solution is you would provide, like DStjhb said, at least some pics with the types of datasets that the models are gonna be tested on and just get a really big and diverse - to your understanding of the level of diversity that you actually expect - not to our guesses - set of data - and do that in a more or less consistent manner as to what % of data in the training set is gonna correspond to what % of data in the testing set. You guys didn't actually include a lot of diverse data and I think the models that are gonna win are probably mostly just gonna show great results on some data similar to the training and testing data with a bonus to any people who, for example, only included BBBC06 dataset into their training loop, but not any other datasets. Because, anyone who actually expected the unexpected had most likely hurt his perf by much, including due to time wastage on chasing the wind.", "votes": null }, { "id": "313695", "postDate": "04/13/2018 17:55:00", "content": "

I wonder, doesn't that introduce division by zero into evaluation https://www.kaggle.com/c/data-science-bowl-2018#evaluation 😆

", "rawMarkdown": "I wonder, doesn't that introduce division by zero into evaluation https://www.kaggle.com/c/data-science-bowl-2018#evaluation 😆", "votes": null }, { "id": "313701", "postDate": "04/13/2018 18:02:40", "content": "

I'm genuinely confused why there is so much distress about the stage 2 data set when it's clear we have not announced which images are scored. It's possible everyone will still be upset that we didn't design the challenge to their liking but at least wait until you see how it was designed before getting too worked up ❤

\n\n

I'm curious whether some who complained the loudest might do the best when the final scores come in!

\n\n

Heng, thanks for infusing the challenge with goodwill and a spirit of learning and community! I'm very indebted.

", "rawMarkdown": "I'm genuinely confused why there is so much distress about the stage 2 data set when it's clear we have not announced which images are scored. It's possible everyone will still be upset that we didn't design the challenge to their liking but at least wait until you see how it was designed before getting too worked up ❤\n\nI'm curious whether some who complained the loudest might do the best when the final scores come in! \n\nHeng, thanks for infusing the challenge with goodwill and a spirit of learning and community! I'm very indebted.", "votes": null }, { "id": "313719", "postDate": "04/13/2018 18:40:47", "content": "

Part of the challenge, I guess, was to train the model to really know, (as far a computer program can 'know'), what a nucleus looks like based on absolute features, as opposed to how different a nucleus looks from everything else in specific image types based on relative features. E.g. You want the model to be oriented more towards: ' a nucleus is usually the middle of a cell and may or may not have a visible nucleolus inside', as opposed to: 'a nucleus is round and brighter than background in grey images, or purple in not-grey images'. A very difficult task that wasn't too predictable, especially given the training data. However, despite all my criticism and devil's advocacy, I do reiterate that I have thoroughly enjoyed this competition.

", "rawMarkdown": "Part of the challenge, I guess, was to train the model to **really** know, (as far a computer program can 'know'), what a nucleus looks like based on **absolute** features, as opposed to how different a nucleus looks from everything else in specific image types based on **relative** features. E.g. You want the model to be oriented more towards: ' a nucleus is usually the middle of a cell and may or may not have a visible nucleolus inside', as opposed to: 'a nucleus is round and brighter than background in grey images, or purple in not-grey images'. A very difficult task that wasn't too predictable, especially given the training data. However, despite all my criticism and devil's advocacy, I do reiterate that I have thoroughly enjoyed this competition.", "votes": null }, { "id": "313725", "postDate": "04/13/2018 18:57:57", "content": "

I was initially distressed because I thought my model scored 0 points and I would end up at the bottom of LB... All because the stage 2 image data was so different and my model does not generalize at all in the way the organizers wanted. Since I did not have any domain knowledge in biology and fairly new to deep learning, computer vision and data science, I just tried to fit Mask RCNN as best as I could. But it turned out that it is exactly what the organizers did not want. I learned a lot but I was also hoping to get a medal so I can brag about it to my wife and kids. They've been complaining that I'm spending too much time on this competition.... But my model got 0 points, I thought. 2+ month of hard work and nothing to show for it, I thought.

\n\n

Later I realized that it was just my ignorance and my tendency to jump to conclusion. I'm now just curious about how well Heng's solution would rank in private LB.

", "rawMarkdown": "I was initially distressed because I thought my model scored 0 points and I would end up at the bottom of LB... All because the stage 2 image data was ***so*** different and my model does not generalize at all in the way the organizers wanted. Since I did not have any domain knowledge in biology and fairly new to deep learning, computer vision and data science, I just tried to fit Mask RCNN as best as I could. But it turned out that it is exactly what the organizers did not want. I learned a lot but I was also hoping to get a medal so I can brag about it to my wife and kids. They've been complaining that I'm spending too much time on this competition.... But my model got 0 points, I thought. 2+ month of hard work and nothing to show for it, I thought. \n\nLater I realized that it was just my ignorance and my tendency to jump to conclusion. I'm now just curious about how well Heng's solution would rank in private LB.", "votes": null }, { "id": "313957", "postDate": "04/14/2018 06:36:58", "content": "

Hi @See,\nThank you very much for sharing the code! There are few images in the test set that are empty and my model predicted as an empty image. I struggled a lot to submit my predictions since RLE doesn't incorporate zeros. After running your code, I was able to submit it successfully. For the benefit of other fellow participants, it would be useful if you upload the code in the kernel section.

\n\n

Thanks again, \nWarm regards,\nK.Vikraman

", "rawMarkdown": "Hi @See,\nThank you very much for sharing the code! There are few images in the test set that are empty and my model predicted as an empty image. I struggled a lot to submit my predictions since RLE doesn't incorporate zeros. After running your code, I was able to submit it successfully. For the benefit of other fellow participants, it would be useful if you upload the code in the kernel section. \n\nThanks again, \nWarm regards,\nK.Vikraman", "votes": null }, { "id": "314103", "postDate": "04/14/2018 15:58:28", "content": "

@bsp2020 kudos to you, I'm sure your model won't let you down and you can catch up with your life again after the competition. We're similar, forgot to feed our baby daughter because we got worked up too much on Kaggle. This was our first Kaggle competition and we were new to data science and computer vision as well, the only thing we know is how to write code - that helped, I guess. :) I'm still frustrated after having seen our visualized results. It feels like as if the Broad Institute gave us the dog images as the training set, and then we have to predict hotdog images at the 2nd stage. :D

", "rawMarkdown": "bsp2020 kudos to you, I'm sure your model won't let you down and you can catch up with your life again after the competition. We're similar, forgot to feed our baby daughter because we got worked up too much on Kaggle. This was our first Kaggle competition and we were new to data science and computer vision as well, the only thing we know is how to write code - that helped, I guess. :) I'm still frustrated after having seen our visualized results. It feels like as if the Broad Institute gave us the dog images as the training set, and then we have to predict hotdog images at the 2nd stage. :D", "votes": null }, { "id": "314104", "postDate": "04/14/2018 16:02:17", "content": "

Spoken as a biologist, totally deserves an upvote, I have no idea of lots of terms you talked about ;)

", "rawMarkdown": "Spoken as a biologist, totally deserves an upvote, I have no idea of lots of terms you talked about ;)", "votes": null }, { "id": "314108", "postDate": "04/14/2018 16:11:17", "content": "

Just my 2 cents, @Anne, I recall you discouraged people from training external data; however, we expected the 2nd stage test images would look drastically different by googling other nuclei in different shapes and colours. I think training external data shouldn't have been even allowed in the first place if your goal was to find a robust model that can withstand unseen data. Otherwise you may end up getting a winning model only because it's trained on external data. We'll see what the winning model has in store for us. ;)

", "rawMarkdown": "Just my 2 cents, @Anne, I recall you discouraged people from training external data; however, we expected the 2nd stage test images would look drastically different by googling other nuclei in different shapes and colours. I think training external data shouldn't have been even allowed in the first place if your goal was to find a robust model that can withstand unseen data. Otherwise you may end up getting a winning model only because it's trained on external data. We'll see what the winning model has in store for us. ;)", "votes": null }, { "id": "314117", "postDate": "04/14/2018 16:27:15", "content": "

Dear @Anne, @Allen. I asked in an other forum already, and never got an answer, nonetheless it would be very crucial IMO. Here is it:

\n\n

By definition if there is no cell on the image and:

\n\n

a) we predict 0-> the score is 0/0=NaN, although the prediction was good. Are you giving 0 or 1 in this case?

\n\n

b) we predict more than zero than the score 0/x = 0.

\n\n

What is the score on case a)?

\n\n

I appreciate,\nPeter

", "rawMarkdown": "Dear @Anne, @Allen. I asked in an other forum already, and never got an answer, nonetheless it would be very crucial IMO. Here is it: \n\nBy definition if there is no cell on the image and: \n\na) we predict 0-> the score is 0/0=NaN, although the prediction was good. Are you giving 0 or 1 in this case? \n\nb) we predict more than zero than the score 0/x = 0.\n\nWhat is the score on case a)?\n\nI appreciate,\nPeter", "votes": null }, { "id": "314160", "postDate": "04/14/2018 18:45:03", "content": "

Anne,

\n\n

TL;DR It is uncertainty that stresses people, not the images.

\n\n

Details below.

\n\n

I think there is distress because there is uncertainty. The rules say we must use the same models as in stage 1, but then stage1 test labels are disclosed, and retraining is allowed. This makes it very tempting to bend the rules and retrain on data that looks like stage2 test data.

\n\n

What is needed is a clear cut boundary between what is allowed and what is not. I wish the clear cut boundary had been: no retraining in stage2.

\n\n

The other source of uncertainty is the zero score. Few of us explained it, pointed to past competition, but still, people want an 'official confirmation', as if previous competition official confirmation was not enough.

\n\n

A third source of distress is that questions stay unanswered for more than 24 hours. Sure, I can imagine that from organizer perspective 2 hours is not a big deal, but when you're stressed and wait for answer it is really bad.

\n\n

I tried to capture what many think above. It is not just me.

\n\n

Beside this lack of clear cut communication from organizers I really enjoyed that competition, learned a lot of stuff for my first image competition, I teamed with nice people, and I really think Heng Cher Keng will become discussion and competition grandmaster very soon! Thanks for organizing this, despite our grumbling.

", "rawMarkdown": "Anne,\n\nTL;DR It is uncertainty that stresses people, not the images.\n\nDetails below.\n\nI think there is distress because there is uncertainty. The rules say we must use the same models as in stage 1, but then stage1 test labels are disclosed, and retraining is allowed. This makes it very tempting to bend the rules and retrain on data that looks like stage2 test data. \n\nWhat is needed is a clear cut boundary between what is allowed and what is not. I wish the clear cut boundary had been: no retraining in stage2.\n\nThe other source of uncertainty is the zero score. Few of us explained it, pointed to past competition, but still, people want an 'official confirmation', as if previous competition official confirmation was not enough.\n\nA third source of distress is that questions stay unanswered for more than 24 hours. Sure, I can imagine that from organizer perspective 2 hours is not a big deal, but when you're stressed and wait for answer it is really bad.\n\nI tried to capture what many think above. It is not just me.\n\nBeside this lack of clear cut communication from organizers I really enjoyed that competition, learned a lot of stuff for my first image competition, I teamed with nice people, and I really think Heng Cher Keng will become discussion and competition grandmaster very soon! Thanks for organizing this, despite our grumbling.", "votes": null }, { "id": "314883", "postDate": "04/16/2018 12:35:08", "content": "

Again, the images in stage 2 test consist of a subset of images that are actually scored, and a set of images that are there to ensure you don't hand-label the entire set or tune your model to the precise images that are scored.

", "rawMarkdown": "Again, the images in stage 2 test consist of a subset of images that are actually scored, and a set of images that are there to ensure you don't hand-label the entire set or tune your model to the precise images that are scored.", "votes": null }, { "id": "314957", "postDate": "04/16/2018 14:57:00", "content": "

Hi @Anne, thanks a lot for the clarification. Makes total sense and this will ensure the competition is fair!

", "rawMarkdown": "Hi @Anne, thanks a lot for the clarification. Makes total sense and this will ensure the competition is fair!", "votes": null }, { "id": "315012", "postDate": "04/16/2018 16:30:10", "content": "

This is indeed true, please see the discussion thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/54575 for an ongoing debate on this matter.

", "rawMarkdown": "This is indeed true, please see the discussion thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/54575 for an ongoing debate on this matter.", "votes": null }, { "id": "315016", "postDate": "04/16/2018 16:32:29", "content": "

@Any of the organizers: \nJust out of curiosity, please advise on the cells the 2 drawings in the Stage 2 test set (170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece.png & 5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2.png) are supposed to represent? Thanks.

", "rawMarkdown": "Any of the organizers: \nJust out of curiosity, please advise on the cells the 2 drawings in the Stage 2 test set (170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece.png & 5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2.png) are supposed to represent? Thanks.", "votes": null }, { "id": "315028", "postDate": "04/16/2018 16:44:18", "content": "

I'm not sure why you are asking?

", "rawMarkdown": "I'm not sure why you are asking?", "votes": null }, { "id": "315062", "postDate": "04/16/2018 17:50:18", "content": "

I just saw this video today: https://datasciencebowl.com/2018dsbtutorial/ (a bit too late) and in retrospect the hint about diversity (and some images shown) was already there.

", "rawMarkdown": "I just saw this video today: https://datasciencebowl.com/2018dsbtutorial/ (a bit too late) and in retrospect the hint about diversity (and some images shown) was already there.", "votes": null }, { "id": "315070", "postDate": "04/16/2018 17:59:01", "content": "

I was able to get an answer from the Kaggle team - if there are no nuclei in an image, submit an empty string.

\n\n

LMK any other threads on this topic so I can reply there.

", "rawMarkdown": "I was able to get an answer from the Kaggle team - if there are no nuclei in an image, submit an empty string.\n\nLMK any other threads on this topic so I can reply there.", "votes": null }, { "id": "315075", "postDate": "04/16/2018 18:04:43", "content": "

@Anne, I think the question is about the score we get if we predict correctly that there is no nuclei on an image. The official scoring definition does not cover that case.

", "rawMarkdown": "Anne, I think the question is about the score we get if we predict correctly that there is no nuclei on an image. The official scoring definition does not cover that case.", "votes": null }, { "id": "315090", "postDate": "04/16/2018 18:28:20", "content": "

And that's why it seems pretty obvious that it's one of those images that won't count.

", "rawMarkdown": "And that's why it seems pretty obvious that it's one of those images that won't count.", "votes": null }, { "id": "315098", "postDate": "04/16/2018 18:37:59", "content": "

@theNickName, congratulations for being hired on Kaggle Staff. Didn't know that. ;)

", "rawMarkdown": "theNickName, congratulations for being hired on Kaggle Staff. Didn't know that. ;)", "votes": null }, { "id": "315110", "postDate": "04/16/2018 18:58:54", "content": "

That question was cleared by @Wendy in another thread: score is 0 for an image that contains no nuclei, whatever you predict for it.

", "rawMarkdown": "That question was cleared by @Wendy in another thread: score is 0 for an image that contains no nuclei, whatever you predict for it.", "votes": null }, { "id": "315148", "postDate": "04/16/2018 19:31:48", "content": "

Could there have been a different option? The only other possible option would be against the evaluation rules, the only other impossible option would be scoring an undefined number.

", "rawMarkdown": "Could there have been a different option? The only other possible option would be against the evaluation rules, the only other impossible option would be scoring an undefined number.", "votes": null }, { "id": "315151", "postDate": "04/16/2018 19:38:42", "content": "

There is, you can define the value of 0/0 as you wish. I would have rewarded those who do not predict nuclei when there is none.

", "rawMarkdown": "There is, you can define the value of 0/0 as you wish. I would have rewarded those who do not predict nuclei when there is none.", "votes": null }, { "id": "315153", "postDate": "04/16/2018 19:39:53", "content": "

As an example of metrics with special case for 0/0 have a look at the SMAPE metrics used in the Web Traffic Forecasting competition. It is not even continuous at 0: https://www.kaggle.com/cpmpml/smape-weirdness

", "rawMarkdown": "As an example of metrics with special case for 0/0 have a look at the SMAPE metrics used in the Web Traffic Forecasting competition. It is not even continuous at 0: https://www.kaggle.com/cpmpml/smape-weirdness", "votes": null }, { "id": "315187", "postDate": "04/16/2018 20:52:19", "content": "

Yes, of course, but who would redefine the rules for the sake of a single meaningless case?

", "rawMarkdown": "Yes, of course, but who would redefine the rules for the sake of a single meaningless case?", "votes": null }, { "id": "315190", "postDate": "04/16/2018 20:59:26", "content": "

Anyways, can't wait to see how y'all tackled this challenge. And who trained on what data and ends up at what place as a result.

", "rawMarkdown": "Anyways, can't wait to see how y'all tackled this challenge. And who trained on what data and ends up at what place as a result.", "votes": null } ], "comments": [ { "id": 312540, "author_name": "weizhezhang", "author_url": "", "post_date": "04/12/2018 00:53:46", "content": "

If I remember correctly, there are some pictures not for scoring which are used to prevent pseudo labeling.

", "votes": null, "replies": [] }, { "id": 312541, "author_name": "antorsae", "author_url": "", "post_date": "04/12/2018 00:54:12", "content": "

Makes you wonder whether we should have prioritized learning microbiology and annotating different datasets rather than fine-tuning off-the-shelf instance segmentation architectures, or training regimes on stage 1 train...

", "votes": null, "replies": [ { "id": 312543, "author_name": "hengck23", "author_url": "", "post_date": "04/12/2018 00:57:15", "content": "

Now i opted for this \"learning microbiology and annotating different datasets\". Actually i submitted a solution not with the highest public LB score. I submission one with LB=0.505 which i think work best on my downloaded cell images.

\n\n

but even that, my downloaded cell images are very different from what i saw in stage2 dataset

", "votes": null, "replies": [] }, { "id": 312545, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 01:02:35", "content": "

Maybe! 😉

\n\n

We intentionally made it difficult for anybody to win by just overfitting ensembles of ensembles of ensembles. We wanted a generalization or robustness challenge.

", "votes": null, "replies": [] }, { "id": 312546, "author_name": "hengck23", "author_url": "", "post_date": "04/12/2018 01:06:11", "content": "

@Allen what is your score range for stage2, using the model you build from stage1? I recall you mentioned there is large overfitting on stage1 in your thread.

", "votes": null, "replies": [] }, { "id": 312547, "author_name": "antorsae", "author_url": "", "post_date": "04/12/2018 01:09:11", "content": "

Funny we attempted (ran out of time for 10 mins) to generate new images using a CycleGAN using our GT as reference. Images came out quite nice, but still since we were using GT from stage 1 the nuclei shapes are nothing like stage 2 so would not have helped anyway.

", "votes": null, "replies": [] }, { "id": 312551, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 01:11:15", "content": "

@Heng My best performing model was in the low 40s. (I didn’t have time to check my other models but I suspect they would’ve done better. )

", "votes": null, "replies": [] }, { "id": 312552, "author_name": "hengck23", "author_url": "", "post_date": "04/12/2018 01:12:28", "content": "

Thanks!

", "votes": null, "replies": [] }, { "id": 312784, "author_name": "rakhlin", "author_url": "", "post_date": "04/12/2018 10:21:47", "content": "
\n

We intentionally made it difficult for anybody to win by just\n overfitting ensembles of ensembles of ensembles. We wanted a\n generalization or robustness challenge.

\n
\n\n

What you actually did, you failed on your part to prepare and provide diverse data needed to create robust models, wasted sponsor money and kagglers time, and turned erstwhile notorious ds bowl into joke.

", "votes": null, "replies": [] }, { "id": 312831, "author_name": "dredata", "author_url": "", "post_date": "04/12/2018 12:35:56", "content": "

I don't disagree with this comment. It's a competitions with a bunch of gotchas and plot twists. However this is not the way you'd actually try to generate a practical solution to a real problem.

", "votes": null, "replies": [] }, { "id": 312897, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 14:08:47", "content": "

@David and @rakhlin Will you feel this way if competitors post scores in the range (say ±0.1 AP) of the phase one leaderboard?

", "votes": null, "replies": [] }, { "id": 312929, "author_name": "dredata", "author_url": "", "post_date": "04/12/2018 14:44:05", "content": "

I'll stick by the statement that if you were trying to come up with a practical solution you wouldn't split up the data in the way it was split up. It will be interesting to see how the final scores come out. My feeling is that the leaders in the 2nd round will be different than in the first round. But we'll see what happens.

", "votes": null, "replies": [] }, { "id": 312937, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 14:50:47", "content": "

@David If a competitor came up with a solution that evaluated to, for example, 0.5 AP on both phase one AND phase two you wouldn’t consider that a practical solution?

", "votes": null, "replies": [] }, { "id": 312942, "author_name": "dredata", "author_url": "", "post_date": "04/12/2018 14:59:03", "content": "

What would the score have been if the data had been split up in a more reasonable manner? You're not making the right comparison. I don't know what the ultimate best score could be but I rather doubt that you can get to it by using the available data in the way it was used in the competition.

", "votes": null, "replies": [] }, { "id": 312945, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 15:02:57", "content": "

@David The competition wasn’t to see who could most overfit their model (that’s a useful competition, but that wasn’t the purpose of this competition) to a specific set of experimental conditions. It was, plainly, to see who could build a model that best adapts to data with unseen experimental conditions.

", "votes": null, "replies": [] }, { "id": 312950, "author_name": "dredata", "author_url": "", "post_date": "04/12/2018 15:11:30", "content": "

Okay the point of my comments is not to be overly critical of the competition. I'm just trying to point out that if you were trying to generate the best possible predictor from a given set of data you wouldn't purposefully exclude classes of data from the training set. That's not how you avoid over training and ensure generalization. That makes a good competition but not smart if you want the best possible solution to a problem.

", "votes": null, "replies": [] }, { "id": 312955, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 15:19:24", "content": "

Yes, of course, but this is a competition. This isn’t a practical problem. You’re absolutely correct that, in the real world, it’d be silly to not use all available data. This was a competition to help unearth strategies to construct and train a model to solve practical problems. I don’t know how you’d design a competition without a challenging holdout that achieved that aim.

", "votes": null, "replies": [] }, { "id": 312956, "author_name": "shiraz14", "author_url": "", "post_date": "04/12/2018 15:21:42", "content": "

@Allen: Indeed, overfitting is (in most instances) an undesirable effect of constant retraining and fine-tuning of a model. However, the data set (in this context) should have been more practically split (with some of the Stage 2 images being introduced in Stage 1 for model training and validation). Introducing a starkly different dataset to an already previously-trained model & expecting a high percentage accuracy (and by high, I'm referring to a score of preferably 0.75++) would seemingly be a little over-ambitious, although this might not be so, if the model had been exposed to a more diverse training set from the start. This would be akin to human learning, where if you are (for example) a trained microbiologist, you would be an expert in a specified field of biology (say, microbial genetics), although if questions on advanced engineering mathematics were posed to you, you would probably be stumped and not be able to answer such questions (if you had no prior learning in this area), without consulting external references/resources. The same applies to a trained ML model, which might work pretty well on a test set similar (though not necessarily identical) to the training set, although the same model might fail drastically if required to predict the outcomes from a completely different test set.

", "votes": null, "replies": [] }, { "id": 312964, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 15:31:52", "content": "
\n

The same applies to a trained ML model, which might work pretty well\n on a test set similar (though not necessarily identical) to the\n training set, although the same model might fail drastically if\n required to predict the outcomes from a completely different test set.

\n
\n\n

Exactly. We’re interested in models that were able to adapt to these conditions. I suspect the winners of the competition will be people or teams that focused entirely on that particular issue.

\n\n

I should add that none of the organizing biologists would describe the images from phase two as “starkly different” than phase one. In fact, there were instances where they eliminated certain images for that reason.

", "votes": null, "replies": [] }, { "id": 312972, "author_name": "shiraz14", "author_url": "", "post_date": "04/12/2018 15:49:47", "content": "

@Allen: Some of the images introduced in Stage 2 were drawings scanned from a textbook, and not even photomicrographs - wouldn't you consider those to be starkly different from phase One (where no drawings were present in both the training and test data sets)? And some of these drawings also had other organelles (such as mitochondria) which might seem confusing to a trained ML model, and be interpreted as nuclei, so unless the organizers (including yourself) had put in these images but did not intend them to be scored, low scores would be inevitable for Stage 2 (even for the eventual winners), which leads us to the initial point that your primary objective for organizing this Competition (to attain robust models for accurately predicting the ROI corresponding to a cell's nucleus) would be unmet.

", "votes": null, "replies": [] }, { "id": 313002, "author_name": "jbfarrar", "author_url": "", "post_date": "04/12/2018 16:45:54", "content": "

My two cents on this thread, is that the organizers have put on a great contest! Engaging problem set!

\n\n

After putting the kind of time we all put into this its natural to be frustrated by the stage 2 data. But the real world is always dealing us new unknowns.

\n\n

The organizers are writing the checks and designed an approach to get what they wanted out of it. At the end of the day the goal of ML is to take known circumstances and translate the learning to new ones. While its hard to learn about circles when all of your data are squares, that is not even close to the situation here. Its a fair fight, and extremely fun to work on.

\n\n

I am still grinding away on the stage 2, and do not expect to be at the top of the leaderboard. But special thanks to @Allen for being a great sport, and a guiding light for all of us!

\n\n

Cheers in advance to the winners. You will have assisted in the solving of a critical problem for humankind.

", "votes": null, "replies": [] }, { "id": 313011, "author_name": "dstjhb", "author_url": "", "post_date": "04/12/2018 16:52:39", "content": "

I agree, it is fun, and we were forewarned; but I did think there might be at least one or two examples of each type of image in the training dataset, even if it was somewhat biased towards one image type or another. It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.

", "votes": null, "replies": [] }, { "id": 313020, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 17:12:44", "content": "
\n

It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.

\n
\n\n

I love this. 👍

\n\n

A goal some of the organizers of the competition and I shared was preventing the competitor with the most resources to ensemble and optimize hyperparameters from dominating. While I ultimately think ensembling and hyperparameter optimization will be key (or even essential), I hope we reduced that problem.

", "votes": null, "replies": [] }, { "id": 313022, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 17:13:28", "content": "
\n

I am still grinding away on the stage 2, and do not expect to be at\n the top of the leaderboard. But special thanks to @Allen for being a\n great sport, and a guiding light for all of us!

\n
\n\n

Thanks, dude! Good luck. ☺️

", "votes": null, "replies": [] }, { "id": 313691, "author_name": "thenickname", "author_url": "", "post_date": "04/13/2018 17:52:09", "content": "

I think what you guys should have actually done if you wanted to find out the best solution is you would provide, like DStjhb said, at least some pics with the types of datasets that the models are gonna be tested on and just get a really big and diverse - to your understanding of the level of diversity that you actually expect - not to our guesses - set of data - and do that in a more or less consistent manner as to what % of data in the training set is gonna correspond to what % of data in the testing set. You guys didn't actually include a lot of diverse data and I think the models that are gonna win are probably mostly just gonna show great results on some data similar to the training and testing data with a bonus to any people who, for example, only included BBBC06 dataset into their training loop, but not any other datasets. Because, anyone who actually expected the unexpected had most likely hurt his perf by much, including due to time wastage on chasing the wind.

", "votes": null, "replies": [] }, { "id": 313719, "author_name": "dstjhb", "author_url": "", "post_date": "04/13/2018 18:40:47", "content": "

Part of the challenge, I guess, was to train the model to really know, (as far a computer program can 'know'), what a nucleus looks like based on absolute features, as opposed to how different a nucleus looks from everything else in specific image types based on relative features. E.g. You want the model to be oriented more towards: ' a nucleus is usually the middle of a cell and may or may not have a visible nucleolus inside', as opposed to: 'a nucleus is round and brighter than background in grey images, or purple in not-grey images'. A very difficult task that wasn't too predictable, especially given the training data. However, despite all my criticism and devil's advocacy, I do reiterate that I have thoroughly enjoyed this competition.

", "votes": null, "replies": [] } ] }, { "id": 312544, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 00:59:44", "content": "

https://youtu.be/1g6Zb98KYJE

\n\n

You can’t say we didn’t warn you! Good luck everybody! 👍

", "votes": null, "replies": [ { "id": 312554, "author_name": "antorsae", "author_url": "", "post_date": "04/12/2018 01:16:33", "content": "

I'd like to candidly ask if for the purpose of getting a great winning model at the end of the competition it would have been better to include at least one instance of each type of test image on stage 1 test; and if not, what is the motivation from the competition organizer's point of view to come up with such vastly different test images?

", "votes": null, "replies": [] }, { "id": 312556, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 01:22:57", "content": "

It’s a fair question. I’m one of many organizers, so I can only provide my personal perspective. I’m interested in accuracy (as described by the competition metric), but I think the far more exciting and far more useful metric is robustness. An accurate model is valuable, but a less accurate model that can adapt to new experimental conditions is truly groundbreaking.

\n\n

I should add that from a user’s perspective (and also a computer vision perspective) the images are more similar than other image segmentation competitions.

", "votes": null, "replies": [] }, { "id": 312600, "author_name": "shiraz14", "author_url": "", "post_date": "04/12/2018 02:57:12", "content": "

Yes indeed, but the imaging modalities used (brightfield & fluorescence) are the same as for stage 1, so the algorithms used by the contestants for Stage 1 should work fine (with the exception of a few tweaks, of course) ... 😉

", "votes": null, "replies": [] }, { "id": 312793, "author_name": "matony114", "author_url": "", "post_date": "04/12/2018 10:46:38", "content": "

May I ask a question? Is there some images that actually don't have any nuclei? like this one?2f25c16e472f0b5f0c14b654a4e78f8f83ad7505f056d48349e1e12e0de98839.png

", "votes": null, "replies": [] }, { "id": 312947, "author_name": "lpachuong", "author_url": "", "post_date": "04/12/2018 15:06:20", "content": "

Now I understood why you \"facepalm\" :D

", "votes": null, "replies": [] } ] }, { "id": 312560, "author_name": "wayne999", "author_url": "", "post_date": "04/12/2018 01:25:45", "content": "

private lb will be really interesting

", "votes": null, "replies": [] }, { "id": 312561, "author_name": "dredata", "author_url": "", "post_date": "04/12/2018 01:28:36", "content": "

Yeah I was worried about this. The idea is to generalize but to do this the training examples need to 'span the space.' I trained on all the data that was posted in the extra data thread that was usable (good ground truth images) and then augmented that. I ended up training on ~27,000 images and after spot checking the 2nd round test, there are still types of images I didn't have to train on. The final results will be interesting and maybe a bit of a crap shoot.

", "votes": null, "replies": [] }, { "id": 312565, "author_name": "dskswu", "author_url": "", "post_date": "04/12/2018 01:43:36", "content": "

What does it means when the submission is still initializing? I am trying to make sure our team made the deadline.

", "votes": null, "replies": [] }, { "id": 312581, "author_name": "ruiforecasting", "author_url": "", "post_date": "04/12/2018 02:08:35", "content": "

There are few images are just very strange and I don't know whether they are real....

\n\n

For example, the image starts with 0fd3ed... and the one starts with 0fd475a...

\n\n

They are just completely black image to me.... are they real cell images for scoring? I am not sure whether the submission will allow 0 masks for some images...

", "votes": null, "replies": [ { "id": 312592, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 02:39:09", "content": "

I just checked. They are not zeroed out. If you equalize them you'll see standard fluorescence noise. Just like with any other image, you need to carefully manage false positives.

", "votes": null, "replies": [] }, { "id": 312811, "author_name": "albuslaev", "author_url": "", "post_date": "04/12/2018 11:31:27", "content": "

Hi, Allen. \nWe are not allowed to change source codes, but our networks predict zeros for some images. What should we put into submission file in case if we strictly want to submit zeros for some image? Current submission format prohibits excluding img_ids with zeros only but zeros only could not be encoded with rle. Thanks

", "votes": null, "replies": [] }, { "id": 312816, "author_name": "seesee", "author_url": "", "post_date": "04/12/2018 11:45:03", "content": "

@Alexander Buslaev I have the same problem. This is what I am using. I just add id + comma and not 0 0 (seems to work):

\n\n
import pandas as pd\nimport argparse\nimport os\nimport shutil\n\n\ndef main():\n  parser = argparse.ArgumentParser()\n  parser.add_argument(\n      '--submission_fn',\n      type=str,\n      default=\"\",\n      help=\"\"\"\\\n      Path to the submission file.\\\n      \"\"\")\n  parser.add_argument(\n      '--sample_submission_fn',\n      type=str,\n      default=\"\",\n      help=\"\"\"\\\n      Path to the sample submission filename.\\\n      \"\"\")\n  args, unparsed = parser.parse_known_args()\n  assert os.path.exists(args.submission_fn)\n  assert os.path.exists(args.sample_submission_fn)\n  sub = pd.read_csv(args.submission_fn)\n  sub_ids = set(sub['ImageId'])\n  sample_sub = pd.read_csv(args.sample_submission_fn)\n  sample_sub_ids = set(sample_sub['ImageId'])\n  missing_ids = sample_sub_ids.difference(sub_ids)\n\n  print(\"Adding %d missing ids.\" % (len(missing_ids)))\n  out_fn = args.submission_fn.replace('.csv', '_all_ids.csv')\n  shutil.copy(args.submission_fn, out_fn)\n  with open(out_fn, 'a') as f:\n    for missing_id in missing_ids:\n      f.write('%s,\\n' % (missing_id))\n\n  print(\"Done! Wrote updated submission to %s.\" % (out_fn))\n\n\nif __name__ == '__main__':\n  main()\n
", "votes": null, "replies": [] }, { "id": 312824, "author_name": "hengck23", "author_url": "", "post_date": "04/12/2018 12:12:43", "content": "

\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"

\n\n

just put image_id without rle (i.e leave it empty, like empty string '')

\n\n

We can use this:

\n\n

The submission process (for stage1 and stage2) is :

\n\n", "votes": null, "replies": [] }, { "id": 313695, "author_name": "thenickname", "author_url": "", "post_date": "04/13/2018 17:55:00", "content": "

I wonder, doesn't that introduce division by zero into evaluation https://www.kaggle.com/c/data-science-bowl-2018#evaluation 😆

", "votes": null, "replies": [] }, { "id": 313957, "author_name": "vikramank", "author_url": "", "post_date": "04/14/2018 06:36:58", "content": "

Hi @See,\nThank you very much for sharing the code! There are few images in the test set that are empty and my model predicted as an empty image. I struggled a lot to submit my predictions since RLE doesn't incorporate zeros. After running your code, I was able to submit it successfully. For the benefit of other fellow participants, it would be useful if you upload the code in the kernel section.

\n\n

Thanks again, \nWarm regards,\nK.Vikraman

", "votes": null, "replies": [] } ] }, { "id": 312618, "author_name": "jfaath", "author_url": "", "post_date": "04/12/2018 03:49:50", "content": "

This is the best one:

\n\n

5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2

\n\n

It's straight out of a textbook. It even has the caption 'Fig 5.' still on it. Notably it was a tiff file even though it was saved with a .png extension. I wonder if it's in the stage 2 set by mistake. I posted a similar one in the pinned data quality thread.

", "votes": null, "replies": [ { "id": 312736, "author_name": "ildoonet", "author_url": "", "post_date": "04/12/2018 08:27:10", "content": "

I guess the organizer put that image on purpose. 'Fig 5.' should NOT be detected if a model REALLY learns what are nuclei cells or not.

", "votes": null, "replies": [] }, { "id": 313012, "author_name": "bsp2020", "author_url": "", "post_date": "04/12/2018 16:53:34", "content": "

I'd argue that a good model should not detect any nuclei since there is no real nuclei in that drawing (it's a drawing, not a picture). I suspect that it won't be used for the LB calculation.

", "votes": null, "replies": [] }, { "id": 313014, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 16:56:54", "content": "

\"enter

\n\n

A classic conundrum in computer vision (and philosophy)! (Biology too, maybe!)

", "votes": null, "replies": [] }, { "id": 313017, "author_name": "dstjhb", "author_url": "", "post_date": "04/12/2018 17:01:20", "content": "

Depends how good the drawing is... A camera just captures (a slightly more accurate) picture of the nuclei too, and normally indirectly via some dye or other. However, as I pointed out in another thread, if you are making a drawing of a H&E stained tissue, why not just take a picture under the scope? That would be more useful and objective.

", "votes": null, "replies": [] }, { "id": 313018, "author_name": "agoodman", "author_url": "", "post_date": "04/12/2018 17:06:59", "content": "

@DStjhb I agree. 😉 I also agree it's probably poison.

\n\n

Nevertheless, it’s a cute image and fun to think about in context of the competition.

", "votes": null, "replies": [] }, { "id": 313026, "author_name": "dstjhb", "author_url": "", "post_date": "04/12/2018 17:18:06", "content": "

I'm just really hoping my model tries to segment the figure legend like See--'s. That will make this whole thing worth it whatever the outcome.

", "votes": null, "replies": [] } ] }, { "id": 312779, "author_name": "alexz0", "author_url": "", "post_date": "04/12/2018 10:11:58", "content": "

The image 170bc41b2095177ccc..... is a jpeg, not a png.\nThe art style is similar to the tiff image mentioned above.

", "votes": null, "replies": [] }, { "id": 312817, "author_name": "perlflo", "author_url": "", "post_date": "04/12/2018 11:53:28", "content": "

Actually i don't really grasp the purpose of the challenge - having a generalizing and robust model is fine, it may even be groundbreaking, but part of the images in stage 2 test are just of bad quality, integration times are sometimes much too low or there is just some random noise. In practice, you can't work with these images because actually what you would do in case of fluorescence images was i.e. to stain with antibodies you want to quantify or to detect, see if certain cells/nuclei are positive for a certain antibody or to count the number of FISH probes per nucleus- you would never run and trust an analysis based on images like those...

", "votes": null, "replies": [ { "id": 314104, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/14/2018 16:02:17", "content": "

Spoken as a biologist, totally deserves an upvote, I have no idea of lots of terms you talked about ;)

", "votes": null, "replies": [] }, { "id": 314883, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/16/2018 12:35:08", "content": "

Again, the images in stage 2 test consist of a subset of images that are actually scored, and a set of images that are there to ensure you don't hand-label the entire set or tune your model to the precise images that are scored.

", "votes": null, "replies": [] }, { "id": 314957, "author_name": "vikramank", "author_url": "", "post_date": "04/16/2018 14:57:00", "content": "

Hi @Anne, thanks a lot for the clarification. Makes total sense and this will ensure the competition is fair!

", "votes": null, "replies": [] } ] }, { "id": 312819, "author_name": "dstjhb", "author_url": "", "post_date": "04/12/2018 11:58:59", "content": "

I am trained in biology, and from that perspective, the stage 2 images are not drastically different from what I might have expected to see in my day-to-day work. That said, I focused on training for the range of features I saw in the combined stage 1 train and test images. I expected some zoom changes, but not IF images with cytoplasmic dyes, or radically different cell types (neurons). I was quite selective with my training sets, and kept them quite small.

\n\n

I do worry that the winner is essentially the team with the biggest GPU farm to process the largest and most diverse external dataset comprised of different bioimaging methods. I guess we will have to wait and see.

\n\n

Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.

", "votes": null, "replies": [ { "id": 312820, "author_name": "hengck23", "author_url": "", "post_date": "04/12/2018 12:01:25", "content": "

\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"

\n\n

just put image_id without rle (i.e leave it empty, like empty string '')

", "votes": null, "replies": [] }, { "id": 312823, "author_name": "antorsae", "author_url": "", "post_date": "04/12/2018 12:12:19", "content": "

Which of the external datasets (of course mentioned in the external data thread, to be allowed here) do you think would have helped?

\n\n

IMO incrementing or enriching the external data for a competition makes total sense but on this one (and based on the external labelled datasets I checked on the external data thread) the effort required to get a realistic training set needed to get SOTA results on stage 2 date is orders of magnitude higher than in other competitions. I would love to be proven wrong.

", "votes": null, "replies": [] }, { "id": 312894, "author_name": "dstjhb", "author_url": "", "post_date": "04/12/2018 14:05:47", "content": "

If some people had used relevant training data my point would probably stand, but you're right that there wasn't really anything posted there that would have made too much difference; so the playing field might have turned out pretty level in the end. I guess everyone was finding things that were similar to those provided in stage 1.

\n\n

Thanks for the tip about leaving RLE column blank. I didn't realise we could do that.

", "votes": null, "replies": [] }, { "id": 312944, "author_name": "dstjhb", "author_url": "", "post_date": "04/12/2018 15:02:43", "content": "

One thing I don't really understand is why the fluorescent images with multiple dyes are merged in a single channel? There are some single channel phase images which make sense, but you wouldn't normally choose dyes with overlapping emission spectra to stain multiple cell compartments. Unless it is showing the distribution of a protein or chemical that can translocate between the nucleus and cytoplasm. In that case, this is a bad way to look for nuclei (which I thought was the purpose of the data).

", "votes": null, "replies": [] }, { "id": 313682, "author_name": "thenickname", "author_url": "", "post_date": "04/13/2018 17:33:33", "content": "

Yeah, a lot of images are like a combo of images from different stain spectra from BBBC06 dataset. Reguarding external data, the playing field is actually probably not turned in favour of those who used much of it. 99% of that data, almost all except BBBC06 can only hurt perf on tests. Unless, maybe, you totally knew what the data's gonna be and you preprocessed it accordingly before feeding it into learning. It's generally much too complicated. And if you also thought there's gonna be even more diversity and applied heavy augmentation, like color invertion, etc, you're screwed big time.

", "votes": null, "replies": [] } ] }, { "id": 313701, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/13/2018 18:02:40", "content": "

I'm genuinely confused why there is so much distress about the stage 2 data set when it's clear we have not announced which images are scored. It's possible everyone will still be upset that we didn't design the challenge to their liking but at least wait until you see how it was designed before getting too worked up ❤

\n\n

I'm curious whether some who complained the loudest might do the best when the final scores come in!

\n\n

Heng, thanks for infusing the challenge with goodwill and a spirit of learning and community! I'm very indebted.

", "votes": null, "replies": [ { "id": 313725, "author_name": "bsp2020", "author_url": "", "post_date": "04/13/2018 18:57:57", "content": "

I was initially distressed because I thought my model scored 0 points and I would end up at the bottom of LB... All because the stage 2 image data was so different and my model does not generalize at all in the way the organizers wanted. Since I did not have any domain knowledge in biology and fairly new to deep learning, computer vision and data science, I just tried to fit Mask RCNN as best as I could. But it turned out that it is exactly what the organizers did not want. I learned a lot but I was also hoping to get a medal so I can brag about it to my wife and kids. They've been complaining that I'm spending too much time on this competition.... But my model got 0 points, I thought. 2+ month of hard work and nothing to show for it, I thought.

\n\n

Later I realized that it was just my ignorance and my tendency to jump to conclusion. I'm now just curious about how well Heng's solution would rank in private LB.

", "votes": null, "replies": [] }, { "id": 314103, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/14/2018 15:58:28", "content": "

@bsp2020 kudos to you, I'm sure your model won't let you down and you can catch up with your life again after the competition. We're similar, forgot to feed our baby daughter because we got worked up too much on Kaggle. This was our first Kaggle competition and we were new to data science and computer vision as well, the only thing we know is how to write code - that helped, I guess. :) I'm still frustrated after having seen our visualized results. It feels like as if the Broad Institute gave us the dog images as the training set, and then we have to predict hotdog images at the 2nd stage. :D

", "votes": null, "replies": [] }, { "id": 314160, "author_name": "cpmpml", "author_url": "", "post_date": "04/14/2018 18:45:03", "content": "

Anne,

\n\n

TL;DR It is uncertainty that stresses people, not the images.

\n\n

Details below.

\n\n

I think there is distress because there is uncertainty. The rules say we must use the same models as in stage 1, but then stage1 test labels are disclosed, and retraining is allowed. This makes it very tempting to bend the rules and retrain on data that looks like stage2 test data.

\n\n

What is needed is a clear cut boundary between what is allowed and what is not. I wish the clear cut boundary had been: no retraining in stage2.

\n\n

The other source of uncertainty is the zero score. Few of us explained it, pointed to past competition, but still, people want an 'official confirmation', as if previous competition official confirmation was not enough.

\n\n

A third source of distress is that questions stay unanswered for more than 24 hours. Sure, I can imagine that from organizer perspective 2 hours is not a big deal, but when you're stressed and wait for answer it is really bad.

\n\n

I tried to capture what many think above. It is not just me.

\n\n

Beside this lack of clear cut communication from organizers I really enjoyed that competition, learned a lot of stuff for my first image competition, I teamed with nice people, and I really think Heng Cher Keng will become discussion and competition grandmaster very soon! Thanks for organizing this, despite our grumbling.

", "votes": null, "replies": [] } ] }, { "id": 314108, "author_name": "nicolefinnie", "author_url": "", "post_date": "04/14/2018 16:11:17", "content": "

Just my 2 cents, @Anne, I recall you discouraged people from training external data; however, we expected the 2nd stage test images would look drastically different by googling other nuclei in different shapes and colours. I think training external data shouldn't have been even allowed in the first place if your goal was to find a robust model that can withstand unseen data. Otherwise you may end up getting a winning model only because it's trained on external data. We'll see what the winning model has in store for us. ;)

", "votes": null, "replies": [ { "id": 315012, "author_name": "shiraz14", "author_url": "", "post_date": "04/16/2018 16:30:10", "content": "

This is indeed true, please see the discussion thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/54575 for an ongoing debate on this matter.

", "votes": null, "replies": [] } ] }, { "id": 314117, "author_name": "h938454", "author_url": "", "post_date": "04/14/2018 16:27:15", "content": "

Dear @Anne, @Allen. I asked in an other forum already, and never got an answer, nonetheless it would be very crucial IMO. Here is it:

\n\n

By definition if there is no cell on the image and:

\n\n

a) we predict 0-> the score is 0/0=NaN, although the prediction was good. Are you giving 0 or 1 in this case?

\n\n

b) we predict more than zero than the score 0/x = 0.

\n\n

What is the score on case a)?

\n\n

I appreciate,\nPeter

", "votes": null, "replies": [ { "id": 315070, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/16/2018 17:59:01", "content": "

I was able to get an answer from the Kaggle team - if there are no nuclei in an image, submit an empty string.

\n\n

LMK any other threads on this topic so I can reply there.

", "votes": null, "replies": [] }, { "id": 315075, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 18:04:43", "content": "

@Anne, I think the question is about the score we get if we predict correctly that there is no nuclei on an image. The official scoring definition does not cover that case.

", "votes": null, "replies": [] }, { "id": 315090, "author_name": "thenickname", "author_url": "", "post_date": "04/16/2018 18:28:20", "content": "

And that's why it seems pretty obvious that it's one of those images that won't count.

", "votes": null, "replies": [] }, { "id": 315098, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 18:37:59", "content": "

@theNickName, congratulations for being hired on Kaggle Staff. Didn't know that. ;)

", "votes": null, "replies": [] }, { "id": 315110, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 18:58:54", "content": "

That question was cleared by @Wendy in another thread: score is 0 for an image that contains no nuclei, whatever you predict for it.

", "votes": null, "replies": [] }, { "id": 315148, "author_name": "thenickname", "author_url": "", "post_date": "04/16/2018 19:31:48", "content": "

Could there have been a different option? The only other possible option would be against the evaluation rules, the only other impossible option would be scoring an undefined number.

", "votes": null, "replies": [] }, { "id": 315151, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 19:38:42", "content": "

There is, you can define the value of 0/0 as you wish. I would have rewarded those who do not predict nuclei when there is none.

", "votes": null, "replies": [] }, { "id": 315153, "author_name": "cpmpml", "author_url": "", "post_date": "04/16/2018 19:39:53", "content": "

As an example of metrics with special case for 0/0 have a look at the SMAPE metrics used in the Web Traffic Forecasting competition. It is not even continuous at 0: https://www.kaggle.com/cpmpml/smape-weirdness

", "votes": null, "replies": [] }, { "id": 315187, "author_name": "thenickname", "author_url": "", "post_date": "04/16/2018 20:52:19", "content": "

Yes, of course, but who would redefine the rules for the sake of a single meaningless case?

", "votes": null, "replies": [] }, { "id": 315190, "author_name": "thenickname", "author_url": "", "post_date": "04/16/2018 20:59:26", "content": "

Anyways, can't wait to see how y'all tackled this challenge. And who trained on what data and ends up at what place as a result.

", "votes": null, "replies": [] } ] }, { "id": 315016, "author_name": "shiraz14", "author_url": "", "post_date": "04/16/2018 16:32:29", "content": "

@Any of the organizers: \nJust out of curiosity, please advise on the cells the 2 drawings in the Stage 2 test set (170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece.png & 5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2.png) are supposed to represent? Thanks.

", "votes": null, "replies": [ { "id": 315028, "author_name": "drannecarpenter", "author_url": "", "post_date": "04/16/2018 16:44:18", "content": "

I'm not sure why you are asking?

", "votes": null, "replies": [] } ] }, { "id": 315062, "author_name": "antorsae", "author_url": "", "post_date": "04/16/2018 17:50:18", "content": "

I just saw this video today: https://datasciencebowl.com/2018dsbtutorial/ (a bit too late) and in retrospect the hint about diversity (and some images shown) was already there.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "312531": "Looking at the test2 data, I think my results will be terrible .\n\nI note that there are some images without nuclei? Are there some nerve cells as well?", "312540": "If I remember correctly, there are some pictures not for scoring which are used to prevent pseudo labeling.", "312541": "Makes you wonder whether we should have prioritized learning microbiology and annotating different datasets rather than fine-tuning off-the-shelf instance segmentation architectures, or training regimes on stage 1 train...", "312543": "Now i opted for this \"learning microbiology and annotating different datasets\". Actually i submitted a solution not with the highest public LB score. I submission one with LB=0.505 which i think work best on my downloaded cell images.\n\nbut even that, my downloaded cell images are very different from what i saw in stage2 dataset", "312544": "https://youtu.be/1g6Zb98KYJE\n\nYou can’t say we didn’t warn you! Good luck everybody! 👍", "312545": "Maybe! 😉\n\nWe intentionally made it difficult for anybody to win by just overfitting ensembles of ensembles of ensembles. We wanted a generalization or robustness challenge.", "312546": "Allen what is your score range for stage2, using the model you build from stage1? I recall you mentioned there is large overfitting on stage1 in your thread.", "312547": "Funny we attempted (ran out of time for 10 mins) to generate new images using a CycleGAN using our GT as reference. Images came out quite nice, but still since we were using GT from stage 1 the nuclei shapes are nothing like stage 2 so would not have helped anyway.", "312551": "Heng My best performing model was in the low 40s. (I didn’t have time to check my other models but I suspect they would’ve done better. )", "312552": "Thanks!", "312554": "I'd like to candidly ask if for the purpose of getting a great winning model at the end of the competition it would have been better to include at least one instance of each type of test image on stage 1 test; and if not, what is the motivation from the competition organizer's point of view to come up with such vastly different test images?", "312556": "It’s a fair question. I’m one of many organizers, so I can only provide my personal perspective. I’m interested in accuracy (as described by the competition metric), but I think the far more exciting and far more useful metric is robustness. An accurate model is valuable, but a less accurate model that can adapt to new experimental conditions is truly groundbreaking.\n\nI should add that from a user’s perspective (and also a computer vision perspective) the images are more similar than other image segmentation competitions.", "312560": "private lb will be really interesting", "312561": "Yeah I was worried about this. The idea is to generalize but to do this the training examples need to 'span the space.' I trained on all the data that was posted in the extra data thread that was usable (good ground truth images) and then augmented that. I ended up training on ~27,000 images and after spot checking the 2nd round test, there are still types of images I didn't have to train on. The final results will be interesting and maybe a bit of a crap shoot.", "312565": "What does it means when the submission is still initializing? I am trying to make sure our team made the deadline.", "312581": "There are few images are just very strange and I don't know whether they are real....\n\nFor example, the image starts with 0fd3ed... and the one starts with 0fd475a...\n\nThey are just completely black image to me.... are they real cell images for scoring? I am not sure whether the submission will allow 0 masks for some images...", "312592": "I just checked. They are not zeroed out. If you equalize them you'll see standard fluorescence noise. Just like with any other image, you need to carefully manage false positives.", "312600": "Yes indeed, but the imaging modalities used (brightfield & fluorescence) are the same as for stage 1, so the algorithms used by the contestants for Stage 1 should work fine (with the exception of a few tweaks, of course) ... 😉", "312618": "This is the best one:\n\n5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2\n\nIt's straight out of a textbook. It even has the caption 'Fig 5.' still on it. Notably it was a tiff file even though it was saved with a .png extension. I wonder if it's in the stage 2 set by mistake. I posted a similar one in the pinned data quality thread.", "312736": "I guess the organizer put that image on purpose. 'Fig 5.' should NOT be detected if a model REALLY learns what are nuclei cells or not.", "312779": "The image 170bc41b2095177ccc..... is a jpeg, not a png.\nThe art style is similar to the tiff image mentioned above.", "312784": "> We intentionally made it difficult for anybody to win by just\n> overfitting ensembles of ensembles of ensembles. We wanted a\n> generalization or robustness challenge.\n\nWhat you actually did, you failed on your part to prepare and provide diverse data needed to create robust models, wasted sponsor money and kagglers time, and turned erstwhile notorious ds bowl into joke.", "312793": "May I ask a question? Is there some images that actually don't have any nuclei? like this one?2f25c16e472f0b5f0c14b654a4e78f8f83ad7505f056d48349e1e12e0de98839.png", "312811": "Hi, Allen. \nWe are not allowed to change source codes, but our networks predict zeros for some images. What should we put into submission file in case if we strictly want to submit zeros for some image? Current submission format prohibits excluding img_ids with zeros only but zeros only could not be encoded with rle. Thanks", "312816": "Alexander Buslaev I have the same problem. This is what I am using. I just add id + comma and not 0 0 (seems to work):\n\n import pandas as pd\n import argparse\n import os\n import shutil\n \n \n def main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\n '--submission_fn',\n type=str,\n default=\"\",\n help=\"\"\"\\\n Path to the submission file.\\\n \"\"\")\n parser.add_argument(\n '--sample_submission_fn',\n type=str,\n default=\"\",\n help=\"\"\"\\\n Path to the sample submission filename.\\\n \"\"\")\n args, unparsed = parser.parse_known_args()\n assert os.path.exists(args.submission_fn)\n assert os.path.exists(args.sample_submission_fn)\n sub = pd.read_csv(args.submission_fn)\n sub_ids = set(sub['ImageId'])\n sample_sub = pd.read_csv(args.sample_submission_fn)\n sample_sub_ids = set(sample_sub['ImageId'])\n missing_ids = sample_sub_ids.difference(sub_ids)\n \n print(\"Adding %d missing ids.\" % (len(missing_ids)))\n out_fn = args.submission_fn.replace('.csv', '_all_ids.csv')\n shutil.copy(args.submission_fn, out_fn)\n with open(out_fn, 'a') as f:\n for missing_id in missing_ids:\n f.write('%s,\\n' % (missing_id))\n \n print(\"Done! Wrote updated submission to %s.\" % (out_fn))\n \n \n if __name__ == '__main__':\n main()", "312817": "Actually i don't really grasp the purpose of the challenge - having a generalizing and robust model is fine, it may even be groundbreaking, but part of the images in stage 2 test are just of bad quality, integration times are sometimes much too low or there is just some random noise. In practice, you can't work with these images because actually what you would do in case of fluorescence images was i.e. to stain with antibodies you want to quantify or to detect, see if certain cells/nuclei are positive for a certain antibody or to count the number of FISH probes per nucleus- you would never run and trust an analysis based on images like those...", "312819": "I am trained in biology, and from that perspective, the stage 2 images are not drastically different from what I might have expected to see in my day-to-day work. That said, I focused on training for the range of features I saw in the combined stage 1 train and test images. I expected some zoom changes, but not IF images with cytoplasmic dyes, or radically different cell types (neurons). I was quite selective with my training sets, and kept them quite small.\n\nI do worry that the winner is essentially the team with the biggest GPU farm to process the largest and most diverse external dataset comprised of different bioimaging methods. I guess we will have to wait and see.\n\nAlso not sure how to submit a 'no nuclei found in this image' RLE without getting errors.", "312820": "\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"\n\njust put image_id without rle (i.e leave it empty, like empty string '')", "312823": "Which of the external datasets (of course mentioned in the external data thread, to be allowed here) do you think would have helped?\n\nIMO incrementing or enriching the external data for a competition makes total sense but on this one (and based on the external labelled datasets I checked on the external data thread) the effort required to get a realistic training set needed to get SOTA results on stage 2 date is orders of magnitude higher than in other competitions. I would love to be proven wrong.", "312824": "\"Also not sure how to submit a 'no nuclei found in this image' RLE without getting errors.\"\n\njust put image_id without rle (i.e leave it empty, like empty string '')\n\n\nWe can use this:\n\nThe submission process (for stage1 and stage2) is :\n\n- run model on images and make csv file\n\n- for images not recorded in csv (i.e. zero detection), we fill in the csv file by hand.", "312831": "I don't disagree with this comment. It's a competitions with a bunch of gotchas and plot twists. However this is not the way you'd actually try to generate a practical solution to a real problem.", "312894": "If some people had used relevant training data my point would probably stand, but you're right that there wasn't really anything posted there that would have made too much difference; so the playing field might have turned out pretty level in the end. I guess everyone was finding things that were similar to those provided in stage 1.\n\nThanks for the tip about leaving RLE column blank. I didn't realise we could do that.", "312897": "David and @rakhlin Will you feel this way if competitors post scores in the range (say ±0.1 AP) of the phase one leaderboard?", "312929": "I'll stick by the statement that if you were trying to come up with a practical solution you wouldn't split up the data in the way it was split up. It will be interesting to see how the final scores come out. My feeling is that the leaders in the 2nd round will be different than in the first round. But we'll see what happens.", "312937": "David If a competitor came up with a solution that evaluated to, for example, 0.5 AP on both phase one AND phase two you wouldn’t consider that a practical solution?", "312942": "What would the score have been if the data had been split up in a more reasonable manner? You're not making the right comparison. I don't know what the ultimate best score could be but I rather doubt that you can get to it by using the available data in the way it was used in the competition.", "312944": "One thing I don't really understand is why the fluorescent images with multiple dyes are merged in a single channel? There are some single channel phase images which make sense, but you wouldn't normally choose dyes with overlapping emission spectra to stain multiple cell compartments. Unless it is showing the distribution of a protein or chemical that can translocate between the nucleus and cytoplasm. In that case, this is a bad way to look for nuclei (which I thought was the purpose of the data).", "312945": "David The competition wasn’t to see who could most overfit their model (that’s a useful competition, but that wasn’t the purpose of *this* competition) to a specific set of experimental conditions. It was, plainly, to see who could build a model that best adapts to data with unseen experimental conditions.", "312947": "Now I understood why you \"facepalm\" :D", "312950": "Okay the point of my comments is not to be overly critical of the competition. I'm just trying to point out that if you were trying to generate the best possible predictor from a given set of data you wouldn't purposefully exclude classes of data from the training set. That's not how you avoid over training and ensure generalization. That makes a good competition but not smart if you want the best possible solution to a problem.", "312955": "Yes, of course, but this is a competition. This isn’t a practical problem. You’re absolutely correct that, in the real world, it’d be silly to not use all available data. This was a competition to help unearth *strategies* to construct and train a model to solve practical problems. I don’t know how you’d design a competition without a challenging holdout that achieved that aim.", "312956": "Allen: Indeed, overfitting is (in most instances) an undesirable effect of constant retraining and fine-tuning of a model. However, the data set (in this context) should have been more practically split (with some of the Stage 2 images being introduced in Stage 1 for model training and validation). Introducing a *starkly* different dataset to an already previously-trained model & expecting a high percentage accuracy (and by high, I'm referring to a score of preferably 0.75++) would seemingly be a little over-ambitious, although this might not be so, if the model had been exposed to a more diverse training set from the start. This would be akin to human learning, where if you are (for example) a trained microbiologist, you would be an expert in a specified field of biology (say, microbial genetics), although if questions on advanced engineering mathematics were posed to you, you would probably be stumped and not be able to answer such questions (if you had no prior learning in this area), without consulting external references/resources. The same applies to a trained ML model, which might work pretty well on a test set ***similar*** (though not necessarily *identical*) to the training set, although the same model might fail drastically if required to predict the outcomes from a completely different test set.", "312964": "> The same applies to a trained ML model, which might work pretty well\n> on a test set similar (though not necessarily identical) to the\n> training set, although the same model might fail drastically if\n> required to predict the outcomes from a completely different test set.\n\nExactly. We’re interested in models that were able to adapt to these conditions. I suspect the winners of the competition will be people or teams that focused entirely on that particular issue.\n\nI should add that none of the organizing biologists would describe the images from phase two as “starkly different” than phase one. In fact, there were instances where they eliminated certain images for that reason.", "312972": "Allen: Some of the images introduced in Stage 2 were drawings scanned from a textbook, and not even photomicrographs - wouldn't you consider those to be starkly different from phase One (where no drawings were present in both the training and test data sets)? And some of these drawings also had other organelles (such as mitochondria) which might seem confusing to a trained ML model, and be interpreted as nuclei, so unless the organizers (including yourself) had put in these images but did not intend them to be scored, low scores would be inevitable for Stage 2 (even for the eventual winners), which leads us to the initial point that your primary objective for organizing this Competition (to attain robust models for accurately predicting the ROI corresponding to a cell's nucleus) would be unmet.", "313002": "My two cents on this thread, is that the organizers have put on a great contest! Engaging problem set! \n\nAfter putting the kind of time we all put into this its natural to be frustrated by the stage 2 data. But the real world is always dealing us new unknowns.\n\nThe organizers are writing the checks and designed an approach to get what they wanted out of it. At the end of the day the goal of ML is to take known circumstances and translate the learning to new ones. While its hard to learn about circles when all of your data are squares, that is not even close to the situation here. Its a fair fight, and extremely fun to work on.\n\nI am still grinding away on the stage 2, and do not expect to be at the top of the leaderboard. But special thanks to @Allen for being a great sport, and a guiding light for all of us!\n\nCheers in advance to the winners. You will have assisted in the solving of a critical problem for humankind.", "313011": "I agree, it is fun, and we were forewarned; but I did think there might be at least one or two examples of each type of image in the training dataset, even if it was somewhat biased towards one image type or another. It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.", "313012": "I'd argue that a good model should not detect any nuclei since there is no real nuclei in that drawing (it's a drawing, not a picture). I suspect that it won't be used for the LB calculation.", "313014": "![enter image description here][1]\n\nA classic conundrum in computer vision (and philosophy)! (Biology too, maybe!)\n\n [1]: https://upload.wikimedia.org/wikipedia/en/b/b9/MagrittePipe.jpg", "313017": "Depends how good the drawing is... A camera just captures (a slightly more accurate) picture of the nuclei too, and normally indirectly via some dye or other. However, as I pointed out in another thread, if you are making a drawing of a H&E stained tissue, why not just take a picture under the scope? That would be more useful and objective.", "313018": "DStjhb I agree. 😉 I also agree it's probably poison.\n\nNevertheless, it’s a cute image and fun to think about in context of the competition.", "313020": "> It is nice to feel like my crappy model might actually do better on this dataset, as I didn't make a particularly over-fitting type of ensemble. I don't feel completely 'out of the game' anymore yay.\n\nI love this. 👍\n\nA goal some of the organizers of the competition and I shared was preventing the competitor with the most resources to ensemble and optimize hyperparameters from dominating. While I ultimately think ensembling and hyperparameter optimization will be key (or even essential), I hope we reduced that problem.", "313022": "> I am still grinding away on the stage 2, and do not expect to be at\n> the top of the leaderboard. But special thanks to @Allen for being a\n> great sport, and a guiding light for all of us!\n\nThanks, dude! Good luck. ☺️", "313026": "I'm just really hoping my model tries to segment the figure legend like See--'s. That will make this whole thing worth it whatever the outcome.", "313682": "Yeah, a lot of images are like a combo of images from different stain spectra from BBBC06 dataset. Reguarding external data, the playing field is actually probably not turned in favour of those who used much of it. 99% of that data, almost all except BBBC06 can only hurt perf on tests. Unless, maybe, you totally knew what the data's gonna be and you preprocessed it accordingly before feeding it into learning. It's generally much too complicated. And if you also thought there's gonna be even more diversity and applied heavy augmentation, like color invertion, etc, you're screwed big time.", "313691": "I think what you guys should have actually done if you wanted to find out the best solution is you would provide, like DStjhb said, at least some pics with the types of datasets that the models are gonna be tested on and just get a really big and diverse - to your understanding of the level of diversity that you actually expect - not to our guesses - set of data - and do that in a more or less consistent manner as to what % of data in the training set is gonna correspond to what % of data in the testing set. You guys didn't actually include a lot of diverse data and I think the models that are gonna win are probably mostly just gonna show great results on some data similar to the training and testing data with a bonus to any people who, for example, only included BBBC06 dataset into their training loop, but not any other datasets. Because, anyone who actually expected the unexpected had most likely hurt his perf by much, including due to time wastage on chasing the wind.", "313695": "I wonder, doesn't that introduce division by zero into evaluation https://www.kaggle.com/c/data-science-bowl-2018#evaluation 😆", "313701": "I'm genuinely confused why there is so much distress about the stage 2 data set when it's clear we have not announced which images are scored. It's possible everyone will still be upset that we didn't design the challenge to their liking but at least wait until you see how it was designed before getting too worked up ❤\n\nI'm curious whether some who complained the loudest might do the best when the final scores come in! \n\nHeng, thanks for infusing the challenge with goodwill and a spirit of learning and community! I'm very indebted.", "313719": "Part of the challenge, I guess, was to train the model to **really** know, (as far a computer program can 'know'), what a nucleus looks like based on **absolute** features, as opposed to how different a nucleus looks from everything else in specific image types based on **relative** features. E.g. You want the model to be oriented more towards: ' a nucleus is usually the middle of a cell and may or may not have a visible nucleolus inside', as opposed to: 'a nucleus is round and brighter than background in grey images, or purple in not-grey images'. A very difficult task that wasn't too predictable, especially given the training data. However, despite all my criticism and devil's advocacy, I do reiterate that I have thoroughly enjoyed this competition.", "313725": "I was initially distressed because I thought my model scored 0 points and I would end up at the bottom of LB... All because the stage 2 image data was ***so*** different and my model does not generalize at all in the way the organizers wanted. Since I did not have any domain knowledge in biology and fairly new to deep learning, computer vision and data science, I just tried to fit Mask RCNN as best as I could. But it turned out that it is exactly what the organizers did not want. I learned a lot but I was also hoping to get a medal so I can brag about it to my wife and kids. They've been complaining that I'm spending too much time on this competition.... But my model got 0 points, I thought. 2+ month of hard work and nothing to show for it, I thought. \n\nLater I realized that it was just my ignorance and my tendency to jump to conclusion. I'm now just curious about how well Heng's solution would rank in private LB.", "313957": "Hi @See,\nThank you very much for sharing the code! There are few images in the test set that are empty and my model predicted as an empty image. I struggled a lot to submit my predictions since RLE doesn't incorporate zeros. After running your code, I was able to submit it successfully. For the benefit of other fellow participants, it would be useful if you upload the code in the kernel section. \n\nThanks again, \nWarm regards,\nK.Vikraman", "314103": "bsp2020 kudos to you, I'm sure your model won't let you down and you can catch up with your life again after the competition. We're similar, forgot to feed our baby daughter because we got worked up too much on Kaggle. This was our first Kaggle competition and we were new to data science and computer vision as well, the only thing we know is how to write code - that helped, I guess. :) I'm still frustrated after having seen our visualized results. It feels like as if the Broad Institute gave us the dog images as the training set, and then we have to predict hotdog images at the 2nd stage. :D", "314104": "Spoken as a biologist, totally deserves an upvote, I have no idea of lots of terms you talked about ;)", "314108": "Just my 2 cents, @Anne, I recall you discouraged people from training external data; however, we expected the 2nd stage test images would look drastically different by googling other nuclei in different shapes and colours. I think training external data shouldn't have been even allowed in the first place if your goal was to find a robust model that can withstand unseen data. Otherwise you may end up getting a winning model only because it's trained on external data. We'll see what the winning model has in store for us. ;)", "314117": "Dear @Anne, @Allen. I asked in an other forum already, and never got an answer, nonetheless it would be very crucial IMO. Here is it: \n\nBy definition if there is no cell on the image and: \n\na) we predict 0-> the score is 0/0=NaN, although the prediction was good. Are you giving 0 or 1 in this case? \n\nb) we predict more than zero than the score 0/x = 0.\n\nWhat is the score on case a)?\n\nI appreciate,\nPeter", "314160": "Anne,\n\nTL;DR It is uncertainty that stresses people, not the images.\n\nDetails below.\n\nI think there is distress because there is uncertainty. The rules say we must use the same models as in stage 1, but then stage1 test labels are disclosed, and retraining is allowed. This makes it very tempting to bend the rules and retrain on data that looks like stage2 test data. \n\nWhat is needed is a clear cut boundary between what is allowed and what is not. I wish the clear cut boundary had been: no retraining in stage2.\n\nThe other source of uncertainty is the zero score. Few of us explained it, pointed to past competition, but still, people want an 'official confirmation', as if previous competition official confirmation was not enough.\n\nA third source of distress is that questions stay unanswered for more than 24 hours. Sure, I can imagine that from organizer perspective 2 hours is not a big deal, but when you're stressed and wait for answer it is really bad.\n\nI tried to capture what many think above. It is not just me.\n\nBeside this lack of clear cut communication from organizers I really enjoyed that competition, learned a lot of stuff for my first image competition, I teamed with nice people, and I really think Heng Cher Keng will become discussion and competition grandmaster very soon! Thanks for organizing this, despite our grumbling.", "314883": "Again, the images in stage 2 test consist of a subset of images that are actually scored, and a set of images that are there to ensure you don't hand-label the entire set or tune your model to the precise images that are scored.", "314957": "Hi @Anne, thanks a lot for the clarification. Makes total sense and this will ensure the competition is fair!", "315012": "This is indeed true, please see the discussion thread https://www.kaggle.com/c/data-science-bowl-2018/discussion/54575 for an ongoing debate on this matter.", "315016": "Any of the organizers: \nJust out of curiosity, please advise on the cells the 2 drawings in the Stage 2 test set (170bc41b2095177cccd3d4c8977c619147580f1d93b4fe9701eddd77736d4ece.png & 5390acefd575cf9b33413ddf6cbb9ce137ae07dc04616ba24c7b5fe476c827d2.png) are supposed to represent? Thanks.", "315028": "I'm not sure why you are asking?", "315062": "I just saw this video today: https://datasciencebowl.com/2018dsbtutorial/ (a bit too late) and in retrospect the hint about diversity (and some images shown) was already there.", "315070": "I was able to get an answer from the Kaggle team - if there are no nuclei in an image, submit an empty string.\n\nLMK any other threads on this topic so I can reply there.", "315075": "Anne, I think the question is about the score we get if we predict correctly that there is no nuclei on an image. The official scoring definition does not cover that case.", "315090": "And that's why it seems pretty obvious that it's one of those images that won't count.", "315098": "theNickName, congratulations for being hired on Kaggle Staff. Didn't know that. ;)", "315110": "That question was cleared by @Wendy in another thread: score is 0 for an image that contains no nuclei, whatever you predict for it.", "315148": "Could there have been a different option? The only other possible option would be against the evaluation rules, the only other impossible option would be scoring an undefined number.", "315151": "There is, you can define the value of 0/0 as you wish. I would have rewarded those who do not predict nuclei when there is none.", "315153": "As an example of metrics with special case for 0/0 have a look at the SMAPE metrics used in the Web Traffic Forecasting competition. It is not even continuous at 0: https://www.kaggle.com/cpmpml/smape-weirdness", "315187": "Yes, of course, but who would redefine the rules for the sake of a single meaningless case?", "315190": "Anyways, can't wait to see how y'all tackled this challenge. And who trained on what data and ends up at what place as a result." }, "source": "meta" }